/dev/websphere

Extreme write behind with JMS

I've been working with one of our biggest customers and they have a high quality problem in that under extreme load, the grid (WebSphere eXtreme Scale) is holding up just fine. but there is a problem. The issue is the database the grid runs on top of. They are using write behind to asynchronously write changes made to the grid to the database. This basically will only write changes to the database every 5 minutes or 1000 entries depending on what happens first.

The issue is even with this batching occurring, the database cannot keep up with the write load. They could buy a bigger database but all they really use their database for is a disk based log for whats in the grid. They load the grid from the database and then all applications use the grid as the system of record for that data. The updates made to the data in the grid and written back to the database asynchronously.

If the database cannot keep up even with the batching then it starts to fall behind and more importantly, the buffered changes not yet flushed from the grid also build up in memory.

Buying a bigger database box is also kind of pointless as the whole point of the grid was to avoid having to buy a large SMP box to run the database. Scaling up is expensive and database boxes with their SANs that can keep up with extreme loads are not really what we want to purchase. We want the grid to act as a shock absorber so that a smaller database can be used. But, this is proving a problem at this customer just using normal write behind techniques.

The real problem here is databases are not designed to be used like this. WebSphere eXtreme Scale is basically using a SQL database as a write only disk store and frankly, databases are not very good at this.

So, what is designed as a write only disk based store? Believe it or not, a messaging system is. JMS queues are designed to implement two operations. PUT to the queue and GET from the queue and they can do this very efficiently. More efficiently than a database. This is all the grid really needs here. Flushing the writes from memory to disk through a database doesn't make a lot of sense.

The idea here is to put the JMS queue between the grid and the database. This way, the grid can flush the changes to 'disk' faster than before and then we can apply the changes from the queue at our leisure to the database. This decouples the database and the grid. The grid can now deal with flash loads or tsunamis of load when they occur and flush the changes to disk using the messaging system. The database can apply those changes at it's own slower rate because the JMS queue is buffering the changes in a durable way to disk. Thus, we don't need to spend a lot of money on an expensive vertical or horizontal scaling DBMS.

This is very like a seat belt in a car. We're trying to absorb a shock by stretching it out over a longer period of time. The JMS queue allows this to happen. While the flash load is happening, the queue is able to keep up with the grid and everything stays stable. The database is emptying the queue as fast as it can which is slower than the grid. Once the flash load is gone then the grid goes back to normal load while the DBMS can keep applying changes for another couple of hours if need be.

The implementation of this could be a Loader that writes the changes to the JMS queue. A JMS listener can then pick up the changes are apply them to the database. This is basically like the write behind built in to WXS but implemented instead with an external JMS queue.

Configuring DataSources with DataGrids that front end Databases

I'm seeing this a lot these days. A customer is using IBM WebSphere eXtreme Scale in front of a database. The DBAs are concerned about how many JDBC connections that the grid is requiring. If it's a larger grid then the problem is especially acute. For example, lets say we have a grid with 200 JVMs in it. All of the grids that I have sized recently are in this range. Lets look at different aspects of this problem

1) WXS as side cache

Here, there are no connections from the grid to the database. The application server JVMs do all the database interaction and hence, the grid adds no additional demands on the DBMS.

2) Preloading WXS with data from a DBMS

Here, preloading it can be interesting. The main issue is usually the time to load the grid with the DBMS data. We usually use parallelism when we do this but that means multiple JDBC connections per preload client. The main issue I see with DBMS preload is simply query complexity slowing down the preload process. Simplifying the query and moving join logic in to the grid side preload process seems to be the key to solve this problem. Once you can achieve a high preload speed per single preload thread then you don't need so many JDBC connections to achieve a specific preload speed.

3) Write through WXS to a DBMS

This is probably the worst offender here. Typically, a WXS Loader consists of a TransactionCallback (like this JDBXTxCallback example) and a Loader. The TransactionCallback makes sure a JDBC connection pool is available for the Loader and the TransactionCallback manages JDBC.begin and JDBC.commit and rollback operations. The Loaders share the connection provided by the TransactionCallback for a specific transaction.

Clearly, preloading the grid will enormously reduce the usage of JDBC connections for grid read operations. However, write operations will still consume one JDBC connection per concurrent WXS transaction. Given, the potential throughput performance of such a grid, clearly, the load is usually too much for most databases. A grid server with 20 JVMs on a modern blade can still achieve 20k transactions a second per blade. Lets start with a grid with a 100% write rate for some peaks. Thats 1000 write transactions per JVM. If each write transaction costs 40ms (remember, the DBMS is directly involved with writes here, this isn't the grid, it's the DBMS) then we could do 25 transactions per thread per JVM (1000 / 40ms). Now, how many threads is that? We'd need 40 (1000 / 25 ops per thread) threads per JVM to achieve this amount of throughput in a single JVM. So, the JVM needs a JDBC connection pool of 40 connections (one per thread) too.

If there are 200 JVMs then we need 8000 (200 x 40) JDBC connections for the grid! Now, we can see why the DBAs may be concerned. A write through grid thats write intensive is a great idea for a grid. But, only with write behind. Write through makes NO SENSE AT ALL for this situation. The DBMS will remain the bottleneck and the customer will likely question why use a grid at all if such a solution with write through was proposed. Remember a 10 blade grid with 20 JVMs per blade and a 100% write rate for some workloads would get you here.

So, write through can still use a lot of DBMS connections if the write transaction percentage is high. If it's lower then these numbers can all be revised downwards. For example, if it was 10% then lets do the math again. One box does 20k tx/sec. 10% is 2000/sec. There are 200 JVMs then thats 10/sec per JVM. One or two connections in the DataSource would likely be enough here depending on response time requirements. This gives us 200 JVMs x 2 connections or 400.

It's very important to share a DataSource within a JVM, do not give each JDBCTxCallback it's own DataSource. The example JDBCTxCallback provide in this blog entry (see the link above) accepts a DataSource reference when it's initialized. You can use Spring to wire it in or change the JDBCTxCallback to get one from JNDI when there is an application server hosting the grid containers. This way, even though there is one JDBCTxCallback PER primary shard in a JVM, at least they will all share the same DataSource instance. This would reduce the connection usage dramatically. This is usually only a problem when used a J2SE hosted WXS container. Customers using WXS inside WebSphere ND will likely be using WebSphere's DataSource implementation and this will naturally result in one DataSource per JVM.

4) Write behind WXS to a DBMS

This approach gives the best opportunity to cut JDBC connection usage. Assuming, we preloaded the grid then the connections for reads can likely be ignored. This leaves writes. Write behind decouples the writes to the database from the grid writes. This means that one WXS write does not mean one DBMS write. This was the major problem with write through.

WXS allow DBMS writes to be triggered every X 'dirty' entries in a Map or when 'dirty' entries in a Map are older than a certain time (5 minutes for example). This means we would see much fewer database writes than grid writes. If we have a grid JVM with 5 primary partitions in it (the normal amount when sized correctly) then there are only 5 potential concurrent writes per second in such a JVM. Each partition will send its changes to the DBMS using a single thread. Therefore, the maximum JDBC connections used for write behind is the number of primary partitions in the JVM which in this case would be 5 PER JVM.

Note, that when boxes fail then the # of primary partitions can be more. Thus, size everything with what you expect to be your minimum # WXS container JVMS to figure this out. It's more likely to be 8-10 primaries in this case. Thus, we'd expect 10 JDBC connections per container if the system was extremely write intensive. However, a very high write rate is unusual. Normally, the write rate is much lower (10-20% might even be high). But, a security system built with a grid that updates last login time for example might result in a WXS write per login so be careful sizing this.

We expect in the worse case scenario with write behind to need one JDBC connection per partition. However, this is only the case when every partition writes at the same time. The big factor here is how long does a write behind transaction take against the database or the 'duty cycle'. If it takes 1 second and we're doing one write behind every 5 minutes then we can likely use one shared JDBC connection in a DataSource per JVM, thats a duty cycle of 1 : 300 (300 seconds in 5 minutes). We are unlikely to get much contention for that one connection as there is a 1:300 change of writes from different partitions clashing within a single JVM. If each write behind tx takes 2 minutes then this clearly changes things given there are 5-10 primaries per JVM. 2 minutes would mean a duty cycle of 120:500 or about 25%. You'd need probably one per primary partition here or 10 per JVM.

But, regardless, it's clear that write behind can reduce the DBMS connection usage significantly when the write behind configuration is 'optimal'. Optimal depends on the situation but as a starting point, 5 minutes or 1000 records (T300;C1000) isn't a bad place to start. The size of the DataSource pool for each JVM depends on the duty cycle of the write behind within a single JVM.

The points on sharing a single DataSource within a WXS JVM equally apply here as to write through.

Summary

I hope this blog entry opens peoples eyes in terms of JDBC/WXS interactions. Most applications elect to use a grid because the DBMS is unable to provide adequate response times under peak loads. If the write rate is very low then there is no problem. But, when the write rate is high then write behind is usually the best scenario here. A high write rate application using write through is likely going to still have performance issues as the DBMS will prevent the grid from achieving linear scaling and good response times.

A quality DataSource implementation is a big deal here. The DataSource thats provided with WebSphere Single Server/Base and WebSphere ND is about as fully developed as exists out there. It is a significant improvement (in my humble opinion) over C3P0 or Apache dbcp. Thus, running WXS inside a single server WAS install is something you could consider instead of a J2SE based WXS implementation for this reason alone, the DataSource implementation.

Keeping data in IBM WebSphere eXtreme Scale grids in sync with a database

This is a very common question. A customer wants to keep a copy of important data in a DataGrid, like IBM WebSphere eXtreme Scale. The data is sourced from a database typically. The data is changed by legacy applications in the source database only. The customer wants to know whats the best way to keep the DataGrid in sync as data changes in the database.

A great solution for this is the IBM Infostreams Change Data Capture product. This product installs on the same box as the database and it can watch the database for changes. It watches for changes usually by watching the transaction logs. This is very lightweight and typically has little or no impact on the database process. It can also watch using triggers but this is going to have an impact on the database.

Once CDC spots a problem then it can send a message to a JMS queue to record the fact that a record in a table that is being watched has been modified. The customer then writes an JMS message listener to listen to that queue and apply the change described by the message to the data in the DataGrid.

This is also a great way for buffering changes made to very large datasets when the grid is down for maintenance. We can stop the JMS listener. CDC will still buffer changes on the queue when the listener is disabled. Then we would write a disk snapshot of the grid using wxsutils. Do the maintenance. Restart the grid, reload the snapshot and then enable the JMS listener. The JMS listener will then bring the grid up to date with the changes buffered during the maintenance interval. This will be much faster than a full reload from the source database when the dataset is large or the source database is a smaller database (i.e. it's not that there is a lot of data, the database is just slow getting it all out quickly).

Anyway, this is something I'm going to start recommending every time moving forward. I think it's a great solution, not very invasive and is pretty simple to get working.

Creating disk snapshots of an IBM WebSphere eXtreme Scale grid and why they are really useful

I've been working with a few customers who are using WXS to load large quantities of data in to the grid from a database. This can be quite time consuming as fetching the data from the backend is usually the bottleneck. Loading hundreds of gigabytes of data can take well over an hour. This means that a full grid restart is not something such a customer wants to do a lot given the cost of reloading the data.

The bottleneck here isn't WXS, it's the source data system. Many of the customer scenarios I see use the grid as the system of record. If the grid is down then the data doesn't change. We'll discuss when it does change later.

I've written a utility as part of wxsutils (if you're not using this with wxs then you are missing out...). It will make a disk based snapshot of a grid map. It writes each partition for each map to a file by writing the key/value pairs as JSON strings. Each physical box hosting the grid container JVMs will store all the partitions hosted on that box in files in a particular folder on that boxes file system. The current code does it for the maps you specify. So, just execute the code once for each map you want 'snapshotted'.

Later, the customer can execute the read snapshot code to reload that data set from those files in to the grid. This will happen at grid speed. Each box in the grid will read the local files and load them in to the grid as fast as possible. The bottleneck here will likely be network depending on the boxes.

This approach can save a lot of time in many scenarios:

• The customer wants to cycle the whole grid for maintenance on the WXS runtime or the application artifacts. The customer should snapshot the grid before stopping it and then read the snapshot after restarting the grid post maintenance. This is much faster than a reload from the backend.
• The customers wants during testing to have a grid with a well known start data set. Load it once, snap shot it and then use it to reset the grid every time you want to run a test case from a well known start state.

If the data is changing all the time even when the grid is down then we can deal with that also. Probably the 'best' (and best is a loose word here) way to synchronize a grid with a database is by using a change data capture type product. For example, here is IBMs link. These products watch the transaction logs of a database and allow an application to intercept changes in the data without slowing the database down or using triggers etc. The application listener would then typically make a message about the change and send it to the grid where the grid will process the message and apply the change. This keeps the staleness between the database and the grid to a minimum.

If this listener stores these messages in a JMS queue or MQ series queue then even if the grid is down, these changes are tracked. Thus, when the grid is restarted, do not start the msg listener straight away. Instead do the read snapshot and then start the listener which will down apply all changes since we wrote the snapshot. This allows a delta update of the data in this case keeping the snapshot useful.

You can see how to use this snapshot capability on github at this link.

You will need to add the PerContainerGrid definition to your objectgrid.xml and deployment.xml files and put wxsutils.jar and the jackson json jars also on your container classpaths.

memcached versus DataGrid, dumb server versus smart server...

Memcached is a pretty popular free open source distributed cache. It's pretty unique compared with the commercial data grids in that it has a completely unique design.

Memcached is basically a dumb server. This means it's a process running on a Linux box typically which accepts client requests for keys and returns the data. Updates and deletes are also supported. This is ALL the function in the server, hence, dumb server. The client is the only piece which is 'distributed' or even knows about other servers. The clients typically get a collection of host:ip addresses (they all NEED to use the same list) and the clients hash the key over the set of online servers. If there were 4 servers running then all clients will hash data for a key over these 4 servers. Clients will write data for a key K to a particular server using this hash algorithm. One server.

Now, thats what memcached does out of the box. Note, there is no replication on the server side. If a server fails then the clients will hash keys over 3 servers but this changes everything from a hashing point of view so it's likely you'll get a very high miss rate as it's unlikely keys still hash to the same server as before given the modulo part of the hashing is now 3 instead of 4.

Clients work around this by adding a consistent hashing layer on top of the normal client. This works better and writes may be written to a server and the next server in the hash ring. This slows down writes by 2x but allows some recoverability when a server fails. The next server in the ring has the data from the dead server so it's not as bad as before BUT if this server also fails then now the cache has lost the data. It's not replicated remember, it's just written twice. After the first server failed, the clients write changes to the next server and the following server in the hash ring. The new next server in the ring does not have a copy of all the data. It was never written there. It will just have new changes since the failure. There is no active replication going on here.

Most of the materiel on the web shows various workarounds to try to make memcached 'reliable'. A lot of the facebook engineering blog entries focus on this aspect. What they had to do to make it reliable. This works for facebook but won't work in general because I don't think the majority of customers are structured like they are from a build your own mentality. I was talking to someone from one of the big internet shops last week and they always run two memcached clusters for HA. If a server in cluster A fails then they stop using cluster A and switch to B. This works but thats a 50% loss in hardware because of a single failure.

Smart servers

Commercial data grids like IBM WebSphere eXtreme Scale are very different. They handle making the data fault tolerant on the server side with built-in replication and elastic scale out and scale in. Their clients understand the server side data layout or routing. If a server fails with WebSphere eXtreme Scale then no data is lost. The software actively maintains replicas on other servers. Clients that stored an entry with a key K before will still find it. It just works. You don't need consistent hashing on the client, our client does it using routing tables instead. Adding more server processes causes the product to automatically redistribute the minimum amount of data to leverage the new memory/CPU and network available through the new JVMs.

The smart servers allow proper replication which means that the cache isn't as brittle as the memcached cache when servers come and go. If nothing ever fails then the memcached approach works but the potential cache hit rates over time if multiple failures occur mean it can be a liability. The last facebook entry I read on this showed them crawling the logs to try to recover lost data to make the 'next server' more complete. They get this issue and are jumping through hoops to handle it but again, most commercial customers won't do this. They expect the product to do this sort of stuff like replication and they view it as table steaks.

So, memcached for sure has its uses but it's got some limitations which need a fair bit of work by its users to handle correctly if its not exactly what you want or risk some bad scenarios as the cache hit rate climbs. It's very, very fast but you gotta remember, it's not doing much either... No transactions, no consistency, no replication, no isolation and so on. If you don't need these features then it's pretty fast but if you do need this stuff then it starts to slow as you add code on the client side to try and compensate for the limitations.

The problem with JPA and Java persistence in general...

The Java community is currently focused around JPA as the persistence API but I think it's missing the big picture. JPA is a combination of several facets which themselves need to be separated into separate specifications so that we can get a better programming model.

JPA consists of the data model which can be viewed as the POJOs + annotations for keys and relationships. There is an API for interacting with a data source to find and query entities within that data model and there is a set of meta data for mapping that data model to a SQL database.

The problem is they are all combined in a single spec. This is problematic because it couples together different facets where are useful in their own right. Lets look at the following problems faced by Java developers.

The web developer

Web session data is still stuck in the dark ages. Session data is kept in a flat Map of key/value pairs. It's difficult for a developer to work with a hierarchical data model for the session which is common and the API is dumb as it doesn't automatically detect changes and just persist those to the Session like JPA can do using byte code weaving or straight copy and compare approaches. Why can't a developer pick a data model expressed as a POJO object graph and then find the root of that in a session and then walk it, update it and have changes picked up automatically. If several modules share the session then why should they all need to use the same POJOs? Why can't each module map it's own POJOs to the data model for the session? The current servlet spec or even the proposed ones are sorely lacking in modernizing the specification down this path.

The work flow developer or JBoss Seam or Spring Webflow developer

The same thing applies here. Why don't we have a standard data model which can be persisted to a flow/conversational flow engine. Why isn't the specification for the data model for a flow projectable on to the POJO data models a developer wants to use. The seam and webflow stuff are attempts to modernize the session management in the servlet spec but without a common data model, annotations for capturing these meta models then we'll end up with a separate incompatible spec and redundant tooling for capturing and interacting with this meta data.

The cache or XTP developer

Most distributed caches still work with a single level Map of key/value pairs. WebSphere eXtreme Scale is the exception here in that it can layer a POJO graph model on top. This is our EntityManager API. We modeled this after the JPA spec for the pieces related to key and relationship definitions. We support different applications sharing data kept in a shared cache even when each application uses a different set of POJOs for the same data. The respective applications just use different metadata for the mapping to those POJOs from a master schema stored in the shared data grid. We built a standalone component within WXS called the projector which has a master meta data and it can then project data stored in a TupleStore to POJOs suitably annotated for the mapping and then push any changes from those POJOs back to TupleStores. We currently implemented TupleStores for our internal Maps of key/value tuples but in theory this could be used to represent any TupleStore (flow instance state or HTTP session). We shouldn't have had to write that component, it should be part of a J2SE specification built in to every JVM. Maybe a first step here once any internal hurdles are overcome would be to contribute this component to the community as an initial building block to start to move down this path.

The database developer

This is the ONLY developer catered to by JPA despite the fact that if this spec was split than many facets, it could be reused in a standard way for the other developer types. It's just a mixed up mess that needs to be pulled apart in to standalone specifications which can be combined to form a standard persistence programming model for all types of programmer.

Summary

You can see there are some parts of the Java programming model which are solely lacking. Flows, caches, conversational state, web developers and database developers should all be shared common specifications for data models, the mapping of that to various non SQL and SQL back-ends and support common query engines, projection/unprojection etc. We need to pull our collective heads out of the sand and look around for the big picture, otherwise, competitively, it's not looking great.

We need these common specifications so that our data based tooling in eclipse etc can be done once and then reused for all these purposes. The latest here is the upcoming criteria API. I'd like to be able to reuse this for WebSphere eXtreme Scale queries for example, why is this spec tied to JPA at all. We need a common POJO persistence spec containing abstract queries, factories, and object meta data like keys or relationships. Layered on top of this and its RI component would be JPA, HTTP Sessions, Caching, Web conversation flows and work flows.