/dev/websphere

Configuring DataSources with DataGrids that front end Databases

I'm seeing this a lot these days. A customer is using IBM WebSphere eXtreme Scale in front of a database. The DBAs are concerned about how many JDBC connections that the grid is requiring. If it's a larger grid then the problem is especially acute. For example, lets say we have a grid with 200 JVMs in it. All of the grids that I have sized recently are in this range. Lets look at different aspects of this problem

1) WXS as side cache

Here, there are no connections from the grid to the database. The application server JVMs do all the database interaction and hence, the grid adds no additional demands on the DBMS.

2) Preloading WXS with data from a DBMS

Here, preloading it can be interesting. The main issue is usually the time to load the grid with the DBMS data. We usually use parallelism when we do this but that means multiple JDBC connections per preload client. The main issue I see with DBMS preload is simply query complexity slowing down the preload process. Simplifying the query and moving join logic in to the grid side preload process seems to be the key to solve this problem. Once you can achieve a high preload speed per single preload thread then you don't need so many JDBC connections to achieve a specific preload speed.

3) Write through WXS to a DBMS

This is probably the worst offender here. Typically, a WXS Loader consists of a TransactionCallback (like this JDBXTxCallback example) and a Loader. The TransactionCallback makes sure a JDBC connection pool is available for the Loader and the TransactionCallback manages JDBC.begin and JDBC.commit and rollback operations. The Loaders share the connection provided by the TransactionCallback for a specific transaction.

Clearly, preloading the grid will enormously reduce the usage of JDBC connections for grid read operations. However, write operations will still consume one JDBC connection per concurrent WXS transaction. Given, the potential throughput performance of such a grid, clearly, the load is usually too much for most databases. A grid server with 20 JVMs on a modern blade can still achieve 20k transactions a second per blade. Lets start with a grid with a 100% write rate for some peaks. Thats 1000 write transactions per JVM. If each write transaction costs 40ms (remember, the DBMS is directly involved with writes here, this isn't the grid, it's the DBMS) then we could do 25 transactions per thread per JVM (1000 / 40ms). Now, how many threads is that? We'd need 40 (1000 / 25 ops per thread) threads per JVM to achieve this amount of throughput in a single JVM. So, the JVM needs a JDBC connection pool of 40 connections (one per thread) too.

If there are 200 JVMs then we need 8000 (200 x 40) JDBC connections for the grid! Now, we can see why the DBAs may be concerned. A write through grid thats write intensive is a great idea for a grid. But, only with write behind. Write through makes NO SENSE AT ALL for this situation. The DBMS will remain the bottleneck and the customer will likely question why use a grid at all if such a solution with write through was proposed. Remember a 10 blade grid with 20 JVMs per blade and a 100% write rate for some workloads would get you here.

So, write through can still use a lot of DBMS connections if the write transaction percentage is high. If it's lower then these numbers can all be revised downwards. For example, if it was 10% then lets do the math again. One box does 20k tx/sec. 10% is 2000/sec. There are 200 JVMs then thats 10/sec per JVM. One or two connections in the DataSource would likely be enough here depending on response time requirements. This gives us 200 JVMs x 2 connections or 400.

It's very important to share a DataSource within a JVM, do not give each JDBCTxCallback it's own DataSource. The example JDBCTxCallback provide in this blog entry (see the link above) accepts a DataSource reference when it's initialized. You can use Spring to wire it in or change the JDBCTxCallback to get one from JNDI when there is an application server hosting the grid containers. This way, even though there is one JDBCTxCallback PER primary shard in a JVM, at least they will all share the same DataSource instance. This would reduce the connection usage dramatically. This is usually only a problem when used a J2SE hosted WXS container. Customers using WXS inside WebSphere ND will likely be using WebSphere's DataSource implementation and this will naturally result in one DataSource per JVM.

4) Write behind WXS to a DBMS

This approach gives the best opportunity to cut JDBC connection usage. Assuming, we preloaded the grid then the connections for reads can likely be ignored. This leaves writes. Write behind decouples the writes to the database from the grid writes. This means that one WXS write does not mean one DBMS write. This was the major problem with write through.

WXS allow DBMS writes to be triggered every X 'dirty' entries in a Map or when 'dirty' entries in a Map are older than a certain time (5 minutes for example). This means we would see much fewer database writes than grid writes. If we have a grid JVM with 5 primary partitions in it (the normal amount when sized correctly) then there are only 5 potential concurrent writes per second in such a JVM. Each partition will send its changes to the DBMS using a single thread. Therefore, the maximum JDBC connections used for write behind is the number of primary partitions in the JVM which in this case would be 5 PER JVM.

Note, that when boxes fail then the # of primary partitions can be more. Thus, size everything with what you expect to be your minimum # WXS container JVMS to figure this out. It's more likely to be 8-10 primaries in this case. Thus, we'd expect 10 JDBC connections per container if the system was extremely write intensive. However, a very high write rate is unusual. Normally, the write rate is much lower (10-20% might even be high). But, a security system built with a grid that updates last login time for example might result in a WXS write per login so be careful sizing this.

We expect in the worse case scenario with write behind to need one JDBC connection per partition. However, this is only the case when every partition writes at the same time. The big factor here is how long does a write behind transaction take against the database or the 'duty cycle'. If it takes 1 second and we're doing one write behind every 5 minutes then we can likely use one shared JDBC connection in a DataSource per JVM, thats a duty cycle of 1 : 300 (300 seconds in 5 minutes). We are unlikely to get much contention for that one connection as there is a 1:300 change of writes from different partitions clashing within a single JVM. If each write behind tx takes 2 minutes then this clearly changes things given there are 5-10 primaries per JVM. 2 minutes would mean a duty cycle of 120:500 or about 25%. You'd need probably one per primary partition here or 10 per JVM.

But, regardless, it's clear that write behind can reduce the DBMS connection usage significantly when the write behind configuration is 'optimal'. Optimal depends on the situation but as a starting point, 5 minutes or 1000 records (T300;C1000) isn't a bad place to start. The size of the DataSource pool for each JVM depends on the duty cycle of the write behind within a single JVM.

The points on sharing a single DataSource within a WXS JVM equally apply here as to write through.

Summary

I hope this blog entry opens peoples eyes in terms of JDBC/WXS interactions. Most applications elect to use a grid because the DBMS is unable to provide adequate response times under peak loads. If the write rate is very low then there is no problem. But, when the write rate is high then write behind is usually the best scenario here. A high write rate application using write through is likely going to still have performance issues as the DBMS will prevent the grid from achieving linear scaling and good response times.

A quality DataSource implementation is a big deal here. The DataSource thats provided with WebSphere Single Server/Base and WebSphere ND is about as fully developed as exists out there. It is a significant improvement (in my humble opinion) over C3P0 or Apache dbcp. Thus, running WXS inside a single server WAS install is something you could consider instead of a J2SE based WXS implementation for this reason alone, the DataSource implementation.

Messaging only APIs need to go, JPA needs to step up

I'm getting tired of the need for separate messaging and data access APIs. You could characterize these APIs as:

• Ordered accessible data (OAD)

• Key accessible data (KAD)

Key accessible data arguably has the most capability and has seen the most innovation over the last 5 years. Most notably, the JPA specification has arrived which standardizes the POJO oriented work done in OR mappers like Hibernate and Toplink. We have automatic mapping, relationship management, query support, lots of nice stuff. Now lets look at JMS. It's about as stagnate as you can imagine. Why can't I go all the stuff with OAD as I can with KAD? Do we now need to enhance JMS. NO, NO, NO. We need to force messaging to simply extend the JPA like standards, not compete with them. Messaging is just state, state should be managed the same way whether its state from a message or state from an inmemory database like ObjectGrid or state from a relational database.

For me, this status quo isn't acceptable at all. The JPA specification is unfortunately very short sighted in this area and is too relational database focused. It should be a main specification focusing on entities and relationship between entities and a subspec specializing it to relational databases but a product that works with persistent state should have been able to use JPA APIs but alas, that isn't the case and the JPA 2.0 stuff doesn't look any different. Why couldn't we use the JPA spec for messaging or for non relational databases or as an API to an in memory database directly?

JPA type interfaces could easily support messaging semantics with one little change. OAD and KAD have different but not incompatible locking requirements. OAD needs a way to scan a filtered set of records  (think query) looking for unlocked records and then lock the first one and return it to the application. The JMS receive method does just this. If many threads/consumers go after the top of a queue then the locking model allows this behavior. Find the first unlocked message in FIFO order, lock it and return it.

Why then don't databases provide this locking model, scan for unlocked record and lock? I don't know. We have added it to ObjectGrid for the next release and this simple addition allows an ObjectGrid Map to be used for both a KAD and an OAD. This completely unifies data and persistent messaging and it's very, very cool.

No more is messaging data and key data treated differently, it's the same and it should be. Why can't I annotate a message with a relationship to data outside the message in a TABLE or ObjectGrid entity? It should be absolutely seamless and messages should not be orphans in your data model, a message is just another record type and it should be able to have relationships with other records and this should be supported first class.

I think it's been too long now for this change to happen in the industry. Messaging APIs are dying on the vine, ws-* stuff just redefines the existing APIs with ws-* APIs and transports but does nothing to simplify the life of a programmer. ws-* just provides a new syntatic sugar for the existing APIs and supports more protocols but fundamentally does nothing new for the programmer. We need one data programming model and that data programming model needs to support both KAD and OAD with the same data model, tooling and APIs. I say, split JPA in two specs and extend the base spec with an optional addition to add efficient OAD support to the API.

Total JMS Message ordering

A lot of customers think JMS products provide message order. I haven't found one yet that really does it.

Naively, you'd think that if a single publisher sends messages then a single receiver will process those message in the published order.

We make an MDB with a single listener thread and run it on a single server. If nothing goes wrong then this is possible. But, in the real world, sadly, things do go wrong.

TCP Sockets

First problem is a crashed client where the socket to the JMS server remains open. This can happen when a network breaks or the server physically crashes. The message(s) buffered on that client stay locked until the socket is closed. This can take a while depending on TCP KEEP ALIVE settings, if you haven't tuned this then it's likely 2 hours. In the meanwhile, if you restart your client on a different server and it asks for messages then it gets the next messages. Oops. Out of order. When the socket is eventually closed, the 'trapped' messages are unlocked and delivered to the client. Busted.

Indoubt Transactions

Next up is indoubts. If an application server was processing messages and then crashed or the box it was running on failed then we may have some indoubt transactions. These are transactions interrupted during the XA commit phase. Normally, these transactions will need to be recovered either by a peer cluster member (WAS v6 in 15 seconds) or by restarting that server on the same or a different box (minutes). During this time, if you started receiving messages on a different server then it will receive the messages after the ones trapped by the indoubts. Oops. Out of order. When the transactions are recovered, it may be that the transactions will be rolled back as a result of recovery. Now, the old messages in use by that transaction become available again and are delivered to your client out of order.

Edge Conditions

This kind of requirements comes up a lot but it's complicated to get right as you should be realizing from the above. There are also interesting edge conditions. JMS offers a time to live on messages. If it's not delivered within 5 tries then the message moves to a dead letter queue. But, we want in order messaging. When this happens, do we suspend the queue until an administrator looks at the suspect message and tells the JMS server put it back or discard it? Poison messages would have the same impact. Then, when we want message order do we mean maintain message order per individual publisher or for the order the queue received the messages. What makes the order? Commit time or publish time? It can turn in to a rat hole pretty quickly.

Summary

I don't think people realize what 'I must have message order' means in terms of implementation. You will need application logic to do this correctly for now. I don't know of any product that can reliably do this at the moment. Anyway, it's a heads up.

When is a rollback a commit...

Thought I'd raise this up to the collective conscience. Heres a scenario. A JVM is using a cache in front of a database. It uses the database in one phase mode. It issues commits to the database. Lets assume the database crashes before it processes the commit. This is fine, the app server interprets this as a rollback and we rollback and inform the application.

Now, lets assume the database processed the commit but crashed before returning the ACK on JDBC to the application server. This looks the same as the previous scenario. So, the server rolls back and tells the application it rolled back.

Problem is, when the application reconnects to the database once the database restarts, the cache is now not in sync with thats in the database and general wierdness can now occur because it's not in sync.

Basically, the moral here is that whenever the application sees a rollback due to a stale connection (i.e. a connectivity problem or a DB2/Oracle reconnect) then it should flush/reload the cache or figure out whats the story when it gets a connection again because it's not clear which transactions that were in flight and rolledback by the app server actually commited on the database so the cache cannot be relied upon to be up to date.

Course, if your cache is optimistic then this doesn't apply but if its a write through cache expecting exclusive access then it should definitely be on your radar as a scenario to handle.

Faster optimistic locking with DB2

Just saw this for the first time today. DB2 allows tables to have a hidden column that is basically a 'last updated timestamp'. This column can be used to do overqualified updates when doing optimistic locking. But, an issue with this was that the application needed to read the updated row to get the new timestamp. DB2 has new statements that allow the updated columns to be returned from the insert or update sql statement. This basically halves the number of statements sent to the database when using this kind of mechanism for optimistic locking.
Read about it here  or google for data-change-table-reference

Tips for efficient frequent JMX calls to WebSphere.

Currently using JMX (through the wsadmin utility in WAS) to call JMX MBeans to query runtime state but it's too expensive to do frequently. Starting wsadmin takes too much CPU and it just takes too long to get the data.

My current workaround is to deploy a servlet to the server and the servlet does the JMX query or other commands against the MBean. The servlet parameters provide the necessary data for what I want to do.

I then use a perl script with the perl http client to invoke the servlet and get the answer. This is signficantly more efficient than using wsadmin for these scenarios. Typically takes well under a second to get the data versus 15-30 seconds using wsadmin, best case. I'll see if I can publish the code, it's pretty straightforward to write the servlet and the perl script is also simple.

Worth remembering if you need to do this sort of checking frequently, probably applies to everyones application server and not just WebSphere also. A perl script simply starts a hell of a lot faster than a JVM.

OSGI, not just for clients.

Andrew Binstock recently published an informative article on the uses of OSGI for clients. Andrews article can be viewed here. It's a good introduction on OSGI and you can find another good introduction at this link on OSCAR-OSGI at source forge.

I think OSGI also has a big future on the server side. If we could get the open source and commercial application server vendors using OSGI for their runtime frameworks for servers then a lot of interesting things could happen. Rather than the current situation where every vendor has its own runtime component mechanism and the open source ones are no better here (JBoss with JMX, Apache with GBeans, ...), if we can all agree on OSGI than third parties making plugins for application servers such cache providers, persistence providers should have a much simpler time integrating those products. Maybe products like Tangosol or Gigaspaces could adopt an OSGI based runtime.

Application programmers would benefit from having a standard service registry in which to find these third party plugins. Application servers would be easy to 'upgrade'/integrate with third party function. It has a lot of potential, if we can just get the vendors to agree. I think a positive initial step would be for the open source J2EE servers to adopt OSGI as their runtime framework in lieu of their current private runtimes.

Another thing to think about is how could OSGI impact J2EE, could web modules/ejb modules be bundles. Would this be easier or harder than today. Could JCA RAs be bundles also? Lots of scope here.

If you want to play with OSGI then an OSGI 3.0 is included in IBM Workplace Client Technology, Micro Edition 5.7.1 and this is available for download. An OSGI V3++ is included in Eclipse also.

WebSphere Partitioning Facility (WPF) Programming guide online

The PDF guide to writing J2EE partitioned applications using the new WebSphere Partition Facility (WPF) APIs is online now at this link. These APIs are part of WebSphere Extended Deployment (XD) 5.1.

WebSphere XD ships

We just shipped WebSphere 5.1 XD a couple of weeks ago. XD has many features that should appeal to customers.

The OnDemand Router (or ODR) is a Java proxy server that sits in front of a set of HTTP servers. These HTTP servers can be WAS servers or servers from other vendors such as BEA as well as servers on the LAMP stack (PHP etc). The ODR can 'shape' the traffic on to those boxes so that service level agreements can be met. When used with WebSphere XD servers it can also supports dynamic clusters. A normal cluster is a set of nodes running the application. A dynamic cluster is an application deployed on to a node group. The node group is a set of nodes on which the application can 'potentially' run. Multiple applications can be deployed to the node group. The ODR needs to be configured with a service level agreement for each application. It then decides based on how well the application is meeting it's SLA, how many servers in the node group will run the application. If the application is too slow then it will start the application on more nodes in the node group. If all nodes are currently being used and an application still cannot meet the SLA then lower priority applications will be stopped to 'make room' for the higher priority application. It's pretty cool.

XD also has the new WebSphere Partitioning Facility (WPF). This allows a J2EE application to split it self in to named partitions. A partition is like a named daemon running in the cluster. They are declared programatically by the application at startup and can also be added/removed dynamically at runtime. The partition runs inside the app server. IIOP work can use content based routing to route an IIOP request to the partition that can service it. This allows J2EE applications to have a singleton running in a cluster and then have requests for that singleton routed to it.  JMS requests or requests from middleware like Tibco RV or Reuters Connect can be similarly routed to the server handling that particular data stream. A partition can use threading to dynamically subscribe/unsubscribe to just the feeds that the partition requires.

If a cluster member fails then all partitions running there are failed over to the surviving cluster members very quickly. How quickly depends on tuning but 6 seconds isn't unreasonable.

Partitions can be grouped together using an application grouping scheme and then those groups can be placed on particular cluster members. This allows 'busy' partitions to be placed on more powerful servers or fewer busy partitions can be placed on a single server. Backup servers for these groups can also be specified and whether failback is desired if the original server restarts. This placement logics lets a rule based mechanism be used to determine simple placement. 5-10 rules can manage 9000 partitions in a cluster.

This lets XD be used for very demanding applications such as electronic trading systems or highend batch processing as these applications are typically stateful and which only scale vertically or don't perform well as a stateless server. WPF allows customers use partition such applications and then each partition can be stateful and runs on a cluster member. If the member fails then the partition fails over to another cluster member. WPF provides failover/restart events to the application so that it can react to these events. When WPF is combined with the async beans features from WBI then some pretty serious applications can now be built that leverage the J2EE environment and stateful applications can be made to scale almost linearly horizontally and still be highly available through our failover mechanism thats built in.

WPF and the high availability manager can also integrate with BladeCenter hardware. This allows WebSphere to make sure that a server is really dead if becomes unresponsive before WebSphere actually fails over the partition. This allows WPF applications to be written and not have to worry about a partition running on another server when unexpected events such as a network partition or split brain syndrome occur.

Anyway, it's pretty cool and it's exciting working with some of the big Wall St firms on projects exploiting these features. Enjoy. For more information, check it out here.

Problems with JMS 1.1 and J2EE 1.3 app servers

I wish the JMS vendors would take care and ensure that their clients don't depend on a JMS 1.1 jar being in the class path. For the next 2/3 years, I think most of the deployed servers (i.e. the ones JMS vendors can sell to) will be at J2EE 1.3.

It seems a lot of the JMS 1.1 products do and as a result don't run out of the box on J2EE 1.3 servers as they only bundle the JMS 1.0x jars and classes. The JMS classes are unfortunately inside the J2EE.jar and thus are kind of hard to get at for most people, I think.

The workarounds currently are to expand the j2ee.jar and then unjar the jms 1.1 files in to it and then rejar. This would work from what I can tell but from a support point of view, a J2EE vendor may punt and claim the configuration isn't supported. It may also be possible to simply add a JMS 1.1 jar to the classpath and hopefully it'll find the extra classes but I haven't tried this.

I guess the JMS JCP should have gone out of their way and insisted that a JMS 1.1 compliant JMS product is required to support JMS 1.0.2 clients, i.e. a J2EE 1.3 server.