/dev/websphere

Configuring DataSources with DataGrids that front end Databases

I'm seeing this a lot these days. A customer is using IBM WebSphere eXtreme Scale in front of a database. The DBAs are concerned about how many JDBC connections that the grid is requiring. If it's a larger grid then the problem is especially acute. For example, lets say we have a grid with 200 JVMs in it. All of the grids that I have sized recently are in this range. Lets look at different aspects of this problem

1) WXS as side cache

Here, there are no connections from the grid to the database. The application server JVMs do all the database interaction and hence, the grid adds no additional demands on the DBMS.

2) Preloading WXS with data from a DBMS

Here, preloading it can be interesting. The main issue is usually the time to load the grid with the DBMS data. We usually use parallelism when we do this but that means multiple JDBC connections per preload client. The main issue I see with DBMS preload is simply query complexity slowing down the preload process. Simplifying the query and moving join logic in to the grid side preload process seems to be the key to solve this problem. Once you can achieve a high preload speed per single preload thread then you don't need so many JDBC connections to achieve a specific preload speed.

3) Write through WXS to a DBMS

This is probably the worst offender here. Typically, a WXS Loader consists of a TransactionCallback (like this JDBXTxCallback example) and a Loader. The TransactionCallback makes sure a JDBC connection pool is available for the Loader and the TransactionCallback manages JDBC.begin and JDBC.commit and rollback operations. The Loaders share the connection provided by the TransactionCallback for a specific transaction.

Clearly, preloading the grid will enormously reduce the usage of JDBC connections for grid read operations. However, write operations will still consume one JDBC connection per concurrent WXS transaction. Given, the potential throughput performance of such a grid, clearly, the load is usually too much for most databases. A grid server with 20 JVMs on a modern blade can still achieve 20k transactions a second per blade. Lets start with a grid with a 100% write rate for some peaks. Thats 1000 write transactions per JVM. If each write transaction costs 40ms (remember, the DBMS is directly involved with writes here, this isn't the grid, it's the DBMS) then we could do 25 transactions per thread per JVM (1000 / 40ms). Now, how many threads is that? We'd need 40 (1000 / 25 ops per thread) threads per JVM to achieve this amount of throughput in a single JVM. So, the JVM needs a JDBC connection pool of 40 connections (one per thread) too.

If there are 200 JVMs then we need 8000 (200 x 40) JDBC connections for the grid! Now, we can see why the DBAs may be concerned. A write through grid thats write intensive is a great idea for a grid. But, only with write behind. Write through makes NO SENSE AT ALL for this situation. The DBMS will remain the bottleneck and the customer will likely question why use a grid at all if such a solution with write through was proposed. Remember a 10 blade grid with 20 JVMs per blade and a 100% write rate for some workloads would get you here.

So, write through can still use a lot of DBMS connections if the write transaction percentage is high. If it's lower then these numbers can all be revised downwards. For example, if it was 10% then lets do the math again. One box does 20k tx/sec. 10% is 2000/sec. There are 200 JVMs then thats 10/sec per JVM. One or two connections in the DataSource would likely be enough here depending on response time requirements. This gives us 200 JVMs x 2 connections or 400.

It's very important to share a DataSource within a JVM, do not give each JDBCTxCallback it's own DataSource. The example JDBCTxCallback provide in this blog entry (see the link above) accepts a DataSource reference when it's initialized. You can use Spring to wire it in or change the JDBCTxCallback to get one from JNDI when there is an application server hosting the grid containers. This way, even though there is one JDBCTxCallback PER primary shard in a JVM, at least they will all share the same DataSource instance. This would reduce the connection usage dramatically. This is usually only a problem when used a J2SE hosted WXS container. Customers using WXS inside WebSphere ND will likely be using WebSphere's DataSource implementation and this will naturally result in one DataSource per JVM.

4) Write behind WXS to a DBMS

This approach gives the best opportunity to cut JDBC connection usage. Assuming, we preloaded the grid then the connections for reads can likely be ignored. This leaves writes. Write behind decouples the writes to the database from the grid writes. This means that one WXS write does not mean one DBMS write. This was the major problem with write through.

WXS allow DBMS writes to be triggered every X 'dirty' entries in a Map or when 'dirty' entries in a Map are older than a certain time (5 minutes for example). This means we would see much fewer database writes than grid writes. If we have a grid JVM with 5 primary partitions in it (the normal amount when sized correctly) then there are only 5 potential concurrent writes per second in such a JVM. Each partition will send its changes to the DBMS using a single thread. Therefore, the maximum JDBC connections used for write behind is the number of primary partitions in the JVM which in this case would be 5 PER JVM.

Note, that when boxes fail then the # of primary partitions can be more. Thus, size everything with what you expect to be your minimum # WXS container JVMS to figure this out. It's more likely to be 8-10 primaries in this case. Thus, we'd expect 10 JDBC connections per container if the system was extremely write intensive. However, a very high write rate is unusual. Normally, the write rate is much lower (10-20% might even be high). But, a security system built with a grid that updates last login time for example might result in a WXS write per login so be careful sizing this.

We expect in the worse case scenario with write behind to need one JDBC connection per partition. However, this is only the case when every partition writes at the same time. The big factor here is how long does a write behind transaction take against the database or the 'duty cycle'. If it takes 1 second and we're doing one write behind every 5 minutes then we can likely use one shared JDBC connection in a DataSource per JVM, thats a duty cycle of 1 : 300 (300 seconds in 5 minutes). We are unlikely to get much contention for that one connection as there is a 1:300 change of writes from different partitions clashing within a single JVM. If each write behind tx takes 2 minutes then this clearly changes things given there are 5-10 primaries per JVM. 2 minutes would mean a duty cycle of 120:500 or about 25%. You'd need probably one per primary partition here or 10 per JVM.

But, regardless, it's clear that write behind can reduce the DBMS connection usage significantly when the write behind configuration is 'optimal'. Optimal depends on the situation but as a starting point, 5 minutes or 1000 records (T300;C1000) isn't a bad place to start. The size of the DataSource pool for each JVM depends on the duty cycle of the write behind within a single JVM.

The points on sharing a single DataSource within a WXS JVM equally apply here as to write through.

Summary

I hope this blog entry opens peoples eyes in terms of JDBC/WXS interactions. Most applications elect to use a grid because the DBMS is unable to provide adequate response times under peak loads. If the write rate is very low then there is no problem. But, when the write rate is high then write behind is usually the best scenario here. A high write rate application using write through is likely going to still have performance issues as the DBMS will prevent the grid from achieving linear scaling and good response times.

A quality DataSource implementation is a big deal here. The DataSource thats provided with WebSphere Single Server/Base and WebSphere ND is about as fully developed as exists out there. It is a significant improvement (in my humble opinion) over C3P0 or Apache dbcp. Thus, running WXS inside a single server WAS install is something you could consider instead of a J2SE based WXS implementation for this reason alone, the DataSource implementation.

Javascript based grid operations client on github

I get asked a lot if WebSphere eXtreme Scale includes a utility to examine the contents of a grid and/or allow the grid to be manipulated. I've uploaded the first version of such a utility to my github repository today. 

https://github.com/bnewport/Samples/tree/master/jsoncrud/

I'll add the README from the project here:

This is a command line utility that allows you to work with a remote grid and examine/manipulate it. It's built around a JavaScript shell. When you start the utility then you specify the grid you want to connect to:

> jsonclient.sh -ch localhost -cp 2809 -grid Grid 

I'm assuming you've created a jsonclient.sh script to call "com.devwebsphere.jsoncrud.WXSShell". This is the main class. This will then connect to the grid and start up a JavaScript shell like this: 

Rhino 1.7 release 2 2009 03 22

js> 

At this point there is one global variable defined called 'grid'. You can use this to get instances of Maps in the remote grid. For example: 

js> personMap = grid.getMap("Person", "java.lang.String", "com.billy.Person")

 This stores a Map object instance for Person in the personMap variable. The map is stored remotely in the grid and uses a String for the key with a value using the class com.billy.Person

Lets suppose your Person looks like this:

public class Person implements Serializable 

{

String firstName;

String middleInitial;

String surname;

Date dateOfBirth;

double creditLimit;

...

}

 

The class would have getters and setters define for the attributes. You can insert a Person entry using this:

js> billy = {"firstName":"Billy", "surname":"Newport", "middleInitial":"Z", "creditLimit":"100.0"}

js> personMap.put("bn", billy)

 

This creates an entry in the map with the key "bn" and the value as shown. The value in the remote Map is a Person class instance. The JSON string above needs to have the same corresponding attributes as the Person POJO. We can modify the entry so have a different credit limit like this:

js> p = personMap.get("bn")

js> p.creditLimit = 200

js> personMap.put("bn", p)

 

We can remove the entry like this:

js> personMap.remove("bn")

 

We can invalidate the entry like this:

js> personMap.invalidate("bn")

Preparation Instructions

Add the jars to the grid classpath. Add a map called "Router" to your mapset. It is just used for routing, no data is stored in it. The client doesn't need any application classes. Once this is done then you should be able to connect to the grid and work as described above

 

 

Why you shouldn't overcommit memory on virtualized servers running Java

Virtualization takes multiple physical machines, virtualizes them and then runs them concurrently on a single server. The argument is that they are normally using little CPU so it's ok to run them all on a single box which will then run with a much higher CPU utilization, say 70/80%. The main issue is memory. Most boxes which are targets for consolation are usually older servers with slow CPUs. 5% on such a box might be 1% on a modern box. This means that many such older machines can be consolidated on to a newer more powerful server. However, while CPU is usually plentiful and cheap, the real issue is memory. That much memory can be expensive. DRAM cost goes through the roof when larger capacity DIMMs are required. This means that virtualization vendors typically recommend over committing the server memory. This basically means they are recommending swapping at the hypervisor level. This dramatically cuts the amount of memory required and lowers the TCO of the solution correspondingly.

C programs or scripts tend to respond well to paging. They typically have small working sets (active memory pages) and then rest of the process or virtual machine can be paged out without dramatically impacting the performance of the virtual machine. They typically don't regularly sweep the contents of their address space. Java, however, is a completely different animal. Java Virtual Machines only have one working set size, the whole virtual machine heap. The objects used by a Java application are typically randomly spread all over the address space of a process. Any timers which fire can cause much of the JVM to be paged in by the hypervisor and if a garbage collection is triggered then the whole JVM heap must be paged in from disk as the garbage collector sweeps ALL the memory, a very slow operation. This page in will cause other virtual machines to be paged out then they garbage collect and we get a ping pong effect. This behavior can cause major variances in response times for users of the applications hosted in the JVM. A garbage collection which normally completes in hundreds of milliseconds might take minutes to complete in such an environment and the applications within that JVM are frozen during this time. Many Java applications utilize timers for JDBC connection pooling, time outs, heart beating for cluster management and so on. The paging activity wreaks havoc on these kinds of time sensitive operations and will cause most Java applications to behave very erratically if not fail completely. Any kind of clustering framework will not survive a paging environment due to the timing sensitive nature of heart beating and similar code.

Java Virtual Machine technology needs to be improved to that it can cooperate with the host operating system or hypervisor to minimize its working set which should allow it to be paged more efficiently but garbage collection will likely still remain a tough problem to beat in an environment that is actively paging. Any virtualization vendor will struggle to keep a customer happy in an over committed memory virtualized system. These systems are frequently sold to customers with the assurance that over committing the memory is ok but this usually results in the customer being unsatisfied and then complaining to the Java application vendor about poor performance or unstable behavior. The response to these questions will likely be "Don't overcommit your memory!". The response is usually "But, the virtualization vendor said it was ok otherwise the system becomes very expensive…" and the final response from the application server vendor will be "Yep, thats right…"

Solid state disks for paging might help with the situation. But, any stateful application server with timing sensitive code will likely still encounter some issues in such as environment because of the jitter introduced by paging. WebSphere includes code to check for such jittery environments. It's a simple alarm which fires periodically and simply checks that it is fired when it expected to be fired. If the timer discovers that its several seconds later than it expected then it logs a message indicating CPU starvation is occurring. This only happened in the past on machines whose operating systems were paging OR on machines running hundreds of JVMs which suddenly all get busy and overwhelm the CPU capacity of the server. Imagine 500 threads that are all CPU bound running on a 8 core box. If each thread runs for 200ms when it gains the CPU then one second can service about 40 threads. This means around 12 seconds will elapse for any given thread before it even RUNS again. This is unacceptable for any timeout style operations with resolutions lower than that. We are seeing customers now reporting problems in virtualized servers where they have overcommitted the memory and while the local operating system is not paging, the hypervisor is. The result is the same. Erratic behavior.

In summary, this kind of behavior is not new. It's been seen before on systems which page or are simply overloaded. Either way the system was either out of CPU or out of memory or BOTH. Customers running virtualized systems need to be very careful when overcommitting either resource if predictable application behavior is required.

Lock free data structures, good and bad

I'm playing with making some lock free versions of sync blocks in code I wrote before. The code is currently protected by either sync blocks or read/write locks. Lock free approaches avoid sync blocks altogether and so in theory offer better concurrency in todays multi-core world.

Lets examine what exactly lock free means. Typically, it means having an atomic reference to a shared read only data structure. Threads can get a reference to the data structure and access it without fear of other threads because the data structure is read only.

If a thread wants to change the data structure then it must copy the current data structure and only update the copy with the new changes. Then, it basically does the equivalent of an optimistic database lock. It uses a compare and swap operation which will replace the atomic reference to the data structure with the new one only if no other thread can also modified it since the updating thread read the original data structure.

If the compare and swap succeeds then all is good. If not then the thread must get a reference to the current data structure and try to make the change to that one and then do compare and swap again. It has to keep trying to do this until it succeeds.

Clearly, first of all for read mostly data structures this approach is pretty cool and is a LOT better than a read/write lock. If writes are more frequent then contention will start to occur between writes as they compete to get to the compare and swap first. Frequent writes would also mean frequent copy operations. Once per loop try per thread. If you have a lot of contention AND the data structure is expensive to copy then this is going to burn a lot of CPU in those copy and update loops.

Java 5 has support for this type of lock programming. Look at the AtomicReference class and its compareAndSwap method. The trick to using it is NEVER update a data structure, always copy and then either update while copying or update after copy. Then try change the AtomicReference to the updated copy using compareAndSwap and if it fails then try updating the data structure that the AtomicReference now points to.

Free tools for analyzing heap usage, winner is Eclipse Memory Analyzer

I've been analyzing some heaps lately and am still very disappointed at the tooling for doing this kind of thing in Java. The first step is creating a heap dump. I am using a Sun Java 6 JVM so I used jmap to dump the heap.

The file was pretty big, 650MB. Next step is analyzing it. First attempt was to use jhat in the Sun JVM. This is pretty much useless. It just lets you walk the heap but I can't figure out how to get it to tell me which objects have the most instances etc. I don't see what that is useful for as a result.

Next was the IBM Alphaworks heap analyzer. First issue there was just loading the file. We failed to load it on our Laptops (4GB). Next I used one of my Mac Pros (32GB RAM) using a 64 bit JVM with a 16GB heap and now it loads but it was pretty buggy and was spitting exceptions in the console like nobodies business. We had a lot of trouble just getting it to respond to right clicks for displaying parents etc. So, this isn't great either.

Next, we downloaded the Eclipse Memory Leak Analyzer and this was the best of the lot by far and is what we're using now. You download this and it runs as it's own private eclipse instance. It loaded the file with no issues using defaults so this works on a laptop. It lets you get at all the information and doesn't spit exceptions at you so this is a huge (HUGE) step up on the Alphaworks heap analyzer. This is what we're using now for checking heap usage dumps.

We prefer a dump a file approach rather than live checking with commercial profilers just because it seems to be more stable. Attaching profilers seems to usually cause a crash etc so dumping the heap and then examining offline seems more useful.

SpringSource application server, very nice, shame about the license

First, hats off to Rod, Adrian and the others. It's very nice and very much in line with where most people see this all going in the next few years.

It looks like what others and I have been planning/hoping to do over the next few years. Most of us are looking for an OGSi based distributed platform with a commercial friendly license (EPL, BSD or Apache). I don't think there is an appetite to develop a proprietary OSGi runtime in any camp right now and IBM, through eclipse, has been placing a very good OSGi implementation into real open source as Equinox, i.e. a commercial friend license, no dual licensing fooling around. WebSphere already has an OSGi enabled distributed runtime. I don’t see much of an appetite to develop another one that does almost the same thing as before, I think we’ll use something rather than build it.

The platform should allow middleware or profiles to run on top of it like JavaEE, WebSphere eXtreme Scale, ESBs, Event processing, process engines and so on. The platform should be distributed, i.e. allow me to install a bundle set or profile on a set of machines matching a filter (this is a cluster) and then start/stop machines, do distributed monitoring on it also. Stuff like Wily or JXInsight is pretty cool for monitoring given they don't require code changes as they use aspects to monitor existing code. I'd like to run Oracle and IBM profiles on the same platform at the same time.

I see this as the new JVM, a module or bundle oriented runtime that’s also distributed. I don't want to see this runtime controlled by one vendor and licensed, I'd like to see it available for free as a community project with a commercial friendly license (Apache, EPL, BSD). Most vendors have absolutely no interest in a GPL type license. IBM has already contributed a lot of work towards this end in Eclipse.

I know SpringSource are doing valuable work making OSGi consumable by the mass market. OSGi is too hard for most people. Spring-DM is a great step in the right direction and I think Adrian is standardizing some, if not all of this, in OSGi anyway.

I think the commercial vendors see the value as not in the platform but in the profiles running on top. I expect to see a commoditized runtime for hosting profiles or middleware/applications. Springsource for now seems to be treating the platform as the valuable thing and I just don't see that. Of course, this is what they have so it's valuable to them. IBM and other vendors have a library of profiles like messaging, process flow etc that is why there is a different perspective. I understand why they are doing it but I don't see it being widely adopted by the large commercial vendors. IBM has already contributed a lot of code in Equinox etc under commercial friendly licenses and this will likely continue (I don't speak for IBM in this blog).

I’d like to see monitoring of this also pluggable and I don’t see a need for APIs here. What I’ve seen of Wily and William Louths JXInsight has convinced me that we do not need APIs to do monitoring. Aspects are hands down for Java, the way to go and SpringSource has tackled getting AOP to work in an OSGi environment and hopefully that gets standardized also.

I think monitoring will be a valuable profile on top of the common runtime and I fully expect vendors to be selling this to monitor what-ever people deploy on this infrastructure.

So, to summarize, I don’t see the platform as the valuable thing. I see it as a commodity. I see profiles and monitoring profiles as the valuable thing and I’d like to see a commercial friendly licensed OSGi distributed runtime as the new JVM that vendors build middleware/profiles for. Given, Spring DM is Apache licensed, I can see the extra work in the Spring server being clean roomed and made available with EPL or Apache pretty soon and this will limit the value from selling the SpringSource server. Duplicating higher value profiles like process flow etc is clearly not so easy and this is why these are the valuable things that vendors will continue to charge for. I don't want to trivialize what SpringSource has done, it's very cool and it's needed but given most of its components are Apache 2.0 or EPL then the last gap is a lot simpler than building a Java EE or BPEL flow engine, thats all.

Will conversational flow engines and AJAX help with the coming multi-core problem?

A lot has been said about the coming issues as multi-core processors become mainstream if that hasn't happened already. Processors are getting more cores but individual cores are getting slower. This means that applications will need to be written in a parallel fashion to really take advantage of this and the majority of people will find that difficult, it isn't easy to do.
I've written quite a bit about this but I was thinking yesterday about the impact of all and AJAX and conversational state engines might just a a solution for a lot of web applications.
AJAX basically changes the way web applications are written. Pages before AJAX were monolithic. A single button resulted in a servlet processing the information a page at a time. A page might have had quite a bit of information on it with a lot of fields to validate against a database or caches etc. All of this logic would run on a single thread in the web container. Customer could multi-thread it but that wasn't easy.
AJAX changes this because each field can be validated 'live' using RPC calls to the server that process fields at a time. As you type the various fields on the page, each field can be validated separately using it's own RPC call from the page to the server. This results in many more server RPCs per page than pre-AJAX days but each RPC is much simpler than before also. This allows us to process the page using more smaller RPC and these RPCs can potentially overlap depending on how quickly information is filled in or how the page does things. Breaking up monolithic logic in to potentially parallel streams of smaller units of work is a good recipe for success on todays multi-core processors. Yep, there is more overhead because of the RPCs but it's probably a good trade off, those MIPs have to be used for something, right?
JavaEE does very little for people having to parallelize applications to run well on newer processors. But, Web 2.0/AJAX style applications and the new wave of POJO conversation centric flow engines may provide the building blocks to allow the masses to write parallel code without even realizing it. Conversation centric flow engines should also allow interactions with backend systems to be parallelized also in a natural way thats coded in the same way as people code web flow conversations using frameworks like Spring WebFlow. I can see how as the conversation flow engines like WebFlow evolve then they can also be used to parallelize many tasks in a familiar way and products like ObjectGrid can make that flow reliable and fault tolerant. By evolve, I mean, the conversation needs to be made independent of the web context and instead just be usable in a web context or any other context such as pulling state from several back-ends in parallel and blocking waiting for the results. Conversations may just become the next programming style and this should naturally allow parallel applications to be written easily/tooled and debugged. The current work on closures and conversational state need to be integrated also, lots of similarities and potentially programming model simplifications should be possible by combining both of those.

ActiveCluster and the problem with open source

First, check out the thread on TSS http://www.theserverside.com/news/thread.tss?thread_id=32659. This effort looks like the beginning another overlapping OSS project. And it appears solely due to the fact that JavaGroups uses an LGPL license. IMHO, the Apache license is much more friendly and I'd like to see Bela drop LGPL and switch to an Apache 2.0 license for JavaGroups.

Don't get me wrong, ActiveCluster is a good idea. We have the same thing but more developed inside WebSphere XD and WebSphere 6.0. But, the implementation of this is going to be complicated and take a lot of time because it isn't using the already existing group services provided by JavaGroups. The codehaus posters on the thread say that it works with JavaGroups but only by using JavaGroups as a simple reliable pub/sub transport, not using it's group services which is the real work in making the APIs work.

ActiveCluster will probably eventually implement most of the group services within the JavaGroups protocols with a great deal of pain and debugging. It appears that codehaus will reinvent the wheel because of two different licenses. Rather than making a very thin veneer on top of JavaGroups for a simplified cluster API, codehaus will likely try to implement their own group services layer simply because JavaGroups uses an LGPL license rather than Apache 2.0 and I don't think OSS is any better for it. But, although there are a lot of overlapping OSS projects, it's probably a good thing as there is nothing like competition to find whats best.

In search of wire speed Java serialization

Been playing with very high speed Java comms recently and found this paper with some interesting information and performance metrics on Java serialization. Externalization is like 10 x serializable but thats no surprise. Whats cool in the paper is that it describes a myrinet integration serialization mechanism that is over 10x externalizable which is pretty cool.

J2SE Timer, lets deprecate it.

On the whole, the J2SE Timer looks like a good idea. Who wouldn't want to have a lightweight timer service. But, it's a problem for J2EE or managed server vendors because it wasn't designed correctly.

The problem with it is that it's an implementation rather than a set of interfaces. The J2SE Timer makes its own threads, and assumes that the TimerListener can be just called on that private thread.

When I did the WAS-E AlarmManager, I realized that the listener needed to be called with the same J2EE context as the creator of the alarm. So, if user B running on component C registered the alarm to expire in 50ms, then when the alarm expires, the listener also needs to be invoked as user B with the J2EE environment for component C setup.

I could have used an EJB for the listener but then what about servlet users, why force them to use an EJB? So, servlet customers can use JavaBeans as their listeners and when they are invoked on these asynchronous events, then they have access to the JNDI from the owning webapp, the security credential (if present). The JNDI is important because when an application looks up databases, ejbs or web services then it does this through JNDI. If the JNDI for the creating component was not available then the absolute JNDI name would have to be hard coded (or property) in the application, this defeats the reasoning behind the whole ref thing.

The J2EE 1.4 Timer Beans are similarly flawed as it's too EJB specific. If we'd have an interface based J2SE timer API then arguably it could have used it also. So, timer beans will require servlet customers to now use EJBs to gain access to those services. Now, timer service is a transactional scheduler but the point is it's APIs are very simple and arguably the session bean could have just extended the TimerListener interface or something. Also, the EJB gains access to the TimerService through EJBContext and I'd prefer to have the EJB use the resource-ref mechanism like I did with WorkManagers and the spec does with DataSources/ConnectionFactories. But, it's here, it ain't changing easily.

So, if WebSphere wanted to provide a lightweight timer service to J2EE applications that the J2SE timer APIs are just not suitable because they are an implementation which assumes an unmanaged environment and worse, they are an implementation and not an API so we couldn't even reuse the interfaces and hence, this is why I implemented the AlarmManager interfaces in WAS-E.