/dev/websphere

The dangers of virtualization and datagrids

Virtualization offers quite a few advantages for customers running large data centers but care needs to be taken when using memory replication technologies like DataGrids. The particular problem here is that each virtual machine looks like a real server as far as middleware running within that virtual machine. Each virtual machine has its own IP address for example. If you are keeping a primary copy and a backup copy of critical data in a DataGrid then it's essential that those copies have a few common failure modes as possible.
The first most obvious one is simply make sure they are in different processes. This is quickly followed by make sure they are in separate servers. This is where virtualization gets you in trouble. A DataGrid might mistakenly place the primary and backup copies in two virtual machines running on the same physical server. This clearly is dangerous. A single hardware failure and you'll lose both copies.

WebSphere eXtreme Scales zone support can avoid this problem. Assign each physical server a zone id. A zone id is simply a string. Make sure each JVM in virtual machines running on that physical server uses the same zone identifier. WebSphere eXtreme Scale will then automatically only place copies on a different physical zone from the one containing the primary copy. This neatly avoids the problem as WebSphere eXtreme Scale handles it automatically once you tell it the zones.

This also highlights some of the dangers with virtual machine migration. This would cause a problem here as moving the virtual machine wouldn't change the zone identifier in use automatically. Virtualization is a new technology and many of the supposed features that make it cool take a very simplistic view of the requirements of software running on those virtual machines. They assume they can add/remove CPUs, memory, move boxes without impacting software running on those virtual machines. This is a naive view of the world unfortunately and many customers are believing the hype from the hypervisor vendors. Be suspicious of these claims, if it looks too good to be true then it usually is.

Extreme Scale V6.1.0.3 ships today

This has been an incredible amount of work but it's finally shipped today. I'd recommend all customers look at moving to the new code level as it contains many many functional and non functional improvements. The following covers the major features added in V6.1.0.3.

Productized Write behind support
Enabling write behind support is now just a flag on the BackingMap definition. This causes all changes to a Map to be buffered to replicated buffer when the application commits the transaction. The changes are then flushed through the Loader periodically or when a certain number of changes are buffered. This can reduce the load on the backend significantly as well as improve availability if the backend is down.

Database integration through Hibernate or OpenJPA to provide the actual POJO mapping to the underlying database
Two builtin Loaders are provided. A Loader for Maps and a Loader for Entities. These Loaders use Hibernate or OpenJPA to fetch missing data and write changed data back to a database automatically. Both loaders provide a preload capability which easily allows the grid to be loaded at startup with the data.

SCN support for easier stale data invalidation
This uses the SCN features of Oracle 10g and DB2 9.5 to easily remove stale data from the grid when data in an associated database changes.

OpenJPA and Hibernate L2 cache plugin support.
L2 cache providers are provided for both OpenJPA and Hibernate. Both L2 providers support preloading the data in the cache.

Memory based eviction
This starts evicting entries from memory when the JVM heap level rises to a certain percentage. This is a very convenient way to manage how many entries to keep in memory.

Spring namespace support
We added a variety of spring name space handles to make using Spring to manage our JPALoaders or MapLoaders and evictors. We also added support for shard scoping which allows Spring applications to create an instance of a bean for each partition within a JVM.

command line monitoring utility, xsadmin
This utility allows the state of the grid to be examined easily from the command line. It's provided as a sample with source code and also serves to show how the information displayed is pulled from our JMX MBeans. It allows the membership to be examined, see which partitions are on what JVMs, etc. We'll be adding to this rapidly over the next months.

You can download the fixpack from here.

I'm going to make a short video outlining the above features which will post shortly.

Speaking at DB2 Information on Demand Conference in late October in Las Vegas

I've gotten accepted to speak at the IOD conference during Oct 26-31 in Las Vegas on WebSphere eXtreme Scale and DB2 integration.

Blog entry on database sharding

Check it out here.

Video interview on fractal virtualization, xtp, datagrids on infoq with myself

The link is here