/dev/websphere

"Penny wise and dollar foolish" applies to distributed caching too often

Many times I've had people compare the performance of WebSphere eXtreme Scale to a local hash map and comment that it's slower. Basically, they are comparing a remote hash map call to a local hash map call and yes, the remote one is much slower. But, this is an apples to oranges comparison. Try caching 600GB in the local hash map on every JVM requiring a cache? But, hey, it's faster, right?

Wrong. If you just need a local hash map then you should use one. A distributed cache like extreme scale brings expandible capacity, high availability of data in the cache, a network accessible remote very large cache, cache stays up when the application jvms are cycled and so on. It's different and thats why the fact that a local hash map is faster isn't really valid.

For a real world example, lets look at WebSphere Commerce Server. It can cache HTML page fragments in a local map (Dynacache) and in extreme scale also. If the cached data in present in both caches then the local map is faster but, that's not why we sell extreme scale to commerce customers.

An extreme scale cache is remote. You can cycle the commerce JVMs and when they restart, they are working with a hot cache straight away. This isn't the case with the local cache. It has to be regenerated and this costs of lot of CPU. This regeneration also hits the database during this time. So, we have extra costs of CPU, longer response times and database load if we cycle a commerce JVM and have to rewarm up its page cache. None of these problems exist with extreme scale as the page fragment cache because the cache contents isn't kept in the commerce JVM at all and therefore it survives a commerce JVM restart.

Now, imagine bringing down a box with 10 JVMs on it. With the local cache, the cost is 10x when you restart the box. With extreme scale, you has the cost of restarting the JVMs but no cache warmup costs at all as the cache survives the JVM/box cycle. It's ten times faster because those ten JVMs do not have to warm up their ten independent page fragment caches again or hit the database and the customers using that box see "normal" response times pretty much as soon as they are online versus waiting for the JVM caches to warm up in the conventional dynacache scenario.

Next, lets look at invalidation. If a large site had 100 JVMs and I update the price of an item, then all 100 JVMs need to regenerate the page fragments for that item every time this happens. They all burn CPU, the response time for serving that item is longer and there is database load. With extreme scale, the page fragments are invalidated but are only regenerated once, the first commerce server to request the page notices that it's not there, generates it once, puts it in the cache and now the other 99 JVMs have the new page even though they didn't generate it. This is a massive load reduction when prices or inventory levels are updated or promotions for product categories become active potentially impacting tens of thousands of skus in the system and the page fragments for those impacted skus. This means you can invalidate prices to match competitive prices in a more timely manner without worrying about high CPU spikes or database spikes. This is a major advantage for extreme scale with commerce server over the dynacache implementation.

Lastly, if each JVM has its own independent cache then potentially depending on which JVM a request is routed to, a different cached page fragment could be served, if pages are being invalidated aggressively then the site becomes a bit of a mess in terms of consistency of what users see across the cluster. WebSphere eXtreme Scale is a single logical cache shared by all commerce JVMs, they don't have a private local cache, they always go remote to extreme scale for page fragments. This means they are always consistent and a user will see the same cached page fragments no matter which JVM they are routed to.

So, when examining whether a customer should use websphere extreme scale with commerce server, it's important to focus on the massive response time, cpu savings and database savings benefits around edge cases, not steady state performance. The steady state CPU load of a commerce cluster with no JVM cycling, additions or data invalidations is likely to be a little higher with extreme scale versus a local dynacache. This is expected. If you have a completely static commerce site then dynacache with disk offload is probably a better solution. But, and I expect this is most customers, the content of the site changes and you want:

• users to see consistent page contents
• you want consistent response times if you need to ripple commerce JVMs or add JVMs
• large cpu and database load reductions when prices/inventory levels or promotions change
• more consistent response times means more satisfied customers versus a web site where depending on whats going on (edge cases), customers see widely variable response times.

Then extreme scale is the way to go, hands down.

Chatty cache syndrome and solutions

There are a lot of applications out there that already use local caches like the many open source ones as well as things like Dynacache. These caches are basically local hash maps in the same address space and effectively have zero access time and cost. Cache misses are zero cost and cache hits/updates are practically zero cost also.

Applications faced with this have been written to expect this speed, i.e. zero cost. This means some applications are written to be extremely 'chatty' with the cache during individual requests. The caches tend to be very fine grained and a single application request may request or update hundreds of key/value pairs from the cache. When the cost of each operation is practically zero then this isn't a problem at all.

The trouble starts when you try to plugin a distributed cache. These caches are typically remote. This means all cache interactions involve an RPC and involved serializing the keys and values to wire format and possibly back again and thus are far from zero cost. Comparing the cost of an RPC with a local hash map lookup, it's a pretty huge difference in code path. Maybe 3 or 4 orders of magnitude. If the application is doing 100 lookups per request then before it was basically 100 x zero ms, now it might be 100 x 1ms even for misses.

This difference can then have a dramatic impact on an applications performance if it interacted with the cache in a very chatty manner. I see this a lot in applications written against a local cost cache because if something is free then developers will use it and then abuse it.

There are a couple of solutions to this.

First, we can modify the application to be less chatty. This typically means the application should store data in the cache in a much coarser form. Rather than have a customer split across 10 maps, a single customer map whose value is a denormalized 'customer' data might be better for example.

Another solution is to use a distributed local cache. WebSphere eXtreme Scale can do this using it's multi-master capabilities (V7.1+). This allows you to set up a single partition grid. The grid is hosted in a pair of container JVMs that we'll call a hub. Each client then instead of just connecting to the hub with our client APIs, they would instead create a local grid with the same objectgrid.xml/deployment.xml but also with a local catalog instance in the client JVM. The client then makes a multi-master link to the hub grid catalog. WXS will then copy everything in the hub grid locally. The client then reads and writes against it's local grid which is now like a spoke in a hub and spoke topology. All changes made to the clients grid are replicated asynchronously to the hub. If other clients did the same thing then the hub will ensure those changes are also copied to the other clients. Any collisions can be handled in the hub using the default arbitrator or an application supplied one.

This approach allows clients to have zero cost reads and writes to a cache bounded by JVM heap limits but it's fully replicated across different clients using a hub and spoke architecture. It would use optimistic versioning through our arbitration logic in the hub. Thus, it's extremely fast but the capacity is limited to the data that fits in a single JVM although this can be quite large these days.

But, the bottom line is that treating a cache as a zero cost access resources is a mistake. There are lots of types of caches these days, the caches can be made to scale out linearly from a capacity and throughput point of view but application developers need to realize this fact and either use coarser grained cache lookups or the local approach with some kind of hub/spoke but with a capacity restriction as it's unlikely a client JVM can host 600GB of data. It is possible in this day and age to have such large JVMs but if there are a lot of clients then cost of memory becomes a deciding factor and moves you back to the remote large cache with a less chatty application/cache interface.

WebSphere Commerce Server now supports using WebSphere eXtreme Scale for page fragment caching

IBM WebSphere Commerce Server V7.0 FEP1 now supports using IBM WebSphere eXtreme Scale V7.0 ifix 3 for page fragment caching. This is a big deal for commerce customers. Commerce customers traditionally use dynacache with disk offload to store page fragments. Each Commerce JVM has it's own private page fragment cache and it overflows to a disk file associated with that JVM. This has worked well for many customers but it's time to dramatically improve this.

IBM allowed dynacache to switch to use a remote WXS grid instead of disk offload using a plugin starting with IBM WebSphere ND V6.1.0.25. This means that by adding a couple of lines to the cachespec.xml, applications that used the dynacache APIs can now store their cache in WXS instead of in the usual dynacache mechanisms. This allows these applications to:

• Shared a single cache between multiple ND JVMs. All JVMs see the same cache
• Handle very large caches. Applications can now comfortably cache 100's of gigabytes of data if needed in this shared cache.
• The cache survives ND JVM restarts. You can shutdown and restart a ND JVM and the cache is still hot.
• Items in the cache when invalidates are only refreshes once PER cluster, not once per JVM

These points are very important for Portal or Commerce applications that typically render expensive page fragments. Displaying page fragments from a cache rather than rendering them from sources every time can dramatically speed up an application with corresponding decreases in CPU costs and response times.

A normal disk offload dynacache can do this but had several disadvantages. First, the cache is private per JVM. This means that each JVM is wasting memory (heap memory and file system cache memory) caching the same data and wasting disk space. Each JVM also wastes CPU cycles updating fragments in the cache. The cache is private and every JVM must recreate invalidated page fragments. If there are N JVMs then each fragment is regenerated N times after an invalidation. If a JVM is stopped for a few minutes and then restarted, its fragment cache is stale and is usually cleared and then warmed up from scratch. This means users that are attached to this warming JVM will see poor response times during this interval. There can be hundreds of thousands if not millions of fragments in these caches so warmup times can generate significant disk I/O, application CPU and database load. All these negatively impact the whole cluster, especially the database load. The next issue is that each JVM has an independent cache and potentially this means that depending on which JVM a user attaches to, the user can see a different version of the page fragments. They are not shared.

Most of the benefits of using WXS with WCS/ND/Portal comes from these simple facts:

• The cache data is stored independently from the application JVMs.
• The cache capacity can be very large, 100s of GBs or bigger without requiring disk.
• It's a shared cache, every application JVM sees the same cached fragments, guaranteed, regardless of how long the application JVM has been running.
• When cache data is invalidated, it is regenerated ONCE no matter how many application JVMs are using the cache. Invalidation now has constant cost.
• The cache runs from network attached shared memory (the grid), not disk. It trades disk I/O for network bandwidth.

These facts are a major advantage to WCS, Portal and ND customers who use dynacache for page fragment caching today.

The fact that the cache is now in a separate tier means that you can restart application JVMs and not suffer from cold cache syndrome. The restarted JVM will run with a hot cache immediately when it restarts. You don't need to warm up the JVMs cache. Customers using a newly started JVM will instantly see the same response times as other longer running JVMs are seeing. There is no warm up time as far as the cache goes.

Disk offload was good for when large retailers wanted to store lots of page fragments, typically < 20GB, and there isn't enough memory to have a cache that large PER JVM. However, WXS allows those customers to keep using 32 bit JVMs with good GC pause times and still cache very large amounts of data remotely. The thing to remember here is that you need to cache say 5GB of data then with replication, thats 10GB of data. If you have 10 application JVMs then normally you'd need 50GB of memory total with a conventional cache. When using a remote WXS cache, you only need a total of 5GB with no replication or a total of 10GB with replication no matter how many application JVMs you use. Thats a significant saving on memory. 50GB versus 5GB, a 10x saving with no replication, and 5x with replication. This is a major advantage for customers running applications on virtualization environments where memory is a premium.

The cache is shared between all application JVMs. This means that no matter which application JVM serves a web page, the page will be identical to pages served from other application JVMs. The cache is NEVER stale. Every application JVM is guaranteed to see the same page fragment in the shared cache ALWAYS. This gives a much better customer experience especially during periods where the fragments change frequently such as black Friday or cyber Monday for retailers.

The cache is shared and this means that when a page fragment is invalidated then the next application JVM to look up that page will recreate the fragment and store it in the cache. Once that happens all other application JVMs will now find the new fragment straight away. This means you only recreate a fragment once per cluster, not once per JVM. This is a major CPU saving that increases the more application JVMs you use. Invalidating a page fragment now has constant cost, no matter how big the application JVM cluster is.

A cache that spills to disk requires fast disks. A WXS cache runs out of memory. Memory is faster than disk, even SANs. Customers could look at using a flash or solid state disk for this but you still don't get around the problem that it's a private cache per JVM. You won't get the advantages listed above with a solid state disk.

So, this all sounds too good to be true. Whats the catch? Very little actually. Network bandwidth between the application tier and the cache tier is the main catch. WXS compresses the data in the cache tier and across the network. This gives a saving of around 2.5 or 3:1. If possible, I'd recommend using a separate gigabit network port/subnet for the communication between the application tier and the WXS tier.

The advantages here are significant over the older technology. You can switch to this by basically editing the cachespec.xml file and starting a remote WXS tier. It's very low risk from that point of view, you don't need to change your application, you just edit the cachespec.xml.  If you use dynacache for page fragment caching right now then I think you need to look at this technology. It's relatively easy to try it out and the savings/advantages are significant.

It's important to realize that the bigger your application tier cluster is, the bigger the savings here. It's also easy to scale out the WXS tier by simply adding more JVMs to add more CPU/network/memory to the cache tier. WXS scales absolutely linearly as you do this.