/dev/websphere

Some reasons why WebSphere eXtreme Scale lowers Commerce Server CPU requirements

I've gotten quite a few questions on the just announced WebSphere eXtreme Scale integration with WebSphere Commerce Server. One of the top themes is around CPU usage.

Duplicate Page Rendering

This is the first major saving. A WXS cache is a separate tier and is shared between all the Commerce server JVMs. Once any commerce JVM renders a page fragment and stores it in the WXS tier then that fragment is immediately available to every other Commerce JVM. If there are N Commerce JVMs then that means the page fragment will be rendered once instead of N times. This is a significant source of CPU savings in the following circumstances:

• Commerce JVM cold start

Here when the commerce cluster starts initially, normally every JVM renders every fragment. With WXS, each fragment is generated once only, this allows the commerce cluster to get warm much faster than before with less CPU burned.

• Commerce JVM warm start

Here, when a Commerce JVM is cycled for maintenance or restarted after a period of being down, there is no need to rewarm its cache when using WXS. The cache is already warm when using WXS because the cache doesn't live within the Commerce JVM at all.

• Invalidation scenarios

Here, when fragments are invalidated, fragment only get rendered once instead of N times.

Cache management costs

Commerce Server using a private disk based cache normally. This cache is private that means the burden of managing that cache is borne totally by that Commerce JVM. The bigger the cache, the more management, the more garbage collection pauses, the more private disk I/O and so on. This cost is per JVM so N JVMs means N times this cost.

A WXS cache is not managed by the Commerce JVMs at all, the costs of managing the WXS cache is shared by the collection of JVMs hosting the grid. This removes this CPU cost from the Commerce cluster and makes that CPU available to do more useful things like sell products. In addition to this CPU saving, WXS scales linearly, no one JVM is responsible for the cache management tasks, the tasks are shared in a uniform manner between all the JVMs hosting the data. There is no single bottleneck. If you make the cache larger then add more JVMs to increase capacity, add CPU for cache request servicing and management and network. We can say that the cost of management for a single WXS JVM stays constant as the cache cluster grows in size. We move the cost of managing the cache away from the client application and in to a more scalable tier.

Summary

So, there are two principal CPU savings. 

• WXS eliminates redundant page fragment rendering in startup and invalidation scenarios. 
• WXS removes cache management costs from the client application tier (i.e. commerce) and makes them constant cost per cache tier JVM as the WXS tier grows.

If you have any other questions then please comment and I'll write up responses on the blog.

WebSphere ND 7.0 ships, cool feature: 64 bit pointer compression

WebSphere ND 7.0 shipped this month and one of the features I've been waiting on is the new IBM Java 6 virtual machine. The 64 bit version has a very cool feature called pointer compression. A lot of customers ask about 64 bit support, I get this a lot with WebSphere Extreme Scale (ObjectGrid). The trouble with 64 bit JVMs is that the pointers are all 64 bit and typically customers report their physical memory usage doubling for the same JVM heap size when they go 64 bit. This means that it can take 4Gb of memory to store a 2 GB 64 bit heap which is almost double what it took on 32 bit JVMs. This is the case (AFAIK) with competing JVMs etc today.

The new IBM Java 6 JVM has pointer compression which finally helps with this issue. This allows customers running on the IBM JVM to only use 2Gb of RAM with a 2Gb heap on 64 bit. This can be a massive benefit for ObjectGrid customers and for customers consolidating multiple servers on to boxes. 64 bit applications can now be consolidated with paying a huge memory overhead.

This feature should give platforms with the IBM JVM a pretty major cost advantage and performance advantage over running on platforms without an IBM JVM available.

You can download a trial version from here.

TheServerSide interview with me on XD and asymmetric clustering

http://www.theserverside.com/talks/videos/BillyNewportText/interview.tss

More on WebSphere 6.0 and shared file systems

It's worth pointing out what may be obvious to some but not to others. If you are putting transaction logs on a shared file system then make sure you configure it to do write through, no caching. It must be capable of ensuring that when WebSphere tells the file system to flush, the change goes all the way through to storage and doesn't get buffered in a file system cache somewhere.

Most shared file systems can be configured to work in this mode and indeed MUST be configured in this mode to be safe for use with WebSphere in this model.

May have been obvious, but worth pointing out.

WebSphere 6.0 and NFS V4/SAN FS, a match in heaven

When I started figuring where I wanted WebSphere 6.0 high availability to go at the end of 2002, I decided I wanted high availability to be a lot easier and cheaper to setup than was possible at the time. I also wanted to avoid the need for SANs, shared disk arrays etc. All the applications required external software like Veritas and HACMP as well as switched disks attached with fiber or SCSI to do proper failover and it was slow. It could take minutes to restart the failed server.

It was very complex to setup, lots of steps, lots of scope for mistakes. It was too hard. This wasn't unique to WebSphere. Everybody had this problem. Everybody still has this problem today except us. WebSphere 6.0 is kind of revolutionary in this respect. For the first time, customers will be able to get best of class availability (on ANY platform) out of the box on commodity hardware.

WebSphere 6.0 is designed to take advantage of shared file systems like IBM SAN FS, Windows CIFS, and NFS v4. A cluster that uses a shared file system is almost childs play to make highly available now and will recover indoubt transactions for the failing cluster member in around 11 seconds. This is the current limit using a shared file system although faster times are possible using other configurations.

The first thing is to mount a shared file system at the same mount point on all cluster members. Say /mnt/cluster. Next, create a directory for each cluster member. On each cluster member change the transaction log directory to be /mnt/cluster/serverX, where serverX is the directory for that server. Now, bring up the cluster screen and enable "High Availability" services.

That's it, you're highly available. If server A crashes then WebSphere's HAManager component will detect the failure and tell one of the other servers in the same cluster as server A to recover the server A transaction logs. The other servers can see server A's transaction logs because of the shared file system.

WebSphere uses file locking to ensure only a single server at a time can modify a set of transaction logs. NFS v4 and SAN FS provide fast lock recovery when a share file system client crashes. AIX 5.3 can have its NFS v4 tuned to do this after only 10 seconds. SAN FS also. This means that any file locks held by the dead server are released automatically after this interval. This allows the peer cluster member to safely lock the files and do recovery.

If the failed server restarts then the HAManager will do an orderly failback a second or two after the server starts.

NFS v4 is currently only available on AIX 5.3 and Solaris 10. I read the other day that Linus is looking at adding NFS v4 to the 2.6 Linux kernels later this year. This means that this new level of availability will be available on x86 Linux hardware.

Failure Detection in WebSphere 6.0

WebSphere 6.0 detects failures in one of two ways out of the box. One way uses TCP sockets to detect dead servers. This depends on KEEP_ALIVE tuning to be effective when hard failures occur. A hard failure is a power failure, motherboard failure or network problem. The other uses active heart beating. The default heart beat rate is a heart beat every 10 seconds and if twenty heart beats are missed then we mark the server as suspect. Obviously, this is 200 seconds or 3 minutes. This was done so that WebSphere worked well in smaller development machines etc. It is, of course, tunable and we've had it down as low as 6 seconds (2 second HBs with 3 indicating failure). The recovery time for indoubt transaction recovery is the greater of the failure detection time of WebSphere and the lock recovery time of the shared file system. Even if WebSphere can failover in 4 seconds, if the file system takes 10 seconds to release the locks then recovery will happen when the locks are released, i.e. 10 seconds. So, the file system lock lease time should be used as the heart beat rate for production systems. Again, this is the current best recovery time using a shared file system, faster recovery is possible using configurations that don't use a shared file system and thats another blog entry.

IP Failover

WebSphere 6.0 doesn't need IP failover for TM recovery when it's configured for hot standby. This is a big improvement over previous versions of WebSphere and everyone else. WebSphere now uses logical names for the TM and these are resolved dynamically at runtime to where ever the TM was placed by the HAManager.

2 minute Configuration

I hope customers will start to use this new feature. It's a very simple way to make a cluster highly available at very low cost and anybody can do it. It takes under 2 minutes to setup.

• Export a file system
• Mount the file system on all cluster members
• Make directories for each server
• Change the tran log directory to the shared file system
• Check the HA box on the cluster panel.

Some of you may also realise that even without a shared file system, this can help with other scenarios. When you use vertical clustering, i.e. multiple JVMs per server then the local file system is in effect a shared file system for those JVMs on the same server and WebSphere can be configured to do this kind of recovery here also. This won't handle box failure but will handle JVM panics etc.

SAMBA

I tried to get SAMBA to work. It's supposed to do file locking etc and the solution also works on native Windows client and Windows file servers. But, making SAMBA do file locking like this was beyond me. I still think it's possible but I couldn't get any help from the community here. If SAMBA could be configured to work then this technology would work on any Linux with that SAMBA. If someone from the SAMBA community wants to work with me on this then email me. It's a shame that it appears not to work.

The problem with NFS v3

NFS v3 doesn't work here. The problem is locking. When a server fails, the locks only get released when that server restarts. That takes AGES. Minutes. Or an administrator can manually release the locks. Again, not really what I want. The solution really needs lock leasing to work well. Only SAN FS, NFS v4 or CIFS provide this fast lock recovery with no dependancy on the fail server.

SAN FS

SAN FS offers a ultra high performance shared file system that doesn't require large file server boxes. It uses a unique architecture that means it's as fast as a local file system but is fully shared. You can have a couple of small 2 way boxes as SAN FS servers (for HA) and run a lot of WebSphere or database boxes on the shared file system. It also offers a shared file system that uses lock leasing on older operating systems. NFS v4 is available on the latest and greatest operating systems like AIX 5.3, Solaris 10 and later this year 2.6 linux kernels. But, if you're on an older operating system then you don't have this support built in. SAN FS is available on these older operating systems and will enable you to take advantage of this new WebSphere support on those platforms.

Summary

Thats it. No extra software besides WebSphere needs to be bought and no switched disks per server when using CIFS or NFS v4 (SAN FS requires a SAN and the SAN FS client to be installed). I think this sets a whole new level of ease of use in clustering on the market and puts WebSphere in front of the pack even after WebLogic 9 ships later this year. The HAManager is exploited all over the place in WebSphere 6.0. The new messaging engine uses it for similar failover times. All the critical singletons in the product that used to run in the deployment manager now 'float' and run anywhere in the cell. Customers can, of course, specify policies to tell WebSphere where they would like these services to run and can specify backup servers etc.

So, there it is. WebSphere 6.0 can achieve < 12 second recovery times on commodity hardware when it's used with a shared file system with lock leasing.

JMS Session sharing in WebSphere 5.1

Get asked a lot how we do JMS connection/session sharing in WebSphere. The following applies equally to applications using the embedded JMS provider or generic JMS providers.

1. Begin
2. App looks up JMS connection and gets a session.
3. Later, another app bean again looks up a connection and gets a session.
4. Commit.

This results in a second independant session that is registered with the transaction. WebSphere returns the same session only when the applications asks for it using the same connection each time.

1. Begin
2. Look up JMS Connection
3. Get session
4. ...
5. Get another session from the same connection as before.
6. Commit

Here, the same session is returned for the second session. This is because the connection is reused. If the application creates a second connection then any sessions creates on it are independant of sessions made on other connections.

This means the application should pass the JMS Connection as an argument to any methods on EJBs or POJOs that may need to make a session later on in the current transaction.

This also means MDBs that republish a message to a different queue/topic must use 2PC even when using the same JMS provider as the MDB. This is because there is no way to get the same JMS connection used by the MDB listener and hence you must make a new one and get a new session. This means two sessions and this means 2PC must be used.

If you do the same thing using async beans then this doesn't apply as you make the JMS connection and can share it. The idea here is that you make a startup bean that starts a daemon thread. The daemon thread makes a JMS connection and session, starts a transaction and then asks the session for a message. If no message is received for say 5 seconds then the tx is rolled back and we restart. If we got a message then we process it and make sure to provide the connection and session to any downstream objects. This lets us share the connection and then get 1PC behavior.

So, sessions are only shared when obtained from the same JMS connection.

JMS session pooling in WAS 5.x

Just had a customer have a problem. The problem was this. They are sending two messages within a transaction and they had to be processed in order.

But, sometimes, the MDB on the receiving end was getting the messages out of order. The MDB side was fine, one session in the listener port with ASF disabled to process the messages serially.

So, what was up. The problem was in the way they sent the messages. They had a SLSB method called 'sendMessage' that takes a message. It looks up a QCF, gets a connection, makes a session, a queue senders, sends the message, closes everything.

Normally, JMS guarantees that all messages from a single publisher are ordered. WebSphere does connection pooling on JMS connections and JMS sessions but the way the sendMessage routine is implemented defeats this.

Connections are normally shared using transaction affinity. But, JMS connections don't know about transactions, JMS sessions do. So, the second time the app called sendMessage, WAS sometimes gets a different connection from the pool, different connection means we make a new session and hence two publishers and hence the messages lose their order sometimes as order is only guaranteed when messages originate from a single publisher.

The workarounds are:
1) Set the connection pool size to 1 for the pool, clearly, limits performance for sending so lets try something else :) but it does mean we keep getting the same connection and hence the same session/publisher and hence ordering is restored.
2) The higher levels of the code should create a single connection or session and pass it as a parameter to the sendMessage routine. Thus, the pooling works, the app gets back the same session/queue sender and we have message order.

Passing either the connection or session will work. The connection is good enough because when the app later calls createqueuesession then we look for an existing one for this transaction and use it if we find it otherwise, we make a new one and in the case of the second message, WAS will find the session that it already made for this transaction and reuse it.

Passing the session clearly works also.

So, the recommended solution is create the session further up the stack and pass it to the common send message or simply send the message in place when needed rather than use a SLSB method.

XA and Oracle with WebSphere

I've seen some customers having trouble with Oracle XA and WAS 5.0. The problem manifests when a WAS server restarts and then the recovery of any inflight transactions appears to fail.

The problem (assuming you're using the right versions of Oracle etc) is usually that the users used by the Oracle XA DataSources don't have the necessary permissions. You need to do the following for each Oracle user id that can be used by WebSphere:

grant select on DBA_PENDING_TRANSACTIONS to JOE

Then the JDBC driver at recovery time has the correct permissions to access the Oracle XA system tables with the information.

This isn't actually WebSphere specific, it applies to any JTA provider (other vendors app servers etc).