/dev/websphere

Mobile Applications and FlashLoads, corporations need more modern IT architectures yesterday

I speak all the time to corporate customers with well known and busy web sites. Typically, they see under 200 page views/sec. Maybe, a large one has a million logins per hour. The load curves for those systems usually resemble a repeating bath tub with a peak in the morning then a long trough until lunch time, another peak, then a trough till they leave the office and so on.

The advent of mobile browsers and mobile applications on devices like the iPhone, Android and Blackberry are changing this load profile. The trough isn't so deep any more. There is more average load simply because people on an impulse can bring up a mobile application and check something out. They don't need to be sitting at a desktop anymore.

A higher trough isn't so bad. People who were betting on consolidation will get a slightly smaller payback because as average utilization of servers rise, the potential to consolidate them falls. This is a small negative. The higher trough is small news though compared with whats going to happen to peaks. We have a new phenomenom here, the FlashLoad.

FlashLoads

We've all heard of flash floods. The stars align, the heavens open, and we get severe sudden massive flooding of a piece of land. The advent of wide spread smart phones and the availability of corporate mobile applications to make accessing their infrastructure easier is about to invent a new IT term. The flash load event.

Imagine the following scenario. A bank has a mobile application. It's a big bank, 20 or 30 million customers. They have a successful iPhone application downloaded to 6 million iPhones. Sounds great? But, lets say the stock market crashes. Everyone sees it instantly, CNN's mobile news application sends up real time notifications to their iPhone, they don't even need to be watching TV or listening to a radio. The phone is all they need to be aware of whether Kim Kardashian wore green as soon as she stepped out just now or that there is a stock market crash. What do the people do? Why, they crack open the banks mobile application to check the brokerage application. The problem is all 6 million are doing it at almost the exact same time. They were all synchronized through mobile notification technology and other media if they are observing. It's a flash load.

Corporate IT systems are usually not designed to handle a million concurrent logins let alone 6 million. They are usually not designed to handle 6 million concurrent users pulling up their brokerage account details or entering sell orders from mobile phones at the same time. The velocity of a flash load event will be fierce. The load will ramp up almost vertically if it's triggered by a media event (Cable show, Twitter, Facebook, TV, Radio, mobile notification service). Companies with auto provisioning systems won't be able to keep up. Provisioning more virtual machines takes time, minutes for the virtual machines to start, operating systems to boot, middleware to start up and so on. Meanwhile, you are getting beaten with a massive surge in load and it's likely not going to be pretty. By the time those extra resources are provisioned in, it's already over. Companies that consolidated resources have a lot less headroom than those who don't. You can't expect the headroom when non consolidated resources run at 10% while consolidated ones run at 50-60% load. The headroom just isn't there until the extra provisioned resources arrive and it won't be instantly.

Caching is a possible solution to some of these problems. The closer to the user that you can service a transaction using a cache then the less of the infrastructure you will touch with each 'transaction'. Caches can usually handle a lot of load, tens of thousands of requests/sec even on blades. A little investment here and having the data preloaded or warm in the cache could be the difference between a severe outage and a working site. I don't think it's a question of whether to cache or not, it's a question of cache or crash.

The cloud may not save companies here either. Very few corporate applications can run in a hybrid cloud/private data center mode. The cloud also has limited capacity. If the major banks used Amazon EC2 to host servers, a flash load event may even strain their ability to provision ec2 instances to try keep up. They are all looking for extra capacity at the same time. It's like the recent credit crisis but it's a mips crisis instead.

I think mobile applications will allow customers to check their accounts and details in a much more accessible and convenient way than the Web ever did. But, with the availability of instant information in our society through mobile devices, they also bring the potential for a new phenomenon. The flash load. Be prepared once you start shipping mobile applications for your public facing services. It's great to see a successful smart application getting millions of downloads but remember as your success measured by the number of downloads grows, the potential of a flash load is also rising fast too. Be prepared before you discover the limitations of a traditional multi-tiered architecture. It's not just a matter of adding REST services or similar, you need a flash load strategy also because it's guaranteed to come along eventually. It's cache or crash.

memcached versus DataGrid, dumb server versus smart server...

Memcached is a pretty popular free open source distributed cache. It's pretty unique compared with the commercial data grids in that it has a completely unique design.

Memcached is basically a dumb server. This means it's a process running on a Linux box typically which accepts client requests for keys and returns the data. Updates and deletes are also supported. This is ALL the function in the server, hence, dumb server. The client is the only piece which is 'distributed' or even knows about other servers. The clients typically get a collection of host:ip addresses (they all NEED to use the same list) and the clients hash the key over the set of online servers. If there were 4 servers running then all clients will hash data for a key over these 4 servers. Clients will write data for a key K to a particular server using this hash algorithm. One server.

Now, thats what memcached does out of the box. Note, there is no replication on the server side. If a server fails then the clients will hash keys over 3 servers but this changes everything from a hashing point of view so it's likely you'll get a very high miss rate as it's unlikely keys still hash to the same server as before given the modulo part of the hashing is now 3 instead of 4.

Clients work around this by adding a consistent hashing layer on top of the normal client. This works better and writes may be written to a server and the next server in the hash ring. This slows down writes by 2x but allows some recoverability when a server fails. The next server in the ring has the data from the dead server so it's not as bad as before BUT if this server also fails then now the cache has lost the data. It's not replicated remember, it's just written twice. After the first server failed, the clients write changes to the next server and the following server in the hash ring. The new next server in the ring does not have a copy of all the data. It was never written there. It will just have new changes since the failure. There is no active replication going on here.

Most of the materiel on the web shows various workarounds to try to make memcached 'reliable'. A lot of the facebook engineering blog entries focus on this aspect. What they had to do to make it reliable. This works for facebook but won't work in general because I don't think the majority of customers are structured like they are from a build your own mentality. I was talking to someone from one of the big internet shops last week and they always run two memcached clusters for HA. If a server in cluster A fails then they stop using cluster A and switch to B. This works but thats a 50% loss in hardware because of a single failure.

Smart servers

Commercial data grids like IBM WebSphere eXtreme Scale are very different. They handle making the data fault tolerant on the server side with built-in replication and elastic scale out and scale in. Their clients understand the server side data layout or routing. If a server fails with WebSphere eXtreme Scale then no data is lost. The software actively maintains replicas on other servers. Clients that stored an entry with a key K before will still find it. It just works. You don't need consistent hashing on the client, our client does it using routing tables instead. Adding more server processes causes the product to automatically redistribute the minimum amount of data to leverage the new memory/CPU and network available through the new JVMs.

The smart servers allow proper replication which means that the cache isn't as brittle as the memcached cache when servers come and go. If nothing ever fails then the memcached approach works but the potential cache hit rates over time if multiple failures occur mean it can be a liability. The last facebook entry I read on this showed them crawling the logs to try to recover lost data to make the 'next server' more complete. They get this issue and are jumping through hoops to handle it but again, most commercial customers won't do this. They expect the product to do this sort of stuff like replication and they view it as table steaks.

So, memcached for sure has its uses but it's got some limitations which need a fair bit of work by its users to handle correctly if its not exactly what you want or risk some bad scenarios as the cache hit rate climbs. It's very, very fast but you gotta remember, it's not doing much either... No transactions, no consistency, no replication, no isolation and so on. If you don't need these features then it's pretty fast but if you do need this stuff then it starts to slow as you add code on the client side to try and compensate for the limitations.

Working on bladecenters.

Currently playing a lot with IBM's xSeries blade centers, very cool. I'm running RHAS 2.1 on them and our internal drivers for the next WebSphere etc, doing performance work, proof of concepts etc.

I have them setup pretty simply, I'm using a SAN/NAS box for DASD over iSCSI which works incredibly well over gb ethernet. Running DB2 at 3k tps over iSCSI on a test yesterday using a software iSCSI initiator. Still showing 59mb/sec between the DB2 box and the SAN using iSCSI at 12k io/sec. Pretty awesome for commodity hardware like blades and gb ethernet, no fiber etc. Matches the economics of blades better than fiber channel.

I just got ssh working between the blades and just wanted to do a quick post showing how to set it up.

Basically, run the following command on each machine

ssh -t rsa -N ""

This generates a key pair in your home directory. Copy the pub key to a shared NFS volume. Do this on each machine.

Now, on one machine, in the shared volume, cat all the public keys together using:

cat *.pub > authorized_keys

Now, copy this file to each machine in the ~/.ssh directory for the user using the following command:

scp authorized_keys carreral1:~/.ssh

where carreral1 is your host name. You'll have to enter your password for each server. Remember to also do it for your current machine.

Once the authorized_keys file is in the directory then ssh, scp etc will work without entering the password. This makes using the blade center pretty convenient in a test environment.

I'm using putty as my ssh client on windows. Works pretty well.

I'm currently using NIS for user id and groups etc. I wired the NAS to use NIS also. To avoid a spof, I made one blade the NIS master and all the others NIS clients. The clients keep a copy of the NIS database and hence keep working if the NIS master server fails.

Currently, working to use IBM director, PIXIE etc to make installing the blades easier, switching OS etc on the fly and using the iscsi-init project on source forge to boot the blades from iSCSI volumes rather than local DASD.

cewolf charting is pretty cool

If you're looking for a web based charting library, check out cewolf at http://cewolf.sourceforge.net/

It's pretty cool, easy to use and appears to work fine with WAS 5.1 (this doesn't mean IBM supports it, just that I'm using it with no apparent problems). I haven't tried it with WAS 5.0 yet as I'm using an internal 5.1 driver for what I'm doing. I'm knocking an application together right now using struts 1.1 and this and it's working well. Easy to use and produces lots of nice shiny charts easily.

Check it out. Just follow the instructions in the tutorial to put it in a WAR and it seems to just work. I'll post more feedback as I use it more.

www.billynewport.com works now.

I added this URL to point to the site now also. May be handier than the old one, both URIs still work fine.

JCA, a thread too far.

JCA defines a contract between a connector implementation and a connector container, typically a J2EE server. In the beginning, code in the connector was hands off and could do anything. Slowly, with JCA 1.5, we're seeing this view slowly being modified as the specification tries to make connectors run in a managed environment.

JCA 1.5 adds a managed threading concept and a way to have callbacks to the container (read MDBs). But, there are still issues. Suppose the JCA connector wants to use a database in a transaction along with itself. No can do, this needs to be pushed up to the application and implemented using a callback (read in the MDB).

I think the defiencies will slowly push the JCP to make connectors a full fledged component and when they do that, connectors will end up like a local session bean. There will be extensions required of course, threading, importing XA trans, timer services, safe callbacks etc.

In WAS Enterprise 5.0, I added support for constructs to allow server components to be written. A server component for me is something which has a lifetime that isn't coupled to a client. Application Servers and databases are examples of server components. They have a lot of state, they use threading, timers etc and can respond to requests from their clients.

For me, a normal J2EE application is not a server application, it's like a bunch of stored procedures running in a database. They can do nothing except respond to requests and have no real lifetime outside of the 'respond to request world'. There are a lot of applications, obviously, that are like that, but importantly, there are also lots of applications that are not. If we want J2EE to be a virtual operating system then it needs to support these applications also. I'd like to be able to write a chat server that can deploy on your J2EE infrastructure and be very performant, be manageable, be highly available and scale because of servers provided from the J2EE infrastructure.

I'd like to see server components to be implementable using J2EE. Hence, the additions I pushed for in 5.0E. This added startup beans, threading (pooled and daemons), managed timers, safe callbacks/observer mechanism.

If we have this support then couldn't JCA connectors be implementable using a session bean? Rather than adding support to the JCA spec for threads, managed trans etc. Why not leave the JCA spec alone and instead define a mapping from the connector spec to the EJB session bean. This lets the JCA spec stay J2EE agnostic from an interface point of view and hence be implemented on any Java server product, not just J2EE. But, customers would be able to leverage their skills in J2EE to implement connectors if they choose using the JCA -> Session Bean bridge mapping.

Now, customers can use the normal tools to write connectors in a familiar programming model and toolset. The JCA spec stays simple and the managed stuff slowly making it's way in to JCA, instead comes as an enhancement to J2EE and makes that more useful.

What do you think? Reinvent the wheel or plugin a SB adapter and enhance J2EE.

Presumed Abort Transactions and availability

It's important to understand how transactions work using XA today. There is the part we all know and love, i.e. the XA verbs like start/prepare/commit/rollback. But, what about before we call commit?

When the application server uses an XA connection, the connection gets enlisted in the transaction when the XA start command is issued.

If the application servers gets a commit command from the application when the app server calls prepare and then commit. The application server is responsible for making sure that all resource managers in the transaction either commit or rollback.

But, what about before commit, i.e. between XA start/begin and when the application invokes commit. What happens here.

Well, we're holding locks on database rows, we're making messages on a JMS system invisible because we consumed them. Suppose the application server fails right now.

If the process fails then the OS closes the socket between the app server and the resource manager. The resource manager sees this, knows we didn't reach the prepare stage yet. What does it do? It presumes abort. It rolls back the transaction. This is the 'protocol' called presumed abort. If it doesn't get to prepare/commit then rollback.

Now, there is some stuff to know here. How did the database (lets use a DB in the example) know to rollback the tran? The socket to the client was closed. How did the socket get closed, the client OS did it when the client process (WAS) died as part of the process cleanup.

Seems cool no? Well, what if the power failed. What then? No OS to clean up the socket. Surely TCP has a way to detect sockets with one end dead? Yep, it does. You can tell the OS on the DB side using the KEEP_ALIVE parameter how long to wait on an inactive socket before pinging the TCP stack on the other side. The problem is normally KEEP_ALIVE is like 2 hours.

Thats pretty bad. Your database locks could be held for 2 hours when the app server box fails. This clearly isn't acceptable. So, it's wise to set KEEP_ALIVE to 30/60 seconds or lower. This doesn't really slow down the DB box. It only results in an extra packet being sent when the socket isn't being used for that long, i.e. 30/60 seconds. If the socket is busy then there is no overhead.

This tip applies to any kind of resource manager that uses TCP to attach to clients. A client is an application server here.

So, when setting up highly available systems, don't let a dead app server hold locks any longer than necessary. Set KEEP_ALIVE to an acceptable value and thats the longest connections to the dead client will remain open if the box fails.

2PC = atomicity? Maybe

I recently did a question and answer session with a customer about WebSphere and 2PC. A lot of it was the usual stuff, what is it? whats the performance costs? how do you make it available etc.

The customer was using 2PC to update a DB and send a message. The message triggers a second application which reads the database. Oops, warning bells go off, this will cause problems.

The problem is that 2PC is exactly that, 2 phase commit over multiple resource managers. A resource manager is a JMS provider, a database or a JCA connector. In this scenario, it's MQ series and DB2 for example. So, what happens, the app sends a message, updates the database and commits. App B then receives the message, reads the database and does something. Easy, right? What can go wrong....

Well, this. The first transaction is commited. What does this mean, it means that WAS asks JMS and DB2 "can I commit". They both answer yes and then WAS tells both to commit, JMS first then DB.

So, first JMS is told to commit, immediately the sent message is visible to App B. Then WAS tells the DB to commit, now the records associated with the message are visible. Spotted the problem yet?

Imagine App B gets the message but before WAS has told the DB to commit. The JMS message is visible but the DB changes aren't yet but will be very shortly. Regardless, the MDB receiving the message looks up the record in the DB and gets a record not found because we were unlucky.

So, while 2PC guarantees all changes will occur. There is a window when the changes to the multiple resource managers will be out of sync. This is a problem with ALL application servers. WAS isn't special. This is just the way it is. So, when building applications like the above, expect this. Code for it. If the MDB discovers the data isn't there then rollback. The message will get redelivered and the data will be then the next time probably. You can run in to funny situations like, the DB fails after WAS told JMS to commit. WAS will then continue trying to tell the DB to commit but this fails until the DB is recovered and then it works.

But, given DBs can take from 2/3 minutes to failover to a day depending on how automatic your system is then you need application logic to handle this. Maybe if the record isn't present, use the WAS scheduler or EJB 2.1 timer service to retry in 5 minutes or something. You get the idea. The real world is a difficult place where even what we're taught is bullet proof can do wrong. Understand the technical environment and then your applications will be much more robust.

The problems here are not specific to JMS and DBs. The same thing applies to 2PC with two databases. Changes will be visible in one database before the other. Applications reading data from DB.A which then try and find the corresponding data in DB.B may see old data depending on whether the second DB has been committed by the TM yet or not.

The only way to avoid these issues is to not use 2PC and use a single resource manager and clearly there are times when this just can't be so you need to be aware and design your applications with this in mind to stay out of trouble.