/dev/websphere

Mobile Applications and FlashLoads, corporations need more modern IT architectures yesterday

I speak all the time to corporate customers with well known and busy web sites. Typically, they see under 200 page views/sec. Maybe, a large one has a million logins per hour. The load curves for those systems usually resemble a repeating bath tub with a peak in the morning then a long trough until lunch time, another peak, then a trough till they leave the office and so on.

The advent of mobile browsers and mobile applications on devices like the iPhone, Android and Blackberry are changing this load profile. The trough isn't so deep any more. There is more average load simply because people on an impulse can bring up a mobile application and check something out. They don't need to be sitting at a desktop anymore.

A higher trough isn't so bad. People who were betting on consolidation will get a slightly smaller payback because as average utilization of servers rise, the potential to consolidate them falls. This is a small negative. The higher trough is small news though compared with whats going to happen to peaks. We have a new phenomenom here, the FlashLoad.

FlashLoads

We've all heard of flash floods. The stars align, the heavens open, and we get severe sudden massive flooding of a piece of land. The advent of wide spread smart phones and the availability of corporate mobile applications to make accessing their infrastructure easier is about to invent a new IT term. The flash load event.

Imagine the following scenario. A bank has a mobile application. It's a big bank, 20 or 30 million customers. They have a successful iPhone application downloaded to 6 million iPhones. Sounds great? But, lets say the stock market crashes. Everyone sees it instantly, CNN's mobile news application sends up real time notifications to their iPhone, they don't even need to be watching TV or listening to a radio. The phone is all they need to be aware of whether Kim Kardashian wore green as soon as she stepped out just now or that there is a stock market crash. What do the people do? Why, they crack open the banks mobile application to check the brokerage application. The problem is all 6 million are doing it at almost the exact same time. They were all synchronized through mobile notification technology and other media if they are observing. It's a flash load.

Corporate IT systems are usually not designed to handle a million concurrent logins let alone 6 million. They are usually not designed to handle 6 million concurrent users pulling up their brokerage account details or entering sell orders from mobile phones at the same time. The velocity of a flash load event will be fierce. The load will ramp up almost vertically if it's triggered by a media event (Cable show, Twitter, Facebook, TV, Radio, mobile notification service). Companies with auto provisioning systems won't be able to keep up. Provisioning more virtual machines takes time, minutes for the virtual machines to start, operating systems to boot, middleware to start up and so on. Meanwhile, you are getting beaten with a massive surge in load and it's likely not going to be pretty. By the time those extra resources are provisioned in, it's already over. Companies that consolidated resources have a lot less headroom than those who don't. You can't expect the headroom when non consolidated resources run at 10% while consolidated ones run at 50-60% load. The headroom just isn't there until the extra provisioned resources arrive and it won't be instantly.

Caching is a possible solution to some of these problems. The closer to the user that you can service a transaction using a cache then the less of the infrastructure you will touch with each 'transaction'. Caches can usually handle a lot of load, tens of thousands of requests/sec even on blades. A little investment here and having the data preloaded or warm in the cache could be the difference between a severe outage and a working site. I don't think it's a question of whether to cache or not, it's a question of cache or crash.

The cloud may not save companies here either. Very few corporate applications can run in a hybrid cloud/private data center mode. The cloud also has limited capacity. If the major banks used Amazon EC2 to host servers, a flash load event may even strain their ability to provision ec2 instances to try keep up. They are all looking for extra capacity at the same time. It's like the recent credit crisis but it's a mips crisis instead.

I think mobile applications will allow customers to check their accounts and details in a much more accessible and convenient way than the Web ever did. But, with the availability of instant information in our society through mobile devices, they also bring the potential for a new phenomenon. The flash load. Be prepared once you start shipping mobile applications for your public facing services. It's great to see a successful smart application getting millions of downloads but remember as your success measured by the number of downloads grows, the potential of a flash load is also rising fast too. Be prepared before you discover the limitations of a traditional multi-tiered architecture. It's not just a matter of adding REST services or similar, you need a flash load strategy also because it's guaranteed to come along eventually. It's cache or crash.

Why you shouldn't overcommit memory on virtualized servers running Java

Virtualization takes multiple physical machines, virtualizes them and then runs them concurrently on a single server. The argument is that they are normally using little CPU so it's ok to run them all on a single box which will then run with a much higher CPU utilization, say 70/80%. The main issue is memory. Most boxes which are targets for consolation are usually older servers with slow CPUs. 5% on such a box might be 1% on a modern box. This means that many such older machines can be consolidated on to a newer more powerful server. However, while CPU is usually plentiful and cheap, the real issue is memory. That much memory can be expensive. DRAM cost goes through the roof when larger capacity DIMMs are required. This means that virtualization vendors typically recommend over committing the server memory. This basically means they are recommending swapping at the hypervisor level. This dramatically cuts the amount of memory required and lowers the TCO of the solution correspondingly.

C programs or scripts tend to respond well to paging. They typically have small working sets (active memory pages) and then rest of the process or virtual machine can be paged out without dramatically impacting the performance of the virtual machine. They typically don't regularly sweep the contents of their address space. Java, however, is a completely different animal. Java Virtual Machines only have one working set size, the whole virtual machine heap. The objects used by a Java application are typically randomly spread all over the address space of a process. Any timers which fire can cause much of the JVM to be paged in by the hypervisor and if a garbage collection is triggered then the whole JVM heap must be paged in from disk as the garbage collector sweeps ALL the memory, a very slow operation. This page in will cause other virtual machines to be paged out then they garbage collect and we get a ping pong effect. This behavior can cause major variances in response times for users of the applications hosted in the JVM. A garbage collection which normally completes in hundreds of milliseconds might take minutes to complete in such an environment and the applications within that JVM are frozen during this time. Many Java applications utilize timers for JDBC connection pooling, time outs, heart beating for cluster management and so on. The paging activity wreaks havoc on these kinds of time sensitive operations and will cause most Java applications to behave very erratically if not fail completely. Any kind of clustering framework will not survive a paging environment due to the timing sensitive nature of heart beating and similar code.

Java Virtual Machine technology needs to be improved to that it can cooperate with the host operating system or hypervisor to minimize its working set which should allow it to be paged more efficiently but garbage collection will likely still remain a tough problem to beat in an environment that is actively paging. Any virtualization vendor will struggle to keep a customer happy in an over committed memory virtualized system. These systems are frequently sold to customers with the assurance that over committing the memory is ok but this usually results in the customer being unsatisfied and then complaining to the Java application vendor about poor performance or unstable behavior. The response to these questions will likely be "Don't overcommit your memory!". The response is usually "But, the virtualization vendor said it was ok otherwise the system becomes very expensive…" and the final response from the application server vendor will be "Yep, thats right…"

Solid state disks for paging might help with the situation. But, any stateful application server with timing sensitive code will likely still encounter some issues in such as environment because of the jitter introduced by paging. WebSphere includes code to check for such jittery environments. It's a simple alarm which fires periodically and simply checks that it is fired when it expected to be fired. If the timer discovers that its several seconds later than it expected then it logs a message indicating CPU starvation is occurring. This only happened in the past on machines whose operating systems were paging OR on machines running hundreds of JVMs which suddenly all get busy and overwhelm the CPU capacity of the server. Imagine 500 threads that are all CPU bound running on a 8 core box. If each thread runs for 200ms when it gains the CPU then one second can service about 40 threads. This means around 12 seconds will elapse for any given thread before it even RUNS again. This is unacceptable for any timeout style operations with resolutions lower than that. We are seeing customers now reporting problems in virtualized servers where they have overcommitted the memory and while the local operating system is not paging, the hypervisor is. The result is the same. Erratic behavior.

In summary, this kind of behavior is not new. It's been seen before on systems which page or are simply overloaded. Either way the system was either out of CPU or out of memory or BOTH. Customers running virtualized systems need to be very careful when overcommitting either resource if predictable application behavior is required.

Why Java is important to VMware but may not matter long term

VMWare has a problem. Consolidation is the usual angle thats used to sell it. Most boxes run at 5% load so ideally you want to consolidate 15 machines or so on to a single box pushing its utilization to 75%. But, that means you need memory for 15 virtual machines on the box. Thats a lot of memory and memory isn't cheap.

So, they usually say overcommit the memory and everything will be ok. Thats not true (especially for Java applications, we run into this ALL the time, don't believe the ok to overcommit advice) but thats another blog post on why swapping and Java DO NOT MIX right now. VMWare needs a way to reduce the cost of RAM so that it lowers the cost of the platform and lets people host as many virtual machines as possible on a single box without breaking the bank on memory cost.

A normal unvirtualized box runs an operating system and the OS hosts processes. One per application maybe. Memory utilization is pretty efficient. All processes can share libraries in memory, file system caches and the memory of the operating system is amortized over many processes.

This isn't the case with VMWare, a hypervisor. Each virtual machine runs its own copy of the operating system with its own file system cache. There is usually no sharing of memory between virtual machine instances (although thats changing). This means a virtualized box is inherently less memory efficient than a single operating system instance running the same applications. The multiple operating system instances are costing VMWare a lot of memory. A hypervisor may be more CPU efficient but it's not as memory efficient as a single operating system running the same applications. You can enable swapping to avoid the need for so much memory but Java applications don't work well with swapping so lets assume we are not swapping. Now, there are advantages to using a hypervisor but we are talking about the resources needed to run X applications here and memory factors in.

VMWare could get in to the operating system business but thats going to fail. Windows and Linux basically own that space. Microsoft is busy adding hypervisor capabilities to their own stack which will cost less than VMWare so hosting Microsoft applications is probably not a long term business for them.

I said at the beginning that Java is important to VMWare. Normal applications are built against a specific operating system thats evolving. VMWare is unlikely to be able to host normal applications without an operating system because they can't emulate the operating systems. They would need to host the operating system as a virtual machine and as we said earlier, thats not memory efficient.

Now, there are a lot of Enterprise applications written in Java. This means that if VMWare could host Java applications natively, i.e. if VMWare can host a Java virtual machine without an operating system then potentially they can be as memory efficient as an operating system. They can use shared memory between JVM instances for JITed code. They would only need to implement the Java API which is basically portable across machines and operating systems. This is significantly easier than emulating Linux or Windows. A Java skin gives them a way to host Enterprise Java applications natively and cut the memory overhead a hypervisor brings to the table by removing the overhead of an operating system.

A hypervisor has advantages over a native operating system in that it can move virtual machines from one box to another and suspend/resume them. Application or virtual machine mobility is a big selling point for a hypervisor. It's possible that VMWare sees Java as a way to lower the cost of virtualizing Enterprise applications. However, operating system zones and WPARs offer application mobility also so this advantage is not such a big deal anymore. A big advantage WPAR/Zones have over virtual machine mobility is that you're not bound to an operating system with WPAR/Zones. You can upgrade an application by moving it from a Linux Version N to a Linux Version N+1 machine. Moving a VMWare image doesnt upgrade the OS, the OS is still the same after you move the virtual machine. You'd need to upgrade it in the virtual machine which is more hassle than just moving a WPAR/Zone.

This native Java Virtual Machine puts them in a similar situation to Azul. Azul can run standard Java code but unless the applications are certified to be supported on it then customers are reluctant to do it and vendors are reluctant to support their products on that configuration mostly because of support/testing cost. A VMWave JVM would suffer the same consequences. Application server vendors would have to certify and support it to move applications over and that can be a tough thing to get them to do.

Buying Spring gives VMWare Spring TC and Spring DM. These are Java application containers which can probably run 80% of departmental and many enterprise Java applications (Servlets, JDBC, JPA, JMS). It's not JavaEE certified but it's probably enough to run most Java applications. If customers can move Java applications to such a container and hosting them on a virtualized environment then potentially you can lower VMWares TCO as it'll require less memory to host them.

I'm actually skeptical that VMWare offers many advantages over just running an operating system natively on the box for Java applications. Best case, VMWare would match the memory footprint of the operating system and it would likely run the applications a little slower. Application mobility is still an issue for operating system hosted applications but technologies like Solaris Zones or AIX WPARs are starting to address application mobility without the memory cost of a full virtualized environment such as VMWare.

So, to summarize. VMWare could host Java applications natively to lower TCO. This would allow them to host multiple Java applications on a single box with similar TCO best case to a conventional operating system. Hosting commercial application servers like this would likely be a support problem but as I said, many Java applications would run on a native Spring TC environment. Application mobility between boxes was a big advantage for hypervisors such as VMWare but operating system features such as Solaris Zones and AIX WPARs are rapidly eroding that advantage because applications installed in a zone or WPAR are basically portable anyway between machines and have the advantage of not being bound to an operating system version as is the case with a virtual machine image. Once customers figure this out then they may decide that using a conventional operating system with WPARs or Zones is actually a more cost efficient and easier path to swallow than picking up a new platform based on a hypervisor which would not be as memory efficient as a normal operating system would be. 

Will VMWare ultimately fail because what customers really want is not virtual machine mobility, they want application mobility?

DataGrids can increase value and efficiency for Z/OS customers

Many customers host critical applications or databases on mainframes. These services are critical for those enterprises and those companies want to make sure the money spent on those services is used efficiently. Many applications in a company will leverage those mainframe hosted services. Each of those applications is probably developed independently within the company.

Problem: Portals and independent applications cause redundant requests on mainframes

Lets say the requests being made from those applications are starting to slow down the mainframe. A common reason for this is portals. Portals present information from different applications on one screen. This makes lots of sense for customers in that they get an integrated view of their situation but this causes headaches for the company. Why?

These applications are probably all fetching the same data from the mainframe. Things like customer profiles, account balances. If there are 5 applications used in the portal then this information will be retrieved 5 times. The same information. This clearly means the mainframe is having to execute the same request 5 time in a row and with customers paying for MIPs to run those requests, this isn't acceptable.

Localized solution: Silo/application based caches

Application level development teams usually do the following when they see a 'slow' backend. Each application team adds a cache. This appears to help within the application teams world. This eliminates redundant requests within the application but the mainframe will still see 5 profile requests in the above example because the caches are application private. Application private caches are therefore not really the answer. Whats needed is an enterprise level solution that addresses the problem for all applications across the Enterprise.

Correct Solution: ESB/Service level DataGrid enablement (cross application)

We need a shared cache thats accessible by all applications and that new ones can utilize without writing any code. This cuts development and testing costs and this would reduce the profile fetch to exactly one which is what it should be. This is exactly what DataGrids like ObjectGrid are designed to do. A data grid can be created and then shared between applications accessing the mainframe. All requests go through this layer and this eliminates any redundant requests.

No-one wants RIP and WRITE

The next issue is that while this sounds great, no one wants to rewrite 5 applications to check a datagrid. The issue now becomes where can they splice in a datagrid without changing any code. Most organizations use SOA now and probably were using SOA before they even knew it was called SOA :) This can help greatly with minimizing "rip and write" to add this capability. The mainframe usually exposes its services using an ESB of some kind, like MQ/390 or a similar product. This allows a new service to be written that has the same interface as the old one but acts as a proxy to the existing mainframe services. The proxy checks the datagrid for already serviced requests before sending them to the mainframe. This allows the DataGrid to eliminate redundant requests without changing any code in the existing silo applications or on the mainframe.

Where should this new proxy service be hosted? zLinux makes a great location. Host the DataGrid in a collection of zLinuz VMs or host it on a distributed platform. The choice would be the customers depending on the specifics.

This shows the value of including a DataGrid component in an ESB rather than in an application. The ESB provides a relatively cheap way of incorporating DataGrid services to eliminate redundant requests to a Z/OS system without incurring high development costs. The time taken to implement this is also considerably shorter than having to line up five development teams on different schedules to do this work.

Summary

So, to summarize. We're seeing this requirement from many customers looking to increase the efficiency of their mainframe applications and databases. Application aligned development teams and portals are aggravating the issue due to different schedules, priorities etc between the teams. Customers that are already using ESBs have the opportunity to leverage DataGrid services to cache service requests and achieve efficiencies at a relatively low cost and implement this quickly. Customers that are not using ESBs may still have some common architectural layer in to which DataGrid services can be exploited. If you don't have an ESB or similar architecture then this is a good reason to start working on adding one because once you have the architecture then enterprise wide efficiencies can become a lot more attainable. Another way of looking at this is that directly accessing shared resources like databases or data serving applications using low level APIs (SQL/JDBC/JCA) doesn't scale. Applications/client accessing the same data through data services scales better because you get more control than you would using SQL/JDBC or other approaches but thats another blog entry.

Making an Amazon EC2 image for ObjectGrid

I'd been doing this anyway for some scale up testing but with the story on infoq about EC2, I figured I'd do a post. The trick with amazon ec2 cost wise is limit the amount of data transfered in and out of  the virtual machines as the bandwidth is the killer component of the cost. When I first did ObjectGrid testing on EC2, I'd start the virtual machines, scp (secure copy) up objectgrid and an ibm jvm to each one which is costing me money per mega byte. This obviously not so smart so heres the right way to do it.

First, create a virtual machine using your favorite prebuild image. I used fedora 4 + mysql + apache. I then uploaded objectgrid and the IBM java 6 JVM. The virtual machine has two disks in it. One has the operating system on it and the other (/mnt) is for data. It's important to unzip these in the root directory and not in /mnt. The /mnt disk isn't part of the image so anything there won't be available in the cloned virtual machines using your new image. Once they are up there then upload any scripts you need or application jars, spring jars what ever. Put everything in a directory on the main file system. Don't use /mnt for anything related to your new image.

Test it and now we need to tell Amazon to create a new private image of this. This new image includes the existing amazon prebuilt image AND the new stuff you uploaded. The first thing to do is to get your amazon account number, key id and secret key. Keep them handy.

First, create a directory called ami from /mnt. Run the following command

ec2-bundle-vol -d /mnt/ami -k /mnt/PrivateKey.pem -c /mnt/509Certificate.pem -u AccountId

You can see I also uploaded my private and 509 certificate pem files to /mnt also. Don't put them in / as then anyone with your image has them. /mnt is not included in the image that this makes. This creates the image after around ten minutes in the /mnt/ami directory and splits it in to chunks for upload to Amazon S3 where the image has to be stored.

Next use this command to upload this new image:

ec2-upload-bundle -b bnewport-ami-og-image -m /mnt/ami/image.manifest.xml -a KeyId -s SecretKey --retry

This uploads the image to an S3 bucket called bnewport-ami-og-image. Use a different bucket name for each image you want to make. I needed to use the -retry parameter as the upload would almost always fail on one of the pieces. The retry parameter fixes this. Now, you have made an image and you have uploaded it to S3. Now, you need to tell Amazon EC2 about it using this command which needs to be run from your laptop.

ec2-register bnewport-ami-og-image/image.manifest.xml

Once you have done this then you can see the image when you do a 'ec2-describe-images' command. You can then start the private image using the 'ec2-run-instance' command and use the parameters to start as many instances as you can afford to do. I start a catalog server on two of the instances and then start my application JVMs on all the machines using the host names for the two catalog servers. I wrote shell scripts for my mac to execute ssh commands to start/stop all the instances using the host names of the running instances that I extract from ec2-describe-instances and awk.

Thats it, this is a pretty cool way to do scale testing on an application. Start em up when you need them and then stop them when you are done.

Goverance and lowering costs in data centers and development

I think many customers have cost problems in their data centers because of the complexity of the application platforms in use. Whats an application platform?

• Hardware platform
• Operating system configured to run on that hardware
• Application hosting middleware (WebSphere, Weblogic etc)

These three factors have many combinations in data centers. Obviously, you will have quite a few types of server. This has gotten worse with the blade type solutions because they are revised frequently when compared with the big iron SMP boxes seen in data centers before. The operating system needs the right device drivers and may need patches because of bugs. The exact patches required may change if different applications are run on it.

Application hosting middleware also varies. Application A may be tested on WAS 6.0.2.7 and Application B might run on WAS 5.1.7 and the operating system used for 6.0.2.7 may not be supported for WAS 5.1.7. And so on.

This, obviously, is a major pain for customers. The testing and maintenance required for these various combinations cannot be underestimated.

What is governance?

Goverance is the simplification of this problem. How can it help?

Mandate a hardware level

This is greatly simplified through virtualization techniques available now. If virtualization is not available then it means mandating a specific box type with specific PCI cards at specific versions etc. This can be difficult. The virtual route is the way to go if at all possible.

Operating system mandate

Standardize on one operating system image. Obviously, with virtualization, this is possible as the virtual processors, network cards, video cards and storage cards are the same no matter what level of hardware the virtualization software is hosted on. This means a single operating system image is even possible.

Application server platform mandate

Specify all J2EE apps run on WAS 6.0.2.9 or some level and only change it infrequently. Many development teams won't like this, of course, because of some feature. If an application won't run on the standard platform then the operations side must figure out why and rev the platform. This may include the whole stack (OS image, app platform).

Summary

So, governance is and will become a necessary evil in order to control costs and add reliability. The fewer stacks you run, the more tested they will be and the more reliable they will be. Lots of platforms means less testing per platform. It's pretty simple. Plus, companies can provide developers with virtual machines with the stack installed. Developers can use products like EMCs VMPlayer to run those images on the desktop without the need to spend two days downloading products, fixes, installing them etc. Just copy a VM image from a network drive and use it, done. Takes as long as the copy does.

Goverance has the potential to offer significant productivity gains in the development area as well as in the operations area. Companies should look at making goverance a priority and try to use it to lower costs as well as improve productivity both in the data center and in the development teams.

The value of machine virtualization.

Customers on distributed platforms have had forms of virtualization for a while now. Storage virtualization has been the main tool through the use of SANs. While this form of virtualization clearly offers cost savings, it's limited in terms of its impact. Machine virtualization has the potential to offer dramatically increased savings over whats possible through just storage. Products like IBMs LPAR technology, VMware and other competitors can potentially offer significant operations costs saving to customers employing them. Obviously, this is a technology that can be misused but thats for the next post. I wanted to go over the main reasons for the much larger potential cost savings through machine virtualization in this post. These will all be well known to Z/OS, 390 or iSeries customers as they have had them for some time now. We're seeing this technology become available on commodity boxes now for the first time.

One of the promises of virtualization is lower costs. Heres where I think the main cost savings lie.

Less physical boxes in the data center

Obviously, there are fewer physical boxes required. You may be able to get 5 to 10 x reduction depending on how much risk you take. The more virtual machines on a physical box, the more risk you can that you'll overload it. But, less physical boxes means less data center cost. Data centers cost a lot of money in terms of power, physical space, construction costs etc.

Administration savings

A virtual machine is as expensive as a physical one from a continous administration point of view. It's still an operating system image which requires the same administration as a physical one. The main saving is that the virtual machine is fixed, the hardware is always the same regardless of the physical platform. Virtualized devices like network cards, displays, storage controllers are constant. When a new type of physical box arrives, you just deploy the virtual image on to it, you don't need to make a new image or install new device drivers etc. You just need to make sure your virtualization software can run on the new physical server. This is the main administrative benefit. You only really have one hardware platform to worry about, the virtual one and it doesn't change. Obviously, it's also cheaper to migrate an application from one box to another and creating a virtual image is a lot cheaper than creating a real one because the OS install is identical. You just copy the image and update host names. Storage virtualization would allow you to easily copy existing drives but if the new box has different hardware specs then the copied image won't run. This is a thing of the past with machine virtualization, the hardware is always the same.

Hardware standardization in a market where new hardware is released almost constantly.

Standardization of 'hardware platform' is the main benefit and most of the cost benefits come from this capability.

Vendors pushing a virtualization marketing story frequently use over provisioning as the main evil to eliminate with this technology. However, overvirtualization (I think this is my own term, I haven't seen it used before) is a potential evil of this technology. I'll discuss this in the next post.