/dev/websphere

WebSphere eXtreme Scale sizing/configuration presentation

This is a deck that we use to educate customers on sizing/configuring a grid optimally. This may have a lot of what some people would consider obvious information but it's useful to write it all down regardless.

Download WXS Sizing worksheet Podcast

New trial version improves on usability

We just updated the trial version of WebSphere eXtreme Scale so that it's easier to get a grid running after downloading it. I think the new gettingstarted folder will allow people to have a working grid within ten minutes of downloading. The update allows a single grid of JVMs to be started and includes a client to perform insert/get/update/remove operations on that grid. Please post any feedback here or on the forum.

It should now be as simple as download the trial, set JAVA_HOME to your current JVM and then run the provided scripts in gettingstarted.

Epidemic based testing for high quality software

We have a pretty huge test bucket for WebSphere eXtreme Scale, it been growing now for a couple of years and it current takes a couple of days just for the auto FVT bucket to run and then we do the testing that we haven't automated yet. Quality is a big deal in the XTP/DataGrid business so it's always on my mind, more so than performance or function. Everyone wants perfect software with no bugs. We can strive towards that goal with larger and larger test case buckets written by developers and do code reviews etc but sometimes bugs still get out. The bugs are harder to come by these days and I'm finding the study of epidemics is an interesting approach to how testing happens.

This reminds me of a TED talk on how the World Health Organization eliminated Smallpox. It was too expensive to immunize everybody. The approach they took instead was to look for outbreaks and then surround the outbreak and vaccinate everybody within that area. Slowly but surely, the outbreaks faded away and today (hopefully) the world is small pox free.

So, you have a large piece of software, you have developers making test cases by the thousands and systems test scenarios as well. Despite all of this, every now and then a bug pops up. When this happens, we do forensics on the issue, code reviews on the code again, improve test cases but we also do scenario specific testing around the use case where the fault occurred. This is additional testing because an 'outbreak' occurred. We build a test and then stress the system in that area where the fault occurred. 

Everyone wants perfect software, we could mathematically prove the software correct so that we know it's perfect but thats not a realistic goal. Even the space shuttle with a zero fail engineering approach has failed several times. I think the current approach we're taking is very like the small pox approach but we're starting with a very well tested system to begin with. Despite this, if we still find issues then we make test cases surrounding the use case where the failure occurred and then stress it and tease out anything else that may be lurking. We use tools to closely examine the system while it's being stressed hoping to find additional opportunities to improve the software.

The aircraft industry also uses epidemic based testing. They have an amazing quality control system during the design and operation of aircraft and DESPITE this, they still have crashes. When the crash occurs, they examine the wreckage, look for the fault and then improve the aircraft design around the area that failed. Then, they wait for the next fault...

This epidemic based test methodology is everywhere now whether we've called it this or not. This cycle of surround the event, stress it, improve it and then wait for next event seems to be the normal course of events now even for the most critical of systems that we create today.

Best practises on building data models for elastic scaling

I just read an excellent summary of the principles of doing this at this site. The points especially relevant to achieving this for WebSphere eXtreme Scale are the following.

• Duplicate data, don't normalize it

Here, this is how common data is handled. The comments are a great example. Duplicate the comments in to each partition and the partition is then keyed by the main key. This allows logic for the main key to be handled within a single partition without having to talk with other partitions which are almost 100% going to involve network IOs.

• Group data for concurrent reads

Here, group related data needed for the partitioned entity underneath this object. WXS provides a tree schema for each partitionable entity. Placing all needed data linked to this tree keeps it all local and eliminates network hops to fetch it. This is really an amplification of the "Duplicate data" rule.

• Structure data around how it will be used

Model the data in a form compatible with the business logic to be executed on it. This makes writing the logic fast and keeps the data close the to logic. This avoids joins.

• Compute attributes at write time

Add extra attributes with commonly calculated values, don't use queries to calculate them, update the total attribute when something changes and just query it. Assuming the queries are more frequent than the updates then this saves a lot of time.

• Create large entities with optional fields

This again is to avoid the small entities created when using a fully normalized model. Normalizing means joins and joins are expensive so try to avoid them if at all possible.

• Define schemas in models

The framework like WXS can't manage these denormalized models automatically for you. You'll need a model which knows how to do this and does it automatically when changes are written to the model. This model can run inside the grid collocated with the data so it's going to run fast.

• Place a many-to-many relation in the entity with the fewest number of elements

This basically says that rather than having a model which has a Company has a collection of employees, have a model with companies and employees with a list of companys. The list in the latter case is MUCH smaller than the other way.

• Avoid unbounded queries

This is kind of dangerously obvious but if you have a tera byte of data in a grid, don't ask for a sorted list of all records and send it back to my client app. The app will die. Bound it to the top 10 or 20 items.

• Avoid contention on datastore entities

This kind of goes without saying. If you use a single record in the grid all the time then it's going to bottleneck there so try to avoid or rather don't do this.

Summary

The linked article is pretty cool and summarizes much of what we already knew about the best practises on designing for DataGrids. So, here it is, enjoy.

Elastic scaling with WebSphere eXtreme Scale, a killer feature for Cloud and virtualized environments.

Hosting applications on platforms such as Amazon EC2, its competitors or on private clouds almost requires elastic scaling. Elastic scaling means that adding or subtracting virtual machines to/from the application farm is handled by the application in a transparent way.

The easiest scenario here is a completely stateless application. Here, adding or removing a machine and then incorporating it to the router will achieve that effect but such applications are rare as a database is usually involved.

Databases are typically not elastic. An Oracle, DB2 or mysql database typically runs on a single machine and besides moving the machine to a bigger machine, that's as far as that goes. The good news here is modern machines are very fast so that can still go a long way. Techniques like database sharding where the data is split between several databases with a directory database storing the record locations or some consistent hashing approach is next usually in terms of solving the database but this is extra work also and it doesn't handle failover etc.

A combination of WebSphere eXtreme Scale and a database is an interesting scenario which easily adds elastic scaling to the persistence tier as we'll now see. WebSphere eXtreme Scale provides out of the box elastic scaling for both HashMap style or queue based XTP. Adding and removing 'boxes' will result in automatic and transparent incorporation of the change in to the grid. WebSphere eXtreme Scale can use OpenJPA or Hibernate to interface between the grid and the database to seamlessly incorporate data preload, lazy read, sync write or write behind write behaviors of data. Failover and high availability are inherent in a product like WebSphere eXtreme Scale. WebSphere eXtreme Scales zone support can also ensure replicas are running in different Amazon EC2 zones providing out of the box higher availability levels without the cost of a fully mirrored system.

We see this pattern of WXS as system of record with write behind to a conventional database as a very common pattern. The application tier makes service requests on WXS for the data and WXS can be scaled elastically depending on the load/capacity requirements. More machines means more memory, CPU and network bandwidth. This elastic scaling architecture allows hosted applications on public or private clouds to scale seamlessly by leveraging WXS to provide elastic persistence services with the database running behind it on a VM that can be restarted quickly on a failure.

A typical Amazon EC2 setup might be an EBS (Elastic block store) volume where the database logs and data are stored. An AMI for the database and an elastic IP. An AMI is a virtual machine definition. This includes the operating system, installed sofware, scripts etc. An instance of the database AMI is started which creates the actual virtual machine using that AMI, the user then attaches the EBS volume and elastic IP to that virtual machine instance. Now we have a running database. If the virtual machine fails then we restart the AMI on a new instance, connect the EBS and elastic IP to the new virtual machine instance and keep going. Downtime should be under 5 minutes if its automated. Backups are a piece of cake with EBS as a customer can create a snapshot of the EBS volume at any time and then start another database instance using that snapshot to run offline reports or they could simply push the snapshot to S3 for archiving. Whats an elastic IP? Amazon EC2 virtual machines receive random IP address when they start. We need our database in a well known place so we can connect to it. An elastic IP is a public static IP address that you can ask Amazon for and then associated with a virtual machine. You can move it from a virtual machine to another if needed (failover for example).

Next is the persistence tier. This runs a collection of identical AMIs with the WXS stack inside. Whats in the stack? Linux, JVM, WXS, Hibernate/OpenJPA, database jdbc driver, any application POJOs and xml files for the grid. These virtual machines don't need elastic IP addresses as WXS virtualizes the collection as a single resource for the application. We'd recommend running at least three of these to lower the odds of a double failure. This stack has the Maps for all the data and each Map is configured to use our JPALoader and say OpenJPA to connect the map with the database. The database is configured to use the elastic IP as the database host name. The WXS AMIs all connect to a pair of WXS catalog servers running on two virtual machines with two elastic IPs. The customer may try to load all the data in to WXS, just load the hot set in to memory initially or do lazy loading where data is only loaded when it's first referenced. If the dataset is larger than will fit in memory then our heap level triggered evictor can help there. It will evict data on an LRU basis when the JVM heap reaches 70% full. This keeps the system from running out of memory even when failures scale down the grid capacity. Fixed map limits or TTLs will NOT protect you in an elastic system. Our heap level eviction will.

Next is the application tier. This can be running in the same virtual machines as the WXS stack. This would work if the WXS and application tiers scaled linearly together. The application tier might just be a tomcat JVM or WebSphere CE. If its a web application then WXS can provide HTTP session failover as well as the persistence service.

The database tier doesn't need to be elastic as the persistence tier services the application tier directly and uses the database only for long term storage. The database is also useful for handling reports etc as standard reporting tools will work with a database but none will work with a DataGrid type product. Even ones with SQL APIs may have substantial trouble running complex queries on a partitioned datagrid so a database is probably still best bet for reports. The database will be slightly behind the data in the grid but only a few seconds.

Lastly, a routing tier will accept incoming requests and spray them on to the application tier. This could be as simple as round robin DNS but given the need for elastic IPs then I'd expect some kind of proxy server to be running there, probably two for redundancy. The routers would have elastic IPs and DNS would be configured to use those.

WXS needs a couple of JVMs running the catalog service (this manages WXS) and these can be run in a separate couple of virtual machines or inside say the first two WXS virtual machines. They need an elastic IP address. The application is configured to connect to the grid using these IP addresses. The catalog service manages laying out the grid and clients boostrap to it to obtain a connection to the grid.

This example architecture is fully elastic out of the box. Adding virtual machines will elastically scale the application and persistence tier. The database tier doesn't need elastic scaling because of the WXS based persistence tier. It's implicitly highly available given the nature of WXS providing the persistence service and a manual mechanism for restarting the database would work as even if the database was down, WXS could service all reads and updates. Clearly, reads are only serviceable if they were already cached in the WXS tier. When the database restarts then the pending dirty writes will be flushed to the database.

I hope this shows the value of elastic scaling in middleware products like WebSphere eXtreme Scale. Virtualized environment provide a very simple production environment of homogenous machines and lowers the complexity of trying to do this. It can be done on virtual environments in the cloud or private environments in a company. Without elastic scaling in the persistence tiers then scaling out the elastic application tier will only work so far. The database will be a big issue unless you've built your own sharding layer and you still need to handle failover or replication to make it all fault tolerant. The WXS based persistence tier is significantly less difficult to deploy because it has fully elastic scaling. This means you can focus on the business logic and leave this aspect to middleware making your project more successfull, cheaper to developer and gives you a fast time to market.

The problem with JPA and Java persistence in general...

The Java community is currently focused around JPA as the persistence API but I think it's missing the big picture. JPA is a combination of several facets which themselves need to be separated into separate specifications so that we can get a better programming model.

JPA consists of the data model which can be viewed as the POJOs + annotations for keys and relationships. There is an API for interacting with a data source to find and query entities within that data model and there is a set of meta data for mapping that data model to a SQL database.

The problem is they are all combined in a single spec. This is problematic because it couples together different facets where are useful in their own right. Lets look at the following problems faced by Java developers.

The web developer

Web session data is still stuck in the dark ages. Session data is kept in a flat Map of key/value pairs. It's difficult for a developer to work with a hierarchical data model for the session which is common and the API is dumb as it doesn't automatically detect changes and just persist those to the Session like JPA can do using byte code weaving or straight copy and compare approaches. Why can't a developer pick a data model expressed as a POJO object graph and then find the root of that in a session and then walk it, update it and have changes picked up automatically. If several modules share the session then why should they all need to use the same POJOs? Why can't each module map it's own POJOs to the data model for the session? The current servlet spec or even the proposed ones are sorely lacking in modernizing the specification down this path.

The work flow developer or JBoss Seam or Spring Webflow developer

The same thing applies here. Why don't we have a standard data model which can be persisted to a flow/conversational flow engine. Why isn't the specification for the data model for a flow projectable on to the POJO data models a developer wants to use. The seam and webflow stuff are attempts to modernize the session management in the servlet spec but without a common data model, annotations for capturing these meta models then we'll end up with a separate incompatible spec and redundant tooling for capturing and interacting with this meta data.

The cache or XTP developer

Most distributed caches still work with a single level Map of key/value pairs. WebSphere eXtreme Scale is the exception here in that it can layer a POJO graph model on top. This is our EntityManager API. We modeled this after the JPA spec for the pieces related to key and relationship definitions. We support different applications sharing data kept in a shared cache even when each application uses a different set of POJOs for the same data. The respective applications just use different metadata for the mapping to those POJOs from a master schema stored in the shared data grid. We built a standalone component within WXS called the projector which has a master meta data and it can then project data stored in a TupleStore to POJOs suitably annotated for the mapping and then push any changes from those POJOs back to TupleStores. We currently implemented TupleStores for our internal Maps of key/value tuples but in theory this could be used to represent any TupleStore (flow instance state or HTTP session). We shouldn't have had to write that component, it should be part of a J2SE specification built in to every JVM. Maybe a first step here once any internal hurdles are overcome would be to contribute this component to the community as an initial building block to start to move down this path.

The database developer

This is the ONLY developer catered to by JPA despite the fact that if this spec was split than many facets, it could be reused in a standard way for the other developer types. It's just a mixed up mess that needs to be pulled apart in to standalone specifications which can be combined to form a standard persistence programming model for all types of programmer.

Summary

You can see there are some parts of the Java programming model which are solely lacking. Flows, caches, conversational state, web developers and database developers should all be shared common specifications for data models, the mapping of that to various non SQL and SQL back-ends and support common query engines, projection/unprojection etc. We need to pull our collective heads out of the sand and look around for the big picture, otherwise, competitively, it's not looking great.

We need these common specifications so that our data based tooling in eclipse etc can be done once and then reused for all these purposes. The latest here is the upcoming criteria API. I'd like to be able to reuse this for WebSphere eXtreme Scale queries for example, why is this spec tied to JPA at all. We need a common POJO persistence spec containing abstract queries, factories, and object meta data like keys or relationships. Layered on top of this and its RI component would be JPA, HTTP Sessions, Caching, Web conversation flows and work flows.