/dev/websphere

Implementing global indexes on WebSphere eXtreme Scale

We get asked a lot how to create global indexes with WebSphere eXtreme Scale (Scale). Scale usually keeps K/V pairs in a large distributed hash table (DHT). Customer want to index attributes of the value and then run queries to find all the Keys with values match some criteria based on the index.

You can, of course, index the attribute and then run a parallel agent to run a query on each partition and then union the results. That works but isn't great from a throughput scaling point of view as every server in the grid is doing a part of the total query. The throughput is then limited to the throughput of the slowest box in the grid. It'll work but it's not great. What it does well is to speed searching over the records, as the larger the grid, the more processors are running the search.

But, there is a better way. It's not so difficult to implement a global index. This is a grid wide index for a specific attribute. This is implementable on top of a DHT (like WebSphere eXtreme Scale) and makes it possible to find all matching keys in a single operation using this type of index. Basically, we make a Map whose key is the search term and the value is the list of keys containing an attribute with that search term. Thus, an index lookup results in a Map.get(value) returning the list of keys. The list of keys can then use WXSUtils.getAll to bulk fetch the results quickly and now we have a better solution. The throughput of this system is much better than a parallel search implementation. Each index lookup results in one RPC to exactly one server. Thus, one server can do N of those look ups. M servers can do M x N lookups and so now we have a better response time as well as linear scaling on index look ups from a throughput point of view.

The code for this is available on my github repository in the wxsutils jar with full source code. It allows three types of index to be created. An exact match one (==), an index which matches any attribute prefixed by a symbol (LIKE XXXX%) and finally one which matches any records whose attribute contains the query anywhere (LIKE %XXX%).

I'll post a blog entry shortly showing some code on how to use these indexes. The Test cases on github also include code showing how it works.

Why would you want a global index? Lets suppose you wanted to find all matching products containing a substring anywhere in the product name. The SubStringIndex in this code allows you to easily index that attribute in a Map containing these records and then find all matching products in low single digit milliseconds or so (depending on the processor), end to end. WebSphere eXtreme Scale can maintain that response time as you scale up the grid by adding boxes. More boxes means larger indexes are allowed and there is more CPU/network for requests, linear scaling. Remember, the query throughput is M x N which M is the number of servers in the grid.

The classes to look at in github are in the package "com.devwebsphere.wxssearch". The IndexManager is the main class from which you obtain an Index instance to search an individual attribute or you can run multiple attribute searches directly using the IndexManager. There are annotations to annotate which attribute on the value object you want to index. The annotations specify the kind of index as well as index configuration properties.

SearchSOA article on WebSphere eXtreme Scale using Microsoft OData for REST

Read the article here at this link.

Bloom Filters for efficient fraud detection with IBM WebSphere eXtreme Scale

I've written and uploaded a Bloom Filter implementation and uploaded it as part of the wxsutils library I keep on github. You can look at the source code here. If you want to use it with your WebSphere eXtreme Scale project just grab the jar from here and add it to your class path.

A bloom filter is best explained elsewhere in terms of how it works. The use cases for where it's useful are worth going over again here though.

Cassandra uses a bloom filter to track all the keys where are stored on disk in a space efficient data structure, i.e. the bloom filter. It uses this to avoid disk I/Os that would otherwise be used looking up keys definitely not on the disk.

Another more customer friendly example is a fraud detection scenario. Lets say you're a credit card company that has the spending history over 6 months of all your customers. You want to catch purchases to vendors that are not in that history. One way would be to keep all the customer vendor history in a database or in a datagrid and check if a new transaction is for an existing relationship. If not then you'd escalate it for scrutiny. For a large credit card company, this can easily be billions of rows. Lets say 200 million credit cards with 200 vendors each? That could be a lot of memory.

A better approach would be to create a custom bloom filter for each customer. The bloom filter is initialized for the number of actual vendors the customer deals with, maybe 200 vendors on average. Each vendor is then added to the bloom filter. This filter is a byte array when this is done. It's about 256 bytes. Pretty small to store 200 vendor strings, no? This filter can then be loaded in to a DataGrid and when a new transaction comes in, the filter for that customer can check with a 97% accuracy whether the vendor was used before by the customer. This is very accurate and was achieved with a huge memory saving and we can improve the accuracy further at the expense of memory if needed. This is the beauty of bloom filters. This example can be used for:

• Tracking unusual telephone numbers for cell phone fraud
• Cities/zip codes that the card was used in
• Categories of vendor the customer deals with (i.e. electronics is common but not motorsports)
• and so on.

Basically any time you want a check a list of things for whether something is present or not present then it's a very memory efficient way to do this.

Another use case would be a security system that wants to track every password every used by a user and try to prevent them using one every again or prevent a customer from using banned passwords. A bloom filter could be used to track existing passwords or the list of banned passwords and allow those lists to be checked quickly without taking a lot of memory or disk space. It has huge advantages in the existing password case as the passwords are not physically stored, the bloom filter is like a fixed size set of password hashes.

IBM WebSphere eXtreme Scale customers can create Maps keyed by customer name whose value is a Bloom Filter. A WXS agent can be used to quickly check if a vendor or some related entity probably not in the 'list' of valid things for that customer. A batch job can generate the bloom filters from the actual vendor data and the grid can be preloaded with the resulting bloom filters periodically. The actual vendor data stays out of the grid.

Security is also improved with a Bloom filter. The grid is not storing any sensitive information. The actual vendors the customer deals with are NOT there. The bloom filter is just a bit array that can be used to check if a vendor is not there, it's not a list of vendors that could be hacked.

Bloom Filters are a very useful data structure to save memory as well as CPU path when checking lists of things for exclusion. Applications can use DataGrids such as IBM WebSphere eXtreme Scale to build extremely scalable, high performance systems exploiting techniques like BloomFilters for fraud detection as well as other scenarios.

Memory usage/sizing in IBM WebSphere eXtreme Scale

UPDATED to include compressed 64 bit numbers.

I figured this was worth doing quickly so here are some numbers for the overhead in WebSphere eXtreme Scale for storing 1m objects in a grid. I'm using a single JVM grid in this case but this applies no matter how many JVMs.

I'm storing 1 million Person objects with the following structure:

    String firstName;

    String surname;

    String street;

    String city;

    String zipcode;

I'm populating these strings with some small sample strings. I'm using a key which is just a person# string.

I did the test with 32 bit and 64 bit Sun Java 6 server JVMs on my macbook. I pulled heap usage using the jmap utility supplied with the JDK (jmap -histo:live pid).

WebSphere eXtreme Scale can store object values in two ways, as POJOs or as a compact serialized form (more efficient than normal serialization, like a byte packed version of the data in the value, only available in 7.0+). The byte mode is enabled by having a copyMode="COPY_TO_BYTES" on the backingmap definition in objectgrid.xml. Lets look at the heap size after a system GC in 64 bit JVM mode:

• 64 bit POJO    782.7MB
• 64 bit BYTE    381.0MB

So, you can see the BYTE representation is much more compact than the POJO version, about double the density.

Lets look at when we use -XX:+UseCompressedOops

• 64 bit POJO 592.0M
• 64 bit BYTE 299.4M

You can see that compressed mode significantly reduces the 64 bit overhead by about 25%.

If we do the same thing with 32 bit JVM (-d32 on Mac) then we see

• 32 bit POJO    523.6MB
• 32 bit BYTE    261.0MB

You can see 64 bit has quite a large overhead on Sun. Compressed 64 bit isn't so bad but still isn't as good as straight 32 bit. This is why we recommend 32 bit JVMs with heap sizes under 2.5Gb. It's just most efficient from a memory point of view.

In 32 bit mode, here is a break down for the POJO mode:

•  num     #instances         #bytes  class name
• ----------------------------------------------
•    1:       6016578      230933320  [C
•    2:       6017049      144409176  java.lang.String
•    3:       1000000       80000000  com.ibm.ws.objectgrid.plugins.CacheEntryHeapFactory$HeapEntry
•    4:       1000000       32000000  sample.bench.Person
•    5:        342796       19860816  [Lcom.ibm.ws.xs.util.AbstractMapEntry;

The WebSphere eXtreme Scale runtime uses around 80 bytes for each cache entry and we use a optimized hybrid multi-level hash map taking 19MB total or 19 bytes per entry. So, thats 99 bytes per entry.

The BYTES model looks like:

•  num     #instances         #bytes  class name
• ----------------------------------------------
•    1:       1005960      108283256  [B
•    2:       1000000       80000000  com.ibm.ws.objectgrid.plugins.ByteArrayCacheEntryFactory$ByteArrayEntry
•    3:       1016742       25359464  [C
•    4:       1017199       24412776  java.lang.String
•    5:        342802       19861344  [Lcom.ibm.ws.xs.util.AbstractMapEntry;

So, our overhead here is 80 bytes per byte array entry, the values are stored as byte[]s in the 108MB chunk and the hybrid hash map takes again 19MB. So, the overhead is still 99 bytes per entry but each entry is stored more efficiently than in POJO mode and it is this saving which makes the big difference in the heap sizes seen above.

Applications interacting with the grid remotely typically use byte mode. Applications running collocated with the grid for highest performance typically use POJO mode in combination with other features but thats another blog entry.

The WXS code about to be released drops the overhead down further to 64 bytes rather than 80 and that will be available soon.

Each partition primary/replica shard is kept in its own area and we've had that design from the beginning because if we kept all data in all shards in a JVM in the same hash map then splitting them is expensive so we never did it that way.

Hopefully, this helps customers size heaps and memory. Obviously, we are continuously working to improve memory density so we can change this when ever we choose to so individual class names we use are not guaranteed at any time.

memcached versus DataGrid, dumb server versus smart server...

Memcached is a pretty popular free open source distributed cache. It's pretty unique compared with the commercial data grids in that it has a completely unique design.

Memcached is basically a dumb server. This means it's a process running on a Linux box typically which accepts client requests for keys and returns the data. Updates and deletes are also supported. This is ALL the function in the server, hence, dumb server. The client is the only piece which is 'distributed' or even knows about other servers. The clients typically get a collection of host:ip addresses (they all NEED to use the same list) and the clients hash the key over the set of online servers. If there were 4 servers running then all clients will hash data for a key over these 4 servers. Clients will write data for a key K to a particular server using this hash algorithm. One server.

Now, thats what memcached does out of the box. Note, there is no replication on the server side. If a server fails then the clients will hash keys over 3 servers but this changes everything from a hashing point of view so it's likely you'll get a very high miss rate as it's unlikely keys still hash to the same server as before given the modulo part of the hashing is now 3 instead of 4.

Clients work around this by adding a consistent hashing layer on top of the normal client. This works better and writes may be written to a server and the next server in the hash ring. This slows down writes by 2x but allows some recoverability when a server fails. The next server in the ring has the data from the dead server so it's not as bad as before BUT if this server also fails then now the cache has lost the data. It's not replicated remember, it's just written twice. After the first server failed, the clients write changes to the next server and the following server in the hash ring. The new next server in the ring does not have a copy of all the data. It was never written there. It will just have new changes since the failure. There is no active replication going on here.

Most of the materiel on the web shows various workarounds to try to make memcached 'reliable'. A lot of the facebook engineering blog entries focus on this aspect. What they had to do to make it reliable. This works for facebook but won't work in general because I don't think the majority of customers are structured like they are from a build your own mentality. I was talking to someone from one of the big internet shops last week and they always run two memcached clusters for HA. If a server in cluster A fails then they stop using cluster A and switch to B. This works but thats a 50% loss in hardware because of a single failure.

Smart servers

Commercial data grids like IBM WebSphere eXtreme Scale are very different. They handle making the data fault tolerant on the server side with built-in replication and elastic scale out and scale in. Their clients understand the server side data layout or routing. If a server fails with WebSphere eXtreme Scale then no data is lost. The software actively maintains replicas on other servers. Clients that stored an entry with a key K before will still find it. It just works. You don't need consistent hashing on the client, our client does it using routing tables instead. Adding more server processes causes the product to automatically redistribute the minimum amount of data to leverage the new memory/CPU and network available through the new JVMs.

The smart servers allow proper replication which means that the cache isn't as brittle as the memcached cache when servers come and go. If nothing ever fails then the memcached approach works but the potential cache hit rates over time if multiple failures occur mean it can be a liability. The last facebook entry I read on this showed them crawling the logs to try to recover lost data to make the 'next server' more complete. They get this issue and are jumping through hoops to handle it but again, most commercial customers won't do this. They expect the product to do this sort of stuff like replication and they view it as table steaks.

So, memcached for sure has its uses but it's got some limitations which need a fair bit of work by its users to handle correctly if its not exactly what you want or risk some bad scenarios as the cache hit rate climbs. It's very, very fast but you gotta remember, it's not doing much either... No transactions, no consistency, no replication, no isolation and so on. If you don't need these features then it's pretty fast but if you do need this stuff then it starts to slow as you add code on the client side to try and compensate for the limitations.

Use IBM WebSphere eXtreme Scale to get the most out of your SOA investment!

Many customers have now used SOA best practices to rebuild their infrastructure. But, it's surprising how many of the customers are leaving a lot of potential savings on the table. A SOA style architecture with an ESB provides a great infrastructure to quickly achieve sizable savings and further demonstrate the value of SOA to the business.

I think a lot of customers think of extreme performance when they think about products like IBM WebSphere eXtreme Scale. Millions of transactions per second. Hundreds of servers and so on. "We're not facebook or google, we don't need this". They may wrongly conclude it's not for them. But, the reality is there are very few customers with those kinds of requirements. However, there are many, many more customers with conventional type applications doing a couple of hundred transactions/sec or less who could readily use WebSphere eXtreme Scale to save them a lot of money for little investment especially when they have already made the investment to go SOA.

WebSphere eXtreme Scale can be readily used as a very advanced caching technology with immense capacity and very low response times. We regularly talk with customers who back ends systems are under pressure. These backend systems range from simple databases, to SAP systems, mainframes running DB/390, CICS, IMS or other valuable enterprise applications. Todays economic climate means belts are tightening so there is little money available to expand the capacity of these backend systems even as additional load is known to be coming.

At the same time even if business is flat, budget controls are at best flat. Many customers report having to run with flat or smaller budgets for backend systems even while adding more demands on them in the form of additional applications leveraging the backend systems.

Most customers estimate that there are many redundant requests being handled by backend systems. One customer with an ESB fronted mainframe estimated 25% of all service requests are duplicates of requests issued within 24 hours, another indicated 50%. Some use cases are as high as 80%. Adding a large cache to such as ESB can capitalize on this and effectively reduce the load on those backend systems by that amount. This results in better response times and cost savings in terms of load on the backend systems. This means your backend can handle 25% more load without a capacity increase.

An ESB cache is an extremely cost effective way to introduce WebSphere eXtreme Scale in to an organization. Add a mediation to the ESB. It doesn't change the front end applications invoking services on the bus NOR does it change the backend system. It just means adding a 'mediation' to the ESB which checks the cache first before hitting the backend. WebSphere eXtreme Scale can handle evicting the data.

Could you do this with a normal, non distributed cache? I don't think so. That 25% saving takes 24 hours of caching to do. That can be a lot of data, more than will fit in a single JVM. You also need a shared cache so that no matter which server received the initial service request, ALL the servers benefit from the cache as soon as a single server caches something. This is crucial to getting the most out of this pattern. A conventional in process cache CANNOT do this. You need a network attached distributed cache. WebSphere eXtreme Scale makes 10GB, 100Gb or even Tb caches possible with very little work. This kind of use case is usually very straightforward to implement and can usually be readily integrated in to a SOA application.

So, what are you waiting for? You had the foresight to envision a SOA architecture and you invested the money to deploy it, exposing backends as services, deploying an ESB and so on. Make sure you get the most out of that investment and deploy WebSphere eXtreme Scale to get you that kind of saving too. This is low hanging fruit. We had a recent customer achieve 500K USD a MONTH in savings after deploying IBM WebSphere eXtreme Scale to take load of a mainframe.

Conversational state in IBM WebSphere eXtreme Scale: PER_CONTAINER grids

We added this type of grid starting with V6.1.0.3, prior to this we had 'normal' grids which are a fixed number of partitions and the key of a Map hashed to one of those partitions. We place the partitions on the set of online container JVMs and automatically scale them out or in if JVMs are added and removed. This grid works very well for key based grids, where the applications uses a key object to locate data in the grid.

PER_CONTAINER grids are different. You specify the grid uses PER_CONTAINER using the placementPolicy attribute in the deployment xml file. Here, you specify how many partitions you want per container JVM that you start. Remember, normally, you specify how many partitions total in the grid, not per container. If the number of partitions is 5 then when you start a container JVM, WXS will create 5 new anonymous partition primaries on that JVM and will create any replicas on the other container JVMs already running. Here is a sequence:

• Start a JVM, it gets 5 primaries. (P0-P4)
  JVM#1 -> P0, P1, P2, P3, P4 
• Start another JVM, it gets 5 primaries (P5-P9). Replicas for P0-4 are placed on it and replicas for P5-9 go on JVM#1.
  JVM#1 -> P0,P1,P2,P3,P4,R5,R6,R7,R8,R9
  JVM#2 -> P5,P6,P7,P8,P9,R0,R1,R2,R3,R4
    
•  Start another JVM, it gets 5 primaries (p10-p14). The replicas for p10-p14 are placed on JVM#1 and JVM#2 and some of the replicas for P0-P9 are moved to JVM#3 to balance the load.
  
  
  

  JVM#1 -> P0,P1,P2,P3,P4,R7,R8,R9,R10,R11,R12

  JVM#2 -> P5,P6,P7,P8,P9,R2,R3,R4,R13,R14

  JVM#3 -> P10,P11,P12,P13,P14,R5,R6,R0,R1
    

This continues as more JVMs are started. The grid will always create 5 new partition primaries on each new JVM and will then place replicas for those on the existing JVMs and will then balance the replicas out so the new JVM gets it's share. But, WXS will NEVER move a primary in PER_CONTAINER mode. It only moves replicas for balancing.

The partition numbers are arbitrary and have nothing to do with keys. You cannot do key based routing with this type of grid. If a JVM was to be stopped then the partition ids which were created for that JVM are not longer in use. This means there is a gap in the partition ids. There would no longer be partitions 5 to 9 if JVM#2 was to die in the above example. There would only be 0-4 and 10-14. This is why key based hashing doesn't work here. 

So, what's it good for? It's good for things like HTTP Session replication or application session state. Here, a HTTP router assigns a HTTP Session to a servlet container. The servlet container needs to create a HTTP Session for it and will choose one of the five local partition primaries for the session. The 'id' of the partition chosen is then stored in the cookie. The servlet container now has local access to the session state which means zero latency access to the data for this request as long as session affinity is maintained. Any changes to the partition are replicated by WXS.

Why not just have one partition per JVM? 5 or more likely 10 is good because think about what happens if a JVM fails. We want the burden of the JVM failing to be spread across the cluster 'evenly'. If there was one partition then when the JVM fails then one JVM has to pick up the entire load, the one where the replica was. This is bad because the load there may double. If there is 5 partitions per JVM then 5 JVMs pick up the load of the JVM which lowers the impact on each JVM by 80%. By using multiple partitions per JVM, we are lowering the impact on the replica substantially. Another way to look at it is if a JVM suddenly spiked then the replication load of that JVM is spread over 5 JVMs, which is better than one.

The next issue I hope some have noticed is that this means every time a JVM starts, we may five more partition primaries and 5 more replicas. Over time, we just keep making partitions and they never go away. This isn't how it works. When a JVM starts, it gets five partition primaries. These are 'home' primaries. They exist on the container which created them. If the JVM fails then the replicas become primaries and WXS will create 5 more replicas to maintain high availability (unless auto repair is turned off!). The new primaries are in a different JVM than the one that creates the partitions. They are foreign primaries. The application should never place new state or sessions in a foreign primary. They should be allowed to 'drain', sessions get evicted etc. Eventually, the foreign primary has no entries and WXS automatically deletes it and its associated replicas. The foreign primaries purpose is to allow existing sessions to still be available but it's not for new sessions.

How does a client interact with such a 'key' less grid then? The client just begins a transaction and then stores data in the grid. The keys are meaning less. The client can ask the Session for a SessionHandle object. This is a serializable handle which allows the client to get back to the same partition later. WXS picks a partition for the client from the list of home partition primaries. It will never return a foreign primary partition. This SessionHandle could be serialized in a HTTP cookie or similar device and then later on receiving it again, it can be converted back in to a SessionHandle and provided to the WXS API to obtain a Session bound to the same partition again. You cannot use Agents to interact with a PER_CONTAINER grid for now.

This is different than a normal FIXED_PARTITION or hash grid because the client stores data in a place in the grid, gets a handle to it and uses the handle to access it again next time. There is no application supplied key here.

Obviously, WXS is not making a new partition for each 'session'. Therefore, the keys used to store data in the partition should be unique within that partition. Maybe the client generates a unique SessionID and then uses that as the key to find information in Maps in that partition. Multiple client sessions will be assigned to the same partition so the application needs to make sure different keys are used to store session data in that partition for each partition.

I used 5 partitions as an example here but the numberOfPartitions parameter in the objectgrid.xml can be used to specify this, it just means the number of partitions per container rather than per grid. The number of replicas is specified in the normal way.

This also works with multiple data centers. If possible WXS will return a SessionHandle to a partition whose primary is located within the same zoned/data center as that client. The client can specify the zone as a parameter to the JVM or using an API. The client zone id can be set using serverproperties or clientproperties

The PER_CONTAINER style of grid suits applications which store conversational type state rather than database oriented data. The key to access it is some kind of conversation id and is not related to a specific database record or something like that. It provides higher performance (because the partition primaries can be collocated with the servlets for example), easier configuration (no need to worry about how many partitions are needed for how many JVMs), works very well in multiple data center scenarios (zone based routing). It provides a very useful extra tool in the bag for effectively using WebSphere eXtreme Scale.

Running iLog jRules in WebSphere eXtreme Scale

Quite a few customers have been looking for a fault tolerant rules engine. I've been working with Victor Moore, a Distinguished Engineer working in Telco on this and we have something now which looks good. Fault tolerant here means that the a specific rules engines needs to run in exactly one Java Virtual Machine and fail over to another if its current JVM fails or stops. The working memory for that rules engine needs to be stored in replicated memory so that if the JVM hosting it fails then there is no lost data.

Clients wanting to access the rules engine need a way to add or remove items from the working memory of the rules engine and also interrogate the working memory looking for results if needed. Of course, the rules engine can execute Java code as a result of rules firing within the current JVM. WebSphere eXtreme Scale handles routing these client requests in a transparent fashion to the JVM currently hosting the rules engine.

The prototype (for Telco and Chemical industries but could be used in any industry) allows the customer to define a set of rules engines. The prototype then runs these rules engines in as many JVMs as are required. WebSphere eXtreme Scale manages the rules engines availability and ensures each rules engine is running in exactly one of the JVMs and we wrote some code to back the working memory of the rules engine with a memory replicated Map. If a JVM failure occurred then we fail over the rules engines that were in that JVM to the JVMs with the replicas of the working set. WebSphere eXtreme Scale would then create additional replicas to achieve full fault tolerance automatically. 

WebSphere eXtreme Scale handles making sure the rules engines are spread evenly across the set of JVMs and handles moving them around to ensure this balance when additional JVMs are added or JVMs fail, this is elastic scaling. The prototype could easily be extended to allow rules engines to be added or removed while it's all running.

If anyone is interested in seeing the code then email myself or Vic and this will likely end up as an article/blog entry with the sample code over the next few weeks. The prototype basically gives you an XTP business rules grid.

Logging failed write behind updates in IBM WebSphere eXtreme Scale

Write behind is a technology to write updated records asynchronously to a backend such as a database. The application can 'commit' changes to replicated memory only and then periodically, all updated records are written to the database in a large batch. This eliminates the database from the response time for updates which improves performance and drastically lowers the load on the database by combining lots of small transactions in to a few big ones.

Clearly, once the application updates the data then the writes to the database must succeed. Sometimes, there are problems. Unforeseen interactions with the data or just application bugs. When the write behind logic cannot update a record then it inserts an entry in a special Map associated with the map holding the failed write behind update. The application needs to watch this map and handle these errors. Usually, the application will just log them to disk, it may try to reconcile the problem also. It all depends on whats possible in the application.

I'm attaching sample source code showing how to write such a watcher. The watcher should be added to your objectgrid.xml file like this:

        <objectGrid name="Grid">

        <bean id="TransactionCallback" className="purequery.loader.PQTxCallback">

            <property name="connecturi" type="java.lang.String" value="jdbc:db2://172.16.129.133:50000/SAMPLE"/>

            <property name="username" type="java.lang.String" value="db2admin"/>

            <property name="password" type="java.lang.String" value="db2admin"/>

            </bean>

                     

        <bean id="ObjectGridEventListener" className="utils.WriteBehindDumper" />

You can see the ObjectGridEventListener bean thats been added which is the write behind watcher or dumper is what I've called it here.

It basically interacts over the Maps for all primary shards in a JVM looking for ones with write behind enabled. If it finds one then it tries to log up to 100 bad updates. It keeps watching a primary shard until it's moved to a different JVM. All applications using write behind MUST use a watcher similar to this one as otherwise the JVMs will run out of memory as this error map is never evicted.

Download WriteBehindDumper

IBM PureQuery Loader for WebSphere eXtreme Scale

I wrote this a couple of weeks ago for the redis API I'm working on. It implements a generic purequery Loader which works for any POJO following some simple rules. The POJO must have public attributes with scalar types. The attributes must have the same name as the database column. Any attributes used as keys must have an Id annotation. The key and value POJOs must share the same key attributes with the id annotations. So, each map has two POJOs, a key and a value. Both have all the key attributes annotated with @Id (the WXS one, not the JPA one), the value POJO also has the non key attributes.

The objectgrid.xml must specify the PQTxCallback class as the transaction callback and must specify the connection uri, database user name and password to be used.

The objectgrid.xml should have something like this:

        <objectGrid name="Grid">

        <bean id="TransactionCallback" className="purequery.loader.PQTxCallback">

            <property name="connecturi" type="java.lang.String" value="jdbc:db2://172.16.129.133:50000/SAMPLE"/>

            <property name="username" type="java.lang.String" value="db2admin"/>

            <property name="password" type="java.lang.String" value="db2admin"/>

            </bean>

The Map using the Loader looks like this:

            <backingMap name="list-item-string-long" writeBehind="T5;C900" lockStrategy="PESSIMISTIC" evictionTriggers="MEMORY_USAGE_THRESHOLD" ttlEvictorType="LAST_ACCESS_TIME" timeToLive="600" pluginCollectionRef="list-item-string-long"/>

Here is an example loader specification:

     <backingMapPluginCollection id="list-item-string-long">

        <bean id="Loader" className="purequery.loader.GenericPQLoader">

            <property name="tableName" 

                      type="java.lang.String" 

                      value="LISTITEMSTRINGLONG"/>

            <property name="className" 

                      type="java.lang.String" 

                      value="data.list.ListItem"/>

        </bean>

    </backingMapPluginCollection>

The classpath looks like this:

Classpath = /Users/bnewport/Desktop/ObjectGridTest/objectgrid.jar:$DEVDIRpq21/db2jcc_license_cisuz.jar:$DEVDIR/pq21/db2cc_license_cu.jar:$DEVDIR/pq21/db2jcc.jar:$DEVDIR/pq21/pdq.jar:$DEVDIR/pq21/pdqmgmt.jar:$DEVDIR/apache-tomcat-6.0.14/lib/tomcat-dbcp.jar

I'm using dbcp from tomcat as a J2SE connection pool/DataSource.

Source code is here:

Download Pqloader