/dev/websphere

Distributed grids as backing store for Lucene Indexes - possible solution

I'd tried to use a distributed grid with Lucene before. I was at a customer with the competitors doing a face off type engagement. We (the competitors and myself) both implemented Directory implementations for Lucene on top of our respective caching products. The implementation I wrote is on github here. The customer had a very large lucene index in memory, abouit 30GB and they were seeing very large GC pauses (minutes) using the RAMDirectory supplied with Lucene. The plan was to use a grid based directory to move the indexes in to a shared remote grid.

This proved much slower than the RAMDirectory the customer was already using. Lucene was too 'chatty' with the Directory and this means lots of RPCs to the grid. The Directory abstraction is too low level. We tried solutions like near caches but depending on the query, the near cache could need to be very big thus making the solution not viable. This ultimately meant that the customer didn't go forward with either grid as a directory store.

However, I was reading about Solandra recently and they have what looks like a better approach. The problem we had was that just implementing a Directory is too low-level. There ends up being too many RPCs between the Lucene engine and the grid. Solandra implements a Cassandra store underneath Lucene. They did not implement a Cassandra directory which would have had the same issues as I just discussed. Instead, they implemented a higher level plugin. They moved up to implement IndexReader and IndexWriter and this allows them to push some of the search down in to Cassandra's native index/query support rather than just use it as a block store. This seems to reduce the number of RPCs between the store and the lucene engine considerably.

This looks like a much more performant approach and one that should work well with a data grid also. If I get time then I may try to implement this with a datagrid and post results.

Apache Lucene and DataGrids, a good match?

I'm not so sure anymore. I was recently investigating this with a large customer using a big set of Lucene indexes, about 25GB. They had the index on disk and were copying the index in to a RAMDirectory. This resulted in fast search times but the JVM heap was very large, well over 40GB. Garbage collection pauses were long because of the large heap.

So, the plan was to store the index in a remote IBM WebSphere eXtreme Scale grid instead. Lucene makes this pretty easy. I was able to write a Lucene directory store for WXS and measure the performance. In theory, we'd be storing the 25GB in a collection of JVMs. Lets say 7 JVMs with heaps in the 9GB range. The data would be replicated normally so we're storing about 50GB of data in 63GB of JVM heap. The clients running the lucene engine and the Lucene plugin for WXS (wxslucene) would now just need a small amount of memory and everything is great.

But, performance can be a problem. We found we needed a near cache on the lucene side to have acceptable performance. This shouldn't be a surprise. The cost in the RAMDirectory to find a block is practically zero. Fetching that block from a network remote JVM simply costs more and when the baseline is zero, even 2-3ms is a lot slower. It's hard to beat zero.

We enabled a near cache which helped significantly with some queries but other queries touched a lot of blocks in the indexes and the game then was to make the near cache bigger and bigger to have higher hit rates. This worked for some simpler queries but more complex queries still caused high near cache misses and resulted in poor performance.

We could have simply make the near caches large enough to hold most of the blocks but thats kind of pointless as the whole point of the exercise was to reduce the amount of memory per client JVM, not use the same as before PLUS the memory requirements of the data grid. We found a pretty dramatic fall off in performance once the hit rate fell even a small amount from the high 90s. It would be better described as a cliff for some of the queries. It was very sensitive to the hit rate being extremely high.

So, in the end, we were unable to get the desired level of performance for the more complex queries and a competitor had the same conclusion with their directory implementation. After speaking with Shay Banon (Apache Compass, Elastic Search, Lucene expert) he basically agreed and said the way to go with Lucene is to shard it and keep the search engine close to the data. Then do parallel queries across all shards keeping the data and the processing within a single box as much as possible and then combine the results. This is basically what google does after all so this shouldn't be a surprise but it was a useful exercise.

I've seen a Cassandra based store for Lucene, Lucassandra or something, also but it's going to have the same problems as mentioned above except it'll be even slower than a memory datagrid probably (don't start benchmarking this for pete sake, moving a byte[] from memory to a socket is faster than anything thats disk based as a general rule).

The code for the WXS lucene directory is here. It may work well for certain types of Lucene query but I couldn't say that in general that it's always better and I think the same applies to anybodies datagrid directory implementation unless there is something else going on that can be played to.

Apache Lucene Directory update

I've checked in what looks like a pretty good Directory implementation for now. The current version is configurable in terms of whether to use block compression and block size itself.

I've changed the way WXS maps are used to use dynamic Maps. I defined two Map templates, one for file meta data and another for chunks (the file data). The plugin now makes a new pair of Maps for each directory. All the file meta data maps are named "FileMetaData."+dirName and chunk maps are "ChunkMap."+dirName. The reason for this change is to enable better metrics. I had all the chunks and file meta data in one Map before. This made it hard to see per directory metrics on map sizes and the like. Separating the chunks and meta data to by directory maps make it easier to analyze this.

Compression has substantial memory and network bandwidth savings, I've seen as high as 90% in indexes that I have available to me but it does burn more CPU. It's the classic trade off.

Implementation details of the Apache Lucene WebSphere eXtreme Scale store

I used 4 maps.

MetaData Map

This map uses the file name as the key and the value is a FileMetaData object. The FileMetaData object has things like lastmodificationtime and number of bytes.

Chunk Map

This has a String key and a byte[] value. The String Key is the fileName + "#" + block number. Every file is stored as a set of blocks. Each block has an entry in the chunk map. The chunks are fixed size. The size is currently defined in GridOutputStream as a constant. 

Directory Map

This stores Lucene Directory objects. The key is the directory name and the value is a Set of Strings. Each string in the set is a file name used by this directory.

Lock Map

This uses a String key which is a combination of the Directory name and the file name. The value is irrelevant, I use a Boolean. If an entry exists in the map for a file then someone has locked it. Thus, the locking protocol just attempts to insert a 'lock' in the map for a given file. This is only possible if there is no current entry. Unlock simply removes the entry for a file. Locks can be given a lease behavior by simply adding a TTL evictor on this Map. You could set the default TTL to be 10 minutes so worst case a locked file would be automatically unlocked after 10 minutes. Possibly an improvement here would be to 'touch' the lock every N reads or writes of data to the file and use a LAST_ACCESS_TIME eviction policy. This would likely prevent the lock being removed/invalidated if it's still in use.

No near cache is used. I used my wxsutils library to interact with the grid because it simplifies the application programming to just simple put/remove static operators.

Apache Lucene Store plugin for IBM WebSphere eXtreme Scale

I'm written a lucene store plugin for IBM WebSphere eXtreme Scale. It's a new Lucene Directory class that uses WXS for the indexes in a remote data grid. I used my wxsutils jar also. You can find it with source code here:

You can see a test case for it here. The test case makes a grid within the test JVM. For real usage, start a remote grid with the configuration xmls in the test/resources directory and then use WXSUtils to connect to it instead of startTestServer.

Any one using this with any issues then make a bug on github for me. I'll be continuing testing myself over the next week regardless looking for any issues.

Make sure you put the wxsutils*.jar file on the grid classpath. Every container JVM you start needs this jar on the classpath. You don't need the wxslucene jar on the grid classpath.

The command line args I use for starting the grid containers is:

-Djava.util.logging.config.file=/Users/ibm/Documents/Development/Samples/wxslucene/src/test/resources/logging.properties -Djava.endorsed.dirs=/Users/ibm/Documents/Development/og71/lib/endorsed -server -Xmx512M -Dcom.sun.management.jmxremote