All my future posts will be appearing over there, so be sure to check it out and update your feed subscriptions:

http://outerthought.org/blog/

You can find it on the lilycms.org site: hbase indexing library.

I am looking into exploiting the sorted nature of HBase tables to build indexes. See this interesting presentation by Ryan Barrett (or these slides by Joe Gregorio) for how Google does the same for App Engine’s datastore on Bigtable.

HBase identifies rows by a key, the key is a byte array, and it keeps the rows sorted on these byte[] keys. If we want to index more high-level data types like strings, number or dates, we will have to figure out how to map these onto bytes so that sort order is maintained as desired. This post is about exactly this problem, but first I will quickly go into the basics of building an index with HBase.

Building an index with HBase

HBase keeps rows sorted lexicographical by row key, and allows to do range scans (from-to) over these rows. Lexicographical sorting means sorting like in a dictionary, where the corresponding characters of two words (here bytes in byte arrays) are compared from left to right.

Suppose we have stored entities in HBase, one HBase-row per entity, and each of these entities has a property Country. Now we want to build a secondary index for these entities, in a different HBase table, to allow querying these entities by their Country property. The row keys in this index table would be composed of the Country property and the key of the row that uses this property value. In the example below, the target row key is shown as a number, but it can be any byte array.

If we want to find all entities whose Country is France, we can use an HBase scanner to find all rows starting with France. Technically, you would set the start row for the scanner to France and stop the scanning by using a RowFilter with a BinaryPrefixComparator on the end value, here again France.

We are not limited to equals searches. We can also do range searches (e.g. from ‘Belgium’ to ‘France’) or prefix searches (e.g. all entities whose country name starts with a B).

Note that we have no use for HBase columns here: all information we need is stored in the row key. The indexed row has to be part of the row key, otherwise we could have duplicate row keys which is not possible. Since HBase actually requires at least one column, we need to store a dummy column.

The same mechanism also allows to build composite indexes, where the row key is build up of multiple properties. Let’s extend our previous index with a new property Category.

The Category property is multi-valued, as exemplified by the entity 4 which occurs two times in the index.

We can again use HBase scanners to search on this index. With composite indexes, you do not necessarily need to search on each field, but you have to use the index-fields from left to right, and only the rightmost field can be used for range or prefix searching. In our example, you could equal-search on just country, equal-search on the combination of country and category, range/prefix scan on country, or equal-search on country in combination with a range/prefix scan on category.

When you do not search on all fields of a composite index, or when you do a range search and the field is multi-valued (like in the Category example), then the index can return the same row multiple times. In the example above, searching for France without a condition on Category would return row 4 two times, and not grouped together. This can be annoying for the consumer of the index results.

Merge joins

If we use indexes such that their returned row keys are unique (by searching on all fields of a composite index, or by not using range-scans for multi-valued properties) and sorted by row key (this last point is automatically assured by HBase), then we can easily merge-join results from multiple indexes! Merge-joins are nicely explained in this presentation by Brett Slatkin.

From the logical index model to bytes

As mentioned before, HBase row keys are plain byte arrays, and HBase determines the row order by byte comparison. This means that to compare two byte arrays, HBase compares the corresponding bytes, from left to right. All corresponding bytes being equal, shorter arrays compare as being smaller than larger arrays.

If we want to build indexes on data types such as strings, numbers or dates we will have to map them onto bytes in such a way that when HBase performs byte comparison, the order is maintained the same as when we would have compared the logical data types.

For composite indexes, we will need to pad the key entries with zeros so that corresponding values align in the byte arrays. So if the country field could be 10 bytes wide and the category field 3 bytes, index entries could look like:

France0000A002
Belgium000C001

The zeros represent bytes with all bits set to zero, so that in comparisons they will always be smaller than anything else.

Note that for strings, this means that you’ll have to decide on beforehand how long the indexed string value can get.

Update: as Chris points out in the comments, this padding is not necessary, a well-chosen separator will also do the trick. After all, we only need to compare corresponding values if all the values more to the left are already equal.

String sorting

When we want to keep an index of string values, we need to convert those strings first to bytes to be able to store them in HBase row keys. If this conversion is done using UTF-8, then according to Wikipedia, “Sorting of UTF-8 strings as arrays of unsigned bytes will produce the same results as sorting them based on Unicode code points”.

Often we will not want our strings to be sorted by Unicode points, since we want to have for example é or E sorted before f. Note that this ordering is important because we want to do range scans, for equals searches only it does not matter what comes first.

Another aspect besides the order of the index is that, when searching on an index, one will often (but not always) want to ignore certain spelling variations like missing accents on characters.

Ignoring case

Ignoring the casing can be solved by translating all strings to their lowercase variant. Lowercasing a string in Java is a locale-dependent operation, though there are only a few locale’s for which this really makes a difference: Turkish, Azerbaijani and Lithuanian.

Normalizing

Sometimes there are multiple possible Unicode representations for the same visual character. A typical example is that characters with accents can be represented either as a single Unicode character or as the combination of the base character and a combining accent character. Java’s Normalizer class can canonicize these different forms.

Simplify strings

A possible solution for accented characters is to remove the accents, and more in general to reduce the text to plain ASCII. Lucene does this, we can re-use the code of their ASCIIFoldingFilter.

Collator

The standard Java solution for Locale-sensitive sorting is to use the Collator class. As far as sorting is concerned, it does exactly what we want. And it is even possible to materialize this ordening to something we can use as HBase row key: via the Collator you can get access to a CollationKey, which “converts a String to a series of bits that can be compared bitwise against other CollationKeys”. And you can get access to these bits using CollationKey’s toByteArray() method.

This sounds like an ideal solution, though there are some things to be aware of:

• The collation byte arrays are rather long: seems like it uses 6 to 8 bytes per character, plus some global overhead.
• The inverse translation, from collation bytes to string is not supported. This is not really needed for our indexing purpose, but might be handy for debugging indexes.
• The algorithm for the construction of the collation key bits is not specified as part of the API, so it might differ between JVM implementations or JVM versions.
• While the Collator offers optimal sorting, it does not help if you want to search ignoring accents. But the reverse is true too: if you want to perform exact case-sensitive searches, while also having locale-sensitive and case-insensitive sorting, then the collator solution is perfect.
• The collation key of a shorter string is not a prefix of the collation key of a longer string, so if you want to search on a prefix of the string, this is not possible. I find this an important disadvantage.

String sorting conclusion

There is no obvious choice for a default fits-all solution, so for my HBase-indexing purpose I am looking into making the string to byte conversion pluggable.

Integer sorting

If we compare the byte representation of two integers, will this behave such that the smaller integer is considered smaller than the larger integer?

To know the answer to this question, we need to know that the binary representation of an integer is two’s complement.

Ignoring the two’s complement for a moment, in plain binary numbers, the more significant bits are more to the left, so HBase’s left-to-right comparison will automatically do the right thing.

The first bit of a two’s complement integer is a sign bit: 1 for negative numbers, 0 for positive numbers. 1 is larger than 0, so with byte-based comparison this would mean negative numbers are considered larger than positive numbers. This can be easily solved by flipping that bit.

In the negative number range, 1000 is smaller than 10, so bigger is smaller. However, because in two’s complements the bits of all negative numbers are inverted, the order will be fine.

So in conclusion, to make integers compare correctly, we only need to flip the sign bit.

Float sorting

Floats are a more interesting problem then integers. For one, floating point numbers are an approximate representation, so equals searches will be a problem. So let us assume that these indexes will be only used for range searches, possibly with a small epsilon range. In composite indexes, they will hence only be usable as the last field.

Java uses the float representation as defined by IEEE:

[1 sign bit][8 exponent bits][23 mantissa bits]

The sign bit is again 1 for negative numbers and 0 for positive numbers. The exponent and mantissa are such that the most significant bits are to the left. The exponent is unsigned. In the mantissa each place represents a negative power of 2: 2^-1, 2^-2, 2^-3, … similar to the decimal system. See this document for more details on the float format.

With this representation positive floats will compare correctly. We only have to flip the sign bit so that positive numbers will be larger than negative numbers.

For negative numbers, in contrast with the two’s complement integers, the bits are not inverted. Simply flipping all the exponent and mantissa bits will get exactly the behavior we need.

Note that there is a lot more to say about floats: there is a way to encode positive and negative infinity, there is something called subnormal numbers, and there is a positive and negative zero. The float representation is designed such that all these will be sorted fine without further consideration. A special case is NaN, not a number, for which there are multiple representations possible, and these will be sorted before or after negative respectively positive infinity.

Decimal sorting

To sort BigDecimals, or also for floats, we can use the decimal string representation of the numbers. A simple example is:

055.23
124.359

Note that you have to pad your numbers with leading zeros in order to have them ordered correctly.

There are some difficulties with this approach though.

First, this will not work as-is for negative numbers: -1 is larger than -2. This can be solved by changing 1 to 9, 2 to 8, and so on. Then there is still the problem that 3.33 is larger than 3.333, but lexicographically the longer string will be sorted after the shorter one. This can be solved by suffixing each negative number with something that is larger than any digit, like the character ‘a’.

Second, you need to know on beforehand to know how large your numbers can get. And if they can get really large, you will end up with large strings. However, inspired by studying the floats encoding above, I think we can do a decimal equivalent of the floating point’s exponent-mantissa approach. With a bit of Googling, I found this post by Steven A Rowe which is about the same idea.

Update:

There is actually no need for a string-like approach for decimals, since HBase does not use strings anyway. I have probably been looking too much at Lucene lately.

As the BigDecimal javadoc says: “A BigDecimal consists of an arbitrary precision integer unscaled value and a 32-bit integer scale”. We can simply use this unscaled value as mantissa, it is available as (two’s complement) bytes via BigDecimal.unscaledValue().toByteArray(). The scale of a BigDecimal is the number of digits to the right of the decimal point. For the exponent, we are rather interested in the number of digits to the left, which can be computed via BigDecimal.precision() – BigDecimal.scale().

Given all this, we can build a byte array in the same way as for floats: a sign bit, some exponent bits, and a variable number of mantissa bits. As in IEEE floats, we can offset the exponent so that it becomes unsigned. For the rest, just invert bits with a similar reasoning as for floats.

Date sorting

The widely used ISO 8601 date-time format is designed such that lexicographic order corresponds to the chronological order, so we can just use that format. It only uses plain ASCII characters, so the string-to-byte conversion does not pose a problem.

But again, there are some things to be aware of.

First, we should normalize all our date-times to the same timezone, preferably UTC, in which case the ISO 8601 string ends on a Z.

Second, the sorting will not be correct for negative years. This could be solved in a similar way as for decimals.

Update:

Dates can of course also be treated as integer/longs.

Ascending and descending indexes

By scanning over an index, the matching rows will be automatically returned in the order of the values we indexed on. HBase cannot scan in reverse order, so if you would like to be able to retrieve results in reverse order, you could do so by inverting all the bits in the index row keys.

Status

All the above is preliminary thought work, it might be full of errors and oversights. I hope to put it into practice sometime soon.

The Avro website provides useful info, but I stumbled across the quotes below which I found helpful in better understanding what it is all about.

Representations

Doug Cutting:

Avro’s Java implementation currently includes three different data representations:

• a “generic” representation uses a standard set of datastructures for all datatypes: records are represented as Map<String,Object>, arrays as List<Object>, longs as Long, etc.
• a “reflect” representation uses Java reflection to permit one to read and write existing Java classes with Avro.
• a “specific” representation generates Java classes that are compiled and loaded, much like Thrift and Protocol Buffers.

We don’t expect most scripting languages to use more than a single representation. Implementing Avro is quite simple, by design. We have a Python implementation, and hope to add more soon.

As also mentioned in that thread, the reflect representation is less interesting from a performance point of view.

And as noted here, In Java, the Encoder/Decoder API also now double as an Event-based API.

The binary format

Doug Cutting:

A record’s fields are serialized in the order that the fields occur in the records schema, with no per-field annotations whatsoever. For example, a record that contains a string and an int is serialized simply as a string followed by an int, nothing before, nothing between and nothing after.

Standardizing a format rather than an API

Doug Cutting:

Thrift fundamentally standardizes an API, not a data format. Avro fundamentally is a data format specification, like XML.

Schema evolution & skipping data while reading

Doug Cutting:

Avro supports schema evolution. In Avro, the schema used to write the data must be available when the data is read. (In files, it is typically stored in the file metadata.)

If you have the schema that was used to write the data, and you’re expecting a slightly different schema, then you simply keep those fields that are in both schemas and skip those not. This is equivalent to Thrift and Protocol Buffer’s support for schema evolution, but does not require manually assigning numeric field ids.

This feature can also be used to support projection. If you have records with many large fields, but only need a single field in a particular computation, then you can specify an expected schema with only that field, and the runtime will efficiently skip all of the other fields, returning a record with just the single, expected field.

The file format

From the javadoc of DataFileWriter class:

[DataFileWriter] Stores in a file a sequence of data conforming to a schema. The schema is stored in the file with the data. Each datum in a file is of the same schema. Data is written with a DatumWriter. Data is grouped into blocks. A synchronization marker is written between blocks, so that files may be split. Blocks may be compressed. Extensible metadata is stored at the end of the file. Files may be appended to.

If you know a bit of Hadoop MapReduce, you will understand why these block-markers are interesting.

Besides the DataFileWriter, you can also use the DatumWriter/DatumReader classes for serializing just one record without the overhead of storing the schema.

RPC socket server status

Doug Cutting:

Brian McCallister wrote:
> Is the socket server in avro intended as an example/dev option or a
> productiony server?

It’s more of an example.  The wire protocol is probably fine for production, but the existing client and server are pretty dumb.  There’s no connection pooling, async server-side i/o etc.

The Java HTTP-based version performs much better than the socket version in my micro-benchmarks.  It gets connection pooling for free, and it’s easy to configure it for an async server that can support more connections than threads. Today that’s the most production-worthy implementation.

They are thinking about a standard transport for Avro though, and there is some talk about using Netty.

Utf8 strings

Something that you will notice soon enough when using the API or when using the SchemaCompiler to generate classes is that Avro does not use the String class to represent strings, but rather its own Utf8 class. I suppose this is for performance reasons: this class just stores the UTF-8 bytes without converting them to a String. It is also mutable, allowing to re-use the same instance.

Next weekend (6-7 februari) it is again Fosdem, the conference for open-source developers in Brussels.

This year my Outerthought colleague Steven is organizing a devroom on NoSQL. And my colleague Evert will give a talk on building a CMS using NoSQL technologies. If you are interested in what our future content management solutions will be like, make sure to be there. Entrance is free, no registration is required.

Now the Dynamo paper notes that “even where there is a significant skew in the access distribution there are enough keys in the popular end of the distribution so that the load of handling popular keys can be spread across the nodes uniformly through partitioning.”

The BigTable paper writes about a relatively small table (~500 GB) in Google Earth that “must serve tens of thousands of queries per second per datacenter with low latency. As a result, this table is hosted across hundreds of tablet servers and contains in-memory column families.”

Apparently in real-world scenarios things turn out OK. Still, what would happen when you have a very small data set accessed by a very large amount of users? Probably it would be a bit crazy to touch the database for every read, and this could be solved by caching in the application layer.

Applications where you have the same problem for writes seem much less common to me, and so a specific solution can be engineered for that.

Scaling the relational model across machine boundaries does not work very well, but in a distributed storage system scenario, the local storage can still be provided by an RDBMS.

A highly available distributed store uses a set of nodes with a set of techniques (replication, consistent hashing, vector clocks, hash trees, hinted handoff, …). But at the node-local level, the data still needs to be stored somehow.

Some (Dynamo depending on config, PNUTS) store the data in an RDBMS (MySQL), which then serves as a btree-based key-value store. In this case, they are simply used because they exist, are ready-to-use and known-to-work.

Others (Cassandra, HBase/BigTable) use a simple better-performing solution based on SSTables (or similar), log files and memtables.

Cassandra and HBase are also examples of how it still makes sense to provide node-local, non-scalable finer-grained structures, namely the columns within column families. Similarly, within a distributed store the relational features could still be exploited at the local level.

Let’s call the systems nodes and the partitions virtual nodes.

This is something which comes back in:

• Consistent hashing: each node gets assigned many of the hash buckets
• Katta (distributed Lucene): each search server can handle a variety of index shards
• Bigtable or HBase: each region server can handle a variety of regions

In these examples, the hash bucket, the index shard and the region are the virtual nodes.

The advantages include:

• being able to handle nodes with different capabilities (a heterogenous set of nodes): more powerful nodes get assigned more of the virtual nodes
• when a node goes down:
  • the virtual nodes that it was responsible for can be re-assigned to multiple other nodes
  • or from another point of view, the replica’s of the virtual nodes managed by this node will be on a variety of other nodes, so the load of the killed node will be automatically spread over multiple other nodes
• when a new node is added, it can take over the responsibility for some of the virtual nodes, and this can be done one by one to allow for warm up

Speaking of recurring patterns, there is also clear similarity in the way Lucene handles index updates and the way HBase & Bigtable handle updates:

• Lucene first buffers index updates in memory, and when certain limits are reached, flushed them to a new file on disk called an index segment. This is instead of trying to update the existing index in-place. Searching is done by searching over all index segments. When the number of segments gets large, some of them are merged together in the background.
• HBase/Bigtable does something similar with MapFiles/SSTables: new data is occasionally flushed to a new immutable MapFile (basically a file of key-value pairs sorted by key with an additional index containing the positions in the file of a subset of the keys, to avoid a full sequential scan). When looking for something, all flushed MapFiles are consulted until the item is found. A background process merges the MapFiles when there are too many of them.

• in modern systems, [...] random access to memory is typically slower than sequential access to disk. Note that random reads from disk are more than 150,000 times slower than sequential access; SSD improves on this ratio by less than one order of magnitude.
• [...] the highest-speed local network technologies have now surpassed most locally attached disk systems with respect to bandwidth, and network latency is naturally much lower than disk latency. As a result, the performance cost of storing and retrieving data on other nodes in a network is comparable to (and in the case of random access, potentially far less than) the cost of using disk. Once a large dataset has been distributed to multiple nodes in this way, however, a huge advantage can be obtained by distributing the processing as well—so long as the analysis is amenable to parallel processing.