cruppstahl’s blog

This release fixes a few minor bugs and adds a performance for In-Memory Databases: Users can now request a pointer to the stored record data by specifying the flag HAM_DIRECT_ACCES to ham_find, ham_cursor_find[_ex] and ham_cursor_move. Thus they can overwrite the record data without calling ham_insert or ham_cursor_overwrite. Also, a few flags were removed and the interface for ham_get_key_count was changed - please see the README.

As usual everything’s available for download here.

chris Coding, Libraries, hamsterdb

After months of work and receiving/integrating tons of patches, i’m happy that 1.1.0 is finally released!

I have increased the minor version because major new functionality was added and the database file format is not backwards compatible (however, 1.1.0 can read/write databases from 1.0.x).

Most of the patches were contributed by Ger Hobbelt (http://www.hobbelt.com, http://www.hebbut.net). Thank sa lot for the great work!

What’s new?

• “Approximate matching” - retrieves the nearest neighbour if a key is not found. Useful i.e. for time-based lookup (timestamp used as index)
• Major performance improvements, esp. in freelist management and in the caching
• Data Access Modes optimize runtime behaviour for performance depending on your Database characteristics (i.e. “read/write” vs “sequential inserts”)
• New functions to get runtime information (ham_env_get_parameter,  ham_get_parameter)
• A new function ham_get_key_count() returns the number of keys
• several bugfixes

As usual everything’s available for download here.

What’s on the plate for the next release? I have more patches from ger to merge, i will clean up/refactor some parts of the code and i want to give direct access to records in in-memory databases (means you can retrieve the pointer to the stored record instead of a copy, and you can directly modify/overwrite it).

The long-term plans for hamsterdb are to merge the functionality of hamsterdb Transactional Storage into hamsterdb Embedded Storage. I decided that i don’t have the resources to develop two different databases, and since Embedded Storage already has many users and i get frequent feedback it makes sense to concentrate my energy to it.

chris Coding, Libraries, hamsterdb

I’m really happy (and a little bit proud) that Tony Bain mentions hamsterdb in his newest post.

What else is on the plate? I’m currently merging lots of patches submitted by Ger Holland with bug fixes, performance improvements and new functionality (”Approximate Matching”) form hamsterdb Embedded Database. I hope we can make a release in the next two weeks.

Afterwards, i’ll work on the next release for hamsterdb Transactional Database - performance improvements and reclaiming deleted disk space.

chris Coding, hamsterdb

I’m really happy to announce the first release of hamsterdb Transactional Storage - an ACID compliant, concurrent key/value store.In a nutshell:

• ACID compliant transactions in a lock-free architecture; Transactions do not hold locks during their life-time. Conflicts are resolved in memory, without accessing the disk
• Fast; performance-relevant functions are moved to the background
• Logical, idempotent logging and Recovery
• Multithreaded; every handle can be used from arbitrary threads
• portable ANSI-C code (Linux/Unix/Win32)
• Easy to use

Early adaptors can download it here:

http://hamsterdb.com/public/dl/hamsterdb2-0.0.1.tar.gz

The linux/unix version is stable - but win32 still has some issues (although it works most of the time). This release is not intended for production use.

The next releases will improve performance, improve win32 and add more features. See the TODO file in the source package for more details.

I also wrote a Whitepaper to explain the architecture of hamsterdb Transactional Storage. It’s available for download here:

http://hamsterdb.com/public/dl/hamsterdb2_technical_overview.pdf

chris Coding, Libraries, hamsterdb

I just found a really cool trick to get a list of all predefined gcc macros:

echo | gcc -dM -E -

Found it here: http://www.willus.com/mingw/_predefined.shtml

chris Coding

This release fixes a minor bug in ham_cursor_move; if the second parameter (”record”) is NULL and compression is enabled, hamsterdb crashes. Also, the unittest-dependency to cppunit was removed.
Sources, CHANGELOG and precompiled win32 libraries are available for download.

chris Coding, Databases, Libraries, hamsterdb

In order to allow linking of hamsterdb with more open source projects, i added FOSS license exceptions. These exceptions allow linking with BSD licensed libraries, Apache, LGPL and others.

Here’s the legal stuff and the list of licenses: http://hamsterdb.com/licensing/exceptions

I’m absolutely willing to add more licenses upon request.

chris Coding, Libraries, hamsterdb

Immediately after releasing 1.0.7, a user informed me of a weird behaviour which appeared after deleting database keys.

It took me about 20 hours to track it down… sometimes, when deleting a key (and it’s an extended key which does not fit into the index page and therefore was stored in an overflow area), the database page has to be merged with other pages because it’s getting too empty. And in one (rare, but nevertheless dangerous) case the overflow area was deleted, but there was still a key pointing to it. And from that moment on the index was in an inconsistent state.

Everything’s available for download.

chris Coding, Libraries, hamsterdb

1.0.7 is out! it fixes a bug on big-endian architectures - they were not able to open database files which were created on little-endian architectures.

From the README:

This release fixes a bug on big-endian architectures; Databases from little- endian architectures were not loaded correctly, and vice versa. A typo in the Tutorial was fixed.

Everything’s available for download.

chris Coding, Libraries, hamsterdb

Currently, it looks as if i stopped working on hamsterdb (and Ohloh’s “Decreasing year-over-year development activity“) really makes me feel bad!) - but that’s far from true. I spend every minute on hamsterdb2, the next generation. I will continue to maintain hamsterdb and i even hope that i have time to release a bugfix release today or even tomorrow.

As i wrote in an older blog entry, hamsterdb will focus on embedded devices with limited hardware; it will offer high configurability and a limited feature set, optimized on throughput and performance. (Therefore i currently do not plan to improve transaction support for hamsterdb).

In contrast, hamsterdb2 will be for modern (preferably multi-core) machines and modern applications. Here’s a list of buzzwords:

- multithreaded - hamsterdb2 will not be just threadsafe, but it will actively encourage the use of multiple threads. Performance intensive operations are processed in the background. Every API handle (for Environments, Databases and Transactions) can be used from multiple threads in parallel.

- transactional - hamsterdb2 will support ACID transactions and multiple isolation levels. Isolation levels can be set individually for each transaction. Transactions itself are kept in memory - whenever a Database is modified, the Database “diff” is stored in-memory; transaction conflicts are resolved in-memory, without disk access. Same applies to UNDO operations. Committed transactions are periodically flushed to disk in the background.

- logging/recovery: since transactions are kept in-memory, they are also written to a logical logfile. In case of a crash, all operations from this logfile are repeated. These operations are idempotent, which means that they can be applied multiple times without side effects. Incomplete/Active transactions are recreated (there’s an API function to enumerate all active transactions). The index file is based on a B+Tree with atomic algorithms and therefore does not need its own logfile.

- Simplified API - i think the API of hamsterdb was already quite simple to use. Nevertheless i managed to improve it a little bit.

- The current state? I’m currently working on the recovery algorithm. Transactions and the index files are already working, but the index files need improvements (they currently only work for fixed-size keys). There’s no freelist management yet - empty pages are not reused. And i only implemented the SERIALIZE isolation level - but others will be simple to do. Currently, there’s no support for duplicate keys, record numbers and cursors.

Feel free to post questions/comments!

chris Coding, hamsterdb