October 23, 2013 1

hamsterdb: release 2.1.3 (unstable)

By in Coding, Databases, hamsterdb, Libraries

This release is “unstable” because the file format will change again in the next few releases. (A manual upgrade with ham_export/ham_import works.) There are no other known issues/bugs/limitations. If that doesn’t bother you then please download!

I’m copy/pasting the README:

This release comes with a major architectural change – the transformation
towards a key/value engine with modern column-store features: cache-optimized,
compressed data structures, reduced I/O and even basic schema support.

hamsterdb is now able to create an optimized Btree for your key data. These
Btrees require less space on disk and therefore less I/O. Also, their layout is
better optimized for CPU caches, since keys and records are now stored
separately from each other. More keys therefore fit into the CPU cache and
lookup operations are faster. (This layout is similar to the “PAX” layout in
the academic database literature.)

To use this functionality, application developers can now specify the type of
key that they insert into a database (i.e. HAM_TYPE_UINT64, HAM_TYPE_REAL32)
Under the hood, hamsterdb will create a Btree using C++ template policies
and template types specifically for this key.

Existing binary arrays are still supported (HAM_TYPE_BINARY), and can either
have fixed length or variable length. This binary type uses a built-in compare
function. For user-supplied callback functions, use HAM_TYPE_CUSTOM.

See the documentation of ham_env_create_db in hamsterdb.h for more details.

This release is the first of at least 3 unstable releases, in which the file
format will frequently change. If this does not bother you, then feel free
to upgrade. See the section “Migrating files from older versions” below on how
to migrate database files from 2.1.2 and older to 2.1.3.

The next few releases will rewrite the handling of extended keys and
duplicate keys by providing a new Btree node layout for variable-length keys.
Also, in-node compression will be implemented, i.e. for prefix compression
and dictionary compression.

This release also introduces the new tool “ham_bench” to benchmark and test
hamsterdb configurations; see the wiki for documentation:

https://github.com/cruppstahl/hamsterdb/wiki/Evaluating-and-Benchmarking

As usual, files are available for downloading.

October 20, 2013 0

Introducing ham_bench

By in Coding, hamsterdb
In a couple of days hamsterdb 2.1.3 will be released. Actually i wanted to release this weekend, but now i have delays because the MSVC 2008-compiler does not like some of the template code, and i have not yet found a workaround for these problems.
2.1.3 has some massive architectural changes, giving hamsterdb “column-store” abilities. I’ll write about that in a few days. For now i’d like to introduce you to a new tool which you can use for benchmarking; the tool is called ham_bench.

From the documentation:

“”"
hamsterdb comes with a handy tool “ham_bench” which is used for benchmarking and testing a myriad of configurations and data patterns. If you wonder which database configuration is the fastest for you then simply fire up ham_bench, describe your data and your operations through a few command line parameters and look at the results (or the generated image files).

hamsterdb’s performance depends on its configuration, the different APIs that you use, whether your application mainly concentrates on reading or writing (or deleting) data, and what kind of data you store. This document describes the various options that you can use to run the tests.
“”"

Read more here:
https://github.com/cruppstahl/hamsterdb/wiki/Evaluating-and-Benchmarking

ham_bench is also used for testing. If you’re curious about the release tests (or want to hack some functionality and run the tests on your own), you can find even more documentation here:
https://github.com/cruppstahl/hamsterdb/wiki/Testing

September 16, 2013 0

hamsterdb now available on OSX Homebrew

By in hamsterdb, Uncategorized

Good news, OSX users! hamsterdb can now be installed via homebrew:
https://github.com/mxcl/homebrew/blob/master/Library/Formula/hamsterdb.rb

Thanks, Johnathan Conley.

August 30, 2013 0

hamsterdb: release 2.1.2

By in Coding, Databases, hamsterdb, Libraries

This release re-implemented AES encryption based on OpenSSL and libcrypto. A few bugs in the build process were fixed (thanks for all the patches!). A new build option –without-tcmalloc was added.

A new internal function ham_env_get_metrics returns metrics and usage statistics that are useful for profiling. When closing the database file, unused space at the end of the file is truncated and the file will shrink.

The client- and server-side implementation of the remote network access has been rewritten. The server now uses libuv (a component of Node.js), which resulted in a performance improvement. libcurl is no longer required as a client-side dependency. Instead of http, a plain tcp connection is now used. To open a remote environment, use the prefix “ham://” instead of the now obsolete “http://”, i.e. “ham://localhost:12345/env.db”.

Unused space at the end of the file is now truncated, and the file size shrinks when the Environment is closed. (This feature is disabled on Win32 if memory mapped I/O is used.)

The file is available for download at http://hamsterdb.com/download.

 

July 2, 2013 0

Some server problems

By in Uncategorized

Some of you noticed that hamsterdb.com and crupp.de were down today (thanks for the mails!) I had server problems and moved to a new server. And since the last backup was already a few days old i had to write the release announcement of 2.1.1 for a second time. Everything should work as expected, although the new server feels a bit slower to me. I’ll continue working on the webpage for the next few days, and there’s a complete rewrite for hamsterdb.com in the pipeline as well.

July 2, 2013 0

hamsterdb: release 2.1.1

By in Databases, hamsterdb, Libraries

2.1.1 is finally available!

This release features some performance improvements, mainly regarding memory management and the integration of tcmalloc. Building against tcmalloc is definitely recommended. Several minor bugs were fixed, mostly related to the build process on Linux and MacOS. Prebuilt libraries for Win32/Win64 are available for download.

This release was quite difficult. I planned to rewrite the freelist and some parts of the caching, expecting to increase performance by a couple of percent. But my assumptions were wrong, and the new versions were slower than the old ones. I wasted about 2 months of work.

As a consequence, i added a new (internal) API to extract performance metrics: ham_env_get_metrics(). I’ll track them in the regression tests, which will immediately scream out if one of the metrics degrade, and i hope they will also help me making better decisions next time when i want to optimize something.

Everything’s available for download here: http://hamsterdb.com/download.

Plans for next release: AES reimplementation, improve client/server performance.

March 2, 2013 0

hamsterdb: release 2.1.0

By in Coding, hamsterdb, Libraries

This release has some performance improvements (mostly related to improvements in mmap-related code) and finally allows using Transactions with In-Memory environments. And there are myriads of API changes which will require you to change your code. See the README for details.

Here’s the announcement:
https://groups.google.com/forum/?fromgroups=#!topic/hamsterdb-announce/m53Nphpio-E

And here are the files:
http://hamsterdb.com/download

December 8, 2012 0

hamsterdb used in Obama’s campaign

By in Coding, hamsterdb

Obama’s team used CloudOpt’s CloudOptimizer, which accelerates uploads
to various cloud providers including Amazon Web Services, Rackspace and others.

CloudOptimizer uses hamsterdb for managing their data. In fact, CloudOpt is one of hamsterdb’s largest commercial users.

Read more about CloudOpt and their products: http://www.cloudopt.com.

Read more about hamsterdb: http://hamsterdb.com.

Read more about Obama’s campain and the technology they used: http://arstechnica.com/information-technology/2012/11/how-team-obamas-tech-efficiency-left-romney-it-in-dust/

December 1, 2012 0

hamsterdb: release 2.0.5

By in Coding, Databases, hamsterdb

This release fixes a few minor bugs, has a few performance improvements and fixes a segmentation fault in the .NET API.

Also, the internal C++ implementation moved into namespace “ham” to avoid conflicts with other symbols.

Files, README and Changelog are ready for download here.

This is quite a “small” release but under the hood the refactoring is ongoing and will result in a next release with many API changes. Over the last 7 years the hamsterdb API has been stable (with only 2 minor changes). It’s about time to clean up some things, get more consistent function names, remove deprecated functions and get rid of functionality that is not used (i.e. AES compression, zlib compression). All in all i hope i can remove about 10% of the code base without really affecting the functionality. It will require that users change their code, though, but the changes won’t be overwhelming.

In addition there are several performance fixes on the TODO list for the next two releases. The mmap implementation will be rewritten, and the freelist will be replaced with a newer implementation which will be faster and result in smaller files as well. The new freelist will require a new file format, therefore databases in 2.0.6 will be incompatible with those of earlier releases (and vice versa). The migration path will be via a command line tool to dump a database into json or another format, and second tool to import this exported data into a database. But i’m open for suggestions and interested in hearing your opinions.

July 23, 2012 0

hamsterdb: release 2.0.4

By in Coding, Databases, hamsterdb, Libraries

This release fixes several issues including a segmentation fault, a few misplaced asserts and a data corruption. It is highly recommended to upgrade.

The number of serious issues that was reported to me over the last couple of weeks was quite astonishing, but on the other hand i had so much encouraging help from those who reported the bugs. Many helped in reproducing and they were really patient. Thanks everyone!

The most serious bug created a data corruption because the first freelist page in the database sometimes was not flushed to disk. This issue was not noticed over the last few years and my tests did not catch it. Now they do.

Everything’s available for download – including 130 megabytes of precompiled Win32/Win64 libraries (for MSVC 2008 and MSVC 2010).
http://www.hamsterdb.com/download

Older Posts