June 26, 2014 11:46 / 1 comments / nosql python unqlite

unqlite python logo

I'm happy to write that I've just released some python bindings for UnQLite, an embedded NoSQL database and JSON document store. UnQLite might be characterized as the SQLite of NoSQL databases, though it's JSON document-store and Jx9 scripting language make it a pretty unique offering. UnQLite is created by Symisc Systems, who are also responsible for Vedis, an embedded Redis-like database (I also wrote some python bindings for vedis). Here is a quick overview of some of UnQLite's features, as described on the project homepage:

In the rest of this post I will show some basic usage of the unqlite-python library. If you'd like to follow along, you can use pip to install unqlite:

pip install unqlite

You can find the project source code hosted on GitHub and the documentation is available on readthedocs.

Initializing a database

To get started, let's open up a python interpreter and create an UnQLite database object. You can specify a filename when instantiating UnQLite, or omit one to use an in-memory database.

>>> from unqlite import UnQLite
>>> db = UnQLite()  # Creates an in-memory database.

Key/Value store

UnQLite can be used as a key/value store and supports a dictionary-style API. It is similar in this regard to the standard library BerkeleyDB module. Here are some example operations you might perform using the key/value APIs:

>>> db['foo'] = 'bar'  # Use as a key/value store.
>>> print db['foo']
bar

>>> for i in range(4):
...     db['k%s' % i] = str(i)
...

>>> 'k3' in db
True
>>> 'k4' in db
False
>>> del db['k3']

>>> db.append('k2', 'XXXX')
>>> db['k2']
'2XXXX'

The database can also be iterated in key-order:

>>> [item for item in db]
[('foo', 'bar'), ('k0', '0'), ('k1', '1'), ('k2', '2XXXX')]

Cursors

For finer-grained record traversal, you can use cursors.

>>> with db.cursor() as cursor:
...     cursor.seek('k0')
...     for key, value in cursor:
...         print key, '=>', value
...
k0 => 0
k1 => 1
k2 => 2XXXX

Cursors also support a couple shortcut methods to simplify common iteration patterns:

>>> with db.cursor() as cursor:
...     list(cursor.fetch_count(3))
...
[('foo', 'bar'), ('k0', '0'), ('k1', '1')]

>>> with db.cursor() as cursor:
...     cursor.seek('k0')
...     list(cursor.fetch_until('k2', include_stop_key=False))
...
[('k0', '0'), ('k1', '1')]

JSON Document Store

In my opinion the most interesting feature of UnQLite is its JSON document store. The Jx9 scripting language is used to interact with the document store, and it is a wacky mix of C, JavaScript and maybe even PHP.

Interacting with the document store basically consists of creating a Jx9 script (you might think of it as an imperative SQL query), compiling it, and then executing it.

>>> script = """
...     db_create('users');
...     db_store('users', $list_of_users);
...     $users_from_db = db_fetch_all('users');
... """

>>> list_of_users = [
...     {'username': 'Huey', 'age': 3},
...     {'username': 'Mickey', 'age': 5}
... ]

>>> with db.compile_script(script) as vm:
...     vm['list_of_users'] = list_of_users
...     vm.execute()
...     users_from_db = vm['users_from_db']
...
True

>>> users_from_db  # UnQLite assigns items in a collection an ID.
[{'username': 'Huey', 'age': 3, '__id': 0},
 {'username': 'Mickey', 'age': 5, '__id': 1}]

This is just a taste of what is possible with Jx9.

Collections

To simplify working with JSON document collections, unqlite-python provides a light API for executing Jx9 queries on collections. A collection is an ordered list of JSON objects (records). Records can be appended or deleted, and in the next major release of UnQLite there will be support for updates as well.

To begin working with collections, you can use the factory method on UnQLite:

>>> users = db.collection('users')
>>> users.create()  # Create the collection if it does not exist.
>>> users.exists()
True

You can use the store() method to add one or many records. To add a single record just pass in a python dict. To add multiple records, pass in a list of dicts. Records can be fetched and deleted by ID.

>>> users.store([
...     {'name': 'Charlie', 'color': 'green'},
...     {'name': 'Huey', 'color': 'white'},
...     {'name': 'Mickey', 'color': 'black'}])
True
>>> users.store({'name': 'Leslie', 'color': 'also green'})
True

>>> users.fetch(0)  # Fetch the first record.
{'__id': 0, 'color': 'green', 'name': 'Charlie'}

>>> users.delete(0)  # Delete the first record.
True
>>> users.delete(users.last_record_id())  # Delete the last record.
True

You can retrieve all records in the collection, or specify a filtering function. The filtering function will be registered as a foreign function with the Jx9 VM and called from the VM.

>>> users.all()
[{'__id': 1, 'color': 'white', 'name': 'Huey'},
 {'__id': 2, 'color': 'black', 'name': 'Mickey'}]

>>> users.filter(lambda obj: obj['name'].startswith('H'))
[{'__id': 1, 'color': 'white', 'name': 'Huey'}]

In the future I may add more Jx9 helpers, but for now hopefully these prove useful.

Transactions

UnQLite supports transactions for file-backed databases (since transactions occur at the filesystem level, they have no effect on in-memory databases).

The easiest way to create a transaction is with the context manager:

>>> db = UnQLite('/tmp/test.db')
>>> with db.transaction():
...     db['k1'] = 'v1'
...     db['k2'] = 'v2'
...
>>> db['k1']
'v1'

You can also use the transaction decorator which will wrap a function call in a transaction and commit upon successful execution (rolling back if an exception occurs).

>>> @db.commit_on_success
... def save_value(key, value, exc=False):
...     db[key] = value
...     if exc:
...         raise Exception('uh-oh')
...
>>> save_value('k3', 'v3')
>>> save_value('k3', 'vx', True)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "unqlite/core.py", line 312, in wrapper
    return fn(*args, **kwargs)
  File "<stdin>", line 5, in save_value
Exception: uh-oh
>>> db['k3']
'v3'

For finer-grained control you can call db.begin(), db.rollback() and db.commit() manually:

>>> db.begin()
>>> db['k3'] = 'v3-xx'
>>> db.commit()
True
>>> db['k3']
'v3-xx'

Thanks for reading

Thanks for taking the time to read this post, I hope you found it interesting. If you have any comments or suggestions for improving the library, please leave a comment. If you find a bug in the library while trying it out, feel free to open a GitHub issue and I will address it.

Below are some links you might find useful:

I would also like to thank GitHub user nobonobo for his/her work on unqlitepy, which I liberally borrowed from while getting started.

Comments (1)

Paul Wray | jul 2014, at 09:01pm

Hi Should this build on Windows 32 bit ? I got a unresolved external symbol initlibunqlite.


Commenting has been closed, but please feel free to contact me