[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
[ANN] guile-wiredtiger 0.8.0
[ANN] guile-wiredtiger 0.8.0
Thu, 16 May 2019 23:05:04 +0200
I am please to announce the release of guile-wiredtiger 0.8.0.
You can find it at:
Or using my guix channel:
$ cat ~/.config/guix/channels.scm
$ guix pull
$ guix package -i address@hidden
Here is the list of changes:
- add support for single bytes column
- fix bug in rollback
- rely on guile-r7rs
- remove null byte added in strings
- add some benchmarks...
- add session-reset
- rename cursor-search to cursor-search?
- improve cursor-search-near to return a symbol or #f
- rename cursor-next and cursor-prev to cursor-next? and cursor-prev?
- %key-not-found is not public anymore, no need.
The main additions are:
- add (wiredtiger pack) lexicographic packing of scheme object
- add (wiredtiger okvs) SRFI-167
- add (wiredtiger nstore) SRFI-168
Eventually, I figured what went wrong. I faced two issues:
- wiredtiger raising WT_ROLLBACK using a single application thread
and a single session which was due to the fact that I did not have
a big enough cache for storing the whole transaction in memory.
This is solved when using okvs with 'cache key set to a "reasonable"
value. With gotofish I set the value to 1GB. It doesn't mean that
the transaction can be 1GB big, it means that wiredtiger will use at
most 1GB to execute a transaction.
- my program leaking memory. I am not sure but it is unlikely that guile
part of the code leaks memory [...] AND I experimented with both Chez
Scheme and Python, they both seems to leak memory. The latter takes
more time but in the end the result is the same. I don't have mongodb
confirmation, to my mind it is again due to a configuration problem.
The default configuration of wiredtiger use one thread for cache
That is there is a single thread dedicated to fighting the growth of
cache using some Least Recently Used algorithm IIRC. Anyway, setting
okvs 'eviction-trigger to 85% (aka. triggers eviction when 85% of the
is filled) and using 4 threads for eviction itself, allows
to complete its mission.
The key word is fine-tuning. That is what makes the database works.
So if you read the above carefully you figured that gotofish can index
wikipedia vital articles level 3 that is 500MB big in two hours. Let's
$ time guile -L . gotofish.scm search GNU
** 0.03677014042867702: data/wikipedia-vital-articles-level-3/Society
and social sciences/Language/Cyrillic_script
Three seconds is not bad since it includes the time necessary to open
the database. Also it is using a USB SSD. By the way, the database
behaves better on SSD without encryption...
gotofish code will prolly end up in guile-wiredtiger repository as an
example. In the mean time, it is available at:
Last but not least, there is still a non-determinist error about locale
that fails to be set, I don't know where it is coming from.
# What the future will bring
Regarding guile-wiredtiger, I hope to keep the interface as is. What I
to do is:
- drop the use of guile-bytestructures OR add support somehow for
pointer in C structs.
- Optimize for the single bytes column using a dedicated set of
- Improve the support of scheme object in (wiredtiger pack)
By the way, I did some testing using guile-next from guix and nothing
I also tried sqlite3 lsm extension but it is (also!) leaking memory. So,
I will not take that route right now. There is also the possibility to
use rocksdb or even postgresql. Anyway, I prefer to continue building
and then redo benchmark and switch database when I see it must be done.
That is SRFI 168 rely on SRFI 167 and it is easy to switch backend. I
with foundationdb. Once you have the okvs interface tested, you just
drop nstore.scm in your project and re-run the tests for nstore.scm.
magic of abstractions :)
- [ANN] guile-wiredtiger 0.8.0,