Re: Announcement stis-engine

[Stefan Israelsson Tampe]

HI all,

I added a new gem, it' a c based serializer and deserializer of guile data
which is relative feature rich and will ping pong
between scheme and c-land in order to be able to be fiber friendly. So
currently the benchmarks are for the following copy
operations,

,time (load (dump (make-bytevector N 1))
1s for ~ 1GB (test of memcpy)

,time (load (dump (make-vector N 1))
1s for ~ 100MB (will be less pipelined)

,time (load (dump (iota N))
1s for ~ 50MB (will create in total N cons cells with 5 bytes)

The scheme based version of these serializers and deserializers was able to
perform 5MB per second, but thats with much higher allocation overhead

This means that we match pythons dump and load functions. I will add those
to my fibers piping tool

Currently supported data structures
all numbers yes even bignums
all immediates  ('(), booleans characters and whatnot)
bytevectors
strings (wide strings and narrow strings)
symbols
keywords
pairs
vectors

the location of the c-code is in
guile-persist:   https://gitlab.com/tampe/guile-persist

(use-modules (persist persistance)) => atom-load-bv and atom-dump-bv is
defined.

This library is used by python-on-guile and I will replace the current dump
and load with this in the end. Also this tool
will be added to the fiber piping system I am working with in stis-engine.
















On Sat, Aug 14, 2021 at 2:07 PM Stefan Israelsson Tampe <
stefan.itampe@gmail.com> wrote:

> Next is to add support for
> 1. self referential data structures like #1 = '(1 2 3 #1), string
> compression
> 2. Structs
> 3 .Goops Classes and Objects
>
> Then I will add
> 1. Lambdas
> and then Continuations to be able to serialize (if you turn off unboxing
> optimizationsas in guile)
>
> After that a toolbox to customize tserializing just as with python
>
> Note all this is supported in python-on-guile, just that the new code is
> better suited for proper streaming
> and the code is much nicer in my personal view so you do have all these
> features already if you use pythons
> dumps and loads commands together with string->fpipe and fpipe->string.
>
> The new thing is essentially stream compression and streams to zmq's.
> Stream compression wll come to
> python as well as the zlib python module is very rudementary
>
>