Re: [Sks-devel] sks-keyserver Fedora Linux

[Jeff Johnson]

On Apr 25, 2011, at 8:29 AM, Robert Hinson wrote:

I tried running the pre compiled sks keyserver on a fedora 14 box.

I do the sks build ./dump/*.pgp -n 10 -cache 100

It craps out after like 2 minutes.  Anyone else experience this problem?

Yes. The problem is in fact described though somewhat obscurely:

You need to adjust the -n and -cache parameters.

This is what I have done to get past "crap out":

The three operations performed while initializing an SKS server

from a dump are in /usr/bin/sks_build.sh:

echo === Running fastbuild... ===

if ! sks fastbuild -n 10 -cache 100; then fail; fi

echo === Cleaning key database... ===

if ! sks cleandb; then fail; fi

echo === Building ptree database... ===

if ! sks pbuild -cache 20 -ptree_cache 70; then fail; fi

Run each of those operations individually and track progress by

putting a tail on the logs produced for each operation.

The resource intensive operations are "fastbuild" and "pbuild" and

each has slightly different parameter sets and tweaks needed.

I'd tell you exactly what parameters I've used but I have forgotten

(and I've done different things on different platforms)

Most definitely adding more memory did not solve the issues (for me).

Then double (or scale by 10x initially) the -n and -cache parameters

and repeat, monitoring the progress using top and paying

particular attention to the VIRT/RSS memory usage.

At "crap out" the process progress ceases (no CPU cycles used) and

you can tell how close you are/were to success by examining the statistics

in the logs.

Another path to an initial load is to copy the database from an existing

running SKS server. This is a bit tricky because of Berkeley DB, the

very obscure command that is needed for success is

db_load -r lsn ...

on each of the tables (Berkeley DB calls these "databases" but they are tables):

These are the "tables" in KDB/* run "db_load -r lsn" on each on a working SKS server:

# ls -al

total 2778072

drwx------ 3 jbj  jbj        4096 Apr 24 20:17 .

drwxr-xr-x 5 jbj  root       4096 Apr 24 20:29 ..

...

-rw------- 1 jbj  jbj  1504172032 Apr 24 20:09 key

-rw------- 1 jbj  jbj   143142912 Apr 24 20:09 keyid

...

-rw------- 1 jbj  jbj        8192 Oct  1  2010 meta

-rw------- 1 jbj  jbj   134213632 Apr 24 20:09 subkeyid

-rw------- 1 jbj  jbj   118718464 Apr 24 20:09 time

-rw------- 1 jbj  jbj       77824 Mar 20 18:42 tqueue

-rw------- 1 jbj  jbj   758652928 Apr 24 20:09 word

The "db_load -r lsn" removes "log sequence numbers" so

that the tables are portable.

The tables index into the dump files so you also need to move

all the dump/* files.

If you are prepared to download multiple Gb's of information, I could

do these operations and hand you an "image" that you could jumpstart

a running SKS instance. Almost certainly it would take a few attempts,

but I have succeeded in setting up a SKS server from an image when

I get annoyed at diddling -n and -cache parameters at 3am.

There's a third path to "works" is loading a truncated dump. There are approximately

200 files (depending on chosen per-dump-file-size) used in initializing

a SKS server. Removing some of those files is another way to achieve

a load in spite of difficulties with "crap out" related (apparently, I do not know) to

the amount of memory resources available.

Please note that removing files from the dump in order to achieve an initial

load WILL lead to a very large update (I've seen >24 hours needed to "catch up")

when you start peering.

Note also that gossip protocols like in SKS key servers are NOT designed for

bulk transport aand are hugely inefficient.

Short answer: A partial load achieved by removing files from the dump is

nothing more than an expedient hack.

I do have private SKS key servers in VM's that could be used if you are

forced to the "expediant hack" route in order to get a SKS keyserver

initially loaded.

hth

73 de Jeff

smime.p7s
Description: S/MIME cryptographic signature