Re: [Sks-devel] sks (fast)build memory/cache problem

[Brian D Heaton]

Stephan,

Having beat my forehead on this one a few weeks ago, I can offer the
following suggestions from JohnC that got me on the right track:

Note:  I used the 5K/file keydump for the initial build.

Add the following to sksconf (in /var/sks or wherever you installed it)
---
pagesize:       128
ptree_pagesize: 16
---

I used the order of operations suggested by JohnC which worked quite
well for me:
---
build
  cp DB_CONFIG KDB/
  clean
  pbuild
  cp DB_CONFIG PTree/
  start 'sks db' and 'sks recon'
---

I did the build with "-n 4" since I saw some swapping with higher values
due to other memory hungry processes on the system.  I think I ran
"-cache 256", but I can't find that item in my notes from the build.

I had some hangs during operations which I believe I've tracked down to
memory issues.  Once I added "diskptree:" to sksconf, they went away and
my keyserver has been running clean and stable for a while.

-Brian


On 6/30/2012 7:29 PM, Stephan Beyer wrote:
> Hi,
> 
> On 30.06.2012 13:31, Stephan Beyer wrote:
>> The DB is already building based on my last keydump.
> 
> No, it was not. The process got stuck after a few minutes or seconds.
> In fact, neither build nor fastbuild works.
> I already "hg bisect"ed the sources and it did not work with any
> version that compiles on my system.
> 
> What happens? sks hangs (deadlocks) at futex(), says strace.
> Interestingly, it seems to depend on the cache size:
> 
> $ sks build -n 2 -cache 56 dump/sks-dump-000{1,2}.pgp
> Loading keys...done
> DB time:  0.04 min.  Total time: 0.04 min.
> 
> works. However, -cache 55 does not work. This is reproducible.
> 
> For 3 dump files (and -n 3), I have to use -cache 87;
> for 4 (and -n 4), it's 117. For 4 and -n 2, even 119 MiB
> are needed.
> If the -cache requirements are climbing this way, I will
> not be able to build the DB by memory restrictions...
> 
> For fastbuild, it is even worse (119 MiB cache do not suffice
> for 2 files).
> 
> What if I do it step by step?
> 
> $ sks build -n 1 -cache 60 dump/sks-dump-0001.pgp
> (works)
> $ sks merge -n 1 -cache 60 dump/sks-dump-0002.pgp
> (works)
> $ strace sks merge -n 1 -cache 60 dump/sks-dump-0003.pgp
> ...
> futex(0x7fc9a4188370, FUTEX_WAIT, 2, NULL
> 
> Interestingly, if I take a look into merge.log now, I can see:
> 
> 2012-07-01 04:13:03 Fatal database error: Bdb.DBError("BDB2034 unable to
> allocate memory for mutex; resize mutex region")
> 2012-07-01 04:13:03 closing database...
> 
> So it is really a memory/-cache issue.
> 
> Any ideas how to build the DB on a "low" memory system?
> Without -cache, it does not work, too, btw.
> 
> Best
>   Stephan
> 
> PS:
>  sks worked all the time on a 1 GiB system (much less free), and
>  the DB built without any problems in 2007 (where the keydump was a
>  little smaller, indeed).
> 
> _______________________________________________
> Sks-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/sks-devel
> 
>