[Sks-devel] Slow syncing (was: Re: Memory Leak in recon server?)

[Arnold]

On 02-02-10 16:05, Phil Pennock wrote:
> On 2010-02-02 at 01:19 +0100, Arnold wrote:
>> If it is not in the number of peers, then what can I do further to make it
>> sync more quickly?
> 
> Do the recon logs contain anything to suggest a cause for not fetching
> all keys?  What about if you bump up the debug level?

As far as I can see / understand, it is spending most of the time examining
blocks of 100 keys (the hashes I assume) and find they are the same.

Current log level is 4 (can set it higher tomorrow if necessary). Below some
lines from my log with update activity. Half an hour before it also found a
difference... The db.log tells me that I've no e-mail peers, which is indeed
the case.

One thing I now noted is that most lines of the form
Requesting 100 missing keys from <ADDR_INET ...:11371>, starting with
1ED038D719CE90EEF95B6F185D191EA7
start with 1D..... or 1E..... Is that OK, or does it mean it is looping
somewhere?

If I miss or misinterpret something, please let me know!


Teun suggested the capacity of my server would be insufficient. Looking at
the memory use and CPU load (see below), I guess it will do fine. The
average network traffic over the past 9 days is about 2% of my capacity,
both upload and download. The statistics at
        http://www.pool.ntp.org/scores/82.95.191.161
show that the server is accurate in time keeping, synchronising over the
net. So I've no network problems either.


Jens suggested off-list to add only his server and set
        http_fetch_size: 500
until my server has caught up. I will try that tomorrow (going to bed now).
For now I have added three more servers. However, the log shows some time
peer A, then some time peer B, etc. So, more peers may indeed not do the
trick, but only spread the traffic across the peers.


Well, thank you all!
   Arnold



$ free
             total       used       free     shared    buffers     cached
Mem:       2068328    2016324      52004          0     154700    1445292
-/+ buffers/cache:     416332    1651996
Swap:       497972        816     497156

$ top
top - 01:50:08 up 9 days,  2:31,  3 users,  load average: 0.41, 0.36, 0.46
Tasks: 153 total,   2 running, 151 sleeping,   0 stopped,   0 zombie
Cpu(s):  0.3%us,  0.3%sy,  0.0%ni, 99.3%id,  0.0%wa,  0.0%hi,  0.0%si,  0.0%st
Mem:   2068328k total,  2016760k used,    51568k free,   154816k buffers
Swap:   497972k total,      816k used,   497156k free,  1445292k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND


32131 debian-s  20   0 35856  31m  28m S  1.3  1.5   0:53.81 sks


13462 list      20   0 10840 7832 2564 S  0.3  0.4   1:37.98 python


13524 mysql     20   0  126m  24m 5140 S  0.3  1.2   5:09.82 mysqld


    1 root      20   0  2100  684  588 S  0.0  0.0   0:08.26 init




In recon.log:

2010-02-02 06:54:54 Requesting 100 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 1ECDEB960744508BCA9D77259DAFB81D
2010-02-02 06:54:56 100 keys received
2010-02-02 06:54:57 Requesting 100 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 1ED038D719CE90EEF95B6F185D191EA7
2010-02-02 06:54:58 100 keys received
2010-02-02 06:55:00 Requesting 83 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 1ED274B269B1A93E7434DD098FA12319
2010-02-02 06:55:01 83 keys received
2010-02-02 06:55:02 Added 3 hash-updates. Caught up to 1265090101.513311
2010-02-02 06:55:39 Recon partner: <ADDR_INET 76.184.64.189:11370>
2010-02-02 06:55:40 Initiating reconciliation
2010-02-02 06:55:43 Reconciliation complete
2010-02-02 06:55:43 10182 hashes recovered from <ADDR_INET 76.184.64.189:11371>
2010-02-02 06:55:46 Requesting 100 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 00C0983B4B12FB65DFE4FB5992282753
2010-02-02 06:55:47 100 keys received
2010-02-02 06:55:49 Added 1 hash-updates. Caught up to 1265090147.860824
2010-02-02 06:55:49 Requesting 100 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 1DE771CA7BDC27F9BF991D9DDD3E3A4A
2010-02-02 06:55:50 100 keys received
2010-02-02 06:55:50 Requesting 100 missing keys from <ADDR_INET
76.184.64.189:11371>, starting with 1DE9463535E7F871D4CCE03BD31B61CE
2010-02-02 06:55:51 100 keys received


In db.log:

2010-02-02 06:54:53 Applying 0 changes
2010-02-02 06:54:56 Applying 0 changes
2010-02-02 06:54:57 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:54:58 Applying 0 changes
2010-02-02 06:55:01 0 potential merges found for keyid 79576865
2010-02-02 06:55:01 1 updates found before filtering
2010-02-02 06:55:01 0 potential merges found for keyid F871868C
2010-02-02 06:55:01 1 updates found before filtering
2010-02-02 06:55:01 0 potential merges found for keyid F30EFBE2
2010-02-02 06:55:01 1 updates found before filtering
2010-02-02 06:55:01 Applying 3 changes
2010-02-02 06:55:01 Adding hash 9BB6A361CD79739D3F2B4259D51CC582
2010-02-02 06:55:01 Adding hash E318F86F4FBF9BED718AFADADCD10E81
2010-02-02 06:55:01 Adding hash FD1CD68BE7B2B5DBAA7A772C562108B8
2010-02-02 06:55:02 Sending LogResp size 3
2010-02-02 06:55:07 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:55:17 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:55:27 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:55:37 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:55:47 <mail transmit keys> error in callback.: Failure("No
partners specified")
2010-02-02 06:55:47 0 potential merges found for keyid FC7D2E6B
2010-02-02 06:55:47 1 updates found before filtering
2010-02-02 06:55:47 Applying 1 changes
2010-02-02 06:55:47 Adding hash 00C0983B4B12FB65DFE4FB5992282753
2010-02-02 06:55:49 Sending LogResp size 1
2010-02-02 06:55:50 Applying 0 changes
2010-02-02 06:55:51 Applying 0 changes

signature.asc
Description: OpenPGP digital signature