Re: [Sks-devel] more sks db problems

[Yaron Minsky]

Chris Kuethe said:
> On Thu, 29 Jan 2004, Yaron Minsky wrote:
>
>> Hmm.  So two things here:
>>
>> first of all, cleandb  seems to throw a gasket when it hits a "bad key",
>> i.e., one that has a nonsensical packet sequence.  This is pretty bad,
>> since it should say something useful rather than just crashing.
>
> maybe delete the key, and let the sync process handle it?

That's possible, but there are some problems with SKS locking up on very
large reconcile-runs, I think due to buffers filling up and causing
deadlock.  Also, it will be much slower, since fastbuild and build do some
optimizations to make insertions into the database faster.

>> But the worse problem is that a bad key got into the database in the
>> first
>> place.  "sks build" and "sks fastbuild" should protect against that, but
>> it's possible that they somehow don't.  Which command (build or
>> fastbuild)
>> did you use for the initial build?  And do you have the corresponding
>> logfile?  If you could make it web-accessible, I could download them and
>> take a look.
>
> i'll have to reload into a new database, but sure... i'll turn up the
> debugging, but the log i posted is everything...
>
> i did a build (not a fastbuild)

I've used fastbuild a lot more than build.  It might be worth trying build.

>> The hashqueries come from someone else who thinks their database is very
>> different from yours, and is requesting, 100-block by 100-block, that
>> you
>> send them the missing keys.  This is how a host with a very different
>> key
>> set gets synchronized with the system.
>
> that's very strange, since it's coming from one of my sync peers that i
> should already be synchronized with. all my diff files are pretty short.

It is weird.  I don't know what to make of it.

>> y
>>
>> Chris Kuethe said:
>> > Thinking I may have bollixed something while loading my last keyring,
>> > I tried a complete rebuild with the keydump from debian/noreply. The
>> > load completed successfully, so I ran cleandb to make the database
>> > ready to start syncing.
>> >
>> > here's what cleandb says:
>> > 2004-01-29 09:01:36 Opening log
>> > 2004-01-29 09:01:36 Opening KeyDB database
>> > 2004-01-29 09:01:37 Keydb opened
>> > 2004-01-29 09:01:37 Database already deduped
>> > 2004-01-29 09:01:37 Merging keys in database
>> > 2004-01-29 09:01:37 Starting key merge
>> > 2004-01-29 09:01:37 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:37 Hash: 4A890E696081B0D3B747143BFB881E79
>> > 2004-01-29 09:01:37 Hash: 600066ED625D84CE88211E0252A675D7
>> > 2004-01-29 09:01:37 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:37 Hash: AAC4EE9160676C62F1BEDF8B74108154
>> > 2004-01-29 09:01:37 Hash: F87E832671B9C61F4ADAABFAC83DBC86
>> > 2004-01-29 09:01:37 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:37 Hash: 6A0F6639D1B70ECA6D8205F7A86D792C
>> > 2004-01-29 09:01:37 Hash: A01F53A7B4FB435470BFCBBF0E07160C
>> > 2004-01-29 09:01:37 10 thousand steps processed
>> > 2004-01-29 09:01:37 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:37 Hash: 1768F356D81C537E811183F5AB07A2AC
>> > 2004-01-29 09:01:37 Hash: 8EC45621CAF6D96D217E2DD7E01AD261
>> > 2004-01-29 09:01:38 20 thousand steps processed
>> > 2004-01-29 09:01:38 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:38 Hash: 3A2D825000E94AD1FF9CC832CF44CDC5
>> > 2004-01-29 09:01:38 Hash: B33E5F8EABE7AD4C465150B07D96983C
>> > 2004-01-29 09:01:38 30 thousand steps processed
>> > 2004-01-29 09:01:38 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:38 Hash: 69EE7077DBB67C5DBE17ADDC6130BED4
>> > 2004-01-29 09:01:38 Hash: A24947DB4E81A5EEC3528D7A11294E60
>> > 2004-01-29 09:01:38 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:38 Hash: 58BE68CE36AF26FD19A948BC298CFF77
>> > 2004-01-29 09:01:38 Hash: 8615A327D1DB0F9541151AA2BA246688
>> > 2004-01-29 09:01:38 40 thousand steps processed
>> > 2004-01-29 09:01:38 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:38 Hash: 1A5E96A986C18513217BE436D39ECA14
>> > 2004-01-29 09:01:38 Hash: F5A20D0DCE2F4B84FDA139C127B12670
>> > 2004-01-29 09:01:38 50 thousand steps processed
>> > 2004-01-29 09:01:39 Multiple keys found with same ID.
>> merge_from_hashes
>> > called
>> > 2004-01-29 09:01:39 Hash: 13E37C592A17EA2A345ED114BEA5D281
>> > 2004-01-29 09:01:39 Hash: 9C2FDB7AC49D748F12937C01463FB8B3
>> > Fatal error: exception KeyMerge.Unparseable_packet_sequence
>> > *exits here*
>> >
>> > so i ran pbuild which claims to have completed successfully:
>> > ...
>> > 2004-01-29 09:09:49 1935000 hashes processed
>> > 2004-01-29 09:09:50 1937671 hashes processed
>> > 2004-01-29 09:09:50 Cleaning Tree.
>> >
>> > yet I get the same error running cleandb... so i'm going to rebuild
>> again,
>> > this time doing a build with just one file used to "build" and then
>> i'm
>> > going to do a cleandb, pbuild and then merge all the keyfiles again to
>> see
>> > if that helps. feh.
>> >
>> > by the way, what's the deal with all hashqueries?
>> >
>> > 2004-01-29 09:32:50 Handling /pks/hashquery
>> > 2004-01-29 09:32:50 100 keys found
>> > 2004-01-29 09:32:54 Handling /pks/hashquery
>> > 2004-01-29 09:32:54 100 keys found
>> >
>> > every time i get one of these i my disk gets beaten on for 30 to 40
>> > seconds
>> > at a time...
>> >
>> > CK
>> >
>> > --
>> > Chris Kuethe, GCIA CISSP: Secure Systems Specialist - U of A CNS
>> >       office: 157 General Services Bldg.    +1.780.492.8135
>> >               address@hidden
>> >
>> >      GDB has a 'break' feature; why doesn't it have 'fix' too?
>> >
>> >
>> >
>> > _______________________________________________
>> > Sks-devel mailing list
>> > address@hidden
>> > http://mail.nongnu.org/mailman/listinfo/sks-devel
>> >
>>
>>
>> |--------/            Yaron M. Minsky              \--------|
>> |--------\ http://www.cs.cornell.edu/home/yminsky/ /--------|
>>
>> Open PGP --- KeyID B1FFD916
>> Fingerprint: 5BF6 83E1 0CE3 1043 95D8 F8D5 9F12 B3A9 B1FF D916
>>
>
> --
> Chris Kuethe, GCIA CISSP: Secure Systems Specialist - U of A CNS
>       office: 157 General Services Bldg.    +1.780.492.8135
>               address@hidden
>
>      GDB has a 'break' feature; why doesn't it have 'fix' too?
>


|--------/            Yaron M. Minsky              \--------|
|--------\ http://www.cs.cornell.edu/home/yminsky/ /--------|

Open PGP --- KeyID B1FFD916
Fingerprint: 5BF6 83E1 0CE3 1043 95D8 F8D5 9F12 B3A9 B1FF D916