Re: Analyzing dumps (Was: 6 million)

[Ángel]

On 2020-05-03 at 11:15 +0200, Kiss Gabor (Bitman) wrote:
> On Sat, 2 May 2020, Wiktor Kwapisiewicz wrote:
> 
> > On 02.05.2020 07:55, Gabor Kiss wrote:
> > > I would create such a programs from the scratch but I cannot
> > > find even the format description of the dump file. :-(
> > 
> > Last time I checked dumps where just packet piles so any OpenPGP tool
> > could read it.
> 
> Thanks again for the hint.
> 
> I wrote a small Perl script to see what is in dump files
> at http://keys.niif.hu/keydump/. (Server is managed by me.)
> 
> I found broken dumps. Certain RFC-4880 packets are truncated. For example
> let's see signatures of key 0x7cec0e7c93115f7e:
> 
> 00483ad0     89 01 22 04 10 01 02  00 0c 05 02 44 cf db 85  |..."........D...|
> 00483ae0  05 03 00 93 89 01 22 04  10 01 02 00 0c 05 02 4d  |......"........M|
> 
> We can see a signature packet starting at 00483ad1.
> (89 01 22 is a typical old style packet header.) Its length should be
> 0x122 octets however it breaks in middle of the second subpacket starting
> at 00483ae0. A new packet starts at 00483ae4 but my simple parser cannot
> detect this and gets confused.
> (Unfortunately such a truncated packet may block the import procedure
> also on a newly set up key server, I guess.)
> 
> I cannot imagine how this dump could be created.
> Could the attacker upload broken packets or is it "sks dump"
> who garbled the dump file? Or file became bad during
> compression/decompression?

My own script can handle them fine. 
There is always a container key around the packets, so it's not a
problem. But yes, sometimes there is a container that claims to have
more data than it has available, particularly around subpackets. Seems
that part is not verified by sks.

I see exactly one error on your first file (0000), on this key of
'Florian Streibelt':
Public Key v4 DSA - Key 3B8EA41F82F61240 - Created 2004-10-22 19:25:43
    #ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed
    # Creation ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 
bytes needed
    - ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed
ERROR: INVALID PACKET: Packet of 70 bytes but at least 1165 bytes needed


While the whole keys are manageable, I do see a large number of errors than I 
used to.

So I took a mattrude dump from 80 days ago. That had 5978288 keys vs 6008063 of 
your dump 
(an increase of 0.5% keys). Then I compared the number of keys with an error in 
both cases.
I considered a key to have an error if it detected a problem anywhere,
regardless of the number of signatures it had (the top one was a key
with 110 diagnostics).

On the February dump, there were 135 "broken keys" out of 5978288
(0,002258%)
On your dump, there are 134 "broken keys" out of 6008063 keys (0,00223%)


So not a problem of your server/dump, and not an increase of garbled
keys, either.

As for the 0x7cec0e7c93115f7e key (Terry Lansdown) I can import it from
your keyserver and use fine, also exporting with no error.
That key lives on keydump-sks-0054.pgp at 5956981 with a length of 29111
bytes.

I can find a funny signature there:
  - Signature v4 sig RSA SHA1
    # Creation 2006-08-02 00:53:57
    # SigExpiration 111d21h47m45s
ERROR: INVALID PACKET: Packet of 290 bytes but at least 8710 bytes
needed

This seems the opposite to your problem, though.

that is followed by 
  - Signature v4 sig RSA SHA1
    # Creation 2006-08-15 12:17:53
    # SigExpiration 14d
    - Issuer 9710B89BCA57AD7C


Interestingly, pgpdump does get confused by it

> Old: Signature Packet(tag 2)(290 bytes)
>         Ver 4 - new
>         Sig type - Generic certification of a User ID and Public Key 
> packet(0x10).
>         Pub alg - RSA Encrypt or Sign(pub 1)
>         Hash alg - SHA1(hash 2)
>         Hashed Sub: signature creation time(sub 2)(4 bytes)
>                 Time - Wed Aug  2 00:53:57 CEST 2006
>         Hashed Sub: signature expiration time(sub 3)(4 bytes)
>                 Time - Tue Nov 21 21:41:42 CET 2006
>         Sub: reserved(sub 1)(15 bytes)
>         Sub: additional decryption key(sub 10) WARNING: see CA-2000-18!!!(-1 
> bytes)
>                 Class - Unknown class(09)
>                 Pub alg - ElGamal Encrypt-Only(pub 16)
>                 Fingerprint - 97 10 b8 9b ca 57 bf 6b 5b 5b 4f 74 8d 81 87 63 
> 79 34 9b 83 
>         Sub: unknown(sub 46, critical)(13301 bytes)
>         Hash left 2 bytes - c6 6e 
>         RSA m^d mod n(13079 bits) - ...
>                 -> PKCS-1
> New: unknown(tag 56)(11 bytes)
> Old: Signature Packet(tag 2)(until eof)
>         Ver 218 - unknown


gpg --list-packets view is slightly different:


> :signature packet: algo 1, keyid 9710B89BCA57AD7C
>         version 4, created 1154472837, md5len 0, sigclass 0x10
>         digest algo 2, begin of digest 47 45
>         hashed subpkt 2 len 4 (sig created 2006-08-01)
>         hashed subpkt 3 len 4 (sig expires after 14d0h0m)
>         subpkt 16 len 8 (issuer key ID 9710B89BCA57AD7C)
>         data: [2047 bits]
> :signature packet: algo 1, keyid 0000000000000000
>         version 4, created 1154472837, md5len 0, sigclass 0x10
>         digest algo 2, begin of digest b7 6d
>         hashed subpkt 2 len 4 (sig created 2006-08-01)
>         hashed subpkt 3 len 4 (sig expires after 111d21h47m)
>         subpkt 1 len 15 (?)
>         subpkt 10 len 0 (additional recipient request)
> WARNING: PGP versions > 5.0 and < 6.5.8 will automagically encrypt to this 
> key and thereby reveal the plaintext to the owner of this ARR key. Detailed 
> info follows:
>         subpkt 10 len 4294967295 (too short: buffer is only 8690)
>         subpkt 9 len 9 (key expires after 8y301d11h27m)
>         subpkt 91 len 90 (?)
>         subpkt 37 len 128 (?)
>         subpkt 14 len 167 (?)
>         critical subpkt 87 len 6 (?)
>         subpkt 95 len 73 (?)
>         critical subpkt 100 len 157 (experimental / private subpacket)
>         subpkt 9 len 95 (key expires after 109y205d21h52m)
>         subpkt 68 len 160 (?)
>         data: [MPI_NULL]


I then extracted the key and performed a gpgsplit. Those above are 
000013-002.sig and 000014-002.sig

If you compare them, these two signature packets are extremely similar:

13:
00000010  03 00[12 75 00 00 0a 09  10 97 10 b8 9b ca 57 ad  |...u..........W.|
00000020  7c 47 45 07 ff 78 8c c9  b5 f5 81 3f f0 1e ea 97  ||GE..x.....?....|
00000030  73 79] bf 6b 5b 5b 4f 74  8d 81 87 63 79 34 9b 83  |sy.k[[Ot...cy4..|

14:
00000010  03 00[93 89 01 22 04 10  01 02 00 0c 05 02 4d 98  |....."........M.|
00000020  35 09 05 03 00 12 75 00  00 0a 09 10 97 10 b8 9b  |5.....u.........|
00000030  ca 57]bf 6b 5b 5b 4f 74  8d 81 87 63 79 34 9b 83  |.W.k[[Ot...cy4..|

This second one is identified by pgpdump as
Old: Signature Packet(tag 2)(290 bytes)
        Ver 4 - new
        Sig type - Generic certification of a User ID and Public Key 
packet(0x10).
        Pub alg - RSA Encrypt or Sign(pub 1)
        Hash alg - SHA1(hash 2)
        Hashed Sub: signature creation time(sub 2)(4 bytes)
                Time - Wed Aug  2 00:53:57 CEST 2006
        Hashed Sub: signature expiration time(sub 3)(4 bytes)
                Time - Tue Nov 21 21:41:42 CET 2006
        Sub: reserved(sub 1)(15 bytes)
        Sub: additional decryption key(sub 10) WARNING: see CA-2000-18!!!(-1 
bytes)
                Class - Unknown class(09)
                Pub alg - ElGamal Encrypt-Only(pub 16)
                Fingerprint - 97 10 b8 9b ca 57 bf 6b 5b 5b 4f 74 8d 81 87 63 
79 34 9b 83 
        Sub: unknown(sub 46, critical)(13301 bytes)

That CA-2000-18 reference is about CERT Advisory about Additional Decryption 
Keys on unhashed subpacket
https://resources.sei.cmu.edu/asset_files/whitepaper/2000_019_001_496188.pdf



00000000  89 01 22 04 10 01 02 00  
00 0c → 12 hashed subpacket bytes
05 02 44 cf db 85 05
    ^ 2 = Signature Creation Time:  Wed Aug  2 00:53:57 CEST 2006
05 03 00 93 89 01
    ^ 3 = Signature Expiration Time: 111 days later     

22 04 
Unhashed packet length of 8708 (invalid)

10 length of 16 bytes
 01 Subpacket type  1 (reserved)
 02 00 0c 05 02 4d 98 35 09 05 03 00 12 75 00
00 invalid length of 0 bytes
 0a Subpacket type 10 additional decryption key (Placeholder for backward 
compatibility in rfc4880)
 etc.


However, if we discard the first 20 bytes of this signature, we get a quite 
differen signature:
> Old: Signature Packet(tag 2)(290 bytes)
>       Ver 4 - new
>       Sig type - Generic certification of a User ID and Public Key 
> packet(0x10).
>       Pub alg - RSA Encrypt or Sign(pub 1)
>       Hash alg - SHA1(hash 2)
>       Hashed Sub: signature creation time(sub 2)(4 bytes)
>               Time - Sun Apr  3 10:51:21 CEST 2011
>       Hashed Sub: signature expiration time(sub 3)(4 bytes)
>               Time - Sun Apr 17 10:51:21 CEST 2011
>       Sub: key flags(sub 27)(critical)(51 bytes)
>               Flag - This key may be used to certify other keys
>               Flag - This key may be used to sign data
>               Flag - The private component of this key may be in the 
> possession of more than one person
>       Hash left 2 bytes - b8 de 
>       RSA m^d mod n(54585 bits) - ....
> 

Those key flags are certainly weird, and the stated mpi is larger than 
expected, but these signature times make much more sense.

So it seems that at a given point, a mid-written signature got replaced by a 
different one, and so we ended up with this broken signature item.


Best regards