Re: [Discussion] Side-by-side dump format

[Jose E. Marchesi]

Hi.

> Something I would find really useful while writing pickles and looking
> at corrupt data is a side-by-side dump format that shows the value
> level information next to the bytes, to quickly debug where things get
> misaligned.
>
> I noodled around with a particular format of this here (I have also
> attempted to attach it to this email):
> https://jade.fyi/poke-dump-demo.html
>
> It seems that this mockup is more legible than staring at the
> information separately, so that is a positive. If Poke were a GUI, I
> could see this information being most easily used by mousing over the
> bytes, or clicking them, to go right to the field, sort of analogously
> to how https://typst.app does their live preview of typesetting. For
> static display, it is probably best to put the information as close as
> possible without interrupting the visual flow of the hexdump. An
> alternate format could perhaps be to put the information under each
> line of hexdump, but I am a little bit worried that would make it hard
> to follow which pieces are hexdump and what they correspond to.
>
> One challenge that this hits is that I can see it being pretty easy to
> get too much information in one line. Also, the syntax for byte arrays
> is a little bit bulky :)
>
> The rules I used for producing this were:
> - A field takes the first colour not yet appearing in that line
> - That colour is retained across multiple lines if the field spans
> multiple lines
> - Arrays of primitives are treated as one item, sub structures are
> treated as new items for each field inside
> - The only thing that's considered is things generating bytes, rather
> than the structure. It may be possible to insert structure markers
> into the dump though; not sure how feasible that would be.
>
> Things that are unclear:
> - Where to put fields that appear in the bytes out of definition order?
>
> Any thoughts on other ways to do this?

This would be a good feature that we definitely want to have in poke.
Your idea of not interrupting the flow of bytes by intercalating field
information is a nice one.

As far as I can see, the main problem/caveat here is that since we have
a "static display", we have to include all the information that may be
useful for the user, all at once.  That leads to these very long lines.

But I think it doesn't have to be like that.

Consider this:

  (poke) dump :val PCAPNG_Shb @ 0#B
  76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
  00000000: 0a0d 0d0a 6000 0000 4d3c 2b1a 0100 0000 typ, len, byte_order, 
major, minor
  00000010: ffff ffff ffff ffff 0400 3a00 4564 6974 opts
  00000020: 6361 7020 2857 6972 6573 6861 726b 2920
  00000030: 342e 302e 3520 2847 6974 2076 342e 302e
  00000040: 3520 7061 636b 6167 6564 2061 7320 342e
  00000050: 302e 352d 3129 0000 0000 0000 6000 0000 len2, assertion

Note how this includes just the names of he fields of the entities of
the dumped value (PCAPNG_Shb @ 0#B) without nesting.  Suppose the
example is colorized in the same way you suggest.  Now, how to achieve
the nesting?

Two ideas which can be complementary:

1. We could install terminal hyperlinks in both the bytes and the field
   names, so that when the user clicks on them, a byte dump of the given
   field is executed ("execute" hyperlink).  Then the user can just
   dump-val the top-level value, and click her way down.

2. We can have the dump-val command to populate a global array of values
   every time it gets run, to print only one level, and to include
   indexes like this:

   (poke) dump :val PCAPNG_Shb @ 0#B
   76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
   00000000: 0a0d 0d0a 6000 0000 4d3c 2b1a 0100 0000 1=typ, 2=len, 
3=byte_order, 4=major, 5=minor
   00000010: ffff ffff ffff ffff 0400 3a00 4564 6974 6=opts
   00000020: 6361 7020 2857 6972 6573 6861 726b 2920
   00000030: 342e 302e 3520 2847 6974 2076 342e 302e
   00000040: 3520 7061 636b 6167 6564 2061 7320 342e
   00000050: 302e 352d 3129 0000 0000 0000 6000 0000 7=len2, 8=assertion

   Lets say that then the user wants to dump-val `opts'.  Could be
   quickly done like this:

   (poke) dump :val 6
   76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
   00000010: ffff ffff ffff ffff 0400 3a00 4564 6974 9=opts[0]
   00000020: 6361 7020 2857 6972 6573 6861 726b 2920
   00000030: 342e 302e 3520 2847 6974 2076 342e 302e
   00000040: 3520 7061 636b 6167 6564 2061 7320 342e
   00000050: 302e 352d 3129 0000 0000 0000 6000 0000 10=endofopt
   (poke) dump :val 9
   76543210  0011 2233 4455 6677 8899 aabb ccdd eeff 10=typ, 11=len, 12=value
   00000010: ffff ffff ffff ffff 0400 3a00 4564 6974
   00000020: 6361 7020 2857 6972 6573 6861 726b 2920
   00000030: 342e 302e 3520 2847 6974 2076 342e 302e
   00000040: 3520 7061 636b 6167 6564 2061 7320 342e
   00000050: 302e 352d 3129 0000 0000 0000 6000 0000 13=padding

   At any point the user can also print the full value of a field just
   by referring to the global array updated by `dump':

   (poke) dumpvals[9]
   PCAPNG_Option {
   ...
   }

   Note that the field indexes (and entries in global array) would be
   kept and accumulated until `dump' gets passed a non-index in :val.
   This allows the user to easily go up.

We can also support a mode (configuration variable) that activates a
nested mode:

   (poke) dump :val PCAPNG_Shb @ 0#B
   76543210  0011 2233 4455 6677 8899 aabb ccdd eeff
   00000000: 0a0d 0d0a 6000 0000 4d3c 2b1a 0100 0000 1=typ, 2=len, 
3=byte_order, 4=major, 5=minor
   00000010: ffff ffff ffff ffff 0400 3a00 4564 6974 6=opts, 7=opts[0], 8=typ, 
9=len, 10=value
   00000020: 6361 7020 2857 6972 6573 6861 726b 2920
   00000030: 342e 302e 3520 2847 6974 2076 342e 302e
   00000040: 3520 7061 636b 6167 6564 2061 7320 342e
   00000050: 302e 352d 3129 0000 0000 0000 6000 0000 11=padding, 12=len2, 
13=assertion

where colors will serve to determine where each field belongs.

I have not included the ASCII dump part for brevity.
WDYT?

As for,

> - Where to put fields that appear in the bytes out of definition
>   order?

I would say, wherever they are stored?  The implementation of the above
would be done in Poke (see pickles/ios.pk) and I think we have enough
features in the language to determine the name of the fields and their
offsets as generic `any' values.  See pickles/diff.pk for an example of
such code.