Re: Help fixing NativeMessaging host: read 32-bit message length in native byte order

[Emanuele Torre]

On Sat, Jun 24, 2023 at 10:01:30AM -0400, Greg Wooledge wrote:
> The only way to read this value would be to have some external program
> do it, and spit out a text encoding that bash can read.  Given that
> the 4 bytes are the prefix of a larger data stream, we need to ensure
> that *only* those 4 bytes are read, and that the rest of the stream
> is untouched.  The tool of choice for that is dd(1).
[...]
> Putting the pieces together, you might write something like this:
> 
> length=$(dd bs=4 count=1 | od -An -td4)
> length=$((length))      # trim leading spaces
> IFS= read -rN"$length" json

One small note: dd(1) will run  bs=4 count=1  as a single call (count=1)
to the read(2) system call with a buffer size of 4 (bs=4), and then
print out what was read.

read(2) with a buffer size of 4, will return with up to 4 bytes (but at
least one), not necessarily exactly 4.

This means that if the input source is "slow", e.g. if the writer to the
pipe is writes in chunks of less than 4 bytes, and there happens to be
not enough data on the pipe when you are reading, you may read only 1,
2, or 3 bytes from the input, and leave the rest.

  $ { /bin/printf '\1'; /bin/printf '\2\3\4' ;} |
  > dd 'bs=4' 'count=1' 2>/dev/null |
  > od -An -tc
   001
  $ { /bin/printf '\1\2'; /bin/printf '\3\4' ;} |
  > dd 'bs=4' 'count=1' 2>/dev/null |
  > od -An -tc
   001 002
  $ { /bin/printf '\1\2\3\4\5' ;} |
  > dd 'bs=4' 'count=1' 2>/dev/null |
  > od -An -tc
   001 002 003 004

  $ { /bin/printf '\1\2'; /bin/printf '\3\4'; echo hello ;} |
  > { dd 'bs=4' 'count=1' of=>(od -An -td) 2>/dev/null; od -An -c ;}
           513
   003 004   h   e   l   l   o  \n
  $ { /bin/printf '\1\2\3\4'; echo hello ;} |
  > { dd 'bs=4' 'count=1' of=>(od -An -td) 2>/dev/null; od -An -c ;}
      67305985
     h   e   l   l   o  \n

FreeBSD's and GNU's implementation of dd(1) support specifying
iflag=fullblock  to resolves this problem.

  $ { /bin/printf '\1\2'; /bin/printf '\3\4'; echo hello ;} |
  > { dd 'bs=4' 'count=1' 'iflag=fullblock' of=>(od -An -td) 2>/dev/null
  >   od -An -c ;}
      67305985
     h   e   l   l   o  \n

It makes dd(1) call read(2) more times until the input has reached end
of file, or it has returned all the neccesary bytes for that count.

NOTE: iflag=fullblock is not available on OpenBSD and some other
systems, and, as of issue 7, it is not specified by POSIX though it is
planned for issue 8 <https://austingroupbugs.net/view.php?id=406>.

So, in the end, you can use something like this.

  length=$(dd iflag=fullblock bs=4 count=1 | od -An -td4)
  length=$((length))      # trim leading spaces
  IFS= read -rN"$length" json

o/
 emanuele6