Re: Help fixing NativeMessaging host: read 32-bit message length in native byte order

[guest271314]

So, in the end, you can use something like this.

  length=$(dd iflag=fullblock bs=4 count=1 | od -An -td4)
  length=$((length))      # trim leading spaces
  IFS= read -rN"$length" json

That still winds up showing "Invalid byte sequence in conversion input"
when echo'ing the output to a file.

Note, I am not opposed to using Perl's unpack in a Bash script. I am just
trying to determine if this is possible at all using only Bash. If not then
a shell script combining Perl with Bash will suffice. A Python version
https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_python.py
uses unpack

    def getMessage():
        rawLength = sys.stdin.buffer.read(4)
        # if len(rawLength) == 0:
        #    sys.exit(0)
        messageLength = struct.unpack('@I', rawLength)[0]
        message = sys.stdin.buffer.read(messageLength).decode('utf-8')
        return json.loads(message)


I passed a C++ version
https://github.com/guest271314/NativeMessagingHosts/blob/main/nm_cpp.cpp
through this https://codepal.ai/language-translator/cpp-to-bash Web site
and it spit out this (the first time I observed dd being applicable to this
case), unforthunately the code doesn't work as expected

#!/bin/bash

getMessage() {
  length=0
  read -n 4 length
  message=$(dd bs=1 count=$length 2>/dev/null)
  echo -n "$message"
}

sendMessage() {
  length=$(echo -n "$1" | wc -c)
  echo -n "$(printf '%04x' $length)"
  echo -n "$1"
}

while true; do
  message=$(getMessage)
  sendMessage "$message"
done

On Sat, Jun 24, 2023 at 7:48 AM Emanuele Torre <torreemanuele6@gmail.com>
wrote:

> On Sat, Jun 24, 2023 at 10:01:30AM -0400, Greg Wooledge wrote:
> > The only way to read this value would be to have some external program
> > do it, and spit out a text encoding that bash can read.  Given that
> > the 4 bytes are the prefix of a larger data stream, we need to ensure
> > that *only* those 4 bytes are read, and that the rest of the stream
> > is untouched.  The tool of choice for that is dd(1).
> [...]
> > Putting the pieces together, you might write something like this:
> >
> > length=$(dd bs=4 count=1 | od -An -td4)
> > length=$((length))      # trim leading spaces
> > IFS= read -rN"$length" json
>
> One small note: dd(1) will run  bs=4 count=1  as a single call (count=1)
> to the read(2) system call with a buffer size of 4 (bs=4), and then
> print out what was read.
>
> read(2) with a buffer size of 4, will return with up to 4 bytes (but at
> least one), not necessarily exactly 4.
>
> This means that if the input source is "slow", e.g. if the writer to the
> pipe is writes in chunks of less than 4 bytes, and there happens to be
> not enough data on the pipe when you are reading, you may read only 1,
> 2, or 3 bytes from the input, and leave the rest.
>
>   $ { /bin/printf '\1'; /bin/printf '\2\3\4' ;} |
>   > dd 'bs=4' 'count=1' 2>/dev/null |
>   > od -An -tc
>    001
>   $ { /bin/printf '\1\2'; /bin/printf '\3\4' ;} |
>   > dd 'bs=4' 'count=1' 2>/dev/null |
>   > od -An -tc
>    001 002
>   $ { /bin/printf '\1\2\3\4\5' ;} |
>   > dd 'bs=4' 'count=1' 2>/dev/null |
>   > od -An -tc
>    001 002 003 004
>
>   $ { /bin/printf '\1\2'; /bin/printf '\3\4'; echo hello ;} |
>   > { dd 'bs=4' 'count=1' of=>(od -An -td) 2>/dev/null; od -An -c ;}
>            513
>    003 004   h   e   l   l   o  \n
>   $ { /bin/printf '\1\2\3\4'; echo hello ;} |
>   > { dd 'bs=4' 'count=1' of=>(od -An -td) 2>/dev/null; od -An -c ;}
>       67305985
>      h   e   l   l   o  \n
>
> FreeBSD's and GNU's implementation of dd(1) support specifying
> iflag=fullblock  to resolves this problem.
>
>   $ { /bin/printf '\1\2'; /bin/printf '\3\4'; echo hello ;} |
>   > { dd 'bs=4' 'count=1' 'iflag=fullblock' of=>(od -An -td) 2>/dev/null
>   >   od -An -c ;}
>       67305985
>      h   e   l   l   o  \n
>
> It makes dd(1) call read(2) more times until the input has reached end
> of file, or it has returned all the neccesary bytes for that count.
>
> NOTE: iflag=fullblock is not available on OpenBSD and some other
> systems, and, as of issue 7, it is not specified by POSIX though it is
> planned for issue 8 <https://austingroupbugs.net/view.php?id=406>.
>
> So, in the end, you can use something like this.
>
>   length=$(dd iflag=fullblock bs=4 count=1 | od -An -td4)
>   length=$((length))      # trim leading spaces
>   IFS= read -rN"$length" json
>
> o/
>  emanuele6
>
>