Re: So I took a look at the gps_read section of Dire Wolf

[Jon Schlueter]

man page and example code here

https://gpsd.io/libgps.html

example code uses gpsd_waiting so you are not in a hot spin

    // Wait for data available.
    while (gps_waiting(&gps_data, 5000000)) {
        // will not block because we know data is available.
        if (-1 == gps_read(&gps_data, NULL, 0)) {
            printf("Read error.  Bye, bye\n");
            break;
        }
         if (TIME_SET == (TIME_SET & gps_data.set)) {
            // not 32 bit safe
            printf("%ld.%09ld ", gps_data.fix.time.tv_sec,
                   gps_data.fix.time.tv_nsec);
        } else {
            puts("n/a ");
        }


in the code on dire wolf you referenced.

https://github.com/wb2osz/direwolf/blob/master/src/dwgpsd.c#L275

it has

#define TIMEOUT 15
while(1) {
    if ( ! gps_waiting(&gpsdata, TIMEOUT * 1000000)) {
        DEBUG_PRINT(...);
        info.fix = DWFIX_ERROR;
        SLEEP_MS(5000);
        continue;
    }
    if (gps_read (&gpsdata, NULL, 0) == -1) {
        DEBUG_PRINT(...);
        break;
    }

    // process fix status into info
    // process lat/lon into info

    // set timestamp to time it was processed in the loop
    // Which is different than timestamp of the gps fix or time from the GPS
    info.timestamp = time(NULL);
}

So from the initial read through

The outer loop looks about right but I would question how it would behave
when you have timeouts on waiting for data either initially or any hic-ups
in communication.

Where I would be concerned is that it uses data without checking to see if
that data was actually included in the structure.  See the set logic from the
example code.  The gps_data.set is a bitwise set for what is valid to be read

https://gitlab.com/gpsd/gpsd/-/blob/master/include/gps.h#L2553

a few of the SETs that may be interesting to gate on would be
TIME_SET
ALTITUDE_SET
STATUS_SET
LATLON_SET
SPEED_SET
TRACK_SET

I would also be cautious of trusting logging from a thread in an
application especially one
that is reading data from a socket connection and trying to
write/message out data
at the same time.  Only semi-reliable method for logging is single log
file per thread
written directly with a timestamp included.

I would also recommend looking at how it is queuing up messages.  it
looks like in the
logs it was getting a large backlog of data and messages where it timed out.

Not sure how else to help you.  but I would first look at the
processing logic in dire wolf
to make sure it is actually picking out the right data from the
messages it is processing.

Likely it should be emitting one message per TPV message that gpsd
sends, and the log
looks like the GPS is sending 2 per second along with a couple of
other messages as well
which can be ignored for the most part.  libgpsd parsing libray is
setup to expose great
detail and error handling but you need to implement the checks to make
sure the message
you are processing is actually the one that has data you want,
otherwise it might be
processing old data or prior information reported as new.  To make
debug easier it would be
good to check for TIME_SET and also include the GPS fix timestamp as
well as the local
timestamp to catch any loop errors/issues when you are trying to debug things.

Jon




On Wed, Jan 5, 2022 at 5:44 PM Don Rolph <don.rolph@gmail.com> wrote:
>
> You stated:
>
> >> And we are receiving over 100 identical records which are not in the
> >> gpsd message debug log.
>
> >And yet you complain about a bug in gpsd...
>
> Certainly:  since the same client code works with gpsd 3.17 but fails with 
> all versions of the library on 3.23.2~dev, it seems that it is the change in 
> gpsd which is triggering the problem.  If it were the gpsd client library I 
> would expect different results with different versions of the library, but 
> the results are constant across versions.
>
> The MT3333 unit is an externality, but it also works smoothly with gpsd 3.17.
>
> Like wise Direwolf 1.7 works with gpsd 3.17 but not 322-400 onward.
>
> So the only piece which when changed causes a failure is gpsd.
>
> Now there are multiple possible scenarios:
>
> - gpsd 3.23.2~dev is generating a message format which can't be handled by 
> the gps_read code:  both come from the gpsd team
>
> - there is a bug in the gps_read code (but across all versions?)
>
> - there is a bug in the Dire Wolf implementations of the gps_read code (but 
> it works with gpsd 3.17)
>
> The variable which triggers the problem appears to be the gpsd code.
>
> One can look at the entire routine:
>
> -  https://github.com/wb2osz/direwolf/blob/master/src/dwgpsd.c
>
> but I don't see anywhere in the code where spurious messages can be generated 
> except in the gps_read routine provided by the gpsd team as part of the 
> client library.
>
> Again, I am not a c expert, so I might be confused.
>
> But I did, as requested, demonstrate the failure with gpsd 3.23.2~rev with a 
> client using the lib.so.29 library from the gpsd 3.23.2~r distribution.  The 
> results were consistent with previous testing./
>
> Again, thanks for everyone's help and support!
>
> On Wed, Jan 5, 2022 at 5:16 PM Gary E. Miller <gem@rellim.com> wrote:
>>
>> Yo Don!
>>
>> On Wed, 5 Jan 2022 16:52:33 -0500
>> Don Rolph <don.rolph@gmail.com> wrote:
>>
>> > If one looks at the dire3.log file one sees that the first TPV message
>> > after a 3D fix has been received by the rest of the code well over 100
>> > times.  The messages are all identical.  They do not show up in the
>> > log data from gpsd received by gpspipe -W.
>>
>> Asked and answered.  The Dire worlf code is bad, from a bad
>> stackexchange example.  loop++.
>>
>> > I did a quick scan of the code, and the following seems salient.  The
>> > reading of the gpsd data all occurs in the routine read_gpsd_thread
>>
>> Yes, since there are very few lines of gpsd code in Dire Wolf, it is
>> probably in there, somewhere.
>>
>> > A highly edited version of the source is below:
>>
>> Please don't do that.  The problem is where you refuse to look.
>>
>> > So it appears that the process reads a record and then prints a
>> > record.  To print a new it first has to read a new record.
>>
>> I suggestt you re-read the gps_read() man page.
>>
>> > And we are receiving over 100 identical records which are not in the
>> > gpsd message debug log.
>>
>> And yet you complain about a bug in gpsd...
>>
>> > This would seem to indicate that gps_read is generating the spurious
>> > messages since between the gps_read and the print there is no code to
>> > cause a loop or generate a record.
>>
>> I suggestt you re-read the gps_read() man page.
>>
>> > Now I am not a c specialist, so I may have made an error here. But if
>> > so, the nature of the error should be able to be identified by others.
>>
>> Yup.  And if you had not over simplified the loop, you might see it
>> too.
>>
>> > So do we know for sure that there is not a potential edge condition
>> > which can cause gps_read to generate multiple spurious identical
>> > spurious records?
>>
>> Yup, we know it, documented in fact.  How about you do some RTFM with
>> the examples in "man libgps".  Compare that known good code, with the
>> ode you helpfully removed and ignored.
>>
>> RGDS
>> GARY
>> ---------------------------------------------------------------------------
>> Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703
>>         gem@rellim.com  Tel:+1 541 382 8588
>>
>>             Veritas liberabit vos. -- Quid est veritas?
>>     "If you can't measure it, you can't improve it." - Lord Kelvin
>
>
>
> --
>
> 73,
> AB1PH
> Don Rolph