Re[2]: NTPsec reports excessive jitter from GPSD/KPPS SHM

[Nick Burkitt]

Hi Gary.

Yes, it's (part of) a kernel driver. It works. 

The timestamps produced by KPPS from my driver's events (/sys/class/pps/pp0) are what I would expect, showing a period of 1.000 s, a pulse width of 0.100 s and a jitter of < 0.000005 s. 

Whatever is going wrong is downstream of that - any excessive interrupt latency would be visible in the output of KPPS. CPU frequency scaling is disabled, and frequency is fixed at 866 MHz.

Xilinx Zynq SoC (FPGA + 2 ARM Cortex A9 cores)

Linux kernel 4.19.101 , HZ = 100

Ubuntu 18.04 LTS distro

GPSD 3.20

NTPsec 1.1.8+

U-blox NEO-M8T GNSS module , via USB

KPPS from u-blox timepulse output through custom kernel driver

gpsd is invoked with "/usr/local/sbin/gpsd -n /dev/pps0 /dev/core100/gnss /dev/ttyS0" where /dev/pps0 is /dev/core100/gnss symlinks to /dev/ttyACM1 (u-blox), and /dev/ttyS0 is unconnected.

ntpd is invoked with "/usr/sbin/ntpd -g -N -u ntp:ntp -p /var/run/ntpd.pid"

I've modified use-gpsd-shm (below), and unplugged the Ethernet cable. Now it's just the GPS and PPS. I'll let it run overnight and see if anything positive develops.

Here's what ntpmon reports at the moment:

     remote           refid      st t when poll reach   delay   offset   jitter
 SHM(0)          .GPS.            0 l   63   64  377   0.0000  76.6485   3.3823
*SHM(1)          .PPS.            0 l    5    8  377   0.0000 -25.5235  27.6401
 0.us.pool.ntp.o .STEP.          16 u    -    8    0   0.0000   0.0000   0.0019
 1.us.pool.ntp.o .STEP.          16 u    -    8    0   0.0000   0.0000   0.0019
 2.us.pool.ntp.o .STEP.          16 u    -    8    0   0.0000   0.0000   0.0019
 3.us.pool.ntp.o .STEP.          16 u    -    8    0   0.0000   0.0000   0.0019
 pool.ntp.org    .POOL.          16 p    -    8    0   0.0000   0.0000   0.0019
ntpd ntpsec-1.1.8+ 2020-03-04T20:55:58Z        Updated: 2020-03-06T08:02:58 (4)
 lstint avgint rstr r m v  count rport remote address
      0   1.36    0 . 6 2    623 44368 localhost

==============================================================

root@MPM-4006:/etc/ntp.d# cat use-gpsd-shm

# Simplest possible refclock configuration for sites with a GPS primary source.

#

# Uses the shared-memory driver, accepting fixes from a running gpsd

# instance watching one PPS-capable GPS. Accepts in-band GPS time (not

# very good, likely to have jitter in the 100s of milliseconds) on one

# unit, and PPS time (almost certainly good to 1 ms or less) on

# another. Prefers the latter.

# GPS Serial data reference (NTP0)

refclock shm unit 0 refid GPS noselect

# GPS PPS reference (NTP1)

#refclock shm unit 1 prefer refid PPS

refclock shm unit 1 minpoll 3 maxpoll 3 refid PPS prefer

# The following sets edit modes for GNU EMACS

# Local Variables:

# mode:conf

# End:

-Nick

------ Original Message ------

From: "Gary E. Miller" <address@hidden>

To: address@hidden

Sent: 3/5/2020 7:51:20 PM

Subject: Re: NTPsec reports excessive jitter from GPSD/KPPS SHM

Yo Nick!

 

On Fri, 06 Mar 2020 03:36:43 +0000

"Nick Burkitt" <address@hidden> wrote:

 

Yes, I modified my driver to tell KPPS about the new "clear"

interrupt. That's why KPPS now reports clear events. The code is

below, for what it's worth.

 

So by "driver" you mean kernel driver?

 

The machines (I have five of them) have all been getting NTP time

from Internet sources for the better part of a year, so their system

clocks' drifts should be well characterized by now.

 

Not really. You are using the usa pool and getting horrible

chimers. Plus, everytime you restart, ntpd has to pull the

system time, slowly, to correct. This is because your

SHM(1) poll is too long. Make it shorter so that it

locks first.

 

My guess is that your server is locking to some far away pool server

and getting pulled off, then will take a long time to correct.

 

NTPsec command line:

/usr/local/sbin/ntpd -g -N -u ntp:ntp

 

Ah, good. You do have the "-g".

 

While the ntpmon snapshot I included showed SHM(1) being used,

sometimes ntpd likes SHM(1), and sometimes it doesn't. Right now it's

giving SHM(1) the silent treatment. In a while, it might have a

change of heart.

 

Nothing really matters for the first hour. How long have you been

waiting?

 

 

remote refid st t when poll reach delay

offset jitter

SHM(0) .GPS. 0 l 29 64 377 0.0000

49.6561 2.9519

xSHM(1) .PPS. 0 l 28 64 377 0.0000

-27.0924 7.5853

 

Yep, there it goes:

 

remote refid st t when poll reach delay

offset jitter

SHM(0) .GPS. 0 l 50 64 377 0.0000

30.2184 16.2243

*SHM(1) .PPS. 0 l 48 64 377 0.0000

-40.9404 13.4211

 

Here's a few minutes worth.

 

Which is not useful. The long time constants in ntpd take hours

to settle. And not helpful without seeing the other chimers.

 

Here's a snapshot of ntpmon from a another copy of the device that's

been running for three days. This is using the assert-only KPPS

driver, of course.

 

remote refid st t when poll reach delay

offset jitter

SHM(0) .GPS. 0 l 59 64 377 0.0000

-2.3327 0.0396

xSHM(1) .PPS. 0 l 21 64 377 0.0000

-133.573 3.4322

 

Not helpful without seeing your other chimers.

 

Revised KPPS driver code:

 

And where does that code go? In gpsd? Or?? If it in the kernel then

I can't really say what may be right or wrong with it.

 

Making the unfounded assumption that your "driver code" is correct,

See what happens with the following changes:

 

set SHM(1) to poll faster:

 

refclock shm unit 1 minpoll 3 maxpoll 3 refid PPS prefer

 

And be sure it is at the top of the ntp.conf.

 

Then, pick some chimers that are local to you. The pool ones you

have been getting are not so good.

 

If that does not help, then something strange is going on with your

(unspecified) system. Maybe CPU sleeping, interrupts acting strangely,

CPU clock rate changing. To eliminate those make sure your system

is running the "performance" governor.

 

RGDS

GARY

---------------------------------------------------------------------------

Gary E. Miller Rellim 109 NW Wilmington Ave., Suite E, Bend, OR 97703

address@hidden Tel:+1 541 382 8588

 

Veritas liberabit vos. -- Quid est veritas?

"If you can't measure it, you can't improve it." - Lord Kelvin