TLDR; this bias is ~11–12 µs on average. Disabling the L1 low power state on the RP1 chip reduces it by ~5 µs.

However, it’s not obvious how you measure this. In this post, I want to explain two techniques: one uses PPS echo, an external time interval counter and eBPF; the other uses GPIO polling. I used AI agents (Claude Code Fable 5.1 and Codex GPT-6 Astra) to help me do this, but I wrote this post myself.

PPS echo with a tinyGTC

The kernel PPS subsystem has an echo feature. The idea is that the kernel can immediately after timestamping a pulse edge then generate a pulse on another pin. The purpose is to allow you to calibrate timestamping delay.

Recently Josh Blake developed a new overlay for the Raspberry Pi 5 that implements the echo feature. This is included in the latest Raspberry Pi Kernel, starting with version 1:6.12.70-1+rpt1 (February 2026). If you have the file /boot/firmware/overlays/pps-rp1.dtbo, then you have it available.

To use the overlay, comment out any existing dtoverlay=pps-gpio,gpiopin=18 line and add

dtoverlay=pps-rp1,pin=18,echo

To make use of the echo feature, you need a time interval counter, which is a device that can very accurately measure the time between two pulses. The tinyGTC works well for this. I already blogged about using it with PTP hardware clocks.

To make the measurements, you connect the OUT connector of the tinyGTC to the PPS input of the Pi, and you connect the echoed PPS output from the PI to input A on the tinyGTC. The tinyGTC input and output ports use SMA female connectors and the Raspberry Pi uses Dupont male pins. So you need two cables with SMA male on one end and Dupont female on the other. You can buy suitable cables on eBay or AliExpress (they are called SMA test cables) or, if you are handier than me, you can probably make them yourself. The cables typically have a red and a black Dupont connector. The red connector of the OUT cable should be plugged into physical pin 12. The red connector of the IN cable should be plugged into physical pin 11. The black connectors need to be plugged into GND pins: pin 6 and pin 9 are convenient for this.

You then configure OUT to have an aligned PPS at 1Hz and then measure the NCO against input A. You also need to run a program to enable PPS_ECHOASSERT through the kernel PPS API in order to get an echo. Leaving it overnight I get something like this.

This is showing a delay of ~14.7 µs over approximately 43,000 measurements.

Initially I controlled the tinyGTC through its touchscreen. One nice feature of the tinyGTC is that it has an SCPI interface for programmatic control. It turned out that the agent was able to figure out from the tinyGTC website how to drive it using Python through this interface, with very little input from me; I only had to explain how aligned PPS worked. This allows an agent to do all sorts of interesting experimentation autonomously.

But this does not mean that bias is 14.7 µs, because there is significant elapsed time between the kernel taking the timestamp and the pulse edge being emitted on the output pin.

It is convenient to give names to instants of time:

• IN - the time the physical edge enters the PPS input pin
• STAMP - the time the kernel records the timestamp
• WRITE - the time that the kernel issues the GPIO output instruction for the echo edge
• OUT - the time the physical edge exits the PPS output pin

We are interested in STAMP - IN. But we are measuring OUT - IN. We therefore need to estimate OUT - STAMP.

The time between STAMP and WRITE can be estimated by inserting probes into the kernel using bpftrace, which builds on Linux’s eBPF feature. This requires a bit of care because the probes themselves have overhead, but this can be compensated for. Using bpftrace produces an estimate of ~2 µs for WRITE - STAMP.

The remaining interval we need to estimate is OUT - WRITE, and I do not know any way to measure it. But based on the RP1 datasheet, I estimate it at no more than 1 µs.

So doing the math:

STAMP - IN = (OUT - IN) - (WRITE - STAMP) - (OUT - WRITE) = 14.7 - 2 - 1 = 11.7 µs

These measurements are done with the machine mostly idle and in its default configuration.

The obvious next questions are where is it spending 11.7 µs and can this be reduced.

It turns out that the biggest single contributor to the 11.7 µs is the RP1 wake latency. The RP1 has an L1 low power state, and if the machine is not busy, the RP1 enters this state between each pulse. You can prevent the RP1 entering L1 state by writing 0 to /sys/bus/pci/devices/0002:01:00.0/link/l1_aspm; writing 1 enables it. Disabling the L1 state reduces OUT - IN by ~5.5 µs and it turns out all the delay from L1 is before STAMP. So this reduces STAMP - IN to ~6.2 µs, and also reduces the jitter. You can also see this in the chart in the screenshot: the downward spikes are ~5.5 µs, presumably corresponding to when the RP1 was busy. I subsequently found that the RP1 datasheet documents a wake latency of approximately 5 µs.

Fixing the CPU frequency to 2.4GHz reduces it further by ~1 µs, bringing STAMP - IN down to ~5.2 µs.

GPIO polling

After doing the above, it occurred to me that there might be a simpler way. Over the last month or so I have implemented a new feature for SatPulse which allows it to read PPS timestamps over a serial line. I wanted this to work on macOS (via a USB-serial converter), but macOS does not have kernel timestamping, nor does the kernel provide a system call to notify an application of a change in the state of a modem control line. This meant I had to implement it by polling: repeatedly reading the modem control status to see when the status changed. There is a fancy adaptive/predictive algorithm to keep CPU-usage minimal. I implemented this on Linux. I also did some measurements on Linux comparing polling and kernel timestamping. I have Linux machines with PTM which allows the system clock to be accurate to tens of nanoseconds, so I can make these measurements very accurately. It turns out that the polling method has worse jitter than kernel timestamping but much less bias. This is a little unexpected at first, but obvious after you think about it: with kernel timestamping the timestamp is always after the pulse; with polling we measure before and after and take the midpoint, so any bias is limited to the time taken for the poll (this is assuming a modem status read actually performs a USB transaction).

So my thought was maybe we can do the same thing with GPIO. The idea is to poll the GPIO and at the same time read kernel timestamps, and then compare them.

There are a couple of problems. The portable way to read a GPIO state is via libgpiod, which uses an ioctl to access a GPIO character device (e.g. /dev/gpiochip0), but this cannot be done while the pps-rp1 or pps-gpio overlays are working. A portable solution is to request edge events from the GPIO character device, which are generated in an interrupt handler similarly to how the PPS GPIO driver generates timestamps. This requires unloading the PPS GPIO driver. On a Raspberry Pi, a superior approach is to read the GPIO input register through /dev/gpiomem0 (/dev/gpiomem on earlier Pis), which can work at the same time as the PPS GPIO driver.

The other problem is that reading the GPIO causes the RP1 to be busy, which, as we saw, makes a big difference to the measurement. Claude Fable suggested a clever way to solve this. The idea is to poll every other pulse. Let’s suppose we have kernel timestamps K0, K1, K2 and polled timestamps P0, P2. For K0 and K2, the RP1 will be busy, but for K1 it will be idle. We can infer a P1 as the midpoint between P0 and P2. We can then estimate the bias as K1 - P1.

With RP1 L1 enabled, this gives us a bias estimate of 12 µs against 11.7 µs for the echo/tinyGTC method, and with RP1 L1 disabled 7.3 µs against 6.2 µs for the echo/tinyGTC method. This suggests allowing 1 µs for the OUT - WRITE interval is a bit too much. On the other hand PCIe reads take about 1 µs, which implies an unquantified bias in the poll method of up to 1 µs.

Compensating for the bias

If you know the bias, you can compensate for it. In chrony, if the time pulse is 11 µs late, you would add an offset 11e-6 option to the refclock line.

Update 2026-09-09

I released ppsbias, a program implementing the GPIO polling idea.

I have made an effort to provide a good out-of-the-box experience. After installing SatPulse, no configuration is required. You can run satpulsewb with no arguments, in which case it will print a URL that you can copy and paste into your browser. The URL includes a unique, generated token to provide a modest level of security. (You can use an SSH tunnel if you need more security than this.) If you are running locally, it will also open the browser for you.

SatPulse Workbench has three main areas of functionality: monitoring, configuration and corrections. The interface is tab-based.

There are two tabs devoted to monitoring. The first provides high-level monitoring, providing the things you would expect like a map view and a sky view. The second provides a low-level packet view; instead of showing a linear stream of packets, it aggregates by message type and epoch, which I find makes it much easier to grasp what is going on. The packet tab also allows binary packets to be decoded into a human-readable JSON.

There are similarly two tabs devoted to configuration. The first exposes a high-level configuration model, which is independent of any vendor protocol. The user describes their intended configuration in GNSS terms, and the implementation figures out how to achieve that for the connected receiver.

The second configuration tab provides a low-level configuration model, where the user selects messages to send from a library of message files, organized by vendor. This can be used both for receivers for which high-level configuration has not been implemented, and for configuring things that the high-level configuration model does not cover.

The final tab allows you to pull corrections from either an Ntrip caster or a TCP server and feed them to the receiver. The correction messages, which may be in RTCM or SPARTN format, are parsed before being fed to the receiver, and the tab summarizes them by message type.

SatPulse’s GPS subsystem has support for a wide range of vendors: u-blox, Unicore, Zhongke Microelectronics (using CASIC protocol), Septentrio, Quectel (using PQTM and PAIR protocols), Allystar, NovAtel, ByNav, SinoGNSS (also known as ComNav), Techtotop (also called Taidou, using SDBP protocol). This is not to say that every receiver model from every one of these vendors is supported. u-blox has particularly broad support, covering everything from LEA-6T (released in 2009) through to the latest ZED-X20P. For other vendors, support is more selective. Support for high-level configuration requires additional implementation effort. High-level configuration has been implemented for the full range of u-blox receivers and for the Unicore UM980 series. There is also experimental support for Zhongke Microelectronics receivers; this needs to be enabled by using a --vendor zhongke (or --vendor casic) option. In addition, there are PRs to add support for Allystar, Septentrio and Quectel, which will be merged in due course. Initially, these will also require the use of the --vendor option. The high-level configuration tab is not available when high-level support for a receiver is not implemented, but the other tabs work as normal.

SatPulse Workbench is intended to be complementary to the existing daemon, satpulsed. The daemon is designed to run unattended for months at a time, providing GNSS time to a time server or acting as an RTK base station. The high-level monitoring tab of the workbench has some overlap with the daemon’s monitoring dashboard. I have not yet figured out to what extent I want to harmonize these: the dashboard is currently designed to work well on a mobile phone, whereas the workbench is not.

The functionality of the configuration tabs is also available in the existing satpulsetool gps command-line tool. The web-based interface is often more convenient for interactive use: you do not have to remember a complex command-line syntax and the monitoring tabs allow you to see the result of configuration. The command-line tool remains useful for scripting and also works very well with coding agents.

Surprisingly little code in satpulsewb is new. The backend is built on top of the same GPS Go subsystem that is used by satpulsed and satpulsetool. The UI is an evolution of that used in the experimental Wails-based desktop app, which I previously blogged about. There were two main pieces needed to enable satpulsewb. The first piece was refactoring the Wails backend into two parts: a shell and a session. The session is the bulk of the code and is independent of Wails; it includes the hard part, which is managing concurrency. The shell implements the Wails-specific parts of the backend on top of the session. satpulsewb is implemented as a second shell on top of the session. To test the session abstraction, I also vibe-coded a TUI using Bubble Tea on top of the same session package. It’s not something I want to spend time on now, but the code is available if anybody wants to play with it. The second piece is dealing with the possibility of multiple browser windows accessing the same session. The approach I have implemented is that only one browser window at a time can modify the GNSS receiver. If you open a new browser window, then the previous window is put into a read-only mode.

So why am I shipping the SatPulse Workbench rather than the Wails desktop app? There are two reasons. First, I think having a receiver attached to a headless SBC is an important use case, and the browser-based approach supports this much better than the desktop app. This is a feature that strongly distinguishes SatPulse Workbench from existing vendor evaluation software. Second, packaging and distribution are much easier: satpulsewb is, from a packaging and distribution point of view, the same kind of program as satpulsed and satpulsetool. In particular, it does not depend on any C libraries. This means I can have a single package including all three programs. On Linux, it maintains the important property that a single per-architecture binary will work reliably on all distributions. However, the refactoring means I can maintain the desktop GUI with minimal extra effort. When SatPulse is more mature, and hopefully has a bit more traction, I can jump through the packaging and distribution hoops that are needed for a smooth installation experience for a desktop app (Microsoft Store on Windows, App Store on macOS, per-distro packages on Linux).

Radxa ZERO 3E

I have been experimenting with a Radxa ZERO 3E. Key hardware features

• Pi Zero form factor
• 1.6GHz quad core Cortex-A55 (roughly Pi 3B+ performance)
• 1 to 8GB RAM
• Gigabit Ethernet port
• 40-pin GPIO
• optional PoE HAT
• USB 2.0 type C power port
• USB 3.0 type C port
• micro HDMI video port

I was able to buy a 1GB model a month ago on AliExpress for $24 excluding shipping.

This is quite a compelling hardware proposition for something to connect a GNSS module to compared to what Raspberry Pi has on offer. The strength of Raspberry Pi is in the software ecosystem. But this has Armbian standard support, which as far as I can tell is what you want if you are not using Raspberry Pi.

Waveshare GNSS HATs

I was particularly interested in getting something in the Pi Zero form factor in order to combine it with a HAT because Waveshare make a number of GNSS modules in a Pi Zero form factor designed to plug into the 40-pin GPIO connector on a Raspberry Pi. They have versions using:

• Quectel LC29H
• Quectel L76K
• u-blox NEO-M8T
• u-blox ZED-F9P
• u-blox MAX-M8Q

I tried it with the LC29H(DA) which is RTK capable. Physically they connect together with no problem. The combination is tiny and quite neat.

OS setup

Setting up the OS isn’t hard but is not completely obvious.

I first downloaded an Armbian image. I chose the Minimal, Debian 13 image with the non-vendor kernel 6.18.9. I used Balena Etcher on macOS to write the image.

Armbian setup needs a monitor and keyboard attached.

There are two things that need setting up: the serial connection and the PPS connection.

Serial

The UART on the HAT is wired to pins 8 and 10 on the GPIO header. These are UART2 on the ZERO 3 which shows up as /dev/ttyS2. By default, this is enabled but configured as a serial console. So all that is necessary is to stop it being used as a serial console.

To do this, first edit /boot/armbianEnv.txt to change the line with console=both to console=display.

Then disable the getty service:

sudo systemctl disable --now serial-getty@ttyS2.service
sudo systemctl mask serial-getty@ttyS2.service

PPS

Most Waveshare HATs wire the GNSS PPS output to pin 12. (The L76K does not wire PPS up at all, so it won’t work well for timing applications.) To make PPS work, we therefore have to create a device overlay that wires up pin 12 on the Radxa as a PPS input.

Create a file /tmp/rk3566-zero3e-pps-gpio12.dts:

/dts-v1/;
/plugin/;

/ {
	compatible = "radxa,zero-3e", "rockchip,rk3566";

	fragment@0 {
		target-path = "/";
		__overlay__ {
			pps_gpio12: pps-gpio12 {
				compatible = "pps-gpio";
				pinctrl-names = "default";
				pinctrl-0 = <&pps_gpio12_pin>;

				/*
				 * Physical pin 12 = GPIO3_A3.
				 * Rockchip GPIO specifier is <controller line flags>,
				 * so this is gpio3 line 3, active high.
				 */
				gpios = <&gpio3 3 0>;

				/*
				 * For normal GNSS 1PPS active-high pulses, leave this out.
				 * For falling-edge PPS timestamping, uncomment:
				 *
				 * assert-falling-edge;
				 */

				status = "okay";
			};
		};
	};

	fragment@1 {
		target = <&pinctrl>;
		__overlay__ {
			pps-gpio12 {
				pps_gpio12_pin: pps-gpio12-pin {
					/*
					 * <bank pin mux config>
					 * bank 3, PA3, mux 0 = GPIO, no pull.
					 */
					rockchip,pins = <3 3 0 &pcfg_pull_none>;
				};
			};
		};
	};
};

(ChatGPT was able to create something that worked first time, slightly to my surprise.)

To install this overlay:

sudo apt install -y device-tree-compiler
sudo armbian-add-overlay /tmp/rk3566-zero3e-pps-gpio12.dts

Then reboot

sudo reboot

This creates a /dev/pps0 device.

SatPulse

To use the LC29H with SatPulse, this is enough to get started:

[serial]
device = "/dev/ttyS2"
speed = 115200

[gps]
vendor = "quectel"

[[http]]
listen = ":2000"

Highlights of this release:

• there is a new PHC synchronization architecture, including a GNSS/PHC simulator (which has a CLI)
• it supports seven vendor protocols in addition to UBX
• it is evolving beyond the original PTP use case to become an alternative to gpsd for some server-side use cases, in particular it can now be used as a source of time for an NTP server (either chrony or ntpd-rs) without needing a PHC
• it has had extensive testing on a wide range of hardware

I created a discussion thread where you can ask questions, share experiences and suggest new features.

Generating samples includes the following tasks:

• pulse edge filtering: some Intel NICs generate timestamps for both edges of a pulse; in this case, we have to identify which edges are leading edges
• sample completion: the timestamp for a leading edge marks the top of a second; completing the sample means determining which second that is
• sawtooth correction: a GPS receiver can only generate a pulse on an edge of its internal clock, but there can be an offset between the edge of its internal clock and the top of the second; a timing-grade GPS receiver will output a message for each pulse giving the size of this offset; the sample then needs to be adjusted for this offset

0.1 PHC synchronization architecture

My initial implementation of PHC synchronization followed the approach of the ts2phc program, included in LinuxPTP. The approach consists of a 2-stage pipeline. The initial stage generates samples by combining timestamps from the PHC with time-of-day information from GPS messages. The samples are fed into a second stage, which uses a PI servo to adjust the phase and frequency of the PHC.

This approach evolved to add a monitoring stage to the pipeline between the sample-generation stage and the servo. This monitoring stage had a variety of responsibilities. It determined whether the PHC was in sync with GNSS time, and used this to dynamically update the PTP grandmaster’s clock quality. It also performed outlier detection using a MAD algorithm.

I found two major problems with this pipeline approach. The first problem was that each stage in the pipeline ended up maintaining its own state, but these states were not coordinated.

• the sample-generation stage had an initialization state for analyzing the intervals between edges; this was used with Intel NICs that timestamp both edges of a pulse to ensure that trailing edges were ignored
• the monitoring stage maintained state of whether the PHC was synchronized to GNSS time
• the servo stage maintained state related to deciding whether to step the PHC

This became particularly problematic for the sample-generation stage. It’s important for PHC synchronization to be as reliable as possible. I found that GPS messages were not completely reliable for determining time-of-day information. Perhaps the most common problem is that the GPS emits too many messages for the available serial bandwidth, which causes messages to be delayed or dropped. When in a synchronized state, a more reliable way is to use the PHC, since the PHC will be accurate to within a microsecond or so, but this doesn’t work at all when the PHC is not synchronized. However, the sample-generation stage doesn’t have access to the synchronization state of downstream stages. The sample-generation stage became increasingly complex over time, using ad hoc heuristics to decide whether to prefer information from the PHC or from messages.

This ties into the second main problem. I had very limited ability to test the pipeline as a whole. My main approach was to save the inputs and outputs of the sample-generation stage; I could then replay the inputs to make sure they produced the same outputs. But if the sample-generation stage was affected by the monitoring stage, this would no longer be possible.

The first problem meant that a rewrite was needed: I hadn’t decomposed the problem in the best way. And if I was going to do a rewrite, then I should solve the second problem once and for all, and that meant I needed a simulator.

The most significant open source project in the timing simulation space that I know of is Miroslav Lichvar’s clknetsim. In fact, this project is what made me realise that serious testing needed a simulator. However, clknetsim would not work for SatPulse, because it relies on being able to redirect system calls using LD_PRELOAD, but SatPulse is written in Go and on Linux Go usually produces statically linked executables; system calls are raw kernel syscalls, which do not go through a dynamic library. So that meant I needed to develop my own simulator. Also clknetsim does not handle the GPS side of things.

Simulator

The goal of the simulator is not to be perfect, but to be realistic enough to enable closed-loop testing of synchronization algorithms.

The simulator is initialized with a configuration and performs a simulation for some period of time. The simulator is driven by the progress of simulated time, which represents true time. As simulated time progresses, the simulator emits timestamps and GPS messages. It also implements a PHC interface that can be used to adjust the phase and frequency of the simulated PHC. There is a crucial feedback loop: each timestamp is measured with respect to the PHC and has to take account of any phase and frequency adjustments made through the PHC interface. Another complicating factor is that GPS messages can include sawtooth corrections for the PPS signal and these corrections have to match the timestamps being generated.

When run under a simulator, the code under test produces its normal output, but the simulator can observe the offsets between the simulated true time and the simulated PHC. It can produce a log of these offsets and also generate statistics such as the maximum offset and the Allan deviation. These statistics could only be produced in real-world testing by using a reference clock that tracks UTC with much greater accuracy than a GPS PPS signal. This would require expensive hardware such as a caesium clock or better still, a hydrogen maser; a rubidium clock would not be sufficient.

The configuration includes error models for the PHC oscillator and the GPS PPS signal. Each error model consists of a number of components that describe different sources of error, which are combined additively. For example, the PHC error model has components for white, flicker and random walk FM noise. The GPS PPS error model includes a component for sawtooth error, which is used in generating both the timestamp for the PPS edge and the sawtooth correction in the corresponding GPS message.

In Go, the error models are represented by func(t float64) float64: the return value gives the instantaneous error at simulated time t. The return value for the PHC error model is a frequency error, whereas the return value for the GPS PPS error model is a phase error. This reflects the underlying physical reality that an oscillator is a continuous process whose state at any instant is a rate, whereas a PPS signal is a discrete process whose state for each pulse is a position in time.

The error models can be derived from physical measurements made of the PHC oscillator and the GPS PPS signal. (A PHC oscillator can be measured by making the PHC output a PPS signal while free-running.) In a future post, I will go into more detail about how I made measurements and used them to derive error models.

0.2 PHC synchronization architecture

The approach in 0.2 is modal. There are three modes: reset, converging and tracking. At a high level, these modes work as follows. Reset is the initial mode: its job is to generate a single, reliable sample; it does this by collecting a batch of timestamps and GPS messages. After generating the sample, reset mode will perform a step of the PHC so as to guarantee that the PHC is close to the GNSS time. At that point, it transitions to converging mode. Its job is to aggressively adjust the frequency so as to bring the PHC into as precise as possible alignment with GNSS time. When the offsets between the PHC and timestamps are no longer decreasing, it transitions to tracking mode. Its job is to continually tweak the PHC frequency so as to keep the offsets as small as possible. It remains in tracking mode so long as the offsets indicate that the PHC is still synchronized to GNSS time. If synchronization is lost, it transitions to reset mode.

Each mode is associated with a clock quality notified to the PTP grandmaster: tracking mode is associated with a clock quality representing a synchronized state; reset and converging mode are associated with a clock quality representing an unsynchronized state.

The following table summarizes the operation of the modes.

The key point to notice in the table is that each mode performs its tasks very differently. I want to focus particularly on sample completion, which is the most fundamental part of sample generation. In production, SatPulse should be spending 99.9999% of its time in tracking mode, and in tracking mode sample completion is utterly trivial: the PHC clock will be accurate to a microsecond, so you can just round the timestamp to know which second the timestamp is for.

In contrast, sample completion in reset mode is quite elaborate. If reset mode makes the wrong choice of second, then that will persist throughout the operation of the daemon, so the implementation takes a lot of care to ensure that it is right. It collects time messages and timestamps for several seconds, and then performs multiple consistency and quality checks.

Reset mode has to correlate time messages with timestamps. For time messages, we record the monotonic time at which we read the first character of the message. But these monotonic times cannot be compared directly with the timestamps, which are in the PHC time domain. The natural way to handle this is to record the monotonic time immediately after the timestamp is read. But the Raspberry Pi CM4/CM5 ethernet PHY driver has a quirk which makes this insufficient by itself: the driver can deliver the timestamp to user space up to 0.25s after the pulse occurred. To handle this, we also record the PHC time immediately after reading the timestamp, and then adjust the post-read monotonic time by the difference between the post-read PHC time and the timestamp. We also have to account for the possibility that the PHC is fast or slow. The average interval in PHC time between successive pulses tells us how much PHC time corresponds to one second, and we use this to scale the PHC difference before using it to adjust the monotonic time.

The decomposition of responsibilities is as follows. The main implementation package is phcsync. It has a controller, which is responsible for orchestrating the modes. For each mode, there is a sample-generator and a sample-processor. The sample-generator is responsible for pulse edge filtering, sample completion and sawtooth correction; in tracking mode it uses the pulse width discovered in reset mode. The sample-processor is responsible for outlier detection, and for determining when and how to adjust the PHC and change mode; these PHC adjustments and mode changes are then performed by the controller. The sample-processors for converging and tracking mode share a PI servo implementation; in tracking mode, the servo is initialized using the PHC frequency error discovered in reset mode. The controller feeds samples from the sample-generator to the sample-processor. The controller also synthesizes missing samples and feeds them to the sample-processor. The controller notifies the PTP grandmaster for mode changes that imply a change in clock quality. Thus, as in 0.1, there is a 3-stage pipeline: sample-generator then sample-processor then controller. But the pipeline is driven by the controller, and the sample-generator and sample-processor are mode-specific.

The other implementation package is timemsg, which serves as a bridge between phcsync and the GPS subsystem. timemsg maintains a buffer of recent time-related messages from the GPS receiver. phcsync defines an interface which captures what it needs to know about time messages, and timemsg implements this. Reset mode obviously depends on this interface, but tracking mode also uses it for sawtooth corrections. This separation between phcsync and timemsg was also designed to enable timemsg to be reused for a new feature in 0.2: samples can be provided to an NTP server based on serial timing, without needing a PHC.

The benefits in terms of user-visible features of the 0.2 implementation are relatively modest. Reset mode can disambiguate leading and trailing edges even with a 50% duty cycle. PHC synchronization parameters are now fully configurable using a new [sync] section of the config file. The simulator has a CLI that can be used to tune these parameters, in particular the Kp/Ki constants for the tracking servo.

But the major wins from the new architecture are in terms of improving reliability and providing a foundation for future development. The most important missing feature at the moment is holdover: the modal architecture can accommodate this in a natural way. Sample generation has a clean and principled architecture that solves the problems this had in 0.1; in tracking mode, it does not depend on time messages and so should be more reliable. The most important aspect of the architecture is, I believe, the simulator. This solves the testability problem we had in 0.1 and improves the reliability of 0.2. But it is also crucial for future development: without a simulator, it would be very difficult to develop a reliable implementation of complex features like holdover.

More details:

abondance

ASUS S500SD desktop (Intel B660 chipset) with an Intel i5-12400 and 64 GB RAM. It has two Intel I225-T1 PCIe cards: one is connected to an internal ArduSimple simpleRTK2B M.2, which has a u-blox ZED-F9P, and the other is connected over RS232 to a BG7TBL CM55 GPSDO, which has a LEA-M8T. This supports cross-timestamping with PTM.

brie

MSI Z690I motherboard with an Intel i9-12900K and 32 GB RAM. It has an Intel E810-XXVDA4T PCIe card. GPS is a u-blox EVK-F10T connected by USB.

gubbeen

Raspberry Pi 5 with 16 GB RAM in a Geekworm X1010 expansion board with PCIe slot, which contains an Intel I210-T1. GPS is a u-blox M10050-KB (u-blox M10 standard-precision) connected to UART0.

maasdam

Minisforum MS-01 with an Intel i5-12600H and 32 GB RAM. It has an I226-T1 PCIe card. GPS is a u-blox ZED-F9T-20B in a gnss.store USB dongle. This supports cross-timestamping with PTM.

morbier

Raspberry Pi CM4 with 8 GB RAM and 32 GB eMMC, on the official IO board. GPS is a u-blox LEA-M8F on a Timebeat sandwich board.

pecorino

Raspberry Pi CM4 with 8 GB RAM and 32 GB eMMC, on the official IO board. GPS is a u-blox LEA-6T connected to UART3.

roquefort

Raspberry Pi CM4 with 4 GB RAM and 8 GB eMMC, on a Waveshare PoE board. GPS is a u-blox EVK-X20P connected by USB.

serpa

Raspberry Pi CM5 with 2 GB RAM and NVMe SSD, on a Geekworm X1500 carrier board. GPS is a Unicore UM980 connected to UART0.

taleggio

Raspberry Pi 5 with 4 GB RAM with a Timebeat TimeHAT, which uses an Intel I226-LM. GPS is a u-blox NEO-M9N in the TimeHAT’s M.2 slot.

valencay

Raspberry Pi CM5 with 2 GB RAM and 16 GB eMMC, on the official IO board. GPS is a u-blox LEA-M8T connected to UART0.

valtellina

Raspberry Pi CM5 Lite with 8 GB RAM and NVMe SSD, on the official IO board. GPS is a u-blox NEO-F10N connected to UART0.

wensleydale

Raspberry Pi CM4 with 8 GB RAM and 8 GB eMMC, on the official IO board. GPS is a Unicore UM960 connected to UART0.

The testing uses the Ansible playbooks in the systest directory. It runs both satpulsed and chrony on the target machines, with satpulsed supplying samples for chrony. The main checks are that

• satpulsed logs show that synchronization has been established and not lost; and
• chrony has selected the satpulsed refclock; chrony is also configured to use highly-accurate NTP servers on the LAN, and won’t select the satpulsed refclock if it diverges too much from those NTP servers.

It also uses the clocklog.py script to produce some statistics from the clock.log about the quality of synchronization.

I also have a Grafana dashboard which shows metrics from each test machine.

This is a native app running on macOS. It uses the same GPS library that satpulsed and satpulsetool use. The code is on the desktop-gui branch of the repo.

It uses a tabbed layout, with tabs for:

• monitoring - this is conceptually quite similar to what the Web monitoring app does, but has a richer set of components
• log-level monitoring - this is similar to what you get with the packet log in satpulsetool or with the --packet-log option in satpulsed; but instead of giving a stream of packets, it groups packets by protocol and message ID; it also allows decoding of packets similar to what is provided by satpulsetool decode
• high-level configuration - this allows you to configure the GPS receiver in a high-level, device-independent way, without needing any knowledge of the receiver protocol, similar to satpulsetool gps
• low-level configuration - this uses message files to provide a lower-level device-dependent way, similar to -m and -t options of satpulsetool gps
• corrections - this allows you to pull RTCM corrections from an NTRIP caster or TCP server and feed them to the receiver, with a live view of the corrections stream; I plan to provide this functionality in a future release of satpulsed using a [stream.pull] section in the configuration file

This is built using Wails, which supports Linux and Windows, as well as macOS. But my initial testing has focused on macOS, partly because that’s what I use day-to-day for desktop work, and partly because there’s a gap on macOS, since GNSS vendors provide their proprietary apps only for Windows.

This is still very much in the preview phase: some aspects of the UI are rough, particularly the monitoring tab. With Wails, the UI is written using standard Web technologies (HTML, CSS and JavaScript), which run in the platform’s native webview component. The desktop GUI and the satpulsed Web monitoring app are both written using Preact and Tailwind CSS; this should make it possible in the future for the monitoring tab to share Preact components with satpulsed.

Why bother with writing a GUI at all? I developed the device-independent configuration approach to give the best possible user experience for satpulsed. But it turned out to be a lot more engineering effort than I anticipated. And to be honest, from the point of view of satpulsed, it doesn’t really earn its keep: the added benefit to users is too small to justify the effort that went into it. I added satpulsetool to help unlock the value of device-independent configuration to a wider audience. But satpulsetool can’t fully do this. Most users only have one GNSS receiver, so the value to users of this configuration model isn’t so much that it’s device-independent, but that it avoids requiring the user to know anything about the receiver’s native protocol; and the kind of user that benefits most from this is a non-technical user, who will want a GUI not a CLI.

At the moment, support for GPS receivers in Linux is almost universally based on GPSd. In this post I want to explain some key design choices I made for SatPulse that are different from those made by GPSd. My goal is to help people understand when it might be worth considering SatPulse as an alternative to GPSd. I want to emphasize that SatPulse is not attempting to be a replacement for GPSd. GPSd does what it sets out to do very well, as evidenced by its popularity.

TL;DR: Consider SatPulse for server-side use of a GPS receiver, or when GPS receiver configuration is needed.

Let me start by giving a brief overview of how GPSd works. For a fuller description, see the GPSd chapter from the Architecture of Open Source Applications book. GPSd is both a daemon, written in C, and a suite of related tools. It is centered around a device-independent data model of the information provided by periodic messages emitted by GPS receivers. A single instance of the daemon acts as a multiplexer. Streams of messages are read from multiple sources such as serial devices, converted to the device-independent data model and provided to multiple clients. GPSd’s primary API is a service API: a client runs as a separate process and interacts with GPSd over a TCP socket using a JSON protocol; it also provides a client library API that wraps the service API. In the GPSd architecture, the daemon does not perform application-level work on the data; its role is restricted to multiplexing and conversion into the device-independent data model. GPSd has a zero-configuration philosophy: when a user plugs in a GPS receiver, a GPSd client should work without requiring the user to perform any configuration. GPSd provides only a very limited device-independent abstraction for GPS receiver configuration; most importantly it has the concept of enabling a binary mode, which configures the receiver to output messages using its vendor-specific binary protocol instead of NMEA. Instead it provides a separate Python program, ubxtool, for configuring u-blox receivers using the UBX protocol.

Like GPSd, SatPulse is also both a daemon and a suite of related tools, and it is also centered around a similar device-independent data model. SatPulse is written in Go. Its codebase is comparable in size to GPSd’s. SatPulse compiles into two executables: satpulsed, which is a daemon, and satpulsetool, which is a command line tool. All the application-level functionality that SatPulse provides is included in satpulsed or satpulsetool depending on whether that functionality needs to work as a daemon or not. The daemon is configured using a file in TOML syntax. One instance of the daemon runs for each device to which a receiver is attached. This allows each instance to have its own configuration file, and for systemd to manage the daemon lifecycle. SatPulse does not attempt to discover GPS receivers: it opens a serial device only when explicitly configured to do so. The configuration file has a modular structure, with separate TOML tables to configure each aspect of application-level functionality. The second executable, satpulsetool, is a suite of command-line tools. It uses a subcommand syntax, so for example satpulsetool gps runs the gps tool. These separate tools are bundled into a single executable because the Go language has a runtime that makes executables quite large. The GPS subsystem of SatPulse is structured as a reusable library of Go packages. Both satpulsed and satpulsetool use this library for their GPS-related functionality: satpulsetool does not need satpulsed to be running.

The most significant GPS functionality in SatPulse that goes beyond what GPSd provides is support for GPS receiver configuration. For basic GPS usage, the factory default configuration is often sufficient. But modern GPS receivers even at modest price points have started to offer more advanced features, such as RTK, PPP-HAS or OSNMA, which typically require some configuration to be used. SatPulse provides a device-independent abstraction for GPS receiver configuration. This allows you to choose which specific aspects of the configuration should be changed. So, for example, you can specify that a time pulse should be enabled with a specific pulse width, that the time pulse should be enabled only when the receiver has a lock, and that it should be aligned to the system time of a particular GNSS; or you can specify that the receiver should operate in time mode with specific fixed ECEF coordinates. The changes to be made are given to the configuration engine which figures out how to apply them to a specific receiver. This often involves reading the existing configuration of the receiver, so that only the specified aspects of the configuration are changed. An important part of configuring a GPS receiver is enabling the right set of periodic messages. Each receiver divides up information in different ways. So instead of specifying particular named device-dependent messages (e.g. UBX-NAV-PVT), you specify the needed information in data-model terms (e.g. the time in UTC and position in geodetic coordinates). The engine then enables the best set of messages that provide this information. The configuration engine is part of the library and is used both by satpulsed and satpulsetool. Having satpulsed perform receiver configuration works well because it can infer some aspects of receiver configuration from the application-level configuration. For example, if the configuration file specifies a PHC to be disciplined, then the daemon will ensure a 1PPS time pulse is enabled. But configuration changes that affect the receiver’s non-volatile memory or interrupt receiver operation (such as changing the enabled GNSS constellations) are only done when specifically requested by the user with satpulsetool. Implementing device-independent configuration is significantly more complex than implementing the device-independent data model for periodic messages. In the Go package that implements the device-independent abstractions for the u-blox UBX protocol, about 70% of the code is devoted to configuration. SatPulse also provides an alternative lower-level approach to configuration, which is less challenging to implement: see this blog post for more detail.

The device-independent data model provided by the GPS subsystem can be serialized as JSON. The daemon uses this for its Web dashboard feature. It includes an HTTP server with an endpoint that exposes this data model as JSON-encoded server-sent events (SSE). Another endpoint serves an HTML page which uses JavaScript to connect to the SSE endpoint and display a dashboard. The daemon does not yet expose this data model over a network socket for consumption by third-party applications. This will be straightforward to do, but I want to stabilize the data model first.

However, SatPulse emphasizes a different approach to allowing multiple independent applications to share access to a single receiver, based on providing network access to packet streams. The daemon can be configured to make TCP ports or Unix domain sockets proxy the packets emitted by the receiver, optionally filtering packets by protocol, and optionally also allowing writing to the receiver, with a configurable locking strategy to prevent conflicts between writers. This provides functionality similar to ser2net, but is protocol-aware. Streams of native-protocol packets are already a well-defined wire-format. Exposing this directly on a network endpoint allows receiver sharing without the need to define a new daemon-dependent wire-format. Many applications already exist that can work with such packet streams. Here are three examples.

1. Every vendor that I know of provides an application for configuring their receivers, typically Windows-only. Many of these applications, notably u-center, allow the receiver to be accessed over a TCP socket. These can be used with SatPulse by taking advantage of the read-write feature.
2. IANA has registered the service name nmea-0183 and port 10110 for NMEA over TCP or UDP. GeoClue is a D-Bus service that provides location information. It includes an NMEA backend that uses DNS-based service discovery (RFC 6763 as implemented by Avahi) to discover nmea-0183 services on the network. SatPulse can be configured to expose the NMEA service on port 10110, Avahi can be configured to advertise it and then the GeoClue NMEA backend can discover it and expose it to desktop apps over D-Bus.
3. RTK works by providing the rover’s receiver with a stream of RTCM packets emitted by a base station’s receiver. In a production environment, NTRIP is typically used to move packets from a base to a rover. An RTK base station uses a NTRIP server to provide RTCM packets to an NTRIP caster; an RTK rover uses an NTRIP client to get RTCM packets from the caster. NTRIP support is planned for a future release of SatPulse. But one popular open-source caster, the BKG NtripCaster can pull RTCM data from a TCP port without using NTRIP. This allows SatPulse to be used today as the GPS component of a combined RTK and time server.

We can summarize the key design choices for SatPulse that are different from GPSd as follows:

1. it is written in Go
2. the primary GPS API is a library API
3. the daemon does application-level work
4. the daemon has a configuration file
5. the daemon has a separately configured instance per receiver
6. it provides a device-independent model for GPS configuration
7. it emphasizes packet streams as a foundational layer

So why did I make these choices? They were not independent. The initial PTP use case requires a daemon to do substantial application-level work: it uses timestamp events from the Linux PHC subsystem in combination with messages from the GPS receiver to discipline the PHC and send metadata updates to the PTP daemon. I didn’t want this use case to require two independent daemons. Once the daemon is doing application-level work, then there needs to be a way to configure it. Having one daemon instance per serial device makes things straightforward: each instance can have its own configuration file and systemd can ensure that the daemon is not started until the serial device is ready.

Even with the initial PTP use case, the daemon has significant internal concurrency: reading from the serial device, reading timestamp events, and updating the PTP daemon are naturally concurrent with the main PHC-disciplining loop. Other features like packet proxying or HTTP monitoring involve additional concurrency. I would not attempt implementing a daemon with this level of concurrency except in a modern language like Go, which is memory safe and has language support for concurrency. The implementation of satpulsetool also uses concurrency, although to a lesser extent than satpulsed. Go makes it relatively straightforward to have a library that supports concurrency in a flexible way.

I hope this post has made it clear that GPSd and SatPulse are very different programs, which have made different design choices for good reasons. SatPulse would not be a suitable replacement for many uses of GPSd. But I think there are several use-cases, particularly on the server side, where SatPulse’s more integrated approach and support for GPS configuration offers significant advantages.

In this post, I will explain how to do this, using two different NTP servers: chrony and ntpd-rs.

Hardware

Before we get into the details of configuration, let me say something about GPS hardware. When using a Raspberry Pi, the PPS signal is connected to a GPIO pin on the Pi. This means that the time of the PPS edge is measured in software by the kernel, rather than in hardware. The accuracy you will get is in the microsecond range. Basic GPS receivers can achieve an accuracy of 50ns in normal conditions. So my main hardware advice is that in this context a fancy, expensive GPS receiver will not deliver any measurable benefit.

GPS receivers are available in a Pi HAT form factor, which plugs into the 40-pin GPIO header. These HATs typically wire the GPS PPS signal to pin 12 (GPIO 18). I do not recommend these when using PHC hardware, because the GPS PPS signal needs to be wired to the SYNC_OUT pin on a different header. But when using a normal Raspberry Pi, they are very convenient, although you typically pay a premium for this convenience.

Setup

The following steps can be done exactly as described in SatPulse Setup Guide.

1. Install Raspberry Pi OS
2. Configure the UART
3. Install SatPulse but choose the 0.2 pre-release
4. Verify the GPS serial connection; the device will be /dev/ttyAMA0

Configuration of kernel PPS is different. First, configure pin 12 (GPIO 18) as a PPS pin, by adding the following at the bottom of /boot/firmware/config.txt

dtoverlay=pps-gpio,gpiopin=18

Reboot after this. To verify the PPS signal is working, install pps-tools using

sudo apt install pps-tools

Then do:

sudo ppstest /dev/pps0

It should show PPS events once per second:

trying PPS source "/dev/pps0"
found PPS source "/dev/pps0"
ok, found 1 source(s), now start fetching data...
source 0 - assert 1775392263.000000336, sequence: 170178 - clear  0.000000000, sequence: 0
source 0 - assert 1775392264.000000215, sequence: 170179 - clear  0.000000000, sequence: 0
source 0 - assert 1775392264.999998780, sequence: 170180 - clear  0.000000000, sequence: 0
source 0 - assert 1775392265.999999085, sequence: 170181 - clear  0.000000000, sequence: 0

Exit with Ctrl-C.

Now you need to edit /etc/satpulse.toml. Most important point is to remove the [phc] section. All you need is this:

[serial]
device = "/dev/ttyAMA0"
# fix to match your serial device speed
speed = 9600

[gps]
config = true
vendor = "u-blox"

# Enable HTTP monitoring on port 2000
[[http]]
listen = ":2000"

[ntp]
# Use this for chrony
sock.path = "/var/run/chrony.satpulse.sock"
# Use this for ntpd-rs
# sock.path = "/run/ntpd-rs/satpulse.ttyAMA0.sock"

Uncomment the sock.path line for whichever of chrony or ntpd-rs you use.

You can now start SatPulse using

sudo systemctl start satpulse@ttyAMA0

You can point a browser at port 2000 and you should get a page showing information from the GPS receiver.

Chrony

To configure chrony, first install with

sudo apt install chrony

Then create a file /etc/chrony/conf.d/satpulse.conf with the following:

refclock PPS /dev/pps0 poll 2 lock GPS refid PPS
refclock SOCK /var/run/chrony.satpulse.sock offset 0.1 delay 0.2 refid GPS noselect

The refclock PPS line makes chrony use the kernel PPS API to read PPS samples from /dev/pps0. These are accurate but lack time-of-day information. The refclock SOCK line makes chrony read samples generated by satpulsed. These are inaccurate but include time-of-day. The offset 0.1 corrects for serial messages coming 0.1 second after the top of the second.

Chrony can use the samples from satpulsed to complete the PPS samples. Chrony can also use samples from other NTP servers to complete the samples, so if you want to check that this is really working, you should at least temporarily comment out the pool line from /etc/chrony/chrony.conf.

ntpd-rs

Ubuntu has announced plans to adopt ntpd-rs as its default NTP server, replacing chrony, primarily because of memory safety. Since SatPulse is written in Go, the combination of ntpd-rs and SatPulse provides a fully memory-safe timing stack. Like SatPulse, ntpd-rs uses TOML for its configuration files and supports Prometheus, so the combination makes for a pleasantly harmonious configuration and observability experience.

To install, download a deb file from the Releases page. You need at least version 1.7.2. (The version in Raspberry Pi OS will not work.)

Next, you will need to remove your existing NTP daemon (e.g. systemd-timesyncd or chrony):

sudo apt remove systemd-timesyncd chrony

Then install ntpd-rs

sudo dpkg -i ./ntpd-rs_1.7.2-1_arm64.deb

Now you need to set things up so that ntpd-rs can access /dev/pps0. Create a file /etc/udev/rules.d/99-ntpd-rs-pps.rules containing the line

KERNEL=="pps0", GROUP="ntpd-rs", MODE="0640"

This ensures that when /dev/pps0 is created at boot time it will have a group and permissions that enables ntpd-rs to access it. To make this take effect without booting, do

sudo udevadm control --reload
sudo udevadm trigger /dev/pps0

Now we need to configure ntpd-rs. Put the following in /etc/ntpd-rs/ntp.toml:

[observability]
log-level = "debug"
observation-path = "/var/run/ntpd-rs/observe"

[[source]]
mode = "pps"
path = "/dev/pps0"
precision = 1e-7
accuracy = 1e-6

[[source]]
mode = "sock"
precision = 1e-2
path = "/run/ntpd-rs/satpulse.ttyAMA0.sock"
accuracy = 0.2

[synchronization]
minimum-agreeing-sources = 1

[[server]]
listen = "0.0.0.0:123"

When I tried initially to get this working with version 1.7.1 of ntpd-rs, it didn’t work. I submitted an issue, and within a day one of the developers, David Venhoek, had added a feature to enable this configuration. This was included in version 1.7.2, released a few days ago. It adds the ability to specify the accuracy of sources as distinct from their precision. Accuracy means how close measurements are to true time. Precision means how much measurements vary from each other. The serial timing measurements from satpulsed are quite precise but are extremely inaccurate. Specifying a low accuracy for samples from satpulsed ensures that ntpd-rs uses the PPS samples as the primary source, and uses the satpulsed samples just to complete the PPS samples. But note that if you specify an accuracy of 0.25 or worse, you need to increase maximum-source-uncertainty from its default of 0.25.

Now you can start ntpd-rs using

sudo systemctl start ntpd-rs

You can verify it’s working by using

ntp-ctl status

The release notes have more information.

I would be grateful if you could try it out and let me know how you get on.

I created a discussion thread where you can ask questions and share experiences.