Updates That Actually Land

You might have noticed a board that just... stopped keeping up. Same version week after week, updates coming and going with nothing to show for it. We dug in, and what we found wasn't one bug — it was a whole chain of them, each one quietly undoing the update the last one had almost finished. So we spent this stretch fixing the update path end to end.

The deepest one was the worst kind of funny. A counter that was supposed to track failed installs was counting successful ones instead, and nothing ever reset it. So the first three updates of a board's life would land, and the fourth would get thrown away — along with the evidence that anything had happened at all, and with the working software rolled back a version for good measure. Then it'd repeat. Forever. The new logic keeps a budget tied to the actual file being installed, clears it the moment an install succeeds, and sets suspect files aside instead of deleting them so we can see what went wrong. Best part: boards already stuck in this loop heal themselves. The cleanup rides in on the update path that still works, so nobody needs to touch a device to unstick it.

From there we worked outward. Updates used to restart the device software through a side process that could die quietly — the system would read that as a clean shutdown, decline to restart anything, and leave the board sitting there with nothing running until someone logged in by hand. That's gone; the update now hands off to the supervisor that was already there, plus a keepalive that steps in only when nothing is watching at all. On weak connections, downloads used to start from byte zero every single time, because each attempt cleaned up after itself a little too enthusiastically — a board on a marginal signal could never finish the download that would have fixed its signal problems. Now a partial download survives between attempts, is tracked by the contents it's meant to be rather than a filename, and an interrupted small patch resumes as a small patch instead of falling back to a 220 MB full image.

Two more guardrails round it out. Boards with two system slots were flipping back and forth between two perfectly healthy ones, burning through the very safety mechanism meant to rescue them, because the health check was reading a signal shared by both sides. Now a boot is confirmed from evidence about the slot itself — did the disk really write, does the kernel match the system, did the essential services come up — and a rollback that couldn't possibly change anything is refused outright, then automatically re-enabled the moment a genuine update arrives. And the device now checks for itself that an offered system image is actually newer before accepting it. When it genuinely can't tell, it goes ahead and says so in the log, because a board that wrongly refuses updates is a board no update can ever reach.

We also cleaned up a pile of internal documentation that had drifted years out of date — normally not worth a mention, except one wrong sentence in there had quietly switched off a safety check on scripts. A common operation was being treated as instant when it actually waits on the cloud, which meant scripts could skip waiting for it and fail silently with no warning. That check is back on, with tests pinning it down so it stays that way. All together: updates that arrive, install, and stick.