Peridio builds Avocado OS: a production-grade, Yocto-based Linux distribution for embedded and edge-AI devices, with the fleet layer - OTA, secure boot, device management - to run it in the field. Delivering that means building Yocto images continuously so our customers do not have to. A full image build is hours of compilation, sometimes a day; deploying one we have already built is minutes.
Running that build workload constantly, at a scale most teams never reach, means kernel bugs under it reach us before they reach anyone else.
One did. A bug in XFS took down our build machines mid-build, the filesystem shutting itself down and reporting no space on a disk that was half empty. We had two options: work around it locally, or find the root cause and fix it upstream.
We chose the root cause. It cost roughly two weeks of one engineer's time. What that bought is a one-line kernel patch now merged upstream into XFS, a regression test that will live in the kernel's own test suite, and a precise answer to the question "which of our machines are exposed and which are not."
The workaround would have cost two days and left us not knowing.
This post is about that tradeoff, because it is the one every team maintaining an embedded Linux distribution has to make repeatedly, and the cheap answer is usually the wrong one.
Worth establishing first, because it is the reason this bug was ours to find.
A Yocto image is built from source. Not assembled from packages: compiled, from the toolchain up, for the specific board it will run on. A full build of a non-trivial image is hours on a good machine and can be a day or more on an ordinary one, and it has to be redone whenever the kernel, a layer, or a dependency moves.
That is the tax on building your own embedded Linux, and most teams pay it repeatedly. Avocado OS exists so they do not: it is the production OS itself - a Yocto-based, immutable Linux with a fleet layer on top - and we run the builds that produce it continuously across the board and feature matrix, shipping the artifacts, so the customer's step is a deploy measured in minutes rather than a build measured in hours or days.
The consequence is that we run this workload constantly and at a scale most individual teams never reach. When something in the kernel breaks under it, it breaks for us first. That is the arrangement working as intended, and it is also how we ended up spending two weeks on a filesystem bug.
During a Yocto build of an Avocado OS image, the root filesystem on a build machine shut itself down:
XFS: Corruption of in-memory data detected at xfs_defer_finish_norollShutting down filesystem
The machine had 862 GiB free on a 2 TB drive. The error reported underneath that shutdown was -ENOSPC: out of space.
A filesystem that stops because it thinks it is full, while being half empty, is not a capacity problem. It is a correctness problem, and correctness problems do not go away when you add a disk.
We had several, and all of them would have worked.
Parent pointers are the XFS feature involved. They are a per-directory-entry extended attribute, and the bug only fires when an inode accumulates many of them. Yocto's packaging step hardlinks the same file into many package staging directories, so it generates exactly that pattern. Turning the feature off at mkfs time would have ended the crashes in an afternoon.
So would moving the build filesystem to ext4. So would giving the build machines enough free space that the failing condition became vanishingly unlikely, which, as it turned out, is precisely why our second build node never crashed at all.
Every one of those is a real option and their appeal is worth stating honestly: they are fast, they are low-risk, and the person who picks one is not being lazy. They are choosing a smaller unknown over a larger one.
The problem is what they leave behind. Each ends with the bug still in the kernel, still in every customer's kernel, and with us unable to say whether a given machine is affected. "We stopped seeing it" is not the same as "it cannot happen." When you ship an operating system, that difference is the entire product.
Two weeks, and the first of them was spent being wrong.
The failure looked like hardware. A machine that hard-reboots under load usually is hardware, so the first days went into BIOS settings and component swaps. That was a dead end, and it belongs in this post rather than smoothed out of it: the cost of this kind of work is mostly the wrong turns, and any account that omits them is useless for planning the next one.
Two further failures were found and fixed along the way that turned out to be unrelated to the one we were chasing: a resource-control setting in our own build tooling that could stall a machine, and a kernel watchdog policy that escalated a recoverable stall into a hard reboot. Both were real, both were worth fixing, and neither was the bug.
The bug itself is one missing assignment, and it is worth spelling out because the shape of it is more instructive than the specifics.
The kernel log, incidentally, said "Corruption of in-memory data." Nothing was corrupt. That message is a generic label on one exit path rather than a diagnosis, and it sent the first day of real investigation in the wrong direction.
There is a counter meaning "how many blocks may this operation still use." It belongs to the code that writes a parent pointer, which is the record that a file belongs to a directory, and one of those is written on every hardlink. That is why a packaging step that hardlinks the same file into dozens of staging directories is what found this, and why nothing else on the machine did. One path forgot to set that counter before using it, so it started at zero. The code then subtracted from it.
Subtracting one from zero in that counter does not give minus one: it is unsigned, so it rolls over to the largest value it can hold, about 4.3 billion, the way an odometer wound backwards past zero reads 999999 rather than -1.
Everything downstream then behaved correctly and arrived somewhere absurd. A safety check read 4.3 billion, concluded there was effectively unlimited room, and let the request through. The next layer did its own arithmetic on the same poisoned number and decided the operation could use zero blocks. A third check compared "needs at least one block" against "may use zero blocks", found that impossible, and reported it as out of disk space.
Nothing was out of disk space. The filesystem had no way to tell "impossible because the disk is full" apart from "impossible because I computed nonsense", so it assumed its own bookkeeping was corrupt and shut down rather than risk writing damage to disk. That is the right response to self-contradictory state. It was reasoning correctly from one bad number.
Two assertions exist in the source specifically to catch this. Both are compiled out of every kernel anyone actually runs, which is why this survived long enough to reach us.
One line, merged upstream into XFS's for-next branch. It reaches every distribution from there, including the kernels our customers run whether or not they use our tooling, and we no longer carry a private patch.
What makes it defensible rather than a guess is that XFS already does this correctly elsewhere. When the filesystem replays the same operation from its log after a crash, it sets that counter from the same function we now call. Replaying the work was always right; performing it live was always wrong. The fix makes the normal path behave like the recovery path instead of contradicting it.
The series is six patches: the fix, an unrelated bug found while reading the same code, an assertion that catches this class of failure at the point it happens instead of three layers downstream, and three smaller corrections. A maintainer can take them separately.
We wrote an xfstests case and, more importantly, verified it fails on an unpatched kernel and passes on a patched one. A regression test that has only ever been seen passing proves nothing. Our first version of this one swept the wrong range of free space and would have shipped as a test that could never fail. Catching that was worth the extra day.
The original failure needed a multi-hour build plus a specific coincidence, and destroyed its own evidence when it fired. It now reproduces deterministically on a 512 MiB scratch filesystem in about 0.2 seconds, without taking the machine down. That is the difference between losing an afternoon per data point and running the experiment forty times before lunch.
We can state the conditions exactly: XFS with parent pointers enabled, an inode accumulating many hardlinks, and an allocation group at exactly zero available space at the wrong moment. That lets us answer "is this machine at risk" with a check rather than a guess. Our second build node had never crashed, and we now know why: it had enough free space that the third condition never arose. It was exposed the entire time.
That second machine also settled the question we started with. It runs different hardware, and it reproduced the identical failure down to the event counts. Whatever this was, it was never our hardware.
We saw none, and the shutdown exists precisely to prevent it: the filesystem stops rather than write anything it cannot account for. One unexplained incident left a boot file damaged and needing a rebuild, so the honest claim is "none observed" rather than "none possible."
obvious at the one-week mark. What made it correct was not persistence. It was that the failure sat directly under the product. We build operating system images for other people's hardware; a filesystem corruption bug in the image build path is not an inconvenience to route around, it is the thing we are paid to have already dealt with.
The inverse holds too, and it is worth saying plainly: had this bug been in a developer's local tooling rather than the image build path, the workaround would have been the right call. "Always find the root cause" is not a policy, it is a slogan. The policy is knowing which failures sit under the product and which do not.
The economics also improved as the work went on, which is not intuitive. The most expensive phase was the first week, when the failure was rare and destroyed its own evidence. Once we understood the mechanism well enough to construct it deliberately, the cost per experiment fell by three orders of magnitude. Investment in reproducibility paid for itself inside a day, and it would have paid sooner had we reached for it earlier. That is the lesson worth carrying to the next investigation.
If you run Yocto or any heavy packaging workload on XFS, this is a concrete check rather than a hypothetical:
xfs_info / | grep -o 'parent=[01]'parent=1 means the feature is enabled and the first condition is met. It is the default in recent xfsprogs, so a filesystem created in the last couple of years likely has it. Combine that with a packaging workload that hardlinks heavily and a filesystem that periodically runs close to full, and all three conditions are present.
If that describes your build infrastructure, the fix is merged upstream and the reproducer is on the linux-xfs mailing list: the applied series is here.
And if you would rather not be reasoning about kernel allocator internals at all, that is the arrangement Peridio is: we build and maintain the OS, run the builds, absorb what breaks under them, and ship you an artifact you deploy - and then update over the air - in minutes instead of waiting out a build measured in hours.
The visible part of that is the time. The part worth paying for is the two weeks in this post, which a team building its own images would have spent themselves, or more likely would not have spent at all. The realistic outcome for most teams is not a root cause and an upstream patch. It is a build server that fails every so often, a reboot, a note in a runbook saying to retry it - and no one who owns whether the OS underneath it is correct. That is the part Peridio owns.
Talk to us about your build infrastructure and we will tell you plainly whether this one affects you.