Over the years, the Linux userspace has evolved with minimal cleanup, leading to a tangled integration of operating system components, vendor packages, configuration files, and user data. This accumulation—reflecting decades of innovation while retaining legacy elements—has made it difficult to find the right balance between immutability and composability. As a reaction to this, embedded product development teams have been resorting to leveraging mutable distributions with runtime package management or lean towards container orchestrators like Docker and Kubernetes. While these solutions seemingly offer great benefits in composability they often have long term hidden costs for maintainability, with high complexity and low reproducibility. In response, the Linux userspace community embarked upon on a journey to reorganize the filesystem structure, starting with “The Case for the /usr Merge”.
The userspace filesystem reorganization effort more clearly demarcates the separation of data types enabling partition strategies and tools to compose and describe the system. During this time, systemd has been introducing features that bring container-like benefits directly into the root Linux filesystem. Enhancements such as sysext, confext, and portable-services utilize overlayfs and other systemd based services and tools to extend the core operating system with image-based packages. Composing the system this way, we create a modular, immutable runtime coupled with a unified OS tree, making the architecture more approchable and easy to reason about. This modular approach enhances control over initialization sequencing, bootstrapping of asymmetrical multi-processor architectures, and works nicely with secure systems implementing LUKS encryption and dm-verity checksumming. It also clearly delineates boundaries for different runtime modes, such as manufacturing provisioning, factory reset, and recovery.
These advancements in systemd are incredibly useful for addressing the challenges of package bundling and process management in embedded application development spaces. They enable the ease of development associated with Docker-like containerization while retaining the control and performance expected from native Linux runtimes and unified root filesystems.
An opinionated Linux distribution called Avocado OS, created by Peridio, builds upon these concepts. Avocado OS is a layered, image-based operating system designed specifically for AIoT (Artificial Intelligence of Things) systems. It embraces an image-based approach leveraging A/B updates across all layers to ensure atomicity and predictable states without risking filesystem corruption.
Avocado OS ensures data protection at rest through dm-verity and LUKS encryption. It supports distinct operational modes such as recovery, manufacturing, and factory reset, while facilitating composability through systemd extensions and portable services. Avocado OS prioritizes a first-class development experience by integrating the familiarity of containerized workflows and a broad ecosystem of existing package managers, coupled with Yocto for vendor-supported BSP layers.
The Avocado OS Linux distribution presents an opinionated solution tailored for tackling package management challenges in embedded AIoT deployments on the edge. We’re excited to share this feature-rich embedded distribution, founded on decades of experience developing, deploying, debugging, and deploying embedded products. We aim to accelerate your product development vision so you can focus on creating the next generation of exciting embedded products.
For those interested in learning more about these systemd tools and concepts, we recommend checking out the following resources: