Introduction
In the world of confidential computing, the integrity and verifiability of every component in a trusted computing base (TCB) is critical. To achieve this, we rely on specialized Linux build systems—tools that transform source code, configuration files, and binaries into complete, bootable operating systems. These build systems broadly fall into two categories: source-based systems like Yocto or Buildroot that compile everything from scratch, offering maximum control and reproducibility at the cost of development speed; and binary-based systems like those used by Debian and Fedora, which primarily assemble pre-built packages for faster iteration but introduce external dependencies.
For our TDX images, we’ve been using Yocto Linux to build bit-for-bit reproducible operating systems that users can independently verify. This approach has served us well by providing small images with strong reproducibility guarantees, which are essential for confidential computing environments. However, we’ve encountered significant challenges with developer experience and flexibility that have led us to explore alternatives.
In this post, I’ll explain why we’re considering a transition from Yocto to a tool called Mkosi for our TDX images, how this change would enhance our development workflow while maintaining strong security guarantees, and our approach to ensuring the reproducibility and trustworthiness of the resulting images.
Our Requirements and Why We Chose Yocto
When building confidential computing environments for Ethereum block building and searching (as detailed in our previous posts on building secure Ethereum blocks on minimal Intel TDX confidential VMs and searching in TDX), we had several key requirements for our base operating system.
Our ideal operating system needed to contain only essential components to minimize security risks while supporting our specialized workloads. It had to enable independent verification of the entire image without requiring extensive OS expertise from users. And, above all, it needed to establish trust through technical guarantees like bit-for-bit reproducibility rather than relying on reputation or authority.
These requirements led us to Yocto Linux as our initial solution.
What Yocto Gets Right
Yocto excels in several critical areas that made it our initial choice:
Full Control and Customization
Yocto provides substantial control over the entire operating system stack. It allows us to specify versions and patches for each package, enabling us to build systems that meet our specific requirements. This control extends to components in the build system, supporting customization that would be challenging with other approaches.
Strong Reproducibility Guarantees
Yocto’s deterministic build process produces bit-for-bit identical images across different build environments. Additionally, the self-contained build environment reduces external dependencies that might introduce variability. When run in a minimal Docker environment, we get an identical operating system image on all machines.
Minimal Image Size
Yocto generates optimized, minimal images without unnecessary components. Its granularity allows us to specify exactly what we need down to every included kernel feature and executable on the machine. This helps reduce the attack surface and results in a much smaller TCB.
These capabilities aligned perfectly with our need for minimal, verifiable, and secure images for TDX environments. As a result, Yocto became the foundation for our confidential computing infrastructure.
The Challenges with Yocto
Despite these strengths, our experience with Yocto has revealed several significant drawbacks that have begun to outweigh its benefits:
Developer Experience
Working with Yocto can be incredibly time-consuming. Simple tasks like updating the Rust compiler version in the build environment or adding system packages to an image can take over an hour of build time on the first run. The lengthy feedback loop makes iteration painful, especially since Yocto’s complex DSL (domain specific language) is prone to errors that can be difficult to debug.
Limited Package Ecosystem
If vulnerabilities are discovered in dependencies used by our images, Yocto’s limited official package ecosystem becomes a critical weakness. Many specialized components required for our TDX images are only available through unofficial, community-maintained layers and repositories that aren’t part of the core Yocto Project.
When security vulnerabilities affect these dependencies, we’re at the mercy of these unofficial repositories’ maintenance schedules. If they don’t promptly release patches, our only recourse is to manually backport security fixes ourselves. This process is complex and time-consuming due to Yocto’s lengthy feedback loop, potentially creating an unacceptable security response timeline for builders and searchers using our products.
Auditability Challenges
Our Yocto configuration consists of extensive changes to the default Yocto reference implementation (called “Poky”). Auditing our particular configuration requires understanding the broader Poky context, making the process more complex.
Exploring Alternative Solutions
Given these challenges, we’ve been investigating alternatives that could maintain Yocto’s strengths while addressing its weaknesses. We considered several approaches:
Containerization
One option would be to containerize all of our applications, maintaining a minimal Yocto base. While this approach would solve our flexibility issues, it would introduce additional complexity in the confidential computing context and potentially expand our TCB.
Existing Minimal Distributions
We looked at minimal distributions like Alpine Linux, but they lacked the reproducibility guarantees we require for confidential computing environments.
Mkosi: A Promising Alternative
After exploring these options, systemd’s “Mkosi” tool has emerged as our top contender to replace Yocto. While these tools serve different primary purposes, both can produce reproducible Linux images.
Mkosi (GitHub repo) is an infrastructure automation tool designed to bootstrap existing Linux distributions like Debian or Fedora, similar to HashiCorp Packer. Instead of building a custom Linux distribution from scratch like Yocto does, Mkosi creates images based on established distributions, allowing us to leverage their existing package ecosystems and infrastructure.
Applying Mkosi to Our TDX Environment
To explore this, we’ve built a proof of concept Debian-based BuilderNet Image implementation in Mkosi. This has allowed us to leverage Debian’s mature ecosystem while still giving us control over verification and customization. We’re not alone in recognizing Mkosi’s potential for confidential computing - Edgeless Systems has adopted a similar approach for their needs.
Here’s how our Mkosi-based approach works:
- We select only the essential Debian packages required for our environment
- We remove unnecessary files to minimize the attack surface
- We integrate our BuilderNet-specific requirements and configurations
This straightforward process produces deterministic images that meet our security requirements while simplifying the build pipeline. In our testing, our mkosi configuration consistently generates identical image files across different build environments, even as the underlying Debian repository evolves.
This approach combines the reliability of Debian’s ecosystem with the control we need for confidential computing. Furthermore, the advantages of this method extend beyond just simplifying our workflow.
The Benefits of Switching to Mkosi
By transitioning to a Debian-based Mkosi configuration, we can address the issues we’ve experienced with Yocto:
Improved Developer Experience
- Updates to packages or configuration wouldn’t require dealing with a complex DSL
- Developers could rapidly prototype by updating components in-place on a running image
- We’d have access to Debian’s entire ecosystem, 97% of which is bit-for-bit reproducible
- The image build process would be faster than Yocto’s
Enhanced Security Response
- Debian has a great track record for quickly implementing security fixes, so we wouldn’t have to worry about porting fixes ourselves
- Security updates would be as simple as making a one-line change to switch our image to a new Debian repository snapshot
Improved Auditability
- Mkosi configuration is declarative and compact, making it significantly easier to audit. For example, files like this base configuration file for our buildernet layer are much easier to read than the Yocto alternative.
Third-Party Verification
- Verification would be much faster for users who trust Debian or their reproducibility infrastructure. This is a huge upgrade from Yocto, which takes hours to build from source, requiring a complex setup process and hundreds of gigabytes of disk space.
- With all our required packages now verified as reproducible by Debian’s infrastructure, users have strong assurances without needing to perform time-consuming builds themselves.
- For users preferring complete independence, we could still provide an automated verification path.
Considerations for a Debian-Based Approach
While Mkosi with Debian offers significant advantages, it doesn’t completely satisfy our requirements out of the box. The main challenge is that Mkosi’s default workflow relies on precompiled binary packages from Debian’s archives rather than building everything from source code, which introduces a dependency on Debian’s infrastructure and package management.
Understanding the Debian Security Model
To mitigate these issues, Debian has implemented robust systems to minimize the risk of compromised packages:
- They operate a distributed network of build servers managed by independent, extensively vetted maintainers. This is called the autobuilder network.
- Build jobs are completed randomly across different servers, making it difficult to maintain a backdoor across package updates.
- Their peer review process and security team provide additional layers of protection
This distributed trust model provides a solid foundation, but it still requires adding Debian’s infrastructure to our chain of trust.
Reproducibility Progress in Debian
However, Debian has made significant strides in ensuring package reproducibility through two primary verification systems:
- reproducible-builds.org - Tests building packages in different environments to catch non-determinism in the build process. This project focuses on ensuring that source code reliably produces the same binaries regardless of build environment. However, it does not attempt to build bit-by-bit copies of the exact binaries on the Debian repo. Currently, 97% of Debian packages are reproducible according to this system.
- reproduce.debian.net - A newer project that attempts to rebuild the exact .deb files found in the Debian repository bit-for-bit. On this site, 95% of packages are reproducible.
We’re pleased to report that as of last weekend, thanks to the hard work of the Debian reproducibility team, all packages required for our TDX images are now reproducible in Debian’s verification systems. This development significantly strengthens our confidence in moving forward with the Mkosi approach.
Exploring Package Verification Strategies
With the excellent progress in Debian’s reproducibility efforts, we now have multiple viable approaches for package verification in our trust model. This represents a significant advantage compared to our previous workflow.
Verification Options
- Leveraging Debian’s Infrastructure - Relying on Debian’s robust ecosystem may be sufficient since a compromise would affect systems far beyond our images, and their distributed infrastructure deters targeted attacks. All our needed packages are reproducible, though the reproduce.debian.net server currently lacks granular enough historical data to verify specific Debian repository snapshots.
- Independent Verification - For those preferring additional assurance, we could provide tooling to independently verify packages from source.
- Hybrid Approach - Use Debian’s verification for most packages while focusing our independent verification efforts on security-critical components.
Each approach offers different trade-offs between convenience and trust assumptions. With the Mkosi-based system, we have the flexibility to implement any of these models while maintaining excellent developer experience.
Current Status and Open Questions
We’ve made substantial progress in our exploration:
- We’ve created proof-of-concept Mkosi images that would dramatically improve our development workflow (see our GitHub repository)
- We’ve confirmed that all packages needed for our buildernet image are now reproducible in Debian’s verification systems
- We’ve established a practical path forward that maintains strong security properties while significantly improving developer experience
Invitation for Discussion
This blog post presents our exploration of Mkosi as a promising alternative to Yocto for building TDX images. With recent developments in Debian’s reproducibility efforts, we now have a clearer path toward potential implementation, but we’d like to gather community input before making any final decisions.
We’re seeking feedback on several specific questions:
- Practical Implementation: If you’ve implemented similar verification systems, what approaches have proven most effective in real-world environments?
- Verification Requirements: How important is independent package verification to you as a user of our TDX images? Would Debian’s existing verification infrastructure meet your needs, or would you prefer that we provide additional verification tools?
If you’re working on similar problems or have insights about trust models in reproducible builds, we’d appreciate your perspective. Our goal is to establish a practical security model that maintains strong guarantees while enabling responsive development and timely security updates.
how much time would a pure (trustless wrt debian’s maintainers) reproducibility verification of the ~minimal debian image take, compared to yocto’s ‘hours’?