Hey everyone! Happy (US) Pride Month! 🏳️‍🌈🏳️‍⚧️

I’m excited to announce the first beta release of zb, the build tool I’ve been working on for a year now. 🎉 I’ve been hard at work writing a Lua interpreter in Go from scratch, bootstrapping a consistent userspace for Linux and macOS, writing documentation, and solving esoteric issues with executable code signatures. (Y’know, no big deal. 😅)

If you’ve been following my progress and are itching to give zb a try, you can jump ahead to the Getting Started guide. The rest of this blog post is a quick overview of zb and an update on what’s new since the previous blog post.

What is zb?

zb is a tool for reproducibly building software, similar to Bazel. (See the comparison page if you’re curious to know the differences.) When a software build process is reproducible, it will produce the exact same output when given the same inputs. Reproducibility is a desirable property for a software build process to have: it simplifies debugging, it enables build speed-ups, and it is essential for digital supply chain security. However, reproducibility is a difficult goal to achieve.

Many software build processes are not reproducible because they neglect to ensure the same versions of their development tools or libraries (dependencies) are used across machines. Approaches like virtual machines or containers lock dependency versions across machines, but rely on large base binary images that themselves can’t be reproduced or audited. Anyone who has built a Dockerfile containing an apt-get install command has likely been bitten by a package changing versions on them and causing their software to break. Virtual machines and containers also make common development tasks harder: attaching debuggers, integrating with IDEs, or even inspecting files can be a chore. Presented with such great friction, many developers (understandably) give up on reproducibility. They use other approaches to use dependencies during development that are “close enough” to what’s running in production. “Close enough” is frequently not enough, and complexity ensues.

zb simplifies reproducible builds.

• zb lets users define their build process using Lua.
• zb runs every build step in a lightweight sandbox with limited environment variables and filesystem access to reduce the likelihood of unexpected dependencies. The sandbox doesn’t rely on containers or virtual machines, so zb works even in restricted continuous integration (CI) environments.
• zb stores build artifacts as plain files and directories. so you can run development tools and build artifacts directly from zb’s store. This means you can point your IDE to use the exact compiler or intepreter that your build is using, for example. It also means that zb’s build artifacts can be copied directly into a container or virtual machine image without fuss.
• zb stores all build artifacts and source code in content-addressable storage. The name of a build directory includes a hash of all its contents, so it’s easy to see at a glance whether two versions of a dependency are the same.
• Consistency of build artifact paths across machines enables distributed caching and remote builds.

Development Highlights

Since my previous blog post, zb changed a lot. From a user standpoint, zb now has:

• sandboxing on Linux
• a full Linux userspace for C and Go
• a full macOS userspace for C and Go
• a module system that supports downloading remote libraries
• a standard library that can be updated independently of the zb CLI
• binary releases
• an installer script
• a nascent web UI for build logs

Under the hood, there have been a couple big developments to get zb ready for beta:

• I created a Lua interpreter in Go from scratch. This is a big enough technical development that I’m probably going to write another blog post just about the interpreter. The bottom line is that the new Lua intepreter enabled the module system, concurrent evaluation, and eliminated the need for cgo in the zb CLI. Lots of benefits, and lots of fun to work on!
• I solved a particularly gnarly issue with how content addressing interacts with macOS code signatures on Apple Silicon. At time of writing, this is an unsolved problem in Nix’s content addressing implementation, so figuring this out was exciting. I ended up writing a parser for the Mach-O executable format and had to rework most of the content addressing logic to account for complex file rewrites. For end-users, this means that they can build native binaries on macOS that contain references to the build directory (a very common occurrence for Unix developer tools) and have them work correctly even when the directory gets renamed to account for content addressing.

What’s Next?

zb is still a work-in-progress, but it’s finally at a point where it’s ready for folks to try out. Follow the Getting Started guide to get zb and learn the basics. If you have questions or feedback, see the support page.

If you’re excited to help shape the future of zb, check out the contributing page. In particular, the standard library can benefit from domain experts in building:

• Windows support
• Addressing determinism issues on macOS
• Linux full source bootstrap
• Linux on 64-bit ARM

(Discussion on Hacker News and Bluesky.)