I recently needed to build a custom Lua interpreter in Go.
The exact reasons aren’t important for this blog post,
but neither the reference implementation —
which I will be referring to as “PUC‑Rio Lua” —
nor the other open source Go Lua intepreters I could find were a good fit for my needs.
Building a Lua interpreter ended up being a rather enjoyable months-long side quest.
I’ve had a number of folks ask me to write about the experience
since these sorts of projects
usually highlight interesting aspects of both the implementation language and the interpreted language.
So here we are!
I’m excited to announce the first beta release of zb,
the build tool I’ve been working on for a year now. 🎉
I’ve been hard at work writing a Lua interpreter in Go from scratch,
bootstrapping a consistent userspace for Linux and macOS,
writing documentation,
and solving esoteric issues with executable code signatures.
(Y’know, no big deal. 😅)
If you’ve been following my progress and are itching to give zb a try,
you can jump ahead to the Getting Started guide.
The rest of this blog post is a quick overview of zb
and an update on what’s new since the previous blog post.
I have decided to develop zb,
my experiment in user-friendly reproducible builds,
into a full-fledged build tool.
Although my previous blog post
stated that I would not be developing this tool to production-readiness,
a few things changed my mind:
My personal projects need a dependency management solution,
and I am loath to continue investing in Nix.
I believe that many projects can benefit from a better dependency management/build tool,
especially with the increasing industry-wide focus on supply chain security.
I believe zb could serve that need.
zb is fun to work on!
It’s an interesting challenge to provide a simple interface to a sophisticated build model.
As a build system, zb provides:
A familiar language for configuration.
zb uses Lua to avoid introducing a significant programming language barrier.
Lua has been used in a variety of applications
to provide scripting facilities.
Lua’s small codebase made it straightforward to extend for zb
to include dependency information in every string.
Powerful build features.
In the language of Build systems à la carte,
zb is a suspending scheduler with constructive traces.
This puts zb in the category of the hypothetical “Cloud Shake” detailed in that paper.
As such, zb supports the early cutoff optimization (to speed up builds)
and dynamic dependencies (i.e. Lua configurations can read files from build targets).
Support for non-determinism.
zb is built to handle non-determinism in builds by rerunning nondeterministic build steps
and then reusing work if possible.
A build will not become incorrect if a build step is not perfectly deterministic,
even when sharing build cache among peers.
This reduces friction in migrating a codebase to use zb from other build systems.
File formats compatible with Nix.
Internally, zb uses the same .drv file format and archive format as Nix,
enabling tools built for the Nix store to be reused.
(At the moment, there’s not a clear way to interoperate with Nix derivations,
but this is hypothetically possible.)
Windows support.
I want a build system that works on Windows
in addition to Linux and macOS
so that tooling need not be split across platforms.
Although zb is not ready for production use yet,
I’ve reached a major milestone: zb no longer depends on Nix!
I have written a build backend from the ground up that supports content-addressed derivations
(a long-standing experimental feature in Nix),
and more broadly, uses the “Intensional Model” described in The Purely Functional Software Deployment Model.
This gives zb a strong foundation to leverage going forward.
If you’re interested in trying out zb for yourself,
follow the instructions in the project README.
Try writing your own builds (although keep in mind the known issues)
and discuss any feedback over on GitHub.
My next development target is to finish the Linux userspace, which will make bootstrapping other development tools much easier.
Stay tuned!
(If your business would benefit from a build expert or some extra backend
engineering bandwidth, I'm available for consulting and contract work! See my
freelance website —
256 Lights
— for details.)
In my previous blog post,
I talked about some of the shortcomings of Nix as a project and what I wish to see exist in its stead.
Since I published the blog post, there have been some promising developments:
a constitutional assembly has formed,
a fork of the Nix interpreter has arisen,
and an alternative Nixpkgs ecosystem is beginning to form.
I am enheartened that Nix has a potential future,
but it is too early to tell how these new efforts will play out.
In the meantime, I prototyped an idea I had kicking around while I wrote the blog post.
zb is an experiment in using Lua as an build configuration language
while still using the model described in The Purely Functional Software Deployment Model.
I am quite pleased with the results.
In my view, this is an existence proof for using a more conventional programming language
to reduce the onboarding cost of a technology like Nix.
The README has a more in-depth description of what I built, but in short:
zb uses a fairly stock Lua 5.4 interpreter,
with the notable exception that it includes dependency information in strings,
much like the Nix language.
Lua’s simple interpreter made it straightforward to add this feature.
zb produces .drv files and uses nix-store --realise to build them.
This means that zb is fully interoperable with Nix derivations
(e.g. using nixpkgs derivations in zb derivations),
but also proves that an alternative build backend could be created in the future.
zb uses content-addressed builds to enable the shallow build optimization
described in Build systems à la carte.
I built a source bootstrap that is mostly equivalent to the nixpkgs standard environment
by following the live-bootstrap steps.
This includes a working GCC, GNU Make, Coreutils, and Automake, among many other common programs.
I discovered in this process that this would be the largest time and effort investment.
Even after a month,
I still haven’t gotten a build that gets to the end of the steps.
I want to share this research prototype
in the interest of showing others what is possible in the space,
but I’m not planning on developing it to production-readiness.
If you have feedback on zb,
let me know on the Auxolotl forum,
on the zb repository discussions,
or privately.
(This is a long post.
In the words of Blaise Pascal,
“I have made this longer than usual because I have not had time to make it shorter.”)
I see the Nix project to have passed a critical moment.
I think the Nix project has made great strides in solving some very hard problems,
but my fear is that due to mismanagement,
Nix will not succeed.
In this post, I hope to share my perspective on what a better future for declarative, reproducible builds could look like,
but sadly, I don’t think it involves Nix as it exists today.
It’s hard to know what the future holds,
and I am certain I do not hold the solutions to the problems that exist with Nix.
However, I feel it would be a shame for the progress to be lost.
As background,
on 2024-04-21, a group of concerned contributors and community members published an open letter to the NixOS foundation
detailing Eelco Dolstra’s ill effects on the Nix project’s governance.
On 2024-04-26, Eelco published a blog post
that has caused many package maintainers to leave,
including me.
While I don’t have direct experience with the problems in the letter,
I very much trust the folks who signed the letter.
I am disappointed with how Eelco has chosen to handle the situation.
I personally think this is a watershed moment for Nix, and not for the better.
Such actions set the precedent for how an organization will act in the future,
and given the power structure, there’s not much that can be done.
In my mind, this endangers the health of Nixpkgs moving forward,
and without healthy package maintenance, the Nix project is doomed.
Consider this post a eulogy for a dream.
While the solution-that-is-Nix may evaporate, the problem it solves remains, at least for me.
The rest of this blog post is my attempt at sketching what a suitable replacement would look like,
in the hopes that it inspires someone to succeed where others have not.
Perhaps a replacement is a fork, perhaps it is a wholly new project, perhaps it is the Nix project under better governance.
I don’t know.
