Zombie Zen

Connecting Bash to Nix

By Ross Light

Julia Evans wrote a toot asking:

are there any guides to nix that start from the bottom up (for example starting with this bash script https://github.com/NixOS/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh) and then working up the layers of abstraction) instead of from the top down?

I realized that despite the title, my blog post Nix From the Ground Up misses the mark on providing this type of explanation. While I do think the Nix language is the lowest abstraction layer to learn Nix, I wanted to zoom in on the core derivation abstraction through a tutorial. This way, we can better understand how Nix derivations relate to Bash scripts.

To follow along, I’m assuming you…

  • know Bash scripting.
  • have compiled Unix software before, especially in C.
  • have installed Nix. I used Nix 2.13.2 while writing this tutorial.
  • have a reliable internet connection to download ~100MB. The first few examples don’t need it, but as you’ll see, we will need to download dependencies for our build environment.

All of the source code in this tutorial is released under the Unlicense and is available on GitHub.

Hello World

Let’s start by creating a Hello World derivation using only the Nix language builtins. We’ll start with a very simple shell script that takes the contents of a greeting environment variable and saves it into the file named by the environment variable out. Open up your favorite editor, and copy the following script into a file called greet.sh:

#!/bin/sh

echo "$greeting" > "$out"

Once you’ve done that, be sure to run chmod +x greet.sh. Now, let’s create our derivation in hello.nix:

builtins.derivation {
  name = "hello.txt";
  builder = ./greet.sh;
  system = builtins.currentSystem;

  greeting = "Hello, World!";
}

Files ending in .nix are expressions written in the Nix programming language. Since this is our first one, let’s break this down line by line:

  1. builtins.derivation is a built-in Nix function. The opening brace starts an attribute set: a data type very similar to a JavaScript/JSON object or a Python dictionary. The attribute set is passed to the derivation function as its argument.
  2. name is the full name of the derivation. It’s a required argument for the derivation function. name is used as part of the build output path (called a store path) when we go to build the derivation.
  3. builder is the program that nix-build will run when the derivation is built. It’s also required for the derivation function. It is typically a shell script, but could be the path to any Unix executable.
  4. system is a string specifying the Nix system type that the build can occur on. It’s the last required argument for the derivation function. I’m using builtins.currentSystem here to say “this derivation can build on whatever system you’re currently running on”.
  5. Any other attributes that we pass to derivation are passed along as environment variables to the builder. In this case, we set the greeting environment variable to give our favorite Hello World string.
  6. And finally, the closing brace ends the attribute set literal.

To build this derivation, we run the nix-build program with our hello.nix file:

$ nix-build hello.nix
this derivation will be built:
  /nix/store/ymhh65wy3nr7b9w8jl9kqavz9bq57fhp-hello.txt.drv
building '/nix/store/ymhh65wy3nr7b9w8jl9kqavz9bq57fhp-hello.txt.drv'...
/nix/store/8ny033mhdz8c7187wskdz2k9n83sifbz-hello.txt

When successful, the last line of nix-build’s output will be the store path, which is autogenerated by Nix. This is the same path that will be sent to your builder program as the out environment variable. As a convenience, nix-build will also create a symlink to the output path in the current directory with the name result. You can inspect it yourself:

$ readlink result
/nix/store/8ny033mhdz8c7187wskdz2k9n83sifbz-hello.txt
$ cat /nix/store/8ny033mhdz8c7187wskdz2k9n83sifbz-hello.txt
Hello, World!

A brief aside: the 8ny033mhdz8c7187wskdz2k9n83sifbz part of the store path is based on the hash of the derivation and its inputs. If you are following along on an x86_64-linux machine, you will get the same hashes! This is a key part of how Nix can reuse built derivations across machines. If you’re using a non-Intel or non-Linux machine, the hashes will be different because the system will be different.

We’ve now used nix-build to build a text file, let’s use it to create some more complex build outputs.

Building and Using Derivations for Programs

Let’s make a new derivation that generates a shell script rather than a text file. The convention for derivations that bundle up programs is to create a small version of the Filesystem Hierarchy Standard at the store path. That means we should create an $out/bin directory, and place our program there.

We’ll start by creating a new builder script build-greeter-script.sh:

#!/bin/sh

set -e
mkdir -p "$out/bin"
echo '#!/bin/sh' >> "$out/bin/hello"
echo "echo '$greeting'" >> "$out/bin/hello"
chmod +x "$out/bin/hello"

Run chmod +x build-greeter-script.sh. Then we’ll make a new derivation hello-script.nix (very similar to the last one):

# First attempt, not the final version:

builtins.derivation {
  name = "hello";
  builder = ./build-greeter-script.sh;
  system = builtins.currentSystem;

  greeting = "Hello, World!";
}

Let’s try to build this with nix-build like we did before:

Z nix-build hello-script.nix
this derivation will be built:
  /nix/store/d4yfycw4zb33d5syhzarcm7lcmk6y6yn-hello.drv
building '/nix/store/d4yfycw4zb33d5syhzarcm7lcmk6y6yn-hello.drv'...
/nix/store/yki5b3vqfzpwvkr6k7iwaxpmckfk3l2b-build-greeter-script.sh: line 4: mkdir: not found
error: builder for '/nix/store/d4yfycw4zb33d5syhzarcm7lcmk6y6yn-hello.drv' failed with exit code 127;
       last 1 log lines:
       > /nix/store/yki5b3vqfzpwvkr6k7iwaxpmckfk3l2b-build-greeter-script.sh: line 4: mkdir: not found
       For full logs, run 'nix log /nix/store/d4yfycw4zb33d5syhzarcm7lcmk6y6yn-hello.drv'.

mkdir not found? Huh?

When Nix goes to run a builder program, it sets PATH=/path-not-set by default (source) and sandboxes the build so that derivations can’t use OS-provided programs. In our first example, because we only used the echo command built into the shell, we didn’t encounter this problem. We must explicitly import the standard core Unix utilities we need in our script for the build to succeed. On the surface, not including standard Unix utilities in the default build environment may seem like a bug, but this is an intentional design decision to ensure reproducibility. Reproducible builds mean that you can be very confident that Nix derivations work the same on every machine.

Back to the task at hand: getting mkdir. The standard nixpkgs repository has a derivation for GNU coreutils, so we’ll use that. Amend hello-script.nix as follows:

let pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ae8bdd2de4c23b239b5a771501641d2ef5e027d0.tar.gz") {};
in

builtins.derivation {
  name = "hello";
  builder = ./build-greeter-script.sh;
  system = builtins.currentSystem;

  greeting = "Hello, World!";
  PATH = "${pkgs.coreutils}/bin";
}

The first two lines import nixpkgs into scope as the variable pkgs. I’m using a specific commit for reproducibility. Notice that how we’re also setting the PATH environment variable. Using the pkgs.coreutils derivation in a string interpolation will use its store path and indicate to Nix that pkgs.coreutils is a build dependency of our hello derivation.

Now we can run nix-build again, and it will download coreutils from the public Nix cache before building our hello derivation:

$ nix-build hello-script.nix
this derivation will be built:
  /nix/store/0akibr7h4yyx1hmn1dnjn066gc4yixaw-hello.drv
these 8 paths will be fetched (9.16 MiB download, 40.92 MiB unpacked):
  /nix/store/2w4k8nvdyiggz717ygbbxchpnxrqc6y9-gcc-12.2.0-lib
  /nix/store/76l4v99sk83ylfwkz8wmwrm4s8h73rhd-glibc-2.35-224
  /nix/store/bg8f47vihykgqcgblxkfk9sbvc4dnksa-coreutils-9.1
  /nix/store/bw9s084fzmb5h40x98mfry25blj4cr9r-acl-2.3.1
  /nix/store/jn9kg98dsaajx4mh95rb9r5rf2idglqh-attr-2.5.1
  /nix/store/jvl8dr21nrwhqywwxcl8di4j55765gvy-gmp-with-cxx-stage4-6.2.1
  /nix/store/qmnr18aqd08zdkhka695ici96k6nzirv-libunistring-1.0
  /nix/store/vv6rlzln7vhxk519rdsrzmhhlpyb5q2m-libidn2-2.3.2
copying path '/nix/store/qmnr18aqd08zdkhka695ici96k6nzirv-libunistring-1.0' from 'https://cache.nixos.org'...
copying path '/nix/store/vv6rlzln7vhxk519rdsrzmhhlpyb5q2m-libidn2-2.3.2' from 'https://cache.nixos.org'...
copying path '/nix/store/76l4v99sk83ylfwkz8wmwrm4s8h73rhd-glibc-2.35-224' from 'https://cache.nixos.org'...
copying path '/nix/store/jn9kg98dsaajx4mh95rb9r5rf2idglqh-attr-2.5.1' from 'https://cache.nixos.org'...
copying path '/nix/store/2w4k8nvdyiggz717ygbbxchpnxrqc6y9-gcc-12.2.0-lib' from 'https://cache.nixos.org'...
copying path '/nix/store/bw9s084fzmb5h40x98mfry25blj4cr9r-acl-2.3.1' from 'https://cache.nixos.org'...
copying path '/nix/store/jvl8dr21nrwhqywwxcl8di4j55765gvy-gmp-with-cxx-stage4-6.2.1' from 'https://cache.nixos.org'...
copying path '/nix/store/bg8f47vihykgqcgblxkfk9sbvc4dnksa-coreutils-9.1' from 'https://cache.nixos.org'...
building '/nix/store/0akibr7h4yyx1hmn1dnjn066gc4yixaw-hello.drv'...
/nix/store/zgcm9jxccq0y9nnghppb2z996ig70q3c-hello

Excellent! Now we have built a small directory tree around our script:

$ cat result/bin/hello
#!/bin/sh
echo 'Hello, World!'
$ result/bin/hello
Hello, World!

Building GNU Hello

Now that we understand how to bring derivations into our build environment, let’s finish up by packaging GNU Hello, a C program that uses Autoconf and Make to print greetings to standard out. It doesn’t have any dependencies beyond Make and common Unix tooling.

As in the previous examples, we’ll start off with a Bash script. This one will unpack a tarball, run ./configure, make, and make install. Save the following script into a file called build-gnu-hello.sh and chmod +x build-gnu-hello.sh:

#!/bin/sh

set -e
tar zxf "$src"
cd hello-2.12.1
./configure --prefix="$out"
make
make install

You’ll notice that we don’t download the source code in this script. That’s because Nix does not permit network access during most derivations’ builds. Instead, we can use pkgs.fetchurl from the nixpkgs repository to download the source and check its contents against a hash. This ensures that the build is working with known inputs, and thus keeps the build reproducible.

Let’s create a new derivation:

# gnu-hello1.nix

let pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ae8bdd2de4c23b239b5a771501641d2ef5e027d0.tar.gz") {};
in

builtins.derivation {
  name = "hello";
  builder = ./build-gnu-hello.sh;
  system = builtins.currentSystem;

  src = pkgs.fetchurl {
    url = "mirror://gnu/hello/hello-2.12.1.tar.gz";
    hash = "sha256-jZkUKv2SV28wsM18tCqNxoCZmLxdYH2Idh9RLibH2yA=";
  };
  PATH = "${pkgs.coreutils}/bin:${pkgs.gnused}/bin:${pkgs.gawk}/bin:${pkgs.gnugrep}/bin:${pkgs.gnutar}/bin:${pkgs.gzip}/bin:${pkgs.gcc9}/bin:${pkgs.gnumake}/bin";
}

Run nix-build gnu-hello1.nix followed by result/bin/hello like before. Because this downloads a lot more dependencies from the internet, I’m going to omit the output from this blog post, but you should see our favorite greeting yet again. This time, you can verify it is GNU Hello by running result/bin/hello --version.

While this works, the nixpkgs repository has a wrapper function, stdenv.mkDerivation, that makes this easier. It includes its own builder script, includes a library of utility functions, runs phases provided by environment variables, and provides common tooling for compiling C packages. The Standard Environment chapter of the nixpkgs manual has all the details if you’re interested.

Let’s rewrite our derivation using stdenv.mkDerivation. This time, we don’t need to provide a script: we include Bash snippets as phases.

# gnu-hello2.nix

let pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ae8bdd2de4c23b239b5a771501641d2ef5e027d0.tar.gz") {};
in

pkgs.stdenv.mkDerivation {
  name = "hello";
  src = pkgs.fetchurl {
    url = "mirror://gnu/hello/hello-2.12.1.tar.gz";
    hash = "sha256-jZkUKv2SV28wsM18tCqNxoCZmLxdYH2Idh9RLibH2yA=";
  };

  configurePhase = "./configure --prefix=$out";
  buildPhase = "make";
  installPhase = "make install";
}

(You can build this with nix-build and it should produce another GNU Hello binary.)

Since so many existing Unix programs follow this ./configure && make && make install pattern, this is the default for stdenv.mkDerivation. Thus, we can omit these arguments:

# gnu-hello3.nix

let pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/ae8bdd2de4c23b239b5a771501641d2ef5e027d0.tar.gz") {};
in

pkgs.stdenv.mkDerivation {
  name = "hello";
  src = pkgs.fetchurl {
    url = "mirror://gnu/hello/hello-2.12.1.tar.gz";
    hash = "sha256-jZkUKv2SV28wsM18tCqNxoCZmLxdYH2Idh9RLibH2yA=";
  };
}

If we peek at the official nixpkgs hello derivation, it’s pretty similar, but has a few extra attributes for testing and metadata.

Summary

That was a lot of ground to cover! In this blog post, we saw how we can turn a Bash script into a Nix derivation and run it with nix-build. We also got a small taste of some of the utilities that Nix and nixpkgs give us at build time. Hopefully, this gave you a sense for how derivations work. If you’re interested in learning more, see my other blog post Nix From the Ground Up to get an overview of how derivations fit into the larger picture of Nix tooling.

(Once you’re done, you can remove the result symlink(s) and run nix-store --gc to reclaim the disk space used during this tutorial.)

Posted at
Permalink