Nix From the Ground Up

By Ross Light

I recently spent some time learning Nix after watching this talk by Xe. Nix is a package manager/build system for Linux and macOS. It does a number of things I really like:

Transparent handling of source and binary packages.
Includes a rich central package registry, but you can host your package descriptions or binaries anywhere.
Does not require root and runs alongside any Linux distribution.
Easy to pin or customize versions of individual packages.
Straightforward support for project-specific dependencies.

Nix is a cool piece of tech, but in my opinion, it’s pretty hard to learn (at least at time of writing). I think this is accidental complexity: I was able to be productive with Nix in my personal projects in a few days, but it took a fair amount of research from many different sources. I took a lot of notes, then realized I wanted to publish them to share this knowledge.

So here’s my guide! “Nix From the Ground Up” aims to help explain the concepts behind Nix with a hands-on approach.

This guide assumes you…

know Unix command line basics
know at least one programming language
have familiarity with at least one other package manager like Homebrew or apt

I recommend reading the sections in order, as each section builds on concepts from the previous ones.

I’ll warn you up-front: I am not a Nix expert. Please let me know if you find any part of this post confusing or incorrect, and I will try to fix it.

For posterity, I’m using Nix 2.5.1 with nixpkgs unstable at commit 0c408a087b4751c887e463e3848512c12017be25.

EDIT 2022-01-25: Added a reference to the exact expression that nix-shell -p uses.

The Language
Enter nixpkgs
Installing Software
Shells from Expressions
Overrides and Overlays
Sharing Built Derivations
Conclusion

The Language

The foundation of Nix is its programming language. Until I learned the language, most of the documentation was hard to understand because the underlying mechanisms seemed very opaque.

Fortunately, the language is pretty straightforward: its syntax is pretty similar to Lua, but it is lazily evaluated like Haskell. The nix repl program is helpful for learning the language, which is what I would use to check the basics.

The language has the basics like strings, numbers, and lists:

2 + 2             # returns 4
"foo" + "bar"     # evaluates to "foobar"
[ 2 "foo" true ]  # a list with three items.
                  # Elements are separated by spaces.

Nix’s primary data type is the set, which is like Lua’s table type or the dictionary/map/hash type in other languages. Each name/value pair inside a set is called an attribute.

{}                 # the empty set
{ x = 5; y = 7; }  # a set with two attributes. The trailing semicolon is mandatory.
{ x = 5; }.x       # The typical dot syntax gets the value of an attribute.

The Nix language exists in service of package management, so it has a few domain-specific data types. Notably, inline paths allow referencing files alongside a .nix file:

./foo.txt  # references foo.txt in the directory the REPL is running in
           # or relative to the .nix file the path is written in.
           # This will automatically be made into an absolute path.

./.        # The current directory.
           # Nix paths must include at least one slash.

# Paths can be concatenated with strings:
./foo + "/bar.txt"

Local variables can be introduced with the let ... in construct:

let x = 2; in x + 3  # evaluates to 5

Nix functions are always anonymous, but can be assigned to variables. Function arguments are separated from the function name by a space, like in Haskell.

(x: x + 2) 2  # evaluates to 4

let
  f = x: x + 2;
in
  f 3  # evaluates to 5

And also like Haskell, multiple parameters can be handled by creating higher-order functions:

let
  add = x: y: x + y;
in
  add 40 2  # evaluates to 42

But more commonly, functions take in a set. Individual attributes can be bound to names using pattern matching:

let
  addSet = { x, y }: x + y;
in
  addSet { x = 40; y = 2; }  # evaluates to 42

There’s also a pattern matching syntax for optional arguments:

let
  scale = { x, factor ? 2 }: x * factor;
in
  scale { x = 5; }  # evaluates to 10

The last fundamental part of the Nix language is derivations. Derivations are created using the built-in derivation function, but you should typically invoke a helper function from nixpkgs (which we’ll talk about in a moment) rather than calling derivation yourself. However, under the hood, all helper functions eventually call the built-in derivation function.

# This is a demonstration, not a real example.
derivation {
  system = "x86_64-linux";
  name = "foo-1.2.0";
  builder = ./builder.sh;
  # ...
}

A derivation, upon evaluation, creates an immutable .drv file in the Nix store (typically located at /nix/store) named by the hash of the derivation’s inputs. A separate step, called realisation, ensures that the outputs of the derivation’s builder program are available as an immutable directory in the Nix store, either by running the builder program or downloading the results of a previous run from a cache. Since Nix takes precautions to make the builder invocation hermetic (details in the derivations section of the manual), these outputs can be shared safely between machines of the same OS and architecture. Cool!

It’s important to keep in mind that the Nix language is lazily evaluated. This means that even if you write let mySet = { x = 2 + 2; };, the expression 2 + 2 will not be evaluated until its value is needed. This means that sets can be huge without having to compute all the values.

This covers the basics of the language for the purposes of this guide, but there are a few more syntactic elements and many more built-in functions. For further reference, see the Nix expressions chapter of the Nix manual.

Enter `nixpkgs`

Now that we have a basic grasp of the Nix language, let’s examine the “standard library”: nixpkgs. nixpkgs defines a set that contains a variety of helper functions as well as an entire repository of software. When you installed Nix, it came with a copy of nixpkgs. You can make it available in your nix repl by running:

nixpkgs = import <nixpkgs> {}

You can find out what version of nixpkgs you have installed with the expression:

nixpkgs.lib.trivial.version
# "22.05pre340162.0c408a087b4" for the author

Nix used to have a channel model, but seems to have moved to a build from HEAD model recently. This means that new installations use “unstable”, so the commit is usually more useful information:

nixpkgs.lib.trivial.revisionWithDefault ""
# "0c408a087b4751c887e463e3848512c12017be25" for the author

Each software “package” is represented as an attribute in the nixpkgs set. For example, a derivation for the Go programming language toolchain is available as nixpkgs.go. You can find nixpkgs attributes using search.nixos.org, or you can examine pkgs/top-level/all-packages.nix in the nixpkgs repository itself. I find I usually have the nixpkgs source open in an editor and the nixpkgs manual open in a browser when I’m working on Nix stuff.

Installing Software

So how do we connect all these concepts to installing software? nix-env is a simple command to install derivations in your user environment, which acts similar to other package managers:

nix-env --install --attr nixpkgs.go
# commonly abbreviated to:
nix-env -iA nixpkgs.go

This will place the go tool into your PATH:

which go
# $HOME/.nix-profile/bin/go

This is a symlink to the go binary in the Nix store:

readlink `which go`
# /nix/store/5zvhj5hvy9mpgr7h8bjw3hj4jfnfd9zh-go-1.16.10/bin/go

Since this binary is deliberately not placed in a common path like /usr/local/bin, different users on a system can use different Go versions. However, if two users are using the same version, they’ll use the same binary.

You can update the software in your Nix user environment by running:

nix-channel --update &&
nix-env --upgrade

This is roughly equivalent to Debian’s apt-get update && apt-get upgrade. If you have a multi-user Nix installation, you may need to replace the first command with sudo -i nix-channel --update. Packages will be matched via the derivation name, not the attribute name.

Because the Nix store is immutable, if something goes wrong during the update, you can roll back!

nix-env --rollback

nix-env is great for getting the latest version of packages, but where Nix really shines is providing a concise, declarative language for describing and sharing project-specific environments. So let’s take a look at the next tool: nix-shell.

Shells from Expressions

nix-shell starts a bash shell with a specified set of derivations present. nix-shell --packages will grab attributes from nixpkgs:

$ nix-shell --packages go

[nix-shell:~]$ go version
go version go1.16.10 linux/amd64

You can use exit or Control-D (EOF) to exit the interactive shell. Using --run lets you run a single bash statement instead of an interactive session:

nix-shell --packages go --run 'go version'
# Output:
# go version go1.16.10 linux/amd64

This is just the beginning! nix-shell --expr gives you the full power of the Nix language. The previous invocation is roughly equivalent to:

nix-shell \
  --expr 'with import <nixpkgs> {}; mkShell { packages = [go]; }' \
  --run 'go version'

The nixpkgs.mkShell function evaluates to a derivation that just depends on other derivations. This is necessary because nix-shell starts a bash shell with the dependencies of the derivation resulting from the expression — not the derivation itself. You can look in the Nix source for the exact expression used.

Using an expression gives you the ability to pin to a specific commit of nixpkgs using the fetchTarball built-in function:

nix-shell \
  --expr 'let p = import (
    fetchTarball "https://github.com/NixOS/nixpkgs/archive/0c408a087b4751c887e463e3848512c12017be25.tar.gz"
    ) {}; in p.mkShell { packages = [p.go]; }' \
  --run 'go version'

As you can imagine, this would be unwieldy to type often, but does give you a shell that gives you the exact same version of Go regardless of machine. While you could create shell scripts or aliases that wrap nix-shell --expr, it’s much more convenient to store the expression in a file. Let’s go ahead and create a file called shell.nix with the following contents:

# Define a function that takes an optional parameter, pkgs.
# It defaults to a pinned nixpkgs version.
{ pkgs ? import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0c408a087b4751c887e463e3848512c12017be25.tar.gz") {}
}:

pkgs.mkShell {
  packages = [ pkgs.go ];
}

You can run nix-shell with a file argument and it evaluates the file as a Nix expression. The new trick we’re using here is that if the expression evaluates to a function, like in the example above, nix-shell will call the function with an empty set, then use the result.

nix-shell shell.nix --run 'go version'

Because shell.nix is the convention, we can omit it:

nix-shell --run 'go version'

Using nix-shell, we can maintain a shell.nix file in our project and ensure that we’re always using the same versions of our tools across machines and team members. You can reuse these expressions to provide consistent environments for continuous integration, building Docker images, or deploying cloud VM instances.

Overrides and Overlays

Up to this point, we’ve been using the version of the packages provided in nixpkgs. While nixpkgs will sometimes include multiple versions (e.g. go and go_1_17), there may come a point at which you want to pin to a specific version. The way you do this depends on the package, but you can usually use the overrideAttrs function to create a modified derivation. For example, here’s an expression I’ve written to pin SQLite to 3.36.0:

pkgs.sqlite.overrideAttrs (oldAttrs: {
  version = "3.36.0";
  src = pkgs.fetchurl {
    url = "https://sqlite.org/2021/sqlite-autoconf-3360000.tar.gz";
    sha256 = "vZDD65a+6ZYga4O+cGXJzhmu84w/T7Uwc62g0LabvOM=";
  };
});

In some cases, overriding will not work. This usually happens with more complex helpers like nixpkgs.buildGoModule. In these cases, you can copy the source .nix file from nixpkgs and make modifications yourself. You can use the nixpkgs.callPackage function to import files written in nixpkgs style.

The techniques outlined so far in this section create an isolated package: if another package depends on sqlite, it will use the nixpkgs version, not our pinned 3.36.0 version. If you want everything in nixpkgs that depends on sqlite to also use 3.36.0, you can use an overlay. An overlay is a function that returns a set of overrides for the nixpkgs set, passed in as an argument to the imported nixpkgs function. For example:

let
  # Overlays take two parameters:
  # self: The final nixpkgs set.
  # super: The nixpkgs that the overlay is wrapping.
  sqliteOverlay = self: super: {
    super.sqlite.overrideAttrs (oldAttrs: {
      version = "3.36.0";
      src = self.fetchurl {
        url = "https://sqlite.org/2021/sqlite-autoconf-3360000.tar.gz";
        sha256 = "vZDD65a+6ZYga4O+cGXJzhmu84w/T7Uwc62g0LabvOM=";
      };
    });
  };

  pkgs = import (fetchTarball "https://github.com/NixOS/nixpkgs/archive/0c408a087b4751c887e463e3848512c12017be25.tar.gz") {
    overlays = [ sqliteOverlay ];
  };
in
  # ...

This blog post goes into more detail about overlays and has some helpful diagrams.

Once you start using complex Nix configurations across machines, especially ones with overlays, Nix will likely have to build from source rather than fetch from the NixOS cache. This is unnecessarily slow: because Nix builds are hermetic, you can reuse results from previous builds, even across machines! Nix can access shared stores over HTTP(S), SSH, or S3-compatible storage. The official docs point out there is a hosted service, Cachix, but for my hobby projects, I found the pricing prohibitively expensive. Luckily, setting up a Google Cloud Storage bucket is pretty easy.

If you’re interested in the gory details, see this S3 blog post and the GCS follow-up post for more information. Here’s a quick summary of what I did:

To start, I created a GCS bucket and a Nix signing key:

gsutil mb -l us "gs://${BUCKET_NAME?}" &&
nix-store --generate-binary-cache-key "${BUCKET_NAME?}-1" cache-private.txt cache-public.txt

GCS supports the S3 protocol using HMAC keys for authentication. This required me to create an ~/.aws/credentials file with my user account HMAC key.

To get the closure of the store paths needed to build a shell:

binstores=( $(nix-store -qR $(nix-build --no-out-link shell.nix -A inputDerivation)) )

For the following steps, I needed to enable some experimental commands in the Nix CLI. You can do this with:

mkdir -p ~/.config/nix &&
echo 'experimental-features = nix-command' >>| ~/.config/nix/nix.conf

I signed the closure and uploaded with:

sudo -i nix store sign --key-file cache-private.txt "${binstores[@]}" &&
nix copy --to "s3://${BUCKET_NAME?}?endpoint=https://storage.googleapis.com" "${binstores[@]}"

If you accidentally run nix copy before signing, you can sign the store paths after the fact by passing an undocumented --store option to nix store sign (see NixOS/nix#4221 for details):

nix store sign \
  --store "s3://${BUCKET_NAME?}?endpoint=https://storage.googleapis.com" \
  --key-file cache-private.txt \
  "${binstores[@]}"

Finally, I configured my machines to fetch from the GCS bucket (nix.conf reference):

sudo mkdir -p /etc/nix &&
echo "extra-substituters = s3://${BUCKET_NAME?}?endpoint=https://storage.googleapis.com" |
  sudo tee -a /etc/nix/nix.conf &&
echo "extra-trusted-public-keys = $(cat cache-public.txt)" |
  sudo tee -a /etc/nix/nix.conf

Conclusion

That’s all I’ve got for now. I hope this guide helped you understand how you can use Nix to manage software dependencies. If you’re interested to learn more, head over to the NixOS learning page for more resources. As I said at the beginning, please reach out if you have any feedback on this guide. Thanks for reading!

Posted at 2021-12-29 9:40 PM

Permalink