nix.md

Introduction

Nix and its ecosytem have signifcantly improved the reliability of my software builds and development environment, and made it significantly easier for me to maintain them as well. The software packaging experience is also much more enjoyable than with other package managers I have used in the past. Declarative OS configuration is also a huge win for me, and configuring linux imperativeley seems like using a calculator to do math when you have a computer.

To learn about nix seems like a daunting and confusing task, and it sometimes can be. While there exists a high learning curve, I believe that the benefits of learning nix are well worth the effort. These workshops aim to cover a progression of some of the pratical benefits of nix, and how to use them in your day to day workflow. After completing some of these workshops I'm hoping you be able to experience some of the benefits of nix that I have, and be able to use them in your own projects, and have a better idea on how to get unstuck, and be easier to pattern match and use examples from other projects as well.

The progression of topics covered is as follows

• Installing Nix on non NixOS systems
• Using Nix to install packages
• Ad hoc shells with nix-shell
• Nix Flakes and development shells
• Packaging your own software with Nix
• NixOS and declarative OS configuration
• Creating a NixOS module
• NixOS Integration Tests
• Impermanence Mode

Installing Nix on non NixOS systems

TLDR: use Determinate Systems Installer

curl --proto '=https' --tlsv1.2 -sSf -L https://install.determinate.systems/nix | sh -s -- install

Using Nix to install packages

Search for a package on nixpkgs and install it with the following command.

nix profile install nixpkgs#hello

Ad hoc shells with nix-shell

Create a shell with a specific set of dependencies with the following command.

nix-shell -p python3

install python and run a python script with the following command.

nix-shell -p python3 --run "python3 script.py"

Nix Flakes and development shells

create a directory, and initialize a nix flake with the following command.

mkdir nix-flake-dev-shell
cd nix-flake-dev-shell
nix flake init

edit the flake.nix file to look like the following.

{
  description = "A basic dev shell";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";


  outputs = { self, nixpkgs, flake-utils, ... }:
    (flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
          };
        in
        {

          devShells.default = pkgs.mkShell {
            buildInputs = [
              pkgs.hello
              pkgs.cowsay
            ];
          };

        })
    );
}

Packaging your own software with Nix

all packaging in nix is based on the stdenv.mkDerivation function. This function takes a set of attributes and returns a derivation. A derivation is a set of instructions on how to build a package.

you can learn about the attributes that stdenv.mkDerivation takes in this blog post

packaging a basic C program

create a main.c file with the following contents.

#include <stdio.h>

int main(void){
  printf("Hello, World!\n");
  return 0;
}

create the following flake.nix

{
  description = "A basic C program";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";


  outputs = { self, nixpkgs, flake-utils, ... }:
    (flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
          };
        in
        {

          packages.default = pkgs.stdenv.mkDerivation {
            name = "my-package";
            src = ./.;

            buildPhase = ''
              ${pkgs.clang}/bin/clang -Wall -Wextra -Werror -O2 -std=c99 -pedantic main.c -o hello
            '';

            installPhase = ''
              mkdir -p $out/bin
              cp hello $out/bin
            '';
          };

        })
    );
}

run the following command to build the package.

nix build

run the executable with the following command.

./result/bin/hello

Exploring more packaging options

{
  description = "A basic rust cli";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";


  outputs = { self, nixpkgs, flake-utils, ... }:
    (flake-utils.lib.eachDefaultSystem
      (system:
        let
          pkgs = import nixpkgs {
            inherit system;
          };

          app = pkgs.rustPlatform.buildRustPackage {
          	pname = "app";
            version = "0.0.1";
            src = ./.;

            cargoLock = {
              lockFile = ./Cargo.lock;
            };

            nativeBuildInputs = [
              pkgs.pkg-config
            ];
            PKG_CONFIG_PATH = "${pkgs.openssl.dev}/lib/pkgconfig";

            buildPhase = ''
              cargo build --release
            '';

            installPhase = ''
              mkdir -p $out/bin
              cp target/release/app $out/bin/app
            '';

            # disable checkPhase
            doCheck = false;

          };
        in
        {
          packages.default = app;
        })
    );
}

where to find more info

• languages and frameworks

selected quote from Javascript - finding examples

Getting unstuck / finding code examples If you find you are lacking inspiration for packing javascript applications, the links below might prove useful. Searching online for prior art can be helpful if you are running into solved problems.

Github Searching Nix files for mkYarnPackage: https://github.com/search?q=mkYarnPackage+language%3ANix&type=code

Searching just flake.nix files for mkYarnPackage: https://github.com/search?q=mkYarnPackage+path%3A**%2Fflake.nix&type=code

Gitlab Searching Nix files for mkYarnPackage: https://gitlab.com/search?scope=blobs&search=mkYarnPackage+extension%3Anix

Searching just flake.nix files for mkYarnPackage: https://gitlab.com/search?scope=blobs&search=mkYarnPackage+filename%3Aflake.nix

NixOS and declarative OS configuration

minimal self contained flakes for configuring a NixOS system with tailscale.

{
  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs?ref=nixos-23.11";
  };
  outputs = { nixpkgs }: {
    nixosConfigurations.hello = nixpkgs.lib.nixosSystem {
      system = "x86_64-linux";
      specialArgs = {
        inherit inputs;
      };
      modules = [
        { config, pkgs, ... }: {
          system.stateVersion = "23.05";

          services.tailscale = {
            enable = true;
            authKeyFile = "/tsauthkey";
            extraUpFlags = [ "--ssh" "--hostname" "flakery-tutorial" ];
          };
        }
      ];
    };
  };
}

but you can also import nixos configuration as a module in your flake.

{
  description = "A basic NixOS configuration";

  inputs.nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
  inputs.flake-utils.url = "github:numtide/flake-utils";

  outputs = { self, nixpkgs, flake-utils, ... }:
	(flake-utils.lib.eachDefaultSystem
	  (system:
		let
		  pkgs = import nixpkgs {
			inherit system;
		  };
		in
		{
		  nixosConfigurations.hello = {
			modules = [
			  ./configuration.nix
			];
		  };
		})
	);
}

and the configuration.nix file would look like the following.

{ config, pkgs, ... }:

{
  system.stateVersion = "23.05";

  services.tailscale = {
	enable = true;
	authKeyFile = "/tsauthkey";
	extraUpFlags = [ "--ssh" "--hostname" "flakery-tutorial" ];
  };

}

Basic configuration with tailscale

{ config, pkgs, ... }:

{
  system.stateVersion = "23.05";

  services.tailscale = {
    enable = true;
    authKeyFile = "/tsauthkey";
    extraUpFlags = [ "--ssh" "--hostname" "flakery-tutorial" ];
  };

}

services are cool, there's 10,000 of them searchable

configuring a systemd service

{ config, pkgs, ... }:

{
  systemd.services.hello = {
	description = "Hello World Service";
	wantedBy = [ "multi-user.target" ];
	serviceConfig = {
	  Type = "simple";
	  ExecStart = "${pkgs.hello}/bin/hello";
	};
  };
}

Apply the flakes to your system

sudo nixos-rebuild  switch --flake '.#hello' --refresh

or to apply a configuration from a git repository

sudo nixos-rebuild  switch --flake 'github:reedrichards/flakes#hello' --refresh

Creating a NixOS module

# /etc/nixos/exampleService.nix
{ config, lib, pkgs, ... }:

let
  cfg = config.services.exampleService;
in
{
  options.services.exampleService = {
    enable = lib.mkEnableOption "Example Service";

    message = lib.mkOption {
      type = lib.types.str;
      default = "Hello, NixOS!";
      description = "Message to write to the log file.";
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.exampleService = {
      description = "Example Service";
      after = [ "network.target" ];
      wantedBy = [ "multi-user.target" ];
      script = ''
        while true; do
          echo "${cfg.message}" >> /tmp/exampleService.log
		  echo wrote "${cfg.message}" to /tmp/exampleService.log
          sleep 60
        done
      '';
    };
  };
}

Using a NixOS module

{ config, lib, pkgs, ... }:

{
  imports =
    [
      ./exampleService.nix
    ];

  services.exampleService.enable = true;
  services.exampleService.message = "This is a test message from the example service.";
}

NixOS Integration Tests

useful resources

• nixos manual\
• nixos tests
• nixacademy

basic example with python echo server, shamelessly stolen from nixacademys github

echo-server.py

import socket
import sys

HOST = ""

try:
    PORT = int(sys.argv[1])
except:
    print("Please provide a port number on the command line")
    sys.exit(1)

with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
    s.bind((HOST, PORT))
    s.listen()

    while True:
        conn, addr = s.accept()
        with conn:
            while True:
                data = conn.recv(1024)
                if not data:
                    break
                conn.sendall(data)

create a module for this server

echo-nixos-module.nix

{ config, pkgs, lib, ... }:
let
  cfg = config.services.echo;
in
{
  options.services.echo = {
    enable = lib.mkEnableOption "TCP echo as a Service";
    port = lib.mkOption {
      type = lib.types.port;
      default = 8081;
      description = "Port to listen on";
    };
  };

  config = lib.mkIf cfg.enable {
    systemd.services.echo = {
      description = "Friendly TCP Echo as a Service Daemon";
      wantedBy = [ "multi-user.target" ];
      serviceConfig.ExecStart = ''
        ${pkgs.python3}/bin/python3 ${./echo-server.py} ${builtins.toString cfg.port}
      '';
    };
  };
}

create a test for this module test.nix

{
  name = "Echo Service Test";

  nodes = {
    server = { config, pkgs, ... }: {
      imports = [
        ./echo-nixos-module.nix
      ];

      services.echo.enable = true;

      networking.firewall.allowedTCPPorts = [
        config.services.echo.port
      ];
    };
    client = { ... }: { };
  };

  globalTimeout = 60;

  interactive.nodes.server = import ../debug-host-module.nix;

  testScript = { nodes, ... }: ''
    ECHO_PORT = ${builtins.toString nodes.server.services.echo.port}
    ECHO_TEXT = "Hello, world!"

    start_all()

    server.wait_for_unit("echo.service")
    server.wait_for_open_port(ECHO_PORT)

    client.wait_for_unit("network-online.target")
    output = client.succeed(f"echo '{ECHO_TEXT}' | nc -N server {ECHO_PORT}")
    assert ECHO_TEXT in output
  '';
}

pull it together in a test

flake.nix

{
  description = "Example NixOS Integration Tests";

  inputs = {
    nixpkgs.url = "github:NixOS/nixpkgs/nixos-unstable";
    flake-parts.url = "github:hercules-ci/flake-parts";
    flake-parts.inputs.nixpkgs-lib.follows = "nixpkgs";
  };

  outputs = inputs: inputs.flake-parts.lib.mkFlake { inherit inputs; } {
    systems = [ "x86_64-linux" "aarch64-linux" "aarch64-darwin" "x86_64-darwin" ];
    perSystem = { config, pkgs, ... }: {
      packages.default = pkgs.testers.runNixOSTest ./test.nix;
      checks = config.packages;
    };
  };
}

run the test with the following command

nix build .#default
# or
nix build # for the default package

takes about 5 mins to run on m1 mac

nix build -L .#echo  11.99s user 4.75s system 5% cpu 5:33.55 total

examples from nixos testing

single node etcd

# This test runs simple etcd node

import ../make-test-python.nix ({ pkgs, ... } : {
  name = "etcd";

  meta = with pkgs.lib.maintainers; {
    maintainers = [ offline ];
  };

  nodes = {
    node = { ... }: {
      services.etcd.enable = true;
    };
  };

  testScript = ''
    with subtest("should start etcd node"):
        node.start()
        node.wait_for_unit("etcd.service")

    with subtest("should write and read some values to etcd"):
        node.succeed("etcdctl put /foo/bar 'Hello world'")
        node.succeed("etcdctl get /foo/bar | grep 'Hello world'")
  '';
})

multi node etcd

# This test runs simple etcd cluster

import ../make-test-python.nix ({ pkgs, ... } : let

  runWithOpenSSL = file: cmd: pkgs.runCommand file {
    buildInputs = [ pkgs.openssl ];
  } cmd;

  ca_key = runWithOpenSSL "ca-key.pem" "openssl genrsa -out $out 2048";
  ca_pem = runWithOpenSSL "ca.pem" ''
    openssl req \
      -x509 -new -nodes -key ${ca_key} \
      -days 10000 -out $out -subj "/CN=etcd-ca"
  '';
  etcd_key = runWithOpenSSL "etcd-key.pem" "openssl genrsa -out $out 2048";
  etcd_csr = runWithOpenSSL "etcd.csr" ''
    openssl req \
       -new -key ${etcd_key} \
       -out $out -subj "/CN=etcd" \
       -config ${openssl_cnf}
  '';
  etcd_cert = runWithOpenSSL "etcd.pem" ''
    openssl x509 \
      -req -in ${etcd_csr} \
      -CA ${ca_pem} -CAkey ${ca_key} \
      -CAcreateserial -out $out \
      -days 365 -extensions v3_req \
      -extfile ${openssl_cnf}
  '';

  etcd_client_key = runWithOpenSSL "etcd-client-key.pem"
    "openssl genrsa -out $out 2048";

  etcd_client_csr = runWithOpenSSL "etcd-client-key.pem" ''
    openssl req \
      -new -key ${etcd_client_key} \
      -out $out -subj "/CN=etcd-client" \
      -config ${client_openssl_cnf}
  '';

  etcd_client_cert = runWithOpenSSL "etcd-client.crt" ''
    openssl x509 \
      -req -in ${etcd_client_csr} \
      -CA ${ca_pem} -CAkey ${ca_key} -CAcreateserial \
      -out $out -days 365 -extensions v3_req \
      -extfile ${client_openssl_cnf}
  '';

  openssl_cnf = pkgs.writeText "openssl.cnf" ''
    ions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = digitalSignature, keyEncipherment
    extendedKeyUsage = serverAuth, clientAuth
    subjectAltName = @alt_names
    [alt_names]
    DNS.1 = node1
    DNS.2 = node2
    DNS.3 = node3
    IP.1 = 127.0.0.1
  '';

  client_openssl_cnf = pkgs.writeText "client-openssl.cnf" ''
    ions = v3_req
    distinguished_name = req_distinguished_name
    [req_distinguished_name]
    [ v3_req ]
    basicConstraints = CA:FALSE
    keyUsage = digitalSignature, keyEncipherment
    extendedKeyUsage = clientAuth
  '';

  nodeConfig = {
    services = {
      etcd = {
        enable = true;
        keyFile = etcd_key;
        certFile = etcd_cert;
        trustedCaFile = ca_pem;
        clientCertAuth = true;
        listenClientUrls = ["https://127.0.0.1:2379"];
        listenPeerUrls = ["https://0.0.0.0:2380"];
      };
    };

    environment.variables = {
      ETCD_CERT_FILE = "${etcd_client_cert}";
      ETCD_KEY_FILE = "${etcd_client_key}";
      ETCD_CA_FILE = "${ca_pem}";
      ETCDCTL_ENDPOINTS = "https://127.0.0.1:2379";
      ETCDCTL_CACERT = "${ca_pem}";
      ETCDCTL_CERT = "${etcd_cert}";
      ETCDCTL_KEY = "${etcd_key}";
    };

    networking.firewall.allowedTCPPorts = [ 2380 ];
  };
in {
  name = "etcd-cluster";

  meta = with pkgs.lib.maintainers; {
    maintainers = [ offline ];
  };

  nodes = {
    node1 = { ... }: {
      require = [nodeConfig];
      services.etcd = {
        initialCluster = ["node1=https://node1:2380" "node2=https://node2:2380"];
        initialAdvertisePeerUrls = ["https://node1:2380"];
      };
    };

    node2 = { ... }: {
      require = [nodeConfig];
      services.etcd = {
        initialCluster = ["node1=https://node1:2380" "node2=https://node2:2380"];
        initialAdvertisePeerUrls = ["https://node2:2380"];
      };
    };

    node3 = { ... }: {
      require = [nodeConfig];
      services.etcd = {
        initialCluster = ["node1=https://node1:2380" "node2=https://node2:2380" "node3=https://node3:2380"];
        initialAdvertisePeerUrls = ["https://node3:2380"];
        initialClusterState = "existing";
      };
    };
  };

  testScript = ''
    with subtest("should start etcd cluster"):
        node1.start()
        node2.start()
        node1.wait_for_unit("etcd.service")
        node2.wait_for_unit("etcd.service")
        node2.wait_until_succeeds("etcdctl endpoint status")
        node1.succeed("etcdctl put /foo/bar 'Hello world'")
        node2.succeed("etcdctl get /foo/bar | grep 'Hello world'")

    with subtest("should add another member"):
        node1.wait_until_succeeds("etcdctl member add node3 --peer-urls=https://node3:2380")
        node3.start()
        node3.wait_for_unit("etcd.service")
        node3.wait_until_succeeds("etcdctl member list | grep 'node3'")
        node3.succeed("etcdctl endpoint status")

    with subtest("should survive member crash"):
        node3.crash()
        node1.succeed("etcdctl endpoint status")
        node1.succeed("etcdctl put /foo/bar 'Hello degraded world'")
        node1.succeed("etcdctl get /foo/bar | grep 'Hello degraded world'")
  '';
})

Impermanence Mode

• Erase Your Darlings
• Impermanence NixOS Wiki
• NixOS Installation Guide
• connect to internet with wpa_supplicant

{ pkgs ? import <nixpkgs> { } }:

let
  pythonEnv = pkgs.python311.withPackages (ps: with ps; [
    ipykernel
    pandas
    pymysql
    prophet
    ipywidgets
    notebook
    scikit-learn
    torch
  ]); 
in
pkgs.mkShell {
  buildInputs = with pkgs; with python3Packages; [
    pythonEnv
    # tensordict
  ];
  shellHook = ''
  export DYLD_LIBRARY_PATH=${pkgs.lib.makeLibraryPath [ pkgs.python3Packages.pytorch ]}
  ${pythonEnv}/bin/python3 -m venv  --system-site-packages .venv
  # activate the virtual environment
  source .venv/bin/activate
  export PYTHONHOME=${pythonEnv}

  '';
}