Skip to content

Instantly share code, notes, and snippets.

@harssh
Forked from noelbundick/README.md
Created September 26, 2023 02:58

Revisions

  1. @noelbundick noelbundick created this gist Oct 14, 2021.
    135 changes: 135 additions & 0 deletions README.md
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,135 @@
    # Optimizing Rust container builds

    I'm a Rust newbie, and one of the things that I've found frustrating is that the default `docker build` experience is **extremely** slow. As it downloads crates, then dependencies, then finally my app - I often get distracted, start doing something else, then come back several minutes later and forget what I was doing

    Recently, I had the idea to make it a little better by combining multistage builds with some of the amazing features from BuildKit. Specifically, cache mounts, which let a build container cache directories for compilers & package managers. Here's a quick annotated before & after from a real app I encountered.

    ## Before

    This is a standard enough multistage Dockerfile. Nothing seemingly terrible or great here - just a normal build stage, and a smaller runtime stage.

    ```Dockerfile
    FROM rust:1.55 AS build

    WORKDIR /app

    # The app has 2 parts: an application named "api", and a lib named "game"

    # Copy the sources
    COPY ./api ./api
    COPY ./game ./game

    # Build the app
    WORKDIR /app/api
    RUN cargo build --release

    # Use a slim Dockerfile with just our app to publish
    FROM debian:buster-slim AS app

    COPY --from=build /app/target/release/my-app /

    CMD ["/my-app"]
    ```

    This corresponds to the following build times

    ```shell
    # Let's pre-pull the bases so we don't unnecessarily penalize the first build
    docker pull rust:1.55
    docker pull debian:buster-slim

    # First build from scratch
    time docker build .

    real 5m43.506s
    user 0m1.239s
    sys 0m0.872s

    # Change a file in api/src, and build again
    time docker build .

    real 5m44.731s
    user 0m1.199s
    sys 0m0.938s
    ```

    Wow, 5 minutes. Yes, I'm probably doing `cargo build` outside of Docker and the real effects aren't this drastic, but this is an eternity for my short attention span. This is our baseline - let's see if we can improve it.

    ## After

    Here we're going to keep multistage builds, but we'll make a few changes:

    1. Split layers so that we cache compiled dependencies. Turns out this is harder in Rust than other languages.
    2. Use BuildKit + [cache mounts](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md#run---mounttypecache). This will save us some download time when we have to rebuild dependencies

    ```Dockerfile
    # syntax=docker/dockerfile:1.3-labs

    # The above line is so we can use can use heredocs in Dockerfiles. No more && and \!
    # https://www.docker.com/blog/introduction-to-heredocs-in-dockerfiles/

    FROM rust:1.55 AS build

    # Capture dependencies
    COPY Cargo.toml Cargo.lock /app/

    # We create a new lib and then use our own Cargo.toml
    RUN cargo new --lib /app/game
    COPY game/Cargo.toml /app/game/

    # We do the same for our app
    RUN cargo new /app/api
    COPY api/Cargo.toml /app/api/

    # This step compiles only our dependencies and saves them in a layer. This is the most impactful time savings
    # Note the use of --mount=type=cache. On subsequent runs, we'll have the crates already downloaded
    WORKDIR /app/api
    RUN --mount=type=cache,target=/usr/local/cargo/registry cargo build --release

    # Copy our sources
    COPY ./api /app/api
    COPY ./game /app/game

    # A bit of magic here!
    # * We're mounting that cache again to use during the build, otherwise it's not present and we'll have to download those again - bad!
    # * EOF syntax is neat but not without its drawbacks. We need to `set -e`, otherwise a failing command is going to continue on
    # * Rust here is a bit fiddly, so we'll touch the files (even though we copied over them) to force a new build
    RUN --mount=type=cache,target=/usr/local/cargo/registry <<EOF
    set -e
    # update timestamps to force a new build
    touch /app/game/src/lib.rs /app/api/src/main.rs
    cargo build --release
    EOF

    CMD ["/app/target/release/my-app"]

    # Again, our final image is the same - a slim base and just our app
    FROM debian:buster-slim AS app
    COPY --from=build /app/target/release/my-app /my-app
    CMD ["/my-app"]
    ```

    And the big test - did it help at all? Let's see

    ```shell
    # We have rust / debian pulled from before

    # We need to use BuildKit for these features, so let's turn that on
    export DOCKER_BUILDKIT=1

    # Build from scratch!
    time docker build .

    real 5m51.538s
    user 0m1.209s
    sys 0m0.933s

    # The big moment - change a file in src and rebuild
    time docker build .

    real 0m36.053s
    user 0m0.148s
    sys 0m0.145s
    ```

    Great success! Container build times dropped from `5m44s` to `0m36s`!