Compiling NGINX module as dynamic module for use in docker

README.md

Compiling NGINX module as dynamic module for use in docker

NGINX offers some very convenient Docker containers, but it is not documented how to extend the functionality of those containers without re-compiling NGINX yourself and continuously adapting your Dockerfile to match the ones in https://github.com/nginxinc/docker-nginx/.

This gist offers a way to build a NGINX ‘dynamic module’ against the widely used “nginx:alpine” docker image. Simply copy this Dockerfile & replace the arguments of the “wget” & “tar” & “configure” commands to include the module you want to build, then adjust the COPY command copying the module (.so file). In this example the NCHAN module is used as an example.

If you found this helpful, please leave a comment / reaction here! :love:

Dockerfile

FROM nginx:alpine AS builder

# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0

# Our NCHAN version
ENV NCHAN_VERSION 1.1.15

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
  wget "https://github.com/slact/nchan/archive/v${NCHAN_VERSION}.tar.gz" -O nchan.tar.gz

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
  gcc \
  libc-dev \
  make \
  openssl-dev \
  pcre-dev \
  zlib-dev \
  linux-headers \
  curl \
  gnupg \
  libxslt-dev \
  gd-dev \
  geoip-dev

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
	tar -zxC /usr/src -f nginx.tar.gz && \
  tar -xzvf "nchan.tar.gz" && \
  NCHANDIR="$(pwd)/nchan-${NCHAN_VERSION}" && \
  cd /usr/src/nginx-$NGINX_VERSION && \
  ./configure --with-compat $CONFARGS --add-dynamic-module=$NCHANDIR && \
  make && make install

FROM nginx:alpine
# Extract the dynamic module NCHAN from the builder image
COPY --from=builder /usr/local/nginx/modules/ngx_nchan_module.so /usr/local/nginx/modules/ngx_nchan_module.so
RUN rm /etc/nginx/conf.d/default.conf

COPY nginx.conf /etc/nginx/nginx.conf
COPY default.conf /etc/nginx/conf.d/default.conf
EXPOSE 80
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]

nginx.conf

# Addition: load NCHAN
load_module /usr/local/nginx/modules/ngx_nchan_module.so;

user  nginx;
worker_processes  5; # Addition: ultiple workers for NCHAN
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;

events {
    worker_connections  1024;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

@mtgq this gist provides a way to build modules externally, but I’m not maintaining the gist carefully. Previously this worked, maybe a recent update broke it: you can retry with an older base image.

If you find a fix please let us know in this thread and will add the fix.

Note that I’m not affiliated with NGINX in any way.

my docker version is 20.10.7, build f0df350 , I found another solution https://github.com/nginxinc/docker-nginx/blob/master/modules/README.md, but I don't like this method, I try to solve the error

error!!

yeah same here

Thanks @hermanbanken it is very helpful .
I used it to build spnego dynamic module.

@mtgq @brockoffdev
I fixed the error by removing the -fomit-frame-pointer and -g from CONFARGS

RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} \

Thanks @hermanbanken it is very helpful .
I used it to build spnego dynamic module.

@mtgq @brockoffdev
I fixed the error by removing the -fomit-frame-pointer and -g from CONFARGS

RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} \

Thank you very much. I will try again

This appears to have some linking problems (perhaps because of alpine's use of musl?):

/ # nginx
2021/08/12 18:54:08 [emerg] 7#7: dlopen() "/etc/nginx/modules/ngx_http_remote_passwd.so" failed (Error relocating /etc/nginx/modules/ngx_http_remote_passwd.so: ngx_http_remote_passwd: symbol not found) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/etc/nginx/modules/ngx_http_remote_passwd.so" failed (Error relocating /etc/nginx/modules/ngx_http_remote_passwd.so: ngx_http_remote_passwd: symbol not found) in /etc/nginx/nginx.conf:1
/ # musl-ldd

I couldn't make it work with nginx 1.21.3. But it worked with 1.20.1

FROM nginx:1.20.1-alpine AS builder

# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0

# Our NCHAN version
ENV NGX_CACHE_PURGE 2.5.1

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
  wget "https://github.com/nginx-modules/ngx_cache_purge/archive/refs/tags/${NGX_CACHE_PURGE}.tar.gz" -O ngx-cache-purge.tar.gz

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
    gcc \
    libc-dev \
    make \
    openssl-dev \
    pcre-dev \
    zlib-dev \
    linux-headers \
    libxslt-dev \
    gd-dev \
    geoip-dev \
    perl-dev \
    libedit-dev \
    mercurial \
    bash \
    alpine-sdk \
    findutils

SHELL ["/bin/ash", "-eo", "pipefail", "-c"]

RUN rm -rf /usr/src/nginx /usr/src/ngx-cache-purge && mkdir -p /usr/src/nginx /usr/src/ngx-cache-purge && \
    tar -zxC /usr/src/nginx -f nginx.tar.gz && \
    tar -xzC /usr/src/ngx-cache-purge -f ngx-cache-purge.tar.gz

WORKDIR /usr/src/nginx/nginx-${NGINX_VERSION}

# Reuse same cli arguments as the nginx:alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') &&  \
    sh -c "./configure --with-compat $CONFARGS --add-dynamic-module=/usr/src/ngx-cache-purge/*" && make modules

FROM nginx:1.20.1-alpine
# Extract the dynamic module NCHAN from the builder image
COPY --from=builder /usr/src/nginx/nginx-${NGINX_VERSION}/objs/*_module.so /etc/nginx/modules/

COPY ./nginx/nginx.conf /etc/nginx/nginx.conf

nginx.conf:

# Addition: load ngx_http_cache_purge_module
load_module /etc/nginx/modules/ngx_http_cache_purge_module.so;

# cat /etc/nginx/nginx.conf from nginx:1.20.1-alpine image
user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

For the one who got make: Operation not permitted, after make modules the solution is to upgrade docker

-> alpinelinux/docker-alpine#182 (comment)

Tried to compile and use ngx_http_substitutions_filter_module with your technic:

# See https://gist.github.com/hermanbanken/96f0ff298c162a522ddbba44cad31081
FROM nginx:stable-alpine AS builder

# Download sources
RUN apk update && \
    apk upgrade && \
    apk add --no-cache --virtual .build-deps git gcc libc-dev make openssl-dev pcre-dev zlib-dev linux-headers curl gnupg libxslt-dev gd-dev geoip-dev curl

RUN git clone git://github.com/yaoweibin/ngx_http_substitutions_filter_module.git && \
    curl "https://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -o nginx.tar.gz

# Reuse same cli arguments as the nginx:stable-alpine image used to build
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') \
    mkdir /usr/src && \
    tar -zxC /usr/src -f nginx.tar.gz && \
    SFM_DIR="$(pwd)/ngx_http_substitutions_filter_module" && \
    cd /usr/src/nginx-$NGINX_VERSION && \
    ./configure --with-compat $CONFARGS --add-dynamic-module=$SFM_DIR && \
    make && make install \
    
FROM nginx:stable-alpine
# Extract the dynamic module from the builder image
COPY --from=builder /usr/src/nginx-$NGINX_VERSION/objs/ngx_http_subs_filter_module_modules.o /usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o
COPY nginx.conf /etc/nginx/nginx.conf
COPY default.conf /etc/nginx/conf.d/default.conf

The /usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o file does exists and have a size (not 0) in the final Docker image but when I run the image I have to following error:

$ docker run -it -p 3000:80 subfilters:0.0.1
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf differs from the packaged version
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2022/03/10 13:14:35 [emerg] 1#1: dlopen() "/usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o" failed (Error loading shared library /usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o: Exec format error) in /etc/nginx/nginx.conf:1
nginx: [emerg] dlopen() "/usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o" failed (Error loading shared library /usr/local/nginx/modules/objs/ngx_http_subs_filter_module_modules.o: Exec format error) in /etc/nginx/nginx.conf:1

I have no idea how to fix that 😕

This is amazing, but I feel like this could be slimmed down even more or otherwise explained the other parts. like what if you don't want to use a nginx.conf and only have a default.conf?

This is the error i get

executor failed running [/bin/sh -c CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p')     CONFARGS=${CONFARGS/-Os -fomit-frame-pointer/-Os} &&     mkdir /usr/src &&  	tar -zxC /usr/src -f nginx.tar.gz &&   tar -xzvf "hmnm.tar.gz" &&   HEADER_MORE_VERSION="$(pwd)/headers-more-nginx-module" &&   cd /usr/src/nginx-$NGINX_VERSION &&   ./configure --with-compat $CONFARGS --add-dynamic-module=$HEADER_MORE_VERSION &&     make modules &&     mv ./objs/*.so /]: exit code: 1

Thanks @hermanbanken and everyone else commenting in this gist. 👍
Very helpful resource, in this hellhole 😡 of trying to compile NGINX with additional/patched modules.

In case it helps anyone else, here is my current Docker build to:

• Compile the latest NGINX (works with version 1.23.2)
• Apply patch from ngx_http_proxy_connect_module and add it as a dynamic module
• Copy over the new dynamic module and also the new NGINX binary (due to patching of other modules) from the builder stage.

FROM nginx:alpine AS builder

# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0

# Additional module version
ENV PROXY_CONNECT_VERSION 102101

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
  git \
  gcc \
  libc-dev \
  make \
  openssl-dev \
  pcre2-dev \
  zlib-dev \
  linux-headers \
  libxslt-dev \
  gd-dev \
  geoip-dev \
  libedit-dev \
  bash \
  alpine-sdk \
  findutils

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
  git clone https://github.com/chobits/ngx_http_proxy_connect_module.git

# Reuse same cli arguments as the nginx:alpine image used to build (but remove the erroneous -fomit-frame-pointer)
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') && \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} && \
    mkdir -p /usr/src && \
    tar -zxC /usr/src -f nginx.tar.gz && \
    cd /usr/src/nginx-$NGINX_VERSION && \
    patch -p1 < /ngx_http_proxy_connect_module/patch/proxy_connect_rewrite_${PROXY_CONNECT_VERSION}.patch && \
    ./configure --with-compat $CONFARGS --add-dynamic-module=/ngx_http_proxy_connect_module && \
    make && make install

FROM nginx:alpine
# Copy new dynamic module ngx_http_proxy_connect_module
COPY --from=builder /usr/lib/nginx/modules/ngx_http_proxy_connect_module.so /usr/lib/nginx/modules/ngx_http_proxy_connect_module.so
# Copy new nginx binary, due to patching for ngx_http_proxy_connect_module
COPY --from=builder /usr/sbin/nginx /usr/sbin/nginx

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 8080
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]

Thanks @hermanbanken and everyone else commenting in this gist. 👍 Very helpful resource, in this hellhole 😡 of trying to compile NGINX with additional/patched modules.

In case it helps anyone else, here is my current Docker build to:

great script, here is my script of geoip:

FROM nginx:alpine AS builder

# nginx:alpine contains NGINX_VERSION environment variable, like so:
# ENV NGINX_VERSION 1.15.0

# Additional module version
ENV PROXY_CONNECT_VERSION 3.4

# For latest build deps, see https://github.com/nginxinc/docker-nginx/blob/master/mainline/alpine/Dockerfile
RUN apk add --no-cache --virtual .build-deps \
  git \
  gcc \
  libc-dev \
  make \
  openssl-dev \
  pcre2-dev \
  zlib-dev \
  linux-headers \
  libxslt-dev \
  gd-dev \
  geoip-dev \
  libedit-dev \
  bash \
  alpine-sdk \
  findutils \ 
  automake \ 
  autoconf \ 
  libtool

# Download sources
RUN wget "http://nginx.org/download/nginx-${NGINX_VERSION}.tar.gz" -O nginx.tar.gz && \
    git clone --recursive https://github.com/maxmind/libmaxminddb && \
    git clone https://github.com/leev/ngx_http_geoip2_module

RUN cd libmaxminddb && \
    ./bootstrap && \
    ./configure && \
    make && make install

# Reuse same cli arguments as the nginx:alpine image used to build (but remove the erroneous -fomit-frame-pointer)
RUN CONFARGS=$(nginx -V 2>&1 | sed -n -e 's/^.*arguments: //p') && \
    CONFARGS=${CONFARGS/-Os -fomit-frame-pointer -g/-Os} && \
    mkdir -p /usr/src && \
    tar -zxC /usr/src -f nginx.tar.gz && \
    cd /usr/src/nginx-$NGINX_VERSION && \
    ./configure --add-dynamic-module=/ngx_http_geoip2_module && \
    make && make install

FROM nginx:alpine
# Copy new dynamic module ngx_http_geoip2_module
COPY --from=builder /usr/lib/nginx/modules/ngx_http_geoip_module.so /usr/lib/nginx/modules/ngx_http_geoip_module.so
# Copy new nginx binary, due to patching for ngx_http_geoip2_module
COPY --from=builder /usr/sbin/nginx /usr/sbin/nginx

RUN rm /etc/nginx/conf.d/default.conf
COPY nginx.conf /etc/nginx/nginx.conf

EXPOSE 8080
STOPSIGNAL SIGTERM
CMD ["nginx", "-g", "daemon off;"]