ImageMagick Static Binaries for AWS Lambda

imagemagick.bash

#!/usr/bin/env bash

# Must be run on an Amazon Linux AMI that matches AWS Lambda's runtime which can be found at:
# https://docs.aws.amazon.com/lambda/latest/dg/current-supported-versions.html
#
# As of May 21, 2019, this is:
# Amazon Linux AMI 2018.03.0 (ami-0756fbca465a59a30)
#
# You need to prepend PATH with the folder containing these binaries in your Lambda function 
# to ensure these newer binaries are used.
#
# In a NodeJS runtime, you would add something like the following to the top of
# your Lambda function file:
# process.env['PATH'] = process.env['LAMBDA_TASK_ROOT'] + '/imagemagick/bin:' + process.env['PATH']
#
# This works with both ImageMagick v6.x and v7.x
# version=6.9.10-23
version=7.0.8-45

sudo yum -y install libpng-devel libjpeg-devel libtiff-devel gcc
curl -O https://imagemagick.org/download/ImageMagick-$version.tar.gz 
tar zxvf ImageMagick-$version.tar.gz
cd ImageMagick-$version 
./configure --prefix=/var/task/imagemagick --enable-shared=no --enable-static=yes
make
sudo make install
tar zcvf ~/imagemagick.tgz /var/task/imagemagick/

Instructions for Python 3.8

My use-case was to convert pages in a PDF to PNG via wand. I followed the instructions provided by @chaddjohnson at https://gist.github.com/bensie/56f51bc33d4a55e2fc9a#gistcomment-3133859 but was getting errors when using wand. Following are the instructions on how I was able to resolve those by doing some slight adjustments:

1. Start an EC2 instance and SSH into it. I used the AMI amzn2-ami-hvm-2.0.20210126.0-x86_64-gp2.
2. Download ImageMagick 6.9.11.

   wget https://download.imagemagick.org/ImageMagick/download/ImageMagick-6.9.11-60.tar.gz

3. Extract the folder.

   tar zxvf ImageMagick-6.9.11-60.tar.gz

4. cd into the extracted folder.

   cd ImageMagick-6.9.11-60

5. Edit the policy.xml file to allow PDF to PNG conversion.

   nano config/policy.xml

   I copy-pasted the following content but you can modify it as needed.

   <?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE policymap [
   <!ELEMENT policymap (policy)+>
   <!ELEMENT policy (#PCDATA)>
   <!ATTLIST policy domain (delegate|coder|filter|path|resource) #IMPLIED>
   <!ATTLIST policy name CDATA #IMPLIED>
   <!ATTLIST policy rights CDATA #IMPLIED>
   <!ATTLIST policy pattern CDATA #IMPLIED>
   <!ATTLIST policy value CDATA #IMPLIED>
   ]>
   <!--
     Configure ImageMagick policies.
   
     Domains include system, delegate, coder, filter, path, or resource.
   
     Rights include none, read, write, and execute.  Use | to combine them,
     for example: "read | write" to permit read from, or write to, a path.
   
     Use a glob expression as a pattern.
   
     Suppose we do not want users to process MPEG video images:
   
       <policy domain="delegate" rights="none" pattern="mpeg:decode" />
   
     Here we do not want users reading images from HTTP:
   
       <policy domain="coder" rights="none" pattern="HTTP" />
   
     Lets prevent users from executing any image filters:
   
       <policy domain="filter" rights="none" pattern="*" />
   
     The /repository file system is restricted to read only.  We use a glob
     expression to match all paths that start with /repository:
     
       <policy domain="path" rights="read" pattern="/repository/*" />
   
     Let's prevent possible exploits by removing the right to use indirect reads.
   
       <policy domain="path" rights="none" pattern="@*" />
   
     Any large image is cached to disk rather than memory:
   
       <policy domain="resource" name="area" value="1GB"/>
   
     Define arguments for the memory, map, area, width, height, and disk resources
     with SI prefixes (.e.g 100MB).  In addition, resource policies are maximums
     for each instance of ImageMagick (e.g. policy memory limit 1GB, -limit 2GB
     exceeds policy maximum so memory limit is 1GB).
   -->
   <policymap>
     <!-- <policy domain="resource" name="temporary-path" value="/tmp"/> -->
     <policy domain="resource" name="memory" value="256MiB"/>
     <policy domain="resource" name="map" value="512MiB"/>
     <policy domain="resource" name="width" value="16KP"/>
     <policy domain="resource" name="height" value="16KP"/>
     <policy domain="resource" name="area" value="128MB"/>
     <policy domain="resource" name="disk" value="1GiB"/>
     <!-- <policy domain="resource" name="file" value="768"/> -->
     <!-- <policy domain="resource" name="thread" value="4"/> -->
     <!-- <policy domain="resource" name="throttle" value="0"/> -->
     <!-- <policy domain="resource" name="time" value="3600"/> -->
     <!-- <policy domain="system" name="precision" value="6"/> -->
     <!-- not needed due to the need to use explicitly by mvg: -->
     <!-- <policy domain="delegate" rights="none" pattern="MVG" /> -->
     <!-- use curl -->
     <policy domain="delegate" rights="none" pattern="URL" />
     <policy domain="delegate" rights="none" pattern="HTTPS" />
     <policy domain="delegate" rights="none" pattern="HTTP" />
     <!-- in order to avoid to get image with password text -->
     <policy domain="path" rights="none" pattern="@*"/>
     <policy domain="cache" name="shared-secret" value="passphrase" stealth="true"/>
     <!-- disable ghostscript format types -->
     <policy domain="coder" rights="none" pattern="PS" />
     <policy domain="coder" rights="none" pattern="EPI" />
     <policy domain="coder" rights="read|write" pattern="PDF" />
     <policy domain="coder" rights="none" pattern="XPS" />
     <policy domain="coder" rights="read|write" pattern="LABEL" />
   </policymap>

6. Configure and install ImageMagick.

   ./configure --prefix=/var/task/imagemagick --sysconfdir=/etc --datadir=/usr/share --includedir=/usr/include --libdir=/usr/lib64 --libexecdir=/usr/libexec --localstatedir=/var --sharedstatedir=/var/lib --mandir=/usr/share/man --infodir=/usr/share/info --enable-shared=no --enable-static=yes --with-modules --with-perl=no --with-x=no --with-gslib=no --with-lcms --without-rsvg --with-xml --without-dps --disable-hdri --with-quantum-depth=8 --disable-openmp
   make
   sudo make install

7. Copy the required .so files.

   mkdir lib
   cd /usr/lib64/
   cp -L libbz2.so.1 libexpat.so.1 libfontconfig.so.1 libfreetype.so.6 libgs.so.9 libjbig.so.2.0 libjpeg.so.62 liblcms2.so.2 liblzma.so.5 libpng15.so.15 libtiff.so.5 libxml2.so.2 libMagickCore-6.Q16.so.6 libMagickWand-6.Q16.so.6 libXext.so.6 libXt.so.6 libltdl.so.7 libSM.so.6 libICE.so.6 libX11.so.6 libgomp.so.1 libuuid.so.1 libxcb.so.1 libXau.so.6 libMagickCore-6.Q8.so.6 libMagickWand-6.Q8.so.6 libm.so.6 libz.so.1 libjasper.so.1 /home/ec2-user/lib/
   cp -r ImageMagick-6.9.10/ ImageMagick-6.9.11/ /home/ec2-user/lib/
   cd /home/ec2-user
   tar zcf lib.tar.gz lib/

   Copy the lib.tar.gz file from the server to your local machine.
8. Copy the required binary files.

   cd /var/task/imagemagick
   sudo tar zcf bin.tar.gz bin/
   cp bin.tar.gz /home/ec2-user/bin.tar.gz 

   Copy the bin.tar.gz file from the server to your local machine.
9. Copy the XML files required by ImageMagick.

   cd /etc/
   sudo tar zcf etc.tar.gz ImageMagick-6/
   cp etc.tar.gz /home/ec2-user/etc.tar.gz

   Copy the etc.tar.gz file from the server to your local machine.
10. Close the SSH session.
11. On your local machine, extract the contents of the 3 *.tar.gz files.
12. Download ghostscript from https://github.com/ArtifexSoftware/ghostpdl-downloads/releases/download/gs9533/ghostscript-9.53.3-linux-x86_64.tgz and extract the ghostscript binary into the bin/ folder and rename it to gs. Run chmod +x bin/gs to make it executable.
13. Compress the 3 - lib, bin and etc - folders into a ZIP file. The tree structure of the ZIP file would look like

    file.zip/
    |-- bin
    |   |-- convert
    |   |-- ...
    |   `-- gs
    |-- etc
    |   `-- ImageMagick-6
    |       |-- coder.xml
    |       |-- ...
    |       `-- type.xml
    `-- lib
        |-- ImageMagick-6.9.10
        |   |-- config-Q16
        |   |   `-- configure.xml
        |   `-- modules-Q16
        |       |-- coders
        |       |   |-- aai.la
        |       |   |-- ...
        |       |   `-- yuv.so
        |       `-- filters
        |           |-- analyze.la
        |           `-- analyze.so
        |-- ImageMagick-6.9.11
        |   |-- config-Q8
        |   |   `-- configure.xml
        |   `-- modules-Q8
        |       |-- coders
        |       |   |-- aai.la
        |       |   |-- ...
        |       |   `-- yuv.so
        |       `-- filters
        |           |-- analyze.la
        |           `-- analyze.so
        |-- libICE.so.6
        |-- ...
        `-- libz.so.1
    

    I have used ... wherever the folder contained more than 2 files to denote that there are more files present.
14. Create a Python 3.8 runtime compatible layer on AWS Lambda and use the ZIP created in step 13.
15. Add the layer to your AWS Lambda function code.
16. Update environment variables in your lambda function.

    import os
    os.environ["PATH"] = f"/opt/bin:{os.environ['PATH']}"
    os.environ["LD_LIBRARY_PATH"] = f"/opt/lib:{os.environ['LD_LIBRARY_PATH']}"
    os.environ["MAGICK_HOME"] = "/opt/"
    os.environ["WAND_MAGICK_LIBRARY_SUFFIX"] = "-6.Q8"
    os.environ["MAGICK_CONFIGURE_PATH"] = "/opt/etc/ImageMagick-6/"
    os.environ["MAGICK_CODER_MODULE_PATH"] = "/opt/lib/ImageMagick-6.9.11/modules-Q8/coders/"

Note: If the size of the uncompressed ZIP file is too large and you reach AWS Lambda size limits, remove the binaries that you don't need from the bin/ folder. In my case, I only kept Magick-config, MagickCore-config, MagickWand-config, Wand-config, convert and gs and removed others.

NOTE: I also have a ready-to-use ZIP file uploaded at https://github.com/samkit-jain/aws-lambda-imagemagick-ghostscript

Where to Place this Code ?

16. Update environment variables in your lambda function.

import os
os.environ["PATH"] = f"/opt/bin:{os.environ['PATH']}"
os.environ["LD_LIBRARY_PATH"] = f"/opt/lib:{os.environ['LD_LIBRARY_PATH']}"
os.environ["MAGICK_HOME"] = "/opt/"
os.environ["WAND_MAGICK_LIBRARY_SUFFIX"] = "-6.Q8"
os.environ["MAGICK_CONFIGURE_PATH"] = "/opt/etc/ImageMagick-6/"
os.environ["MAGICK_CODER_MODULE_PATH"] = "/opt/lib/ImageMagick-6.9.11/modules-Q8/coders/"

@R-Harishkumar You may place the code at the top of the Python file.

@samkit-jain

I created the layer with your ZIP file and added it to my lambda, along with the code from step 16,
but when I try to do: from wand.image import Image
I get this error: Unable to import module 'lambda_function': No module named 'wand'

And if I try to package wand myself and add it then I get the error where wand doesn't find the shared library for ImageMagick.

Is there any other step you took to get Wand working in AWS Lambda?

Hi @ClickheadZ Adding the ZIP file as a layer would give you the required libraries that you want at the OS level to communicate with imagemagick, gs, ... You still need the pip package files in your lambda archive. Did you add Wand as a dependency in your lambda function archive? Example here.

@samkit-jain Thanks for the response! I had not added it properly, I repackaged and deployed a zip now and I'm getting a different error, not sure what I did wrong.

Here are the steps I took to package Wand:

• Connect to an EC2 instance
• pip install --target=package wand, pip install --target=package magickwand
• scp the package to my computer, and zip the package along with my lambda function as per AWS zip deployment instructions

In lambda console, my lambda structure now looks like this:

myLambdaFunction
      - magickwand
      - magickwand-0.2.dist-info
      - wand
      - Wand-0.6.7.dist-info
      lambda_function.py

And when i try from wand.image import Image I now get the error message : Unable to import module 'lambda_function': MagickWand shared library not found.\nYou probably had not installed ImageMagick library despite the fact that I have deployed the imagemagick/ghostscript zip as a layer for this lambda and added it. Did I do something wrong in the Wand packaging or is the problem my lambda layer?

@ClickheadZ Facing same issue, were you able to find a fix?

Thanks~ It's work for me !!! But I found a small problem while using it：
curl download url: curl -O https://imagemagick.org/download/ImageMagick-$version.tar.gz , it will return 404 http status...
the ture url is: https://download.imagemagick.org/archive/ImageMagick-$version.tar.xz

@samkit-jain please can you help here , the layer you build is partially working for me as it doent have webp support , i need webp support so I followed the same steps but in docker container on linux , the difference is imagemagick is now updated to version 7 , there is no coders file at the same place as in 6 so i am also not sure what env vars to set in lmabda as well-

I am getting error on lambda ->

{
  "errorMessage": "Unable to import module 'lambda_function': MagickWand shared library not found.\nYou probably had not installed ImageMagick library.\nTry to install:\n  [https://docs.wand-py.org/en/latest/guide/install.html"](https://docs.wand-py.org/en/latest/guide/install.html%22),
  "errorType": "Runtime.ImportModuleError",
  "requestId": "",
  "stackTrace": []
}

this is my docker file ->

FROM amazonlinux:2 as builder
 
# Install development tools and libraries

RUN yum update -y && \

    yum install -y \

        gcc \

        gcc-c++ \

        make \

        autoconf \

        automake \

        libtool \

        git \

        wget \

        tar \

        gzip \

        zip \

        unzip \

        python3 \

        python3-devel \

        python3-pip \

        yum-utils
 
# Create directory structure for the Lambda layer

RUN mkdir -p /opt/python
 
# Install libwebp

WORKDIR /tmp

RUN wget https://storage.googleapis.com/downloads.webmproject.org/releases/webp/libwebp-1.3.0.tar.gz && \

    tar xzf libwebp-1.3.0.tar.gz && \

    cd libwebp-1.3.0 && \

    ./configure --prefix=/opt && \

    make && \

    make install
 
# Install ImageMagick with WebP support

WORKDIR /tmp

RUN wget https://imagemagick.org/archive/ImageMagick.tar.gz && \

    tar xzf ImageMagick.tar.gz && \

    cd ImageMagick-* && \

    ./configure --prefix=/opt \

                --with-webp \

                --disable-static \

                --enable-shared \

                --without-perl \

                --without-x \

                --without-xml \

                --without-pango && \

    make && \

    make install
 
# Install Wand (Python binding for ImageMagick)

RUN pip3 install --target=/opt/python wand
 
# Create the layer structure as expected by Lambda

FROM amazonlinux:2
 
# Install zip in the second stage

RUN yum update -y && \

    yum install -y zip
 
# Copy built libraries and python packages from the builder

COPY --from=builder /opt /opt
 
# Set up a directory for the final zip

RUN mkdir -p /layer
 
# Create layer.zip with the right structure

RUN cd /opt && zip -r9 /layer/layer.zip .
 
WORKDIR /layer

CMD ["/bin/bash"]

 

My lambda is running on python3.10 , i also tried the above with FROM public.ecr.aws/lambda/python:3.10 as builder image but same error.

please can someone help here , Thanks in Advance

您好，您的邮件我已收到，我会尽快与您联系，谢谢来信

@siddhant-grover I have not worked on this project for a long time so can't help with full debugging. However, it looks like the coders file for imagemagick-7 might be somewhere at ImageMagick-7.1.1/modules-Q16/coders or similar.

Also, AWS Lambda now has support for Docker images so you might be better off building a Docker and using that instead of adding a layer.