A quick tutorial on how to generate a Singularity image with loadcaffee

Singularity

Bootstrap: docker
From: ubuntu:16.04

%runscript
    . /torch/install/bin/torch-activate
    exec /bin/bash


%labels
MAINTAINER [email protected]

%post

    # Directories for Sherlock
    mkdir -p /scratch
    mkdir -p /scratch-local
    mkdir -p /share/PI

    apt-get update && apt-get install -y \
            gcc libprotobuf-dev protobuf-compiler \
            luarocks wget git vim cmake

    # Install torch
    git clone https://github.com/torch/distro.git /torch --recursive && cd /torch
    sed -i 's/sudo//g' install-deps
    yes no | ./install.sh
     . /torch/install/bin/torch-activate
    cd / && git clone https://github.com/szagoruyko/loadcaffe && cd loadcaffe
    luarocks make

singularity_loadcaffee.md

Setup

In this example, we will look at a few ways to build a Singularity image (for running on the Sherlock cluster at Stanford) to implement different machine learning toolkits. Here, the library of interest is loadcaffe.

Install Singularity

You can read full instructions here or just install the development branch as follows:

git clone -b development https://github.com/singularityware/singularity.git
cd singularity
./autogen.sh
./configure --prefix=/usr/local
make
sudo make install

Choose your Docker Image

In our case, we are going to start with an Ubuntu operating system and add to it. Remember that you could also choose a Docker image that already has some machine learning tool (eg tensorflow) in it. For example, this will get you into the tensorflow demo in one line:

singularity shell --nv docker://tensorflow/tensorflow:latest-gpu

Building

Create image

The first step to generate a Singularity image is to use create. You will want to also specify an image size to ensure that it is big enough.

singularity create --size 8000 caffe.img

Boostrap

We will next want to bootstrap, meaning creating a definition file that tells Singularity how to build our image. The core of a bootstrap is a specification file, a build file we name Singularity where we write these instructions. For the most basic of images, ours would now look like this (this is for ubuntu)

Bootstrap:docker
From: ubuntu:16.04

You can also specify a registry - the format for the uri is generally registry/container:tag To run the boostrap, we can now do:

sudo singularity bootstrap caffe.img Singularity

and this will build our image! I have provided in this gist two example specification files. The first is simple and builds a basic loadcaffee image, and the second is a more robust, all encompassing "all the deep learning things" image.

It can be run/shelled/or execed from where is necessary:

# These two are the same thing
./caffe.img
singularity run caffe.img 

# Look inside
singularity shell caffe.img

# Look inside and make changes (must have sudo locally)
sudo singularity shell --writable caffe.img

# Execute a command to the container
singularity exec caffe.img echo "Hello World!"

The build specification (Singularity file) that I used to install torch and the loadcaffe library are included in the second file in this gist.