Skip to content

Instantly share code, notes, and snippets.

@sanromd

sanromd/hpc_setup.md

Last active January 29, 2021 12:56
Show Gist options
  • Save sanromd/13d2daf20677d9eea11e72eacaf8028d to your computer and use it in GitHub Desktop.
Save sanromd/13d2daf20677d9eea11e72eacaf8028d to your computer and use it in GitHub Desktop.
Download ZIP
setup an hpc & learning environment using miniconda, openblas, mpich, petsc, and tensorflow
Raw
hpc_setup.md

setting up an hpc environment

preamble

create a sandbox, a place to store sources, compile, and develop

mkdir -p <path to sandbox>\opt\src

replace <path to sandbox> with the desired path

set the profile file path, e.g.

export profile=~/.zshrc

Install basic libraries

sudo apt install apt-transport-https git wget build-essential gnu-standards \
    linux-headers-$(uname -r) linux-image-$(uname -r) \
    libtinfo5 openssl libssl-dev libcupti-dev libnuma-dev libboost-all-dev \
    libtool autoconf automake autotools-dev valgrind valgrind-mpi

miniconda

download & install miniconda [py 3.8: linux | mac]

install jupyterlab and basic dependencies

conda install jupyterlab nb_conda_kernels

create an envrionment for hpc, e.g. numerics

conda create -n numerics -c conda-forge python=3.9 jupyterlab nb_conda_kernels

Note: for best compatibility use python 3.8

MPICH

install

git clone [email protected]:pmodels/mpich.git 
cp mpich
git checkout v3.4.1
./autogen.sh
./configure --enable-shared --with-threads --enable-shared-libs=gcc \ 
    --enable-fast=all --disable-multi-aliases --enable-fortran=all --enable-cxx \
    --enable-romio \
    CFLAGS='-fPIC -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -D_FILE_OFFSET_BITS=64' \
    FFLAGS='-fPIC' \
    --with-device=ch3 \
    --prefix=<MPICH_DIR>

With gcc compilers 10 and above it is necessary to use the Fortran flag -fallow-argument-mismatch.

make -j$(nproc)
make install

post-installation

add MPICH to PATH and LD_LIBRARY_PATH or ld.conf

echo "\n# MPICH\nexport MPICH_DIR=<MPICH_DIR>"\
    "\nexport PATH=\$MPICH_DIR/bin:\$PATH"\
    "\nexport LD_LIBRARY_PATH=\$MPICH_DIR/lib:\$LD_LIBRARY_PATH" | tee -a $profile > /dev/null

alternatively add the libs via ldconfig

sudo sh -c "echo '$MPICH_DIR/lib' > /etc/ld.so.conf.d/mpich.conf"
sudo ldconfig

save and run sudo ldconfig check that ld is correclty configured sudo ldconfig -p | grep mpich if all goes well the output would look like this:

libmpifort.so.12 (libc6,x86-64) => <MPICH_DIR>/lib/libmpifort.so.12
libmpifort.so (libc6,x86-64) => <MPICH_DIR>/lib/libmpifort.so
libmpicxx.so.12 (libc6,x86-64) => <MPICH_DIR>/lib/libmpicxx.so.12
libmpicxx.so (libc6,x86-64) => <MPICH_DIR>/lib/libmpicxx.so
libmpi.so.12 (libc6,x86-64) => <MPICH_DIR>/lib/libmpi.so.12
libmpi.so (libc6,x86-64) => <MPICH_DIR>/lib/libmpi.so

OpenBLAS

install OpenBLAS

git clone https://github.com/xianyi/OpenBLAS
git checkout v0.3.13 -b v0.3.13

make -j$(nproc) CC=mpicc FF=mpifort

make PREFIX=<BLAS_DIR> install

add libs

sudo sh -c "echo '<BLAS_DIR>/lib' > /etc/ld.so.conf.d/openblas.conf"
sudo ldconfig

check that ld is correclty configured sudo ldconfig -p | grep openblas if all goes well the output would look like this:

libopenblas.so.0 (libc6,x86-64) => <BLAS_DIR>/lib/libopenblas.so.0
libopenblas.so (libc6,x86-64) => <BLAS_DIR>/lib/libopenblas.so

reload the enviroment

HDF5

git clone https://github.com/HDFGroup/hdf5.git

cd hdf5
git checkout 1.12/master

./configure --enable-fortran --enable-optimization=-O3 \
  --enable-parallel --enable-shared \
  --prefix=<HDF5_DIR>

make -j$(nproc)
make -j$(nproc) check
make install

add to path

echo "\n# HDF5 \nexport HDF5_DIR=<HDF5_DIR>"\
    "\nexport PATH=\$HDF5_DIR/bin:\$PATH"\
    "\nexport LD_LIBRARY_PATH=\$HDF5_DIR/lib:\$LD_LIBRARY_PATH" | tee -a $profile > /dev/null

add to the library

sudo sh -c "echo '<HDF_DIR>/lib' > /etc/ld.so.conf.d/hdf5.conf"
sudo ldconfig

check that ld is correclty configured sudo ldconfig -p | grep hdf5 if all goes well the output would look like this:

libhdf5hl_fortran.so.200 (libc6,x86-64) => <HDF_DIR>/lib/libhdf5hl_fortran.so.200
libhdf5hl_fortran.so (libc6,x86-64) => <HDF_DIR>/lib/libhdf5hl_fortran.so
libhdf5_hl.so.200 (libc6,x86-64) => <HDF_DIR>/lib/libhdf5_hl.so.200
libhdf5_hl.so (libc6,x86-64) => <HDF_DIR>/lib/libhdf5_hl.so
libhdf5_fortran.so.200 (libc6,x86-64) => <HDF_DIR>/lib/libhdf5_fortran.so.200
libhdf5_fortran.so (libc6,x86-64) => <HDF_DIR>/lib/libhdf5_fortran.so
libhdf5.so.200 (libc6,x86-64) => <HDF_DIR>/lib/libhdf5.so.200
libhdf5.so (libc6,x86-64) => <HDF_DIR>/lib/libhdf5.so

FFTw

get the latest version from fftw. assuming it is in Downloads

tar -xvf fftw-X.X.X.tar.gz && cd fft-X.X.X
./configure
make -j$(nproc)
make -j$(nproc) check
make install

the second step in make is optional.

post-installation

add to path and ld_library_path

echo "\n# fftw \nexport FFTW_DIR=<FFTW_DIR>"\
    "\nexport PATH=\$FFTW_DIR/bin:\$PATH"\
    "\nexport LD_LIBRARY_PATH=\$FFTW_DIR/lib:\$LD_LIBRARY_PATH" | tee -a $profile > /dev/null

add to the library

sudo sh -c "echo '<FFTW_DIR>/lib' > /etc/ld.so.conf.d/hdf5.conf"
sudo ldconfig

check that ld is correclty configured sudo ldconfig -p | grep fft if all goes well the output would look like this:

libfftw3_threads.so.3 (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3_threads.so.3
libfftw3_threads.so (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3_threads.so
libfftw3_mpi.so.3 (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3_mpi.so.3
libfftw3_mpi.so (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3_mpi.so
libfftw3.so.3 (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3.so.3
libfftw3.so (libc6,x86-64) =><FFTW_DIR>/lib/libfftw3.so

numpy

git clone https://github.com/numpy/numpy && cd numpy
git fetch --all --tags

to check aparticular version (e.g. 1.19.4 into a branch with that name) or stay in master to get the latest development.

git checkout v1.19.5 -b v1.19.5

configure

cp site.cfg.example site.cfg

edit site.cfg. uncomment the lines:

[openblas]
libraries = openblas
library_dirs = <BLAS_DIR>/lib
include_dirs = <BLAS_DIR>/include
runtime_library_dirs = <BLAS_DIR>/lib

Then run

python setup.py config

if all goes well the output will look something like this

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

install

Use pip to install, e.g:

pip3 install .

test

cd .. && python -c "import numpy as np; np.__config__.show()"

the output should show the following for blas and openblas.

openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

scipy

prerequisites

get pybind11, which is needed to build scipy

pip3 install pybind11

get the latest scipy and set the configuretion file

git clone https://github.com/scipy/scipy.git scipy
cd scipy
git fetch --all --tags
git checkout v1.6.0 -b v.1.6.0

configure

cp site.cfg.example site.cfg

uncomment the lines

[openblas]
libraries = openblas
library_dirs = <BLAS_DIR>/lib
include_dirs = <BLAS_DIR>/include
runtime_library_dirs = <BLAS_DIR>/lib

run config

python setup.py build

if all goes well the output will look something like this

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

  FOUND:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

install

Then use pip to install, e.g:

pip3 install .

test

cd .. && python -c "import scipy as sp; sp.__config__.show()"

the output should show the following for blas and openblas.

openblas_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
blas_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
openblas_lapack_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']
lapack_opt_info:
    libraries = ['openblas', 'openblas']
    library_dirs = ['<BLAS_DIR>/lib']
    language = c
    define_macros = [('HAVE_CBLAS', None)]
    runtime_library_dirs = ['<BLAS_DIR>/lib']

mpi4py

git clone https://bitbucket.org/mpi4py/mpi4py.git && cd mpi4py

build & install

python setup build
pip3 install .

test

mpiexec -n $(nproc) python test/runtests.py

h5py

get h5py

git clone https://github.com/h5py/h5py.git h5py
cd h5py

build

since we compiled with the latest version of HDF5, then the best is to use master

CC="mpicc" HDF5_MPI="ON" HDF5_DIR=$HDF5_DIR python setup.py build

install

to install the best is to use pip to keep track of installation and metadata

CC="mpicc" HDF5_MPI="ON" HDF5_DIR=$HDF5_DIR pip3 install .

matplotlib, pandas, seaborn

pip3 install matplotlib seaborn bokeh pandas dask

matplotlib tools

pip3 install mpl-scatter-density mpl-tune mpl-sankey mpl-styles mpl-finance \ 
    mpl-font mpl-colors mpl-format mpl-tools mpl-sns-viz
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment