Linux Containers for HPC

From Research Computing @ TAMUQ
Jump to navigation Jump to search


Introduction to Linux Containers

Singularity Containers

Introduction

Installation

Users have various options to install Singularity package on their workstation. The approach can vary based on which platform you are using.

We recommend that you follow installation instructions provided at Sylabs website so that you always find the latest updated installation instruction. These instruction can be found here; Singularity Installation Instruction by SyLabs

We have also made Singularity package pre-installed as a Virtual machine and the VM can be used via Vagrant and VirtualBox.

Linux

On Linux, you can compile Singularity from source code available on Singularity Git Repo.

Step 01: Install required development tools and libraries

For Ubuntu

sudo apt-get update
sudo apt-get install -y build-essential libssl-dev uuid-dev libgpgme11-dev squashfs-tools libseccomp-dev wget pkg-config git cryptsetup debootstrap

For Centos

sudo yum -y update 
sudo yum -y groupinstall 'Development Tools'
sudo yum -y install wget epel-release
sudo yum -y install debootstrap.noarch squashfs-tools openssl-devel libuuid-devel gpgme-devel libseccomp-devel cryptsetup-luks

Step 02: Singularity V3.0 is written in Go Language. You will need to download and install Go Language on your local system.

wget https://dl.google.com/go/go1.13.linux-amd64.tar.gz

sudo tar --directory=/usr/local -xzvf go1.13.linux-amd64.tar.gz

ln -s /usr/local/go/bin/go /usr/local/bin/go
ln -s /usr/local/go/bin/go /usr/local/bin/go

Step 03: Download Singularity source code

via Git

git clone https://github.com/sylabs/singularity.git

via Wget

wget https://github.com/singularityware/singularity/releases/download/v3.5.3/singularity-3.5.3.tar.gz

tar -xzvf singularity-3.5.3.tar.gz

Step 04: Build and Install


cd singularity

./mconfig

cd builddir

make

sudo make install

Windows

This guide walk through the process of download and installing Virtual Box and Vagrant on your Windows system.

Step 01: Download and Install Virtual Box

Virtual Box is a Virtualization software which allows users to run Linux or Windows Virtual machines on any operating system. Refer to this guide to install Virtual Box on your Windows OS Install VirtualBox

Step 02: Download and Install Vagrant

Vagrant automates the process of launching Virtual Machines. You don't have to worry about downloading the OS image and then performing the installation. Vagrant takes care of all the steps and gives you a ready to use virtual machine with very few commands. You can download the Vagrant executable from this link Download Vagrant. Launch the installer and the installation steps are self explanatory.

Step 03: Launch Virtual Machine which has Singularity pre-installed

MAC OS

On MAC you can use brew to setup Virtual Box and Vagrant. Then launch Sylabs provided Virtual Machine for Singularity

Step 01: Install Brew, VirtualBox and Vagrant

/usr/bin/ruby -e "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/master/install)"
brew cask install virtualbox
brew cask install vagrant
brew cask install vagrant-manager

Step 02: Start Virtual Machine with Singularity pre-installed

Step 03: Launch Virtual Machine which has Singularity pre-installed

Launch VM with Vagrant

If you have virtualbox and Vagrant installed, you can quickly start Singularity VM by issuing following commands;

mkdir singularity-vm && cd singularity-vm
vagrant init marif/singularity-3.5.3-ubuntu-bionic64 --box-version 20200413.0.0
vagrant up
vagrant ssh
vagrant@devhost:~$ singularity --version
singularity version 3.5.3

In some cases, users might get error related to VboxManage, in this case do following;

vagrant destroy

Then modify Vagrantfile in the current directory and add following lines after 'config.vm.box_version' entry. Make sure the 'end' aligns with 'config' section as shown below;

  config.vm.box = "marif/singularity-3.5.3-ubuntu-bionic64"
  config.vm.box_version = "20200413.0.0"
  config.vm.provider "virtualbox" do |v|
    v.memory = 4096
    v.cpus = 4
    v.customize [ "modifyvm", :id, "--uartmode1", "disconnected" ]
  end

Once done issue following;

vagrant up

Using Pre-Built containers

Where to find pre-build containers?

Singularity Container Library

Docker Hub

Quay.io

Nvidia GPU Cloud

Interact with Container

SHELL

EXEC

RUN

How to build a container?

Write a definition file

Create sandbox containers

Build production containers (.sif) file

Build on top of existing containers

Ecosystem

Example Project

Porting containers to HPC System