Skip to content

Instantly share code, notes, and snippets.

@Markieta

Markieta/bosh-lite-virtualbox-cf-deployment.md

Last active September 26, 2021 03:35
Show Gist options
  • Save Markieta/2141a631402d97f5daed94ed7f381093 to your computer and use it in GitHub Desktop.
Save Markieta/2141a631402d97f5daed94ed7f381093 to your computer and use it in GitHub Desktop.
Download ZIP
BOSH Lite (VirtualBox) cf-deployment Quickstart 2021
Raw
bosh-lite-virtualbox-cf-deployment.md

BOSH Lite (VirtualBox) cf-deployment Quickstart 2021

This is a quickstart guide to deploying BOSH Lite v2 locally in VirtualBox that adds some context to the bosh-deployment and cf-deployment guides that I think are important to capture in 2021.

Assumptions

These steps are based on running BOSH Lite and cf-deployment using the following 2 host machines. These steps should still apply to other Linux-based systems (slight alterations for macOS):

System OS CPU Memory Storage
B550 Fedora 34 AMD Ryzen 9 5950X (16 cores, 32 threads) 64GB NVMe SSD
ThinkPad X1 Carbon (Gen 6) Fedora 32 Intel Core i7-8650U (4 cores, 8 threads) 16GB NVMe SSD

Prerequisites

Before using this script...

Machine Resources

BOSH Lite + cf-deployment can easily chew up about 50GB of storage from the start. I recommend dedicating at least 100GB to this. Allocating more CPU threads and memory can help speed up the cf-deployment compilation process and deployment time.

Git

Make sure Git is installed on your host:

# Fedora
sudo dnf install git

# Ubuntu
sudo apt install git

BOSH CLI

Download the latest bosh-cli release and move it to your $PATH:

# Linux
sudo mv ~/Downloads/bosh-cli-X.X.X-linux-amd64 /usr/local/bin/bosh
sudo chmod +x /usr/local/bin/bosh

CF CLI

Install the latest cf-cli:

# Fedora
sudo wget -O /etc/yum.repos.d/cloudfoundry-cli.repo https://packages.cloudfoundry.org/fedora/cloudfoundry-cli.repo
sudo yum install cf7-cli

# Ubuntu
wget -q -O - https://packages.cloudfoundry.org/debian/cli.cloudfoundry.org.key | sudo apt-key add -
echo "deb https://packages.cloudfoundry.org/debian stable main" | sudo tee /etc/apt/sources.list.d/cloudfoundry-cli.list
sudo apt-get update
sudo apt-get install cf7-cli

CredHub

Download the latest credhub release, extract and move it to your $PATH:

# Linux
sudo tar zxvf ~/Downloads/credhub-linux-X.X.X.tgz
sudo mv credhub /usr/local/bin/

VirtualBox

As of writing, there is an open issue with BOSH Lite failing to deploy on VirtualBox 6.1. To get around this, I have installed VirtualBox 6.0.

Relatively modern Linux OS releases (Fedora 32+, Ubuntu 21.04+) no longer support VirtualBox 6.0 due to requiring an older package and kernel not provided in the default package managers' repositories.

The workaround for this in Fedora 32+ is to patch the old VirtualBox 6.0 RPM and install and reboot into a legacy kernel from Fedora Copr.

Install kernel long-term 5.4 and boot into it:

sudo dnf copr enable kwizart/kernel-longterm-5.4
sudo dnf install kernel-longterm kernel-longterm-devel # devel required for vboxconfig
sudo reboot # Be sure to select kernel 5.4 in GRUB menu

Download (for Fedora 31) and rebuild VirtualBox 6.0:

sudo dnf install rpmrebuild
rpmrebuild -enp ~/Downloads/VirtualBox-6.0-6.0.24_139119_fedora31-1.x86_64.rpm

Find this line and make the following change:

Requires:      python(abi) = 3.7
Requires:      python(abi) >= 3.7

Save, exit, and accept changes to rebuild the RPM (may take a minute), then install the rebuilt RPM:

sudo dnf install /home/$USER/rpmbuild/RPMS/x86_64/VirtualBox-6.0-6.0.24_139119_fedora31-1.x86_64.rpm

VirtualBox 6.0 will now be installed. If you are receiving errors running /sbin/vboxconfig, ensure you are running the 5.4 kernel (uname -a), have installed kernel-longterm-devel, and virtualization is enabled:

egrep '^flags.*(vmx|svm)' /proc/cpuinfo # Intel and AMD CPUs

Refer to Enabling hardware virtualization support if required.

Script

Please mind the WARNINGS, read through, and adjust values relative to your system's resources and paths. Use what you need from this script (i.e skip the cleanup steps if you want to do this manually).

#!/usr/bin/env bash

###########################################################
# WARNING: This will delete all VMs in VirtualBox
#          whose names contain "sc-" or "vm-"
###########################################################
vboxmanage list vms | while read machine id; do
  if [[ "$machine" =~ sc-* ]] || [[ "$machine" =~ vm-* ]]; then
    vboxmanage controlvm "$id" poweroff
    sleep 2 # poweroff is not always immediate
    vboxmanage unregistervm "$id" --delete
  fi
done

cf_deployment="/home/$USER/cf-deployment"
deployments="/home/$USER/deployments"
workspace="/home/$USER/workspace"

# Clean up directories (from previous failures)
rm -rf ~/VirtualBox\ VMs/sc-* ~/VirtualBox\ VMs/vm-* # VirtualBox VMs
rm -rf ~/.bosh/* ~/.bosh_virtualbox_cpi/*            # BOSH Director VM data
rm -rf $cf_deployment/ $deployments/ $workspace/     # BOSH/CF deployment directories and repositories
cd                                                   # Go home

# Clone and checkout bosh-deployment repo, prepare for Director install
git clone https://github.com/cloudfoundry/bosh-deployment $workspace/bosh-deployment
cd $workspace/bosh-deployment
git checkout fe82b26ef260dc1753e8a14f72cbe20905389eab # Revert to BOSH 271.9.0 (latest had bugs)
mkdir -p $deployments/vbox
cd $deployments/vbox

# Adjust CPU (threads) and memory (MB) values (on the right) to your machine.
# Reserve at least a couple GBs for your host.
sed -i 's@cpus: 4@cpus: 32@' $workspace/bosh-deployment/virtualbox/cpi.yml
sed -i 's@memory: 6144@memory: 51200@' $workspace/bosh-deployment/virtualbox/cpi.yml

# Create BOSH Director VM
time bosh create-env $workspace/bosh-deployment/bosh.yml \
  --state ./state.json \
  -o $workspace/bosh-deployment/virtualbox/cpi.yml \
  -o $workspace/bosh-deployment/virtualbox/outbound-network.yml \
  -o $workspace/bosh-deployment/bosh-lite.yml \
  -o $workspace/bosh-deployment/bosh-lite-runc.yml \
  -o $workspace/bosh-deployment/uaa.yml \
  -o $workspace/bosh-deployment/credhub.yml \
  -o $workspace/bosh-deployment/jumpbox-user.yml \
  --vars-store ./creds.yml \
  -v director_name=bosh-lite \
  -v internal_ip=192.168.50.6 \
  -v internal_gw=192.168.50.1 \
  -v internal_cidr=192.168.50.0/24 \
  -v outbound_network_name=NatNetwork

# Alias and log into the Director
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET=`bosh int ./creds.yml --path /admin_password`
bosh alias-env vbox -e 192.168.50.6 --ca-cert <(bosh int ./creds.yml --path /director_ssl/ca)

# Confirm that it works
bosh -e vbox env

# local route for bosh ssh commands or accessing VMs directly
sudo ip route add 10.244.0.0/16 via 192.168.50.6 # Linux (using iproute2 suite)

# Clone cf-deployment (tagged release to avoid latest bugs)
git clone -b v16.24.0 [email protected]:cloudfoundry/cf-deployment.git $cf_deployment
cd $cf_deployment

cp $deployments/vbox/creds.yml . # Copy creds to $PWD

# Export BOSH and CREDHUB environment variables
export BOSH_CLIENT=admin
export BOSH_CLIENT_SECRET="$(bosh int ./creds.yml --path /admin_password)"
export BOSH_CA_CERT="$(bosh interpolate ./creds.yml --path /director_ssl/ca)" # newline required after this

export CREDHUB_SERVER=https://192.168.50.6:8844
export CREDHUB_CLIENT=credhub-admin
export CREDHUB_SECRET=$(bosh interpolate ./creds.yml --path=/credhub_admin_client_secret)
export CREDHUB_CA_CERT="$(bosh interpolate ./creds.yml --path=/credhub_tls/ca )"$'\n'"$( bosh interpolate ./creds.yml --path=/uaa_ssl/ca)"

# Use correct cloud config for BOSH Lite + cf-deployment
bosh -ne vbox update-cloud-config $cf_deployment/iaas-support/bosh-lite/cloud-config.yml

# Upload required Stemcell
bosh -e vbox upload-stemcell --sha1 f399044d2ebe3351f0f1b0b3f97ef11464d283b4 \
  https://bosh.io/d/stemcells/bosh-warden-boshlite-ubuntu-xenial-go_agent?v=621.125

# DNS config update
bosh -ne vbox update-runtime-config $workspace/bosh-deployment/runtime-configs/dns.yml --name dns

# Deploy CF
# Remove use-compiled-releases.yml to compile releases yourself (slower)
time bosh -ne 192.168.50.6 -d cf deploy \
  cf-deployment.yml \
  -o operations/bosh-lite.yml \
  -o operations/use-compiled-releases.yml \
  -v system_domain=bosh-lite.com

# Authenticate into Cloud Foundry
cf api https://api.bosh-lite.com --skip-ssl-validation
credhub api --server api.bosh-lite.com --skip-tls-validation
cf login -u admin -p "$(credhub get -qn /bosh-lite/cf/cf_admin_password)"

Graceful Shutdown

In case VirtualBox VM shuts down or reboots, you will have to re-run create-env command from above with --recreate flag. The containers will be lost after a VM restart, but you can restore your deployment with bosh cck command. Alternatively Pause the VM from the VirtualBox UI before shutting down VirtualBox host, or making your computer sleep.

Use the following script (adapted from jujoramos) to gracefully pause (savestate to drive) and resume your BOSH Directory VM:

#!/usr/bin/env bash

WORKSPACE_DIRECTORY=~/deployments/vbox
BOSH_LITE_VM_ID=$(cat $WORKSPACE_DIRECTORY/state.json | python2 -c "import json,sys;obj=json.load(sys.stdin);print obj['current_vm_cid'];")

case $1 in
  ssh)
    ssh -i ~/.ssh/bosh-virtualbox.key [email protected]
    ;;
  pause)
    echo "Pausing Bosh_Lite VM with ID $BOSH_LITE_VM_ID..."
    VBoxManage controlvm $BOSH_LITE_VM_ID savestate
    echo "Pausing Bosh_Lite VM with ID $BOSH_LITE_VM_ID... Done!."
    ;;
  resume)
    echo "Resuming Bosh_Lite VM with ID $BOSH_LITE_VM_ID..."
    VBoxManage startvm $BOSH_LITE_VM_ID --type=headless
    echo "Resuming Bosh_Lite VM with ID $BOSH_LITE_VM_ID... Done!"
    sudo ip route add 10.244.0.0/16 via 192.168.50.6 # Linux (using iproute2 suite)
    ;;
  *)
    echo "Usage: bosh_vm {ssh|pause|resume}" ;;
esac
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment