Build an Azure stack to operate NP-series VMs on Azure with Dragen's pay-as-you-go (PAYG) license.
- Sign up for an Azure subscription at this link if you don't already have one.
- Follow these instructions to register resource providers
Microsoft.Network
,Microsoft.Storage
, andMicrosoft.Compute
. - Visit Quotas in Azure Portal, login if needed, and increase
Standard NPS Family vCPUs
in theSouth Central US
region to 10 vCPUs. This allows you to create a singleStandard_NP10s
VM in this region. NP-series VMs are only available in some regions per the FAQs here, but as of 2025 in the US, we've only been able to get them in one region (South Central US
). Based on demand for these SKUs, you may also need to submit a service request and justify your use-case to a person before that quota gets approved. Also note that a quota of 40 vCPUs lets you run 4 NP10 VMs at a time, or 2 NP20 VMs, or 1 NP40 VM. But NP20 and NP40 VMs have more than one FPGA cards, that Dragen does not take advantage of as far as I can tell. So, those would be overkill. - Visit this page, login if needed, and ensure that
Status
is set toEnable
for the Azure subscription you intend to use. This allows programmatic deployment of the VMs that we will use later. - Follow these instructions to generate an SSH key using the Ed25519 algorithm and store it as
~/.ssh/id_ed25519
. We'll use this to SSH into VMs. - Follow these instructions to install Azure CLI, and then run
az login --use-device-code
and follow the instructions to link your subscription. - Accept the Terms of Use for the Dragen PAYG image we will use.
az vm image terms accept --urn illuminainc1586452220102:dragen-vm-payg:dragen-4-4-4-payg:latest
- Follow the instructions to install AzCopy to upload/download data.
- All the commands in this repo were tested on Ubuntu 24.04 in WSL2 with these dotfiles, but you should be fine with Bash in any Linux environment or Zsh on macOS.
Make separate resource groups for networking, storage accounts, and VMs.
az group create --name dgn-net-rg --location southcentralus
az group create --name dgn-st-rg --location southcentralus
az group create --name dgn-vms-rg --location southcentralus
Create a VNet with a subnet and permit SSH connections from our current IP address.
az network nsg create --resource-group dgn-net-rg --name dgn-nsg
az network vnet create --resource-group dgn-net-rg --network-security-group dgn-nsg --name dgn-vnet --address-prefixes "10.10.10.0/24"
az network vnet subnet create --resource-group dgn-net-rg --network-security-group dgn-nsg --vnet-name dgn-vnet --name dgn-sub1 --address-prefixes "10.10.10.0/25" --service-endpoints Microsoft.Storage
az network nsg rule create --resource-group dgn-net-rg --nsg-name dgn-nsg --name AllowSSHInBound --priority 200 --protocol TCP --access Allow --direction Inbound --source-address-prefixes $(curl -s https://icanhazip.com) --source-port-ranges "*" --destination-address-prefixes "*" --destination-port-ranges 22
Create a storage account with hierarchical namespace enabled (ADLS Gen2).
export AZURE_STORAGE_ACCOUNT=dgntestdata
az storage account create --resource-group dgn-st-rg --name ${AZURE_STORAGE_ACCOUNT} --kind StorageV2 --access-tier Hot --sku Standard_LRS --enable-hierarchical-namespace true --min-tls-version TLS1_2 --public-network-access enabled --publish-internet-endpoints false --publish-microsoft-endpoints false --routing-choice MicrosoftRouting
Create blob containers (aka file systems) for input FASTQs, reference data, and output data from Dragen.
az storage container create --account-name ${AZURE_STORAGE_ACCOUNT} --name fqs --auth-mode login --public-access off
az storage container create --account-name ${AZURE_STORAGE_ACCOUNT} --name ref --auth-mode login --public-access off
az storage container create --account-name ${AZURE_STORAGE_ACCOUNT} --name dgn --auth-mode login --public-access off
Set environment variables that allow Azure CLI to access our storage account.
export AZURE_STORAGE_ACCOUNT=dgntestdata
export AZURE_STORAGE_KEY=$(az storage account keys list --account-name ${AZURE_STORAGE_ACCOUNT} --query [0].value --output tsv)
Generate a single-use short-lived SAS token to upload data into the ref
container, and use it to upload the hg38 v5 Graph Reference Genome compatible with Dragen 4.4.
curl -LO https://webdata.illumina.com/downloads/software/dragen/references/genome-files/hg38-alt_masked.cnv.graph.hla.methyl_cg.rna-11-r5.0-1.tar.gz
gzip -d hg38-alt_masked.cnv.graph.hla.methyl_cg.rna-11-r5.0-1.tar.gz
END=$(date -u -d "2 hours" '+%Y-%m-%dT%H:%MZ')
SAS=$(az storage container generate-sas --name ref --permissions cw --expiry $END --output tsv)
azcopy cp hg38-alt_masked.cnv.graph.hla.methyl_cg.rna-11-r5.0-1.tar "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/ref/hg38/?${SAS}" --content-type="application/x-tar"
Similarly, generate another SAS token for the fqs
container to upload your FASTQs. This is demonstrated below using test FASTQs from a dataset I created here.
tar -xf test_trio_wgs.tar --wildcards *_{L001,L002}_{R1,R2}_001.fastq.gz
END=$(date -u -d "20 mins" '+%Y-%m-%dT%H:%MZ')
SAS=$(az storage container generate-sas --name fqs --permissions cw --expiry $END --output tsv)
azcopy cp dad "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/fqs/ajtrio/?${SAS}" --recursive --content-type="text/fastq" --content-encoding="gzip"
azcopy cp mom "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/fqs/ajtrio/?${SAS}" --recursive --content-type="text/fastq" --content-encoding="gzip"
azcopy cp son "https://${AZURE_STORAGE_ACCOUNT}.blob.core.windows.net/fqs/ajtrio/?${SAS}" --recursive --content-type="text/fastq" --content-encoding="gzip"
Set up an SSH configuration that we have determined (with trial and error) will reliably get us into Azure VMs to run long-running scripts.
SSH_USERNAME="azureuser"
SSH_AUTH_KEY="~/.ssh/id_ed25519"
SSH_OPTIONS="-q -i ${SSH_AUTH_KEY} -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null -o LogLevel=QUIET -o ServerAliveInterval=120 -o ServerAliveCountMax=30"
On the subnet and NSG created earlier, create a Dragen 4.4.4 VM with a 1TB OS disk.
SUBNET=$(az network vnet subnet show --resource-group dgn-net-rg --vnet-name dgn-vnet --name dgn-sub1 --query id --output tsv)
NSG=$(az network nsg show --resource-group dgn-net-rg --name dgn-nsg --query id --output tsv)
VMIP=$(az vm create --resource-group dgn-vms-rg --subnet $SUBNET --nsg $NSG --public-ip-address dgn1-pip --name dgn1 --size Standard_NP10s --os-disk-name dgn1-os-disk --os-disk-size-gb 1024 --os-disk-delete-option delete --nic-delete-option delete --image illuminainc1586452220102:dragen-vm-payg:dragen-4-4-4-payg:latest --admin-username ${SSH_USERNAME} --ssh-key-values ${SSH_AUTH_KEY}.pub --query publicIpAddress --output tsv)
Copy over a script that does alignment and variant calling, then run it for one of the test samples.
curl -LO https://gist.githubusercontent.com/ckandoth/4006866209475ae558ead88a53e6b59f/raw/process_fastqs.sh
chmod +x process_fastqs.sh
scp ${SSH_OPTIONS} process_fastqs.sh ${SSH_USERNAME}@${VMIP}:~/
ssh ${SSH_OPTIONS} ${SSH_USERNAME}@${VMIP} "AZURE_STORAGE_ACCOUNT=${AZURE_STORAGE_ACCOUNT} AZURE_STORAGE_KEY=${AZURE_STORAGE_KEY} ./process_fastqs.sh fqs/ajtrio/son ref/hg38 dgn/ajtrio/son"
NOTE: If using the Dragen PAYG 4.3.6 image, we needed to add LANG='en_US.UTF-8'
before the SSH command above because of a bug in the CentOS 7 image that breaks something in the FPGA driver when LANG='C.UTF-8'
, the default LANG
in Ubuntu 24.04 Server. This is no longer an issue in Dragen PAYG 4.3.14 and newer images which use AlmaLinux 8 instead.
Make sure the Dragen output was uploaded into the dgn
container.
az storage blob list --container dgn --prefix ajtrio-son --query [].name --output tsv
Delete the VM and its public IP to save money.
az vm delete --yes --resource-group dgn-vms-rg --name dgn1
az network public-ip delete --resource-group dgn-vms-rg --name dgn1-pip
Now we are ready to orchestrate the creation of VMs and have them analyze multiple samples in parallel and/or in series.
As a policy to reduce long-term storage costs, we can set a blob to Cool
tier if there are no plans to access it within 6 months. And Archive
tier if we can be certain not to access it for at least 2 years. After successfully analyzing and delivering results for a batch of MPG samples, here is how to delete FASTQs and move CRAMs into Cool
tier.
BATCH="ajtrio"
az storage fs directory delete --yes --file-system fqs --name ${BATCH}
az storage blob list --container dgn --prefix ${BATCH} --num-results 500000 --query "[?ends_with(name,'.cram')].name" --output tsv | xargs -L1 az storage blob set-tier --container-name dgn --tier cool --name