Skip to content

Instantly share code, notes, and snippets.

@wroyca
Last active May 13, 2025 17:10
Show Gist options
  • Save wroyca/a251a49fdd69a1468e94306d1ecde9cf to your computer and use it in GitHub Desktop.
Save wroyca/a251a49fdd69a1468e94306d1ecde9cf to your computer and use it in GitHub Desktop.
Create snapshot release on GitHub when using build2 GitHub CI integration with bbot.bindist.upload.
name: snapshot
run-name: snapshot pre-release from build2 CI bindist
on:
# Trigger on completion of any check suite.
#
# This triggering mechanism enables the workflow to automatically detect when
# a potential new build might be available on the CI server. That is, we
# avoid having to periodically poll the CI server.
check_suite:
types: [completed]
# Enable manual workflow triggering.
#
# This provides a fallback mechanism for creating releases even when the
# automatic detection fails. It's also useful during development and testing
# of the workflow itself, and for creating releases from older builds that
# may have been missed by the automated process.
workflow_dispatch:
permissions:
# Write permission is required to create GitHub releases.
#
# This is the minimal permission level needed for publishing release
# artifacts. Note that we avoid requesting broader permissions to follow the
# principle of least privilege.
contents: write
env:
# Base URL for the build2 CI bindist repository.
#
# This is the entry point for all artifact discovery. The URL points to the
# staging environment which contains the most recent builds before they
# potentially get promoted to production.
base_ci_url: https://ci.stage.build2.org/0/bindist
# Package name to monitor and snapshot.
#
# This defines which package's artifacts we're interested in collecting.
# It's used to construct the URL paths for discovering build IDs and
# available platforms.
package: lysis
# Reference platform used for primary availability checks.
#
# This platform is checked first when validating build IDs as it's typically
# one of the most reliable and fastest to complete in the build2 CI system.
primary_platform: debian/debian12
# Retry settings for monitoring platform availability.
#
# We use a relatively long delay between retries to avoid overwhelming the
# CI server with requests, while still being responsive enough to detect
# when new platforms become available. The interval between progress
# updates provides useful feedback during long waits.
retry_delay: 600 # 10 minutes between retry attempts
retry_interval: 300 # 05 minutes between progress updates
# Platforms to monitor and include in the snapshot.
#
# This list defines all platforms that must be available before a release is
# created.
platforms: |
debian/debian11
debian/debian12
fedora/fedora40
fedora/fedora41
archive/debian11
archive/debian12
archive/fedora40
archive/fedora41
archive/macos13.4
archive/windows10
jobs:
snapshot-release:
name: create snapshot release
runs-on: ubuntu-latest
steps:
# Checkout the repository to access workflow files and scripts.
#
# While this step isn't strictly necessary for creating releases from
# external artifacts, having the repository content available allows us to
# include additional information in the release (e.g., README snippets) or
# use custom scripts if needed.
- name: checkout repository
uses: actions/checkout@v4
# Detect the latest valid build ID and package version.
#
# The output from this step defines which specific build artifacts will be
# included in the release.
- name: detect build id and version
id: detect
run: |
# Initialize base CI url for build discovery.
#
# The trailing slash is important for URL construction and proper
# parsing of the directory listing. Note that ?C=M;O=D is used to sort
# the directory listing by date, which helps find the most recent
# builds first.
package_ci_url="${base_ci_url}?C=M;O=D"
echo "::group::discovering latest builds"
echo "retrieving build ids from $package_ci_url"
# Get the most recent build IDs (UUIDs) from the CI server.
#
# We limit to 10 most recent builds for efficiency. This is a balance
# between examining enough history to find a valid build while not
# wasting resources checking very old builds that are unlikely to be
# relevant. UUIDs are extracted from directory links in the HTML.
build_ids=$(curl -fsSL "$package_ci_url" | grep -oP 'href="\K([a-f0-9-]{36})(?=/)' | head -n 10)
build_count=$(echo "$build_ids" | wc -l)
echo "found $build_count recent builds"
echo "::endgroup::"
# Initialize variables for build detection.
#
# We'll populate these as we examine builds. The build_id and version
# must both be successfully determined for a build to be selected.
build_id=""
version=""
mapfile -t platform_list < <(echo "$platforms" | tr ' ' '\n' | sed '/^$/d')
total_platforms=${#platform_list[@]}
echo "::group::validating builds and detecting version"
echo "examining builds for valid package artifacts across $total_platforms platforms"
# Iterate through build IDs from newest to oldest.
#
# This order is important as we want to find the most recent valid
# build. We'll select the first build that meets our criteria and has
# our package available on at least one platform.
for id in $build_ids; do
echo "checking build id: $id"
build_valid=false
build_platforms=()
# Track platform availability for this build.
#
# We need to know which platforms are available and which are
# missing to implement our waiting strategy if needed.
available_platforms_count=0
missing_platforms_count=0
available_platforms=()
missing_platforms=()
# First check primary platform as it's most likely to succeed.
#
# By checking the primary platform first, we can quickly determine
# if a build is likely to be valid without checking all platforms.
# This reduces the number of requests needed to find a valid build.
echo "checking primary platform: $primary_platform"
primary_url="${base_ci_url}/${id}/${primary_platform}/${package}/"
if curl --head --silent --fail "$primary_url" > /dev/null 2>&1; then
# Primary platform is available - this is a good sign.
#
# Using HEAD requests instead of GET reduces bandwidth usage
# as we only care if the URL exists, not its content.
echo "build $id available on primary platform"
build_valid=true
available_platforms_count=$((available_platforms_count + 1))
available_platforms+=("$primary_platform")
build_platforms+=("$primary_platform")
else
echo "build $id not available on primary platform"
missing_platforms_count=$((missing_platforms_count + 1))
missing_platforms+=("$primary_platform")
fi
# Check remaining platforms to get full availability picture.
#
# Even if primary platform is missing, we still check other
# platforms as the build might be valid but just missing that
# specific platform.
for platform in "${platform_list[@]}"; do
# Skip primary platform as we already checked it.
if [[ "$platform" == "$primary_platform" ]]; then
continue
fi
test_url="${base_ci_url}/${id}/${platform}/${package}/"
if curl --head --silent --fail "$test_url" > /dev/null 2>&1; then
echo "build $id available on platform: $platform"
build_valid=true
available_platforms_count=$((available_platforms_count + 1))
available_platforms+=("$platform")
build_platforms+=("$platform")
else
echo "build $id not available on platform: $platform"
missing_platforms_count=$((missing_platforms_count + 1))
missing_platforms+=("$platform")
fi
done
# If this is a valid build (available on at least one platform)
if [[ "$build_valid" == "true" ]]; then
# We've found a build with at least one valid platform.
#
# This is a candidate for our release, but we need to also
# determine its version and check if all platforms are available.
echo "selecting build id: $id"
build_id="$id"
echo "build availability: $available_platforms_count available, $missing_platforms_count missing (out of $total_platforms)"
# Get version from first available platform.
#
# The version should be consistent across all platforms for a
# given build, so we only need to extract it from one platform. We
# try platforms in order of availability until we find one that
# allows us to determine the version.
for platform in "${build_platforms[@]}"; do
version_url="${base_ci_url}/${id}/${platform}/${package}/${package}/"
echo "attempting to detect version from platform: $platform"
# Unlike platform checks, here we need the actual content.
#
# We use a full GET request as we need to parse the HTML to
# extract the version number from the directory listing.
response=$(curl -fsSL "$version_url" 2>/dev/null || echo "FAILED")
if [[ "$response" != "FAILED" ]]; then
# Extract version from directory listing.
#
# The version appears as a subdirectory in the package path.
# We parse the HTML, extract href attributes, filter out
# navigation links, and take the first result.
version=$(echo "$response" | grep -oP 'href="\K[^"/?]+(?=/)' | grep -v '^Parent$' | head -n1)
if [[ -n "$version" ]]; then
echo "detected version: $version from platform: $platform"
break
fi
fi
done
# If we have a valid build ID and version, we can proceed.
#
# Both conditions must be met to have a usable build. If we can't
# determine the version, we must skip this build.
if [[ -n "$version" ]]; then
# If not all platforms are available, implement retry logic.
#
# This is a key feature of the workflow: patience. Instead of
# giving up on a build that's partially available, we wait for
# all platforms to appear, knowing that the build2 CI system
# publishes them asynchronously.
if [[ $missing_platforms_count -gt 0 ]]; then
echo "waiting for all platforms to become available for build $id"
echo "missing platforms: ${missing_platforms[*]}"
# Wait indefinitely until all platforms are available.
#
# This is important for creating complete releases. We don't
# impose an artificial timeout because waiting longer is
# preferable to creating an incomplete release or failing
# entirely.
retry_count=0
while [[ $missing_platforms_count -gt 0 ]]; do
retry_count=$((retry_count + 1))
echo "waiting for platform availability (attempt $retry_count, will retry indefinitely)"
echo "pausing for $retry_delay seconds before next check"
# Sleep with periodic progress updates.
#
# Long sleeps without feedback make it hard to monitor the
# workflow. Break it into smaller intervals to provide
# progress updates during long waits.
remaining=$retry_delay
while [[ $remaining -gt 0 ]]; do
sleep_time=$((remaining > retry_interval ? retry_interval : remaining))
sleep $sleep_time
remaining=$((remaining - sleep_time))
elapsed=$(( (retry_delay - remaining) / 60 ))
remaining_mins=$(( remaining / 60 ))
echo "progress update: $elapsed minutes elapsed, $remaining_mins minutes remaining"
done
echo "rechecking missing platforms"
# Reset counters for missing platforms.
#
# We'll rebuild these lists based on current availability.
temp_missing=("${missing_platforms[@]}")
missing_platforms=()
missing_platforms_count=0
# Check only previously missing platforms.
#
# There's no need to recheck platforms we already know are
# available, so we focus only on the ones that were missing
# in the previous check.
for platform in "${temp_missing[@]}"; do
test_url="${base_ci_url}/${id}/${platform}/${package}/"
echo "checking platform: $platform"
if curl --head --silent --fail "$test_url" > /dev/null 2>&1; then
echo "platform now available: $platform"
available_platforms_count=$((available_platforms_count + 1))
available_platforms+=("$platform")
if [[ ! " ${build_platforms[*]} " =~ " ${platform} " ]]; then
build_platforms+=("$platform")
fi
else
echo "platform still unavailable: $platform"
missing_platforms+=("$platform")
missing_platforms_count=$((missing_platforms_count + 1))
fi
done
echo "updated availability: $available_platforms_count available, $missing_platforms_count missing (out of $total_platforms)"
if [[ $missing_platforms_count -eq 0 ]]; then
echo "all platforms now available"
fi
done
fi
# We've found a valid build with all platforms, exit the loop.
#
# Once we find a suitable build with all platforms available, we
# don't need to check older builds. We prefer the most recent
# complete build.
break
else
echo "could not detect version for build $id, skipping"
build_id=""
fi
fi
done
echo "::endgroup::"
# Verify we found a valid build ID and version.
#
# These checks guarantee we don't proceed with an incomplete or
# invalid build selection. Both build ID and version are required to
# construct valid artifact URLs in subsequent steps.
if [[ -z "$build_id" ]]; then
echo "::error::no valid build found for package '$package'"
exit 1
fi
if [[ -z "$version" ]]; then
echo "::error::could not determine version for build $build_id"
exit 1
fi
echo "final selection: build=$build_id, version=$version"
# Export variables for subsequent steps.
#
# We make these values available both as environment variables and
# step outputs so they can be referenced in later steps.
echo "build_id=${build_id}" >> "$GITHUB_ENV"
echo "version=${version}" >> "$GITHUB_ENV"
echo "build_id=${build_id}" >> "$GITHUB_OUTPUT"
echo "version=${version}" >> "$GITHUB_OUTPUT"
# Discover package artifacts across all platforms.
#
# With the build ID and version determined, we now need to locate the
# exact artifact URLs for each platform. This step verifies that artifacts
# are actually available at the expected locations and collects the URLs
# for the download step.
#
# While we've already checked for platform availability in the previous
# step, this step goes deeper in the directory structure to verify that
# the actual package artifacts exist, not just the platform directories.
- name: discover package artifacts
id: artifacts
run: |
echo "::group::locating package artifacts"
mapfile -t platform_list < <(echo "$platforms" | tr ' ' '\n' | sed '/^$/d')
total_platforms=${#platform_list[@]}
# Initialize arrays for artifact tracking.
#
# We track both combined and platform-specific artifacts to provide
# detailed reporting and support different download strategies if
# needed.
artifact_urls=()
archive_artifact_urls=()
main_artifact_urls=()
# Initialize counters for reporting and validation.
available_platforms=0
available_archive_platforms=0
available_main_platforms=0
echo "checking $total_platforms platforms for artifacts"
# Check each platform for package artifacts.
#
# We need to verify that each platform has the expected package
# directory structure containing binary artifacts.
for platform in "${platform_list[@]}"; do
# Construct the URL where we expect to find artifacts.
#
# The URL pattern follows build2 CI bindist convention:
# base/build-id/platform/package/package/version/
artifact_url="${base_ci_url}/${build_id}/${platform}/${package}/${package}/${version}/"
echo "checking platform: $platform"
# Verify the artifact directory exists.
#
# We use HEAD requests to minimize bandwidth while checking
# directory existence. This is more efficient than downloading the
# full content when we only need to confirm availability.
if curl --head --silent --fail "$artifact_url" > /dev/null 2>&1; then
echo "artifacts available for platform: $platform"
artifact_urls+=("$artifact_url")
available_platforms=$((available_platforms + 1))
# Track archive vs main platforms separately.
#
if [[ "$platform" == archive/* ]]; then
archive_artifact_urls+=("$artifact_url")
available_archive_platforms=$((available_archive_platforms + 1))
else
main_artifact_urls+=("$artifact_url")
available_main_platforms=$((available_main_platforms + 1))
fi
else
# This should not happen since we confirmed all platforms are
# available in the previous step, but we handle it just in case.
#
# There could be race conditions or server-side issues that cause
# a previously available platform to become unavailable.
echo "error: artifacts not available for platform: $platform"
echo "expected url: $artifact_url"
fi
done
# Verify we found artifacts for all platforms.
#
# This is a sanity check that should rarely fail since we already
# waited for all platforms in the previous step. However, it provides
# an additional layer of validation.
if [[ $available_platforms -lt $total_platforms ]]; then
echo "::warning::expected all platforms to be available but found only $available_platforms out of $total_platforms"
echo "this is unexpected since we verified availability in the previous step"
# List missing platforms for debugging.
#
# If there is a discrepancy, we identify exactly which platforms are
# missing to help with troubleshooting.
for platform in "${platform_list[@]}"; do
artifact_url="${base_ci_url}/${build_id}/${platform}/${package}/${package}/${version}/"
if ! curl --head --silent --fail "$artifact_url" > /dev/null 2>&1; then
echo "artifacts missing for platform: $platform"
fi
done
fi
# Verify we found at least one artifact URL.
#
# This is a important check - if we have no artifact URLs at all, the
# workflow cannot proceed to create a meaningful release.
if [[ ${#artifact_urls[@]} -eq 0 ]]; then
echo "::error::no valid artifact urls found"
exit 1
fi
# Report platform coverage for transparency.
echo "platform coverage: $available_platforms/$total_platforms"
echo "main platforms: $available_main_platforms, archive platforms: $available_archive_platforms"
echo "::endgroup::"
# List the actual artifact URLs for debugging.
echo "::group::artifact urls"
printf '%s\n' "${artifact_urls[@]}"
echo "::endgroup::"
# Store artifact URLs and stats for subsequent steps.
#
# These values will be used by the download step to retrieve the
# actual binary packages and by the release step for reporting.
echo "total_artifacts=${available_platforms}" >> "$GITHUB_ENV"
echo "main_artifacts=${available_main_platforms}" >> "$GITHUB_ENV"
echo "archive_artifacts=${available_archive_platforms}" >> "$GITHUB_ENV"
echo "total_artifacts=${available_platforms}" >> "$GITHUB_OUTPUT"
echo "main_artifacts=${available_main_platforms}" >> "$GITHUB_OUTPUT"
echo "archive_artifacts=${available_archive_platforms}" >> "$GITHUB_OUTPUT"
# Export artifact URLs as multi-line environment variable.
#
# This special syntax allows us to pass the entire array of URLs to
# subsequent steps, preserving spaces and newlines properly.
printf "artifact_urls<<EOF\n%s\nEOF\n" "${artifact_urls[*]}" >> "$GITHUB_ENV"
printf "artifact_urls<<EOF\n%s\nEOF\n" "${artifact_urls[*]}" >> "$GITHUB_OUTPUT"
# Download all binary artifacts from the discovered URLs.
#
# This step performs the actual retrieval of binary packages from the
# build2 CI server. It processes each platform, each architecture, and
# each file to create a complete local copy of all available artifacts.
#
- name: download artifacts
id: download
run: |
echo "::group::downloading package artifacts"
# Prepare directory for artifacts.
#
# We create a clean directory to store all downloaded files. This will
# become the source for our GitHub release.
mkdir -p release_binaries
# Convert space-separated URLs to array.
#
# The artifact_urls environment variable contains space-separated
# URLs, which we convert to a proper array for iteration.
mapfile -t urls < <(echo "$artifact_urls" | tr ' ' '\n')
total_urls=${#urls[@]}
# Initialize counters for tracking progress.
#
# These counters help with reporting and validation after the download
# process completes.
processed_urls=0
total_files=0
failed_downloads=0
echo "downloading artifacts from $total_urls platform sources"
# Process each platform URL.
#
# Each URL corresponds to a platform directory that contains
# architecture-specific subdirectories with binary packages.
for base_url in "${urls[@]}"; do
processed_urls=$((processed_urls + 1))
# Extract platform name from URL for reporting.
#
# This makes the logs more readable by showing which platform we're
# currently processing instead of just the full URL.
platform=$(echo "$base_url" | grep -oP '(?<='"$build_id"'/)[^/]+')
echo "::group::processing platform $processed_urls/$total_urls: $platform"
# Get architecture directories for this platform.
#
# Each platform contains one or more architecture subdirectories
# (e.g., x86_64, aarch64) with the actual binary packages.
echo "retrieving architecture directories"
arch_dirs=$(curl -s "$base_url" | grep -oP 'href="([^"?/]+/)"' | cut -d '"' -f 2 | grep -v "Parent" | grep -v "\\?C=")
# Handle case where no architectures are found.
#
# This could happen if the platform directory exists but is empty or
# doesn't contain the expected structure.
if [[ -z "$arch_dirs" ]]; then
echo "warning: no architecture directories found for platform $platform"
echo "::endgroup::"
continue
fi
echo "found architectures: $arch_dirs"
# Process each architecture.
#
# For each architecture, we need to discover and download all
# available binary packages.
for arch_dir in $arch_dirs; do
arch_name=${arch_dir%/} # Remove trailing slash
arch_url="${base_url}${arch_dir}"
echo "processing architecture: $arch_name"
# Get list of files for this architecture.
#
# We fetch the directory listing and extract links to actual
# files, filtering out directories and special files.
echo "retrieving file list"
arch_files=$(curl -s "$arch_url" | grep -oP 'href="[^"?]+"' | cut -d '"' -f 2 | grep -v "/$" | grep -v "Parent" | grep -v "\\?C=" | grep -v "packages.sha256")
# Handle case where no files are found.
#
# This could happen if the architecture directory exists but
# doesn't contain any binary packages.
if [[ -z "$arch_files" ]]; then
echo "warning: no files found for architecture $arch_name"
continue
fi
# Download each file.
#
# We process each file individually, downloading it to our local
# release_binaries directory.
for file in $arch_files; do
# Skip directories and special files.
#
# We only want actual binary packages, not navigation links or
# metadata files.
if [[ "$file" == */ ]] || [[ "$file" == "?C="* ]]; then
continue
fi
filename=$(basename "$file")
full_url="${arch_url}${file}"
dest="release_binaries/${filename}"
echo "downloading: $filename"
# Download with retry and detailed error handling.
#
# The retry logic helps handle transient network issues. We use
# curl's built-in retry functionality for this purpose.
if ! curl -L --silent --show-error --fail --retry 3 --retry-delay 3 -o "$dest" "$full_url"; then
echo "error: failed to download $filename"
failed_downloads=$((failed_downloads + 1))
else
file_size=$(du -h "$dest" | cut -f1)
echo "downloaded $filename ($file_size)"
total_files=$((total_files + 1))
fi
done
done
echo "::endgroup::"
done
# Report download summary for transparency.
#
# This provides a clear overview of what was downloaded and any issues
# that occurred during the process.
echo "::group::download summary"
echo "platforms processed: $processed_urls/$total_urls"
echo "files downloaded: $total_files"
failed_ratio="$failed_downloads/$total_files"
echo "failed downloads: $failed_downloads ($failed_ratio)"
echo "downloaded files:"
ls -la release_binaries/
echo "::endgroup::"
# Verify we downloaded at least one file.
#
# If no files were downloaded, the release would be empty, which is
# not a valid outcome for this workflow.
if [ -z "$(ls -A release_binaries)" ]; then
echo "::error::no artifacts downloaded"
exit 1
fi
# Export download statistics for the release step.
#
# These values will be used in the release notes to provide
# transparency about the release contents.
echo "total_files=${total_files}" >> "$GITHUB_ENV"
echo "failed_downloads=${failed_downloads}" >> "$GITHUB_ENV"
echo "total_files=${total_files}" >> "$GITHUB_OUTPUT"
echo "failed_downloads=${failed_downloads}" >> "$GITHUB_OUTPUT"
# Create a GitHub release with all downloaded artifacts.
#
# This final step packages all downloaded binary artifacts into a GitHub
# release. It provides a consistent, versioned distribution point that's
# directly accessible from the project's GitHub page.
#
# The release includes detailed metadata about the source build, platform
# coverage, and artifact counts to help users understand what they're
# downloading.
- name: create snapshot release
env:
# GitHub token for release creation.
#
# This token is automatically provided by GitHub Actions and gives us
# permission to create releases when combined with the 'contents:
# write' permission declared earlier.
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: |
echo "::group::creating github release"
# Generate snapshot timestamp and name.
#
# Using a timestamp in the release name provides uniqueness and
# chronological ordering when viewing releases.
timestamp=$(date +%Y%m%d-%H%M%S)
release_name="snapshot-$timestamp"
# Count files for reporting.
#
# This gives us the exact number of files included in this release,
# which may differ from total_files if there were download failures.
file_count=$(find release_binaries -type f | wc -l)
echo "creating snapshot release: $release_name"
echo "build id: $build_id"
echo "version: $version"
echo "including $file_count files from $total_artifacts platforms"
# Prepare platform coverage information for release notes.
#
# This provides clear information about whether the release contains
# artifacts from all expected platforms.
platform_coverage=""
platform_count=$(echo "$platforms" | grep -v "^$" | wc -l)
if [[ $total_artifacts -lt $platform_count ]]; then
platform_coverage="Note: This release contains artifacts from $total_artifacts out of $platform_count expected platforms."
else
platform_coverage="This release contains artifacts from all expected platforms."
fi
# Create release notes.
#
# These notes provide metadata about the release, including its source
# build, version, and platform coverage.
cat > release_notes.txt << EOF
Automated snapshot release from build2 CI.
Build ID: $build_id
Version: $version
Main platforms: $main_artifacts
Archive platforms: $archive_artifacts
Total artifacts: $file_count
$platform_coverage
EOF
# Create GitHub release with all artifacts.
#
# We use the gh CLI tool to create the release and upload all files
# from the release_binaries directory. The --prerelease flag marks
# this as a non-production release.
gh release create "$release_name" release_binaries/* \
--prerelease \
--title "Build2 CI Snapshot $timestamp" \
--notes-file release_notes.txt
echo "successfully published pre-release: $release_name"
echo "::endgroup::"
@wroyca
Copy link
Author

wroyca commented May 13, 2025

Important

Rename package name and update platforms to fit your project.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment