Skip to content

Instantly share code, notes, and snippets.

@smoser

smoser/README.md

Last active June 12, 2025 17:41
Show Gist options
  • Save smoser/2c391ca52b50d13c5d1429e0546a23e7 to your computer and use it in GitHub Desktop.
Save smoser/2c391ca52b50d13c5d1429e0546a23e7 to your computer and use it in GitHub Desktop.
Download ZIP
git bisect with melange package build
Raw
README.md

git bisect with melange package build

I spent some time debugging a change in melange behavior and put together a reproduce script. I wanted to save that knowledge off.

After putting together repro.sh, I was able to let git bisect tell me where the change came in.

In this gist there is some general knowledge on how to call melange with everything in place (as of today) and how to call git bisect.

$ git checkout upstream/main 
HEAD is now at d6235bd9 Add a linter for CUDA driver-specific libraries (#2033)

$ git bisect start
status: waiting for both good and bad commits

$ git bisect bad
status: waiting for good commit(s), bad commit known

$ git bisect good v0.19.0
Bisecting: 120 revisions left to test after this (roughly 7 steps)
[ed9c3ca093140a3db074dec7011901f00f224074] build(deps): bump go.opentelemetry.io/otel/exporters/stdout/stdouttrace (#1797)

$ git bisect run ./repro.sh
...

After some spinning around, git bisect showed:

1b272db2a0bb3441553284cc56d87236b4b64c04 is the first bad commit
commit 1b272db2a0bb3441553284cc56d87236b4b64c04
Author: Evan Gibler <[email protected]>
Date:   Thu Mar 13 18:45:22 2025 -0500

    Persist workspace filesystem throughout package builds (#1836)

 .github/workflows/wolfi-presubmit.yaml             |   1 +
 e2e-tests/ipset-build-test.yaml                    |  90 +++++++++++++
 go.mod                                             |   8 +-
 go.sum                                             |  12 +-
 pkg/build/build.go                                 |  17 ++-
 pkg/build/package.go                               |  12 +-
 pkg/build/readlinkfs.go                            | 142 ---------------------
 pkg/build/sca_interface.go                         |   7 +-
 pkg/build/test.go                                  |   5 +-
 pkg/config/schema.json                             |  45 +++++++
 pkg/container/qemu_runner.go                       |   2 +-
 pkg/linter/linter.go                               |   3 +-
 pkg/sca/sca.go                                     |   4 +
 .../testdata/generated/x86_64/shbang-test-1-r1.apk | Bin 4522 -> 5968 bytes
 14 files changed, 179 insertions(+), 169 deletions(-)
 create mode 100644 e2e-tests/ipset-build-test.yaml
 delete mode 100644 pkg/build/readlinkfs.go
bisect found first bad commit

repro-run

This script just loops over calls to repro.sh with each runner and priv/non-priv.

Currently, result is:

Thu, 12 Jun 2025 13:24:16 -0400: PRIV=priv RUNNER=bubblewrap : PASS
Thu, 12 Jun 2025 13:24:17 -0400: PRIV=unpriv RUNNER=bubblewrap : FAIL
Thu, 12 Jun 2025 13:24:26 -0400: PRIV=priv RUNNER=qemu : PASS
Thu, 12 Jun 2025 13:24:33 -0400: PRIV=unpriv RUNNER=qemu : PASS
Thu, 12 Jun 2025 13:24:35 -0400: PRIV=priv RUNNER=docker : PASS
Thu, 12 Jun 2025 13:24:37 -0400: PRIV=unpriv RUNNER=docker : FAIL
Raw
repro-run
#!/bin/bash
# shellcheck disable=SC3043
LOG=/tmp/results
log() {
local now=""
now=$(date -R)
echo "$now:" "$@" >> "$LOG.txt"
echo "$now:" "$@"
}
if [ -z "$QEMU_KERNEL_IMAGE" ]; then
echo "need QEMU_KERNEL_IMAGE set"
exit 1
fi
make melange || { echo "failed make melange"; exit 1; }
export MELANGE="$PWD/melange"
set -o pipefail
rm -f "$LOG.txt"
pairs="priv:bubblewrap unpriv:bubblewrap priv:qemu unpriv:qemu priv:docker unpriv:docker"
for pair in $pairs; do
priv=${pair%:*}
runner=${pair#*:}
env PRIV="$priv" RUNNER="$runner" ./repro.sh 2>&1 | tee "$LOG-$priv-$runner.log"
rc=$?
result=FAIL
if [ $rc -eq 0 ]; then
result=PASS
fi
log "PRIV=$priv RUNNER=$runner : $result"
done
Raw
repro.sh
#!/bin/bash
# shellcheck disable=SC2015,SC2162
fail() {
echo "FAIL FAIL FAIL" "$@"
exit 99
}
untestable() {
echo "untestable" "$@"
exit 125
}
catpkg() {
cat <<"EOF"
package:
name: PACKAGE_NAME
version: "1.0.0"
epoch: 0
description: Test some perms.
copyright:
- license: GPL-2.0-or-later
dependencies:
runtime:
- wolfi-baselayout
environment:
contents:
packages:
- busybox
- wolfi-base
pipeline:
- runs: |
p=${{targets.destdir}}/usr/share/testownership/flist.txt
mkdir -p ${p%/*}
cat > "$p" <<"ENDLIST"
d 0:101 var/spool/mydir/dir1
d 0:102 var/spool/mydir/dir2
d 101:0 var/spool/mydir/dir3
d 101:102 var/spool/mydir/dir4
f 0:101 usr/bin/file1
f 0:102 usr/bin/file2
f 101:0 usr/bin/file3
f 101:102 usr/bin/file4
ENDLIST
- runs: |
set +x
vr() { echo "$" "$@"; "$@"; }
d=${{targets.destdir}}
vr cd "$d"
while read ftype uidgid path; do
dir=${path%/*}
[ -d "$dir" ] || vr mkdir -p "$dir"
case "${ftype}" in
d) vr mkdir "$path";;
f) vr touch "$path";;
*) echo "unknown ftype $ftype on $path"; exit 1;;
esac
vr chown "$uidgid" "$path"
done < usr/share/testownership/flist.txt
test:
environment:
contents:
packages:
- busybox
pipeline:
- runs: |
set +x
fails=0
passes=0
vr() { echo "$" "$@"; "$@"; }
vr cd /
while read ftype uidgid path; do
found=$(stat -c "%u:%g" "$path")
if [ "$found" = "$uidgid" ]; then
echo "PASS: $found $path"
passes=$((passes+1))
continue
fi
echo "FAIL: $path - expected ownership $uidgid found $found"
fails=$((fails+1))
done < usr/share/testownership/flist.txt
[ $fails -eq 0 ] || { echo "FAIL: found $fails/$((passes+fails)) failures"; exit 1; }
echo "PASS: $((passess))/$((passes)) pass"
EOF
}
write_keys() {
cat > "$1".pub <<"EOF"
-----BEGIN PUBLIC KEY-----
MIICIjANBgkqhkiG9w0BAQEFAAOCAg8AMIICCgKCAgEApTm9NTGCr7qbaeOk/4cC
/ANdVLjXydQL4OmT6Zq9ztPkI2SIeVtB4Dmzm1u26BdU99Vda1EjzAYlFUjMc3dU
2KNYl6Gm+KJlByQk/mBmZ05eFcHqCzqJ9Cp2MTFJ1zEzaaPz45/cUAXusWU61xK3
pB82yl+cMwRO9AmjvJ1FMIUJO2uCt1R49eY1MpqdzkAgPbZi+71xKS9uhu5BSlfa
5zGkv6Sl9G070jEBLj+aBDHsqHBhOsIIg2cytZmoDyTvcwLN1nUP/dVYqDykYswG
x5syA/QbjekuoKEljnm7NDJgcQZlFrj+I2oRETHZDO6MWVQa8I+ABp+ILYLlRXAm
mt/C+1FKgEMnU95QOqVqDQvM03djbGafR+/7uTBruOwzdCKGQQHB0KLvyZPKmjrx
sYxRGFZib1TmXYe7dUNMg22moaHyzrEUpbRP3XazgP9ZEuTlbiDf3VjoRaZ9DjJu
3t3YUqz6gSpa5Mf1ZFtI1ro4EyniH1YguyC6XS48uH7rdH3wMlhEwZXNg54OwgUu
a9POo/it5soNh5sxsbsjEYMWVblIuO8114LahMGDPR1D82j5IogkmqE06yR2vg+X
qRwbYAAYVoP8lMYrwBngHScFgNqMPF45fcQmInGGjhaC3wcZXG1XQl3T+eGiTq7g
PAbCbn7zVppQ3776BEaUpscCAwEAAQ==
-----END PUBLIC KEY-----
EOF
cat > "$1" <<"EOF"
-----BEGIN RSA PRIVATE KEY-----
MIIJKQIBAAKCAgEApTm9NTGCr7qbaeOk/4cC/ANdVLjXydQL4OmT6Zq9ztPkI2SI
eVtB4Dmzm1u26BdU99Vda1EjzAYlFUjMc3dU2KNYl6Gm+KJlByQk/mBmZ05eFcHq
CzqJ9Cp2MTFJ1zEzaaPz45/cUAXusWU61xK3pB82yl+cMwRO9AmjvJ1FMIUJO2uC
t1R49eY1MpqdzkAgPbZi+71xKS9uhu5BSlfa5zGkv6Sl9G070jEBLj+aBDHsqHBh
OsIIg2cytZmoDyTvcwLN1nUP/dVYqDykYswGx5syA/QbjekuoKEljnm7NDJgcQZl
Frj+I2oRETHZDO6MWVQa8I+ABp+ILYLlRXAmmt/C+1FKgEMnU95QOqVqDQvM03dj
bGafR+/7uTBruOwzdCKGQQHB0KLvyZPKmjrxsYxRGFZib1TmXYe7dUNMg22moaHy
zrEUpbRP3XazgP9ZEuTlbiDf3VjoRaZ9DjJu3t3YUqz6gSpa5Mf1ZFtI1ro4Eyni
H1YguyC6XS48uH7rdH3wMlhEwZXNg54OwgUua9POo/it5soNh5sxsbsjEYMWVblI
uO8114LahMGDPR1D82j5IogkmqE06yR2vg+XqRwbYAAYVoP8lMYrwBngHScFgNqM
PF45fcQmInGGjhaC3wcZXG1XQl3T+eGiTq7gPAbCbn7zVppQ3776BEaUpscCAwEA
AQKCAgAewJ5s1ihC2PKwRMSjItf1XBL0/+p6EFOdqxvysPB/HIUr6TxVihy6xUKe
ufVTQXR5JGdc4B9PjB0+1uvm2eEa6VxF7LqNMHypVZrlFcA5niZ42jzbliuzHGwf
P9NEkE3HbdKpHUn+QvxoXDUVbtBrZbvm8uC5xiWD2n91TbhhDYHoY1xTDw/shMqF
/q08hEb79YR64om7Zq6lwpa4ZZ3CxkSpvtV8pxApUSGtH/1kpn4r467VJuQ6wOx5
TH4xQ2il+XGiZutxDVKRDahzExJ3+HTYs4ilYy9gil3ZJOMpFe9JWi1ai1gR8131
21dQxt2sirDw8WSc1vibr+9Ie1VT1Y9N9LAMIDQTmmPjGkxE2bgBPb7gyutI9/eR
LISm+vVZmTzYoBT5YnyAN6zLqAK8jnipupDs9rJbHTqOCo3S1tGaDZ4yI3dZsI8J
5qQeWuFC37cjuecIk6+bUek86bLZtYvKCT11gJVK4dGitqzipJ5QwN8bUdqyxP0I
UAA6ObjP+EeQmQmWf/SCAF+7BMCI1xK/M226Asfgaa8ZV8yF1GLTF9SwK2ZbkfK7
PFZ6GC8VcbjgbEbscJsNB5Knt9Rq6nV1Apwg3Na/ueh6UXOljDXgB8rvugztBD+U
Q9bkGPadE4hu5va0InTHsj3MZTmLmsrxS6HmYNIbLWHZyjoVsQKCAQEA1glNixq3
a+FWl4onwim7WWQOjovrvoL1urbMFL5Rz9/NRFxgEzodTK3osyLzj+lvCXs3z9qj
K09JbmCiXO0gtw4Ym1J1gBLsTRuU6+79Z6dSuPd+RThJfCYynoNDxvF3g9U9xFPs
8YU9TU6nupxuiAwxS+wr0XqfyyWV2yj3+5vZEAvmnqJRyG8UbWjXZJbK73vdGNXG
NO2JzrZnNWu3+lstWRIjVNe59xYW/ebArgbSKPkuIRgCF6Ekkgf7bIC/KbVSaCRb
UoQ3TJ15l2qJTN6gJwmRSt7KjDODNHMQIiAviJC6oQwFEqDgO3Tsi25Oy7zc3e8I
pCX6txd6FsEBKwKCAQEAxZ6S3nKXGKovgni55hOidJA/JLHBrcaHR27K2v+5eNSr
IwijVhas7R1YoD7QVNnok8R0H2LpgqGo8qlfZ+hRXU7EurtihEtH0roA0Kdycjgy
6ITNthJkP6Us0DjxxZz7DNULfCiD5KjryiOuukk8jaeillBcA3PESV7wxfLsazzv
ZYG44xnc3EpfIy4X3YGPBOdmwAIntRyRAtkqZK93Zozpx1khCgH2QHWeWQDLirUE
BOTAn8/YI1bLgwjncCwNm0CqUsw+KeGKVNL7dh6g0zz+N0nt62xZSQ+ICgLp34Jq
x5Q1LoWEtZWxsyhqw35VZzWAXZTSNPNc62eoMlkK1QKCAQEAygzwKc/ntSx83QNr
suxIUwOUhlZqDmd8Z21mzHI5sx3XnaAChkQHIrrNgEYPXRyFWJL8Rfhkrl7juBH7
U/P6PTROYFoAWA4FCmluNSdgCni3K6tfhSK0pEE+6cqKsgSCyW4FKA/+DqW0EAWf
axd/dGjmJIDngJHcxR5PEBvltZ0S1QYXCEu9HpRV2U7ufqOnIVpxhp9Pw+k1fXFN
ju2IOQvXjE0GE3KuuCUfQUKTQ/VmcY2BUXZji0Dwz/Y/D9eeHlZwW6VunD/6IBZM
kXbd9DX5Q8DRardUx/Nc+adhZIWzuZy+wYciRsqdwyMGpj1UnxpD+jJLmwHLYwoD
k/VRDwKCAQEAk9/k5q+4gKxYgLj14kQu/Co0FbEtzGsSL1GoWdehYdlBdiWwk2rE
W2nqN37JRbUJZAlNG29YbHerhGlZCte8iqW2cERno7bxN64Xjgq+iNmkvnkhtWxt
t3JR5j22LLka4q9IuM9YYnJEfi3VQXAr75PRBcl8ENk/t+djI7xzTIt8V+iKAOYJ
bo34Qsrq8MVRz8mYCXWTYOLPX+JA5ramEga/qHv5TRPKwNFCfDcsF878Au1pdiyA
/3CjOJohFL7iL5pj2TEcExrB+Fcn82tXzLmHQLgBYHjjFtiYY2Vqu+VLIay1Xupq
M7agEf/cZAzUuzY9p8kEwrqYfU5DfelDQQKCAQAvCaOriHZPOiHf0H9c+AGoKqAO
62mQdKHgmcoPzLcMa1TaLSIGbcdjT3I2vuRF68C0fwNdQPd5EK6W4Ld5EIxlhQzK
CxbUkTY5S+WUiN8CjZ/UDkyBjwcRoRIBUQ3n7FhENmfkaSBryBfUdzEjV9RR4jFo
gXoxV0uvY/j1cOXCq38DyqnxpGs67x8nlM0EEs28Kqw2uQdUloRSWSG684SL1wRb
ZG5bqJUaKogJzUYjb1N4G+BRdjLF3e0PlFRT6KsddIzvCQZ8vRyEIv9NukWR0oGp
KFemtzBSfdIqnlCA5rsuWGr7uv2WfOBuncjHCN9H4eAhLwxR2SZvegcGQX+2
-----END RSA PRIVATE KEY-----
EOF
}
runner=${RUNNER:-bubblewrap}
priv=${PRIV:-unpriv}
escalate=""
if [ "$priv" = "priv" ]; then
escalate=sudo
fi
if [ "$runner" = "qemu" ] && [ ! -f "$QEMU_KERNEL_IMAGE" ]; then
fail "runner qemu requireds QEMU_KERNEL_IMAGE. found '$QEMU_KERNEL_IMAGE'"
fi
melange="${MELANGE}"
buildd="$PWD/build.out"
pkgname="testownership"
if [ -z "$melange" ]; then
make melange || untestable "make melange failed"
melange="$PWD/melange"
else
command -v "$melange" || fail "'$melange' is not executable"
fi
env=(
env "QEMU_KERNEL_IMAGE=$QEMU_KERNEL_IMAGE"
)
common=(
--debug
"--runner=$runner"
--repository-append="$PWD/packages"
--keyring-append=local-melange.rsa.pub
--arch=x86_64
--env-file=build-x86_64.env
--keyring-append=https://packages.wolfi.dev/os/wolfi-signing.rsa.pub
--repository-append=https://packages.wolfi.dev/os
--source-dir="./$pkgname/"
)
buildcmd=(
${escalate:+"${escalate}"}
"${env[@]}"
"$melange" build "$pkgname.yaml"
"${common[@]}"
--license='Apache-2.0'
--signing-key=local-melange.rsa
--namespace=wolfi
--git-repo-url='https://github.com/wolfi-dev/os'
--cache-dir=./melange-cache
--pipeline-dir=./pipelines/
)
testcmd=(
${escalate:+"${escalate}"}
"${env[@]}"
"$melange" test "$pkgname.yaml"
"${common[@]}"
)
sudo rm -Rf "$buildd" || fail "rm $buildd failed"
mkdir "$buildd" &&
cd "$buildd" || fail "cd buildd failed"
export TMPDIR="$PWD"
catpkg > "$pkgname.yaml" || fail "write file failed"
sed -i -e "s,PACKAGE_NAME,$pkgname," "$pkgname.yaml" || fail "sed failed"
cat >build-x86_64.env <<"EOF"
export GOFLAGS=""
export GOTOOLCHAIN=local
export PYTHONHASHSEED=0
EOF
write_keys local-melange.rsa || fail "write kesy failed"
mkdir -p "$pkgname" || fail "mkdir $pkgname failed"
export SOURCE_DATE_EPOCH=1749656955
echo "${buildcmd[@]}"
"${buildcmd[@]}" || untestable "build failed"
apk=$(echo packages/x86_64/$pkgname*.apk)
[ -f "$apk" ] || untestable "didnt build an apk"
echo "melange test with:" "( cd build.out && " "${testcmd[@]}" ")"
expected="101:102"
f1="usr/bin/file4"
tar --numeric-owner -tvf "$apk" > apk.tvf 2>tvf.err &&
tar --numeric-owner -tvf "$apk" "$f1" > apk.file.tvf 2>tvf.err ||
{ cat tvf.err ; fail "tar tvf failed"; }
read perms owner _size _ymd _mmdd _path < apk.file.tvf || fail "read perms failed"
[ "$owner" = "0/0" ] && { echo "ERROR: owner of $f1 is $owner expected $expected"; exit 1; }
echo "PASS: owner of $f1 is $owner ($perms)"
exit 0
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment