I migrated all my packages from downloading off random github URLs to downloading off my server (commit)

Because github will throttle downloads of random archive files, causing intermittent CI failures
To speed up downloading packages, because some dependencies use the slower git+ URLs rather than archive .tar.gz URLs for their dependencies
To ensure building continues to work in the future even if some dependency authors delete their repositories, i.e. to prevent a left-pad incident

Unfortunately, this required updating the hashes of many packages even though their source code is identical. This is because some packages have dependencies that need their URLs updated to use the mirror.

Consider the dependency tree:

myapp
- depends on zls-0.15.0-rmm5fkjqIwDZpmDHyKwxa9K2gcI3FPaGVFPwjYWFBM5B
  - depends on lsp_kit-0.1.0-bi_PL04yCgAxLsF0hH2a5sZKT84MGQaKXouw2jvCE8Nl

I want to host a mirror for zls and lsp_kit. Mirroring lsp_kit is easy, I can upload the .tar.gz file without modifying anything. But to mirror zls, I need to change the url for lsp_kit in its build.zig.zon which means its hash changes:

// zls's build.zig.zon needs to be modified to use the mirror URL
.{
    .name = .zls,
    .dependencies = .{
        .lsp_kit = .{
-           .url = "github",
+           .url = "my-mirror",
            .hash = "lsp_kit-0.1.0-bi_PL04yCgAxLsF0hH2a5sZKT84MGQaKXouw2jvCE8Nl",
        },
    },
}

// now in myapp's build.zig.zon, the both the URL and the hash of zls have to change
.{
    .name = .myapp,
    .dependencies = .{
        .zls = .{
-           .url = "github",
-           .hash = "zls-0.15.0-rmm5fkjqIwDZpmDHyKwxa9K2gcI3FPaGVFPwjYWFBM5B",
+           .url = "my-mirror",
+           .hash = "zls-0.15.0-rmm5fvvpIwB3AOC3DOPZNIpt70PPexOUhee-0AEkAg3a",
        },
    },
}

Possible solutions

#14292 ability to specify additional build dependency mirrors that affect any package in the full dependency tree

This would work as long as outermost mirrors are tried before innermost mirrors. The order for which mirror gets tried first with complex dependency trees might be nondeterministic.

Offering a CLI flag to override all download URLs

Eg zig build --download-url=https://my-mirror/, and it would fetch every package from https://my-mirror/$hash rather than using the original specified URL.

Excluding URLs from the hash of a package

When hashing a package, it could rerender the build.zig.zon file to exclude 'url' fields, which means a mirror could upload the package with the urls of dependencies edited but that when downloaded still resolves to the same hash.

#23236 support torrenting for fetching and serving packages

You could provide a "mirror" by seeding all the packages you use.

#14291 build.zig dependencies: support url mirrors

This would somewhat help, but the upstream package has to cooperate with the mirror for it to work. To provide a mirror for zls & lsp_kit this way, you would have to first create a pull request to zls to add your mirror URLs for the dependencies, then once it is merged you can host a mirror of zls without changing the hash. You couldn't provide a mirror for old versions of zls that haven't added your mirror URLs in the build.zig.zon yet because mirroring the dependencies would require changing the build.zig.zon which changes the hash. Your mirror would likely be second on the list, so it would always try github first despite your mirror. Every time zls updates its dependencies, this would have to be redone.

No change to zig

You could download all the packages from the mirror and manually install them each in dependency order with zig fetch ./package.tar.gz, that way it never fetches from the original URLs. Or you could download one bundle with all the packages and extract it to your zig package cache.

pfgithub/zigmirrors.md

Select an option

No results found