cross-compiling-for-archpower.md

Cross-compiling for ArchPower

Here's how I setup reasonably fast builds for ArchPower 32-bit. There are many ways to do this, but this is what worked for me, and it made my builds much faster than native.

Warning: Cross-compiling could give you different results from compiling on a real POWER/PowerPC box! Usually it's fine, but be cautious about distributing the resulting binaries.

We'll use icecream to coordinate cross-compilation.

Build a toolchain targeting ArchPower

• Create a basic Arch system
  • Not ArchPower, but Arch for amd64 or aarch64, something fast
  • An easy way to do this in isolation is with Distrobox. Eg: distrobox create -n arch-ppc-toolchain -i archlinux
• Install base-devel
• Edit makepkg.conf, set MAKEFLAGS for the right level of concurrency for your system, eg: -j8 for 8 cores
• Clone the archpower repo
• Install cross-compilers from ArchPower
  • Use the packages with powerpc-unknown-linux-gnu- prefix, since we're targeting 32-bit.
    • If you want to target 64-bit, use a different prefix instead
  • In order: binutils linux-api-headers gcc-stage1 glibc-headers gcc-stage2 glibc libxcrypt gcc pacman
  • Build each one using makepkg -si
• Install yay-bin
  • Set defaults: yay --save --answerclean All --answerdiff None --answeredit None --removemake --sudoloop
• Use yay to install icecream-git
• Package the toolchain: /usr/lib/icecream/bin/icecc-create-env $(which powerpc-unknown-linux-gnu-gcc).
  • This produces a tarball. Rename it something like icecc-arch-amd64-ppc32.tgz, and copy it somewhere safe.

Setup icecc hosts

• These should be fast computers, running any distro
• They should be accessible to the network. Native machines on Ethernet are best.
  • You can probably get this working with VM hosts, but you'll need to be very careful with your networking
• Install icecream for your distro. Instructions will vary by distro!
  • For Fedora, install the icecream package, then run the iceccd service
  • For Arch, install icecream from the AUR, then run the icecream service
• Pick one host to run the icecream scheduler
  • On that host, start the scheduler service, eg: icecream-scheduler
• You probably won't have to configure anything special
• You should also install icemon, a GUI program for monitoring icecream

Setup a build box

• This could be a real POWER computer, or a VM
• If you want to use a VM, I recommend:
  • Run it on the same host that runs the icecc scheduler, to simplify networking
  • Configure the scheduler to run on the network the VM is on, using the -n flag. Eg: use the IP of the host on the virbr0 network if you're using libvirt.
  • Run ArchPower 64-bit with a 32-bit userland. This lets you use many virtual CPUs and a fair amount of memory
    • Libvirt can set up a VM easily if you have qemu-system-ppc64 installed
    • Use the archpower-current6432.iso to install, following the KVM instructions
• Once this is installed, run icecream on your VM
  • Install the icecream package
  • Configure icecream.conf. Set ICECREAM_MAX_JOBS=1, so we prefer to use the fast cross-compiler
  • Start the icecream service
• Check your icecream network
  • Open icemon on some computer in your icecream network, and make sure it can find the scheduler
  • Choose View -> Star view. Make sure all your hosts appear
  • Usually this is fine. But if something doesn't look right, you may have to configure icecream on one or more hosts to help them find the scheduler, or make sure the services start up ok.
    • The config files are different on every distro, so good luck!
• Finish configuring icecream
  • In /usr/lib/icecream/libexec/icecc/bin, create prefixed aliases for the compilers. Eg: ln -s /usr/lib/icecream/bin/icecc /usr/lib/icecream/libexec/icecc/bin/powerpc-unknown-gnu-cc, and so on for c++, gcc, g++
  • Copy your icecream toolchain (eg icecc-arch-amd64-ppc32.tgz) somewhere like /var/lib/
• Test icecream
  • Create a small C hello world file
  • export PATH="/usr/lib/icecream/libexec/icecc/bin:$PATH", so icecc wrappers come before real gcc
  • export ICECC_VERSION=x86_64:/var/lib/icecc-arch-amd64-ppc32.tgz (or whatever you named it), so icecc can tell the cross-hosts how to build
  • Run ICECC_DEBUG=debug gcc -c -o hello.o hello.c. You hopefully see messages about distributing the build!
  • If you open icemon, go View -> List, and try to compile something, it should show it building on another host.
• Make your config permanent
  • Set PATH and ICECC_VERSION in your shell config
  • You may also want to set MAKEFLAGS to set parallelism really high, enough to saturate all your icecream hosts
• Then setup for package building
  • Enable whatever pacman repos you want in pacman.conf. I recommend testing and maybe TechFlash's extra packages
  • Clone the archpower repo
  • Set makepkg.conf flags appropriate for your builds. I use !debug !lto so my builds don't take forever

Build things!

• On your build box (or VM), just cd into directories in the archpower repo, and run makepkg!
• Beware that there are still limitations on the speed you get
  • Configuring and linking have to happen natively
  • Non-C languages, like Rust, are harder to distribute
• You may want to setup a repo with repo-add, and make it accessible to your ArchPower hosts
• If you get some new interesting packages working, share your PKGBUILDs with the community please!
  • Here are mine
• Let me know if you come up with any fun improvements to this process