Skip to content

Instantly share code, notes, and snippets.

@daemonhorn

daemonhorn/FreeBSD Arm64 Qemu.md

Last active October 3, 2023 12:20
Download ZIP

Revisions

  1. daemonhorn revised this gist Oct 3, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -63,11 +63,11 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * e.g.: `sh /qemu-data/start_guest.sh fido` will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup. Any device string that matches a substring of `usbconfig` output should work (e.g. `ugen0.4` or `fido` or `Mouse` or `Yubikey`)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd` (loads from `/usr/local/share/qemu/` by default)
    * Simple Example telnet console for **guest** (localhost port **4444**) and **qemu monitor** (localhost port **4445**)
    * tap device cleanup (if not in use)
    * Error checking to ensure that qemu is not already running.
    * Machine Configuration: CPU = 4, RAM = 4GB
    * Machine Configuration: CPU = 4, RAM = 4GB (Customize as desired)
    * Adjust `FreeBSD-14.0-BETA4-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    ```shell
    #!/bin/sh
  2. daemonhorn revised this gist Oct 3, 2023. 1 changed file with 8 additions and 16 deletions.
    24 changes: 8 additions & 16 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -6,7 +6,7 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    <!-- toc -->

    ## Dependancies
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest release as of this writeup is 8.1.0
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest pre-built package release as of this writeup is 8.1.0
    * Sufficient disk space (50+GB) on a mounted FreeBSD Host disk (e.g.: `/qemu-data` in this example)
    * Network environment that allows for multiple mac addresses on one switch port (or vswitch port configuration) for bridging mode

    @@ -19,11 +19,7 @@ fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA4/aarch64/Latest/
    unxz FreeBSD-14.0-BETA4-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA4-arm64-aarch64.qcow2 +40G
    ```
    2. Install qemu (or qemu-nox11 as desired)
    ```
    pkg install qemu
    ```
    3. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name). *Note: 802.11 Wireless (Wi-Fi) host NICs usually will have issues with bridging, so stick with wired ethernet*
    2. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name). *Note: 802.11 Wireless (Wi-Fi) host NICs usually will have issues with bridging, so stick with wired ethernet*
    ```
    ifconfig bridge0 create
    ifconfig bridge0 addm vmx0 edge vmx0 up
    @@ -33,7 +29,7 @@ Make permanant in `/etc/rc.conf` by adding:
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    cloned_interfaces="bridge0"
    ```
    4. Create Qemu network ifup/ifdown scripts `/usr/local/etc/qemu-if[up|down]`
    3. Modify Qemu network ifup/ifdown scripts `/usr/local/etc/qemu-if[up|down]`
    * Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.<br>
    `vi /usr/local/etc/qemu-ifup`
    ```
    @@ -47,11 +43,7 @@ ifconfig $1 up
    ifconfig $1 down
    ifconfig bridge0 deletem $1
    ```
    5. Enable the scripts to be directly executed by root user from qemu using:
    ```
    chmod +x /usr/local/etc/qemu_if*
    ```
    6. Boot Qemu with the pre-installed qcow2 image (text mode)
    4. Boot Qemu with the pre-installed qcow2 image (text mode). Alternatively, see shell script example below.
    ```
    qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -bios edk2-aarch64-code.fd -serial telnet:localhost:4444,mux=on,server,wait=off -vnc :0,websocket=on \
    @@ -60,10 +52,10 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,id=net0
    ```
    7. Login as `root` with no password, immediately set new password using `passwd`. (See below for accessing guest console via telnet)
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.
    5. Login as `root` with no password, immediately set new password using `passwd`. (See below for accessing guest console via telnet)
    6. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    7. Edit configuration file in `/etc/rc.conf` and give a hostname. Verify other settings.
    8. Use the new guest VM of Arm64(aarch64). Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

    ## Example Qemu startup script
    This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data/start_guest.sh`):
  3. daemonhorn revised this gist Oct 2, 2023. 1 changed file with 6 additions and 6 deletions.
    12 changes: 6 additions & 6 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -15,9 +15,9 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    ```
    mkdir /qemu-data
    cd /qemu-data
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA3/aarch64/Latest/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    unxz FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA3-arm64-aarch64.qcow2 +40G
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA4/aarch64/Latest/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2.xz
    unxz FreeBSD-14.0-BETA4-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA4-arm64-aarch64.qcow2 +40G
    ```
    2. Install qemu (or qemu-nox11 as desired)
    ```
    @@ -55,7 +55,7 @@ chmod +x /usr/local/etc/qemu_if*
    ```
    qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -bios edk2-aarch64-code.fd -serial telnet:localhost:4444,mux=on,server,wait=off -vnc :0,websocket=on \
    -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,id=net0
    @@ -76,7 +76,7 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * tap device cleanup (if not in use)
    * Error checking to ensure that qemu is not already running.
    * Machine Configuration: CPU = 4, RAM = 4GB
    * Adjust `FreeBSD-14.0-BETA3-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    * Adjust `FreeBSD-14.0-BETA4-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    ```shell
    #!/bin/sh
    # Safety check.
    @@ -105,7 +105,7 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -serial telnet:localhost:4444,mux=on,server,wait=off \
    -monitor telnet:localhost:4445,mux=on,server,wait=off \
    -display none \
    -drive if=none,file=/qemu-data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -drive if=none,file=/qemu-data/FreeBSD-14.0-BETA4-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,id=net0 \
  4. daemonhorn revised this gist Sep 28, 2023. 1 changed file with 8 additions and 6 deletions.
    14 changes: 8 additions & 6 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -19,11 +19,11 @@ fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA3/aarch64/Latest/
    unxz FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA3-arm64-aarch64.qcow2 +40G
    ```
    2. Install qemu
    2. Install qemu (or qemu-nox11 as desired)
    ```
    pkg install qemu-nox11
    pkg install qemu
    ```
    3. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name)
    3. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name). *Note: 802.11 Wireless (Wi-Fi) host NICs usually will have issues with bridging, so stick with wired ethernet*
    ```
    ifconfig bridge0 create
    ifconfig bridge0 addm vmx0 edge vmx0 up
    @@ -124,7 +124,7 @@ telnet localhost 4444
    ```
    ### Access Qemu monitor console
    Connect over telnet protocol from localhost to the **qemu monitor**.
    *Note: Use Ctrl+] to get a telnet prompt, then `quit` to exit telnet. Typing `quit` at monitor `(qemu)` prompt kills qemu guest.*
    *Note: Use Ctrl+] to get a telnet prompt, then `quit` to exit telnet. Typing `quit` at monitor `(qemu)` prompt non-gracefully kills qemu guest.*
    ```
    telnet localhost 4445
    ```
    @@ -179,8 +179,10 @@ hw.usb.usbhid.enable="1"
    - [X] Document ISO mapping syntax for install media
    - [X] Document Fresh aarch64 install via 13.2 bootonly ISO
    - [ ] Document qemu-guest-agent install and configuration
    - [ ] Document u2f virtual device (emulated and passthrough) - amd64 guest only ? - needs investigation
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [ ] Document u2f virtual device (emulated and passthrough) - ? - needs investigation
    - [ ] Look into porting https://github.com/Agnoctopus/libu2f-emu into FreeBSD to enable in Qemu
    - [ ] Look into other qemu feature build dependancies and meson build flags
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interface.

    ### References
    * Alternate aarch64 UEFI Firmware:
  5. daemonhorn revised this gist Sep 28, 2023. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -105,7 +105,7 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -serial telnet:localhost:4444,mux=on,server,wait=off \
    -monitor telnet:localhost:4445,mux=on,server,wait=off \
    -display none \
    -drive if=none,file=/data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -drive if=none,file=/qemu-data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,id=net0 \
    @@ -152,12 +152,12 @@ qemu-img create -f qcow2 FreeBSD-13.2-Release.qcow2 50G
    ```
    * Make modifications to qemu command line (change/addition from example above) to attach new qcow2 disk and virtual cdrom for ISO
    ```
    -cdrom /data/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso \
    -drive if=none,file=/data/FreeBSD-13.2-Release.qcow2,id=hd0 \
    -cdrom /qemu-data/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso \
    -drive if=none,file=/qemu-data/FreeBSD-13.2-Release.qcow2,id=hd0 \
    ```
    * Start new amd64 aarch64 13.2 guest and start installation by following the prompts.
    ```
    sh /data/start_guest.sh
    sh /qemu-data/start_guest.sh
    telnet localhost 4444
    ```
    * After installation is complete and system reboots, you can remove the `-cdrom...` line from the shell script as desired.
  6. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -193,4 +193,5 @@ hw.usb.usbhid.enable="1"
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
    * https://getlabsdone.com/4-easy-step-to-install-centos-on-kvm/
    * https://getlabsdone.com/4-easy-step-to-install-centos-on-kvm/
    * http://cdn.kernel.org/pub/linux/kernel/people/will/docs/qemu/qemu-arm64-howto.html
  7. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 7 additions and 4 deletions.
    11 changes: 7 additions & 4 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -183,10 +183,13 @@ hw.usb.usbhid.enable="1"
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.

    ### References
    * Alternate aarch64 UEFI Firmware (works):
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (9/22/23)
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4480/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (older)
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/`.
    * Alternate aarch64 UEFI Firmware:
    * Works:
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (9/22/23)
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4480/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (older)
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/`.
    * Does *NOT* work:
    * https://releases.linaro.org/components/kernel/uefi-linaro/latest/release/qemu64/QEMU_EFI.fd
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
  8. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 7 additions and 7 deletions.
    14 changes: 7 additions & 7 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -33,23 +33,23 @@ Make permanant in `/etc/rc.conf` by adding:
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    cloned_interfaces="bridge0"
    ```
    4. Create Qemu network ifup/ifdown scripts
    4. Create Qemu network ifup/ifdown scripts `/usr/local/etc/qemu-if[up|down]`
    * Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.<br>
    `vi /etc/qemu-ifup`
    `vi /usr/local/etc/qemu-ifup`
    ```
    #!/bin/sh
    ifconfig bridge0 addm $1 up
    ifconfig $1 up
    ```
    `vi /etc/qemu-ifdown`
    `vi /usr/local/etc/qemu-ifdown`
    ```
    #!/bin/sh
    ifconfig $1 down
    ifconfig bridge0 deletem $1
    ```
    5. Enable the scripts to be directly executed by root user from qemu using:
    ```
    chmod +x /etc/qemu_if*
    chmod +x /usr/local/etc/qemu_if*
    ```
    6. Boot Qemu with the pre-installed qcow2 image (text mode)
    ```
    @@ -58,7 +58,7 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0
    -netdev tap,id=net0
    ```
    7. Login as `root` with no password, immediately set new password using `passwd`. (See below for accessing guest console via telnet)
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    @@ -72,7 +72,7 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * e.g.: `sh /qemu-data/start_guest.sh fido` will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup. Any device string that matches a substring of `usbconfig` output should work (e.g. `ugen0.4` or `fido` or `Mouse` or `Yubikey`)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * Simple Example telnet console for **guest** (localhost port 4444) and **qemu monitor** (localhost port 4445)
    * Simple Example telnet console for **guest** (localhost port **4444**) and **qemu monitor** (localhost port **4445**)
    * tap device cleanup (if not in use)
    * Error checking to ensure that qemu is not already running.
    * Machine Configuration: CPU = 4, RAM = 4GB
    @@ -108,7 +108,7 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -drive if=none,file=/data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0 \
    -netdev tap,id=net0 \
    -usb \
    -device qemu-xhci,id=xhci \
    ${usb_qemu_cli} \
  9. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -164,7 +164,12 @@ telnet localhost 4444
    ### Yubikey usb host->guest passthrough with hidraw(4) usb driver
    - Untested if only guest has hidraw(4) kernel module loaded. (*should* work)
    - Seems to require `qemu-xhci` virtual usb device, ehci variant was problematic for unknown reason.
    - Works if both host and guest have enabled. Usually second device `/dev/hidraw1` is correct, depends on `kldstat -v` including `hkbd` or not.
    - Works if both host and guest have enabled hidraw kernel module and sysctl. Usually second device `/dev/hidraw1` is correct, depends on `kldstat -v` including `hkbd` or not. Example `/boot/loader.conf`:
    ```
    hidraw_load="YES"
    hkbd_load="YES"
    hw.usb.usbhid.enable="1"
    ```
    - Requires patches to libfido (included in 14.0-RELEASE, and stable after 9/23/23) - libfido2 1.13 for automatic detection
    - Manual workaround for older OS code is to pass device entry explicitly `ssh-keygen -v -K -O device=/dev/hidraw1`
    - Using string `fido` or `yubikey` for device selection and passthrough and above script works.
    @@ -184,4 +189,5 @@ telnet localhost 4444
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/`.
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
    * https://getlabsdone.com/4-easy-step-to-install-centos-on-kvm/
  10. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 9 additions and 2 deletions.
    11 changes: 9 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -161,7 +161,14 @@ sh /data/start_guest.sh
    telnet localhost 4444
    ```
    * After installation is complete and system reboots, you can remove the `-cdrom...` line from the shell script as desired.
    ### TODO
    ### Yubikey usb host->guest passthrough with hidraw(4) usb driver
    - Untested if only guest has hidraw(4) kernel module loaded. (*should* work)
    - Seems to require `qemu-xhci` virtual usb device, ehci variant was problematic for unknown reason.
    - Works if both host and guest have enabled. Usually second device `/dev/hidraw1` is correct, depends on `kldstat -v` including `hkbd` or not.
    - Requires patches to libfido (included in 14.0-RELEASE, and stable after 9/23/23) - libfido2 1.13 for automatic detection
    - Manual workaround for older OS code is to pass device entry explicitly `ssh-keygen -v -K -O device=/dev/hidraw1`
    - Using string `fido` or `yubikey` for device selection and passthrough and above script works.
    ## TODO
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
    - [X] Document USB Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [X] Document ISO mapping syntax for install media
    @@ -174,7 +181,7 @@ telnet localhost 4444
    * Alternate aarch64 UEFI Firmware (works):
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (9/22/23)
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4480/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (older)
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/`.
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
  11. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -124,7 +124,7 @@ telnet localhost 4444
    ```
    ### Access Qemu monitor console
    Connect over telnet protocol from localhost to the **qemu monitor**.
    *Note: Use Ctrl+] to get a telnet prompt, then quit to exit telnet*
    *Note: Use Ctrl+] to get a telnet prompt, then `quit` to exit telnet. Typing `quit` at monitor `(qemu)` prompt kills qemu guest.*
    ```
    telnet localhost 4445
    ```
  12. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 4 additions and 1 deletion.
    5 changes: 4 additions & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -171,7 +171,10 @@ telnet localhost 4444
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.

    ### References
    * Alternate UEFI Firmware (works): https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd
    * Alternate aarch64 UEFI Firmware (works):
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (9/22/23)
    * https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4480/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd (older)
    * FreeBSD emulators/qemu 8.1 port firmware: `edk2-aarch64-code.fd` automatically loaded from `/usr/local/share/qemu/
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
  13. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 3 additions and 3 deletions.
    6 changes: 3 additions & 3 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -162,13 +162,13 @@ telnet localhost 4444
    ```
    * After installation is complete and system reboots, you can remove the `-cdrom...` line from the shell script as desired.
    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
    - [X] Document USB Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [ ] Document ISO mapping syntax and connect/disconnect monitor commands
    - [X] Document ISO mapping syntax for install media
    - [X] Document Fresh aarch64 install via 13.2 bootonly ISO
    - [ ] Document qemu-guest-agent install and configuration
    - [ ] Document u2f virtual device (emulated and passthrough) - amd64 guest only ? - needs investigation
    - [ ] Document Fresh aarch64 install via ISO vs relase QCOW Image
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.

    ### References
    * Alternate UEFI Firmware (works): https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd
  14. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -94,8 +94,8 @@ if [ ! -z ${1} ] ; then
    usb_map_addr=$(echo ${usb_map} | grep -o -e [0-9]$)
    usb_qemu_cli="-device usb-host,hostbus=${usb_map_bus},hostaddr=${usb_map_addr},id=${1}"
    echo "Mapping usb device $(usbconfig | grep -ie ${1}) into the guest."
    echo -n "In qemu monitor, you can inspect with \"info usbhost\" command, "
    echo "or delete the usb device mapping with \"device_del ${1}\""
    echo -n "In qemu monitor, you can inspect attached usb guest devices with \"info usb\" "
    echo "command, or delete the usb device mapping with \"device_del ${1}\""
    fi
    # Cleanup tap0 interfaces that are not in use anymore.
    ifconfig tap0 2>/dev/null | grep -cq -e "Opened by PID" || ifconfig tap0 destroy 2>/dev/null
  15. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 22 additions and 1 deletion.
    23 changes: 22 additions & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -139,7 +139,28 @@ See https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details
    * `info usb` or `info usbhost` for information on usb guest/host devices respectively.
    * `info network` for network configuration and mac address
    * `info chardev` for serial/character device configuration.


    #### Example Fresh Install from ISO
    1. Prepare to install:
    * Download iso file (example is 13.2-RELEASE-AARCH64-bootonly variant )
    ```
    fetch https://download.freebsd.org/releases/ISO-IMAGES/13.2/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso
    ```
    * Create new blank drive image using `qemu-img create`
    ```
    qemu-img create -f qcow2 FreeBSD-13.2-Release.qcow2 50G
    ```
    * Make modifications to qemu command line (change/addition from example above) to attach new qcow2 disk and virtual cdrom for ISO
    ```
    -cdrom /data/FreeBSD-13.2-RELEASE-arm64-aarch64-bootonly.iso \
    -drive if=none,file=/data/FreeBSD-13.2-Release.qcow2,id=hd0 \
    ```
    * Start new amd64 aarch64 13.2 guest and start installation by following the prompts.
    ```
    sh /data/start_guest.sh
    telnet localhost 4444
    ```
    * After installation is complete and system reboots, you can remove the `-cdrom...` line from the shell script as desired.
    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
  16. daemonhorn revised this gist Sep 24, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -150,6 +150,7 @@ See https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details
    - [ ] Document Fresh aarch64 install via ISO vs relase QCOW Image

    ### References
    * Alternate UEFI Firmware (works): https://snapshots.linaro.org/components/kernel/leg-virt-tianocore-edk2-upstream/4989/QEMU-AARCH64/RELEASE_GCC5/QEMU_EFI.fd
    * https://bugs.launchpad.net/qemu/+bug/1849644
    * https://www.qemu.org/docs/master/system/devices/usb.html#connecting-usb-devices
    * https://unix.stackexchange.com/questions/452934/can-i-pass-through-a-usb-port-via-qemu-command-line
  17. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -60,7 +60,7 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0
    ```
    7. Login as `root` with no password, immediately set one using `passwd`. (See below for accessing guest console via telnet)
    7. Login as `root` with no password, immediately set new password using `passwd`. (See below for accessing guest console via telnet)
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.
    @@ -70,7 +70,7 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Command line parameter(optional) to perform qemu **host->guest USB passthrough** based upon usb device string
    * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * e.g.: `sh /qemu-data/start_guest.sh fido` will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup.
    * e.g.: `sh /qemu-data/start_guest.sh fido` will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup. Any device string that matches a substring of `usbconfig` output should work (e.g. `ugen0.4` or `fido` or `Mouse` or `Yubikey`)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * Simple Example telnet console for **guest** (localhost port 4444) and **qemu monitor** (localhost port 4445)
    * tap device cleanup (if not in use)
  18. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -70,6 +70,7 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Command line parameter(optional) to perform qemu **host->guest USB passthrough** based upon usb device string
    * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * e.g.: `sh /qemu-data/start_guest.sh fido` will check the usb device list for a FIDO USB device, and passthrough that device to the guest on startup.
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * Simple Example telnet console for **guest** (localhost port 4444) and **qemu monitor** (localhost port 4445)
    * tap device cleanup (if not in use)
  19. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 7 additions and 4 deletions.
    11 changes: 7 additions & 4 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -131,17 +131,20 @@ telnet localhost 4445
    See https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details.
    * `system_reset` - Reboot the vm
    * `system_pwerdown` - Send Powerdown event (may require qemu-guestd)
    * `device_add` usb-host,hostbus=xxx,hostaddr=yyy,id=friendlyname
    * `device_del` <id> (where id=friendlyname)
    * `info status` to see if the vm is running
    * `info usbhost` for information on usb passthrough devices
    * `device_add` `usb-host,hostbus=xxx,hostaddr=yyy,id=friendlyname`
    * `device_del` `id` (where `id`=friendlyname)
    * `info status` to see if the vm is running/paused
    * `info cpu` to check how many cpus are running/pids
    * `info usb` or `info usbhost` for information on usb guest/host devices respectively.
    * `info network` for network configuration and mac address
    * `info chardev` for serial/character device configuration.

    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
    - [X] Document USB Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [ ] Document ISO mapping syntax and connect/disconnect monitor commands
    - [ ] Document qemu-guest-agent install and configuration
    - [ ] Document u2f virtual device (emulated and passthrough) - amd64 guest only ? - needs investigation
    - [ ] Document Fresh aarch64 install via ISO vs relase QCOW Image

  20. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -128,11 +128,15 @@ Connect over telnet protocol from localhost to the **qemu monitor**.
    telnet localhost 4445
    ```
    #### Example monitor commands
    See https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details.
    * `system_reset` - Reboot the vm
    * `system_pwerdown` - Send Powerdown event (may require qemu-guestd)
    * `device_add` usb-host,hostbus=xxx,hostaddr=yyy,id=friendlyname
    * `device_del` <id> (where id=friendlyname)
    https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details.
    * `info status` to see if the vm is running
    * `info usbhost` for information on usb passthrough devices
    * `info network` for network configuration and mac address
    * `info chardev` for serial/character device configuration.

    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
  21. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -112,6 +112,7 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -device qemu-xhci,id=xhci \
    ${usb_qemu_cli} \
    -daemonize
    echo "Connect to guest console (telnet localhost 4444), or qemu monitor (telnet localhost 4445)"
    ```

    ### Access Qemu guest console
  22. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 4 additions and 4 deletions.
    8 changes: 4 additions & 4 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -15,7 +15,7 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    ```
    mkdir /qemu-data
    cd /qemu-data
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA1/aarch64/Latest/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA3/aarch64/Latest/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    unxz FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA3-arm64-aarch64.qcow2 +40G
    ```
    @@ -34,7 +34,7 @@ ifconfig_bridge0="addm vmx0 edge vmx0 up"
    cloned_interfaces="bridge0"
    ```
    4. Create Qemu network ifup/ifdown scripts
    Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.
    * Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.<br>
    `vi /etc/qemu-ifup`
    ```
    #!/bin/sh
    @@ -55,12 +55,12 @@ chmod +x /etc/qemu_if*
    ```
    qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -bios edk2-aarch64-code.fd -serial telnet:localhost:4444,mux=on,server,wait=off -vnc :0,websocket=on \
    -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA1-arm64-aarch64.qcow2,id=hd0 \
    -drive if=none,file=/qemu_data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0
    ```
    7. Login as `root` with no password, immediately set one using `passwd`.
    7. Login as `root` with no password, immediately set one using `passwd`. (See below for accessing guest console via telnet)
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.
  23. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -33,7 +33,8 @@ Make permanant in `/etc/rc.conf` by adding:
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    cloned_interfaces="bridge0"
    ```
    4. Create Qemu network ifup/ifdown scripts to handle bridge and tap syntax.
    4. Create Qemu network ifup/ifdown scripts
    Example shown is designed to handle bridge and tap devices automatically as the guest is started/stopped.
    `vi /etc/qemu-ifup`
    ```
    #!/bin/sh
  24. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 15 additions and 6 deletions.
    21 changes: 15 additions & 6 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -16,8 +16,8 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    mkdir /qemu-data
    cd /qemu-data
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA1/aarch64/Latest/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    unxz FreeBSD-14.0-BETA1-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA1-arm64-aarch64.qcow2 +40G
    unxz FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA3-arm64-aarch64.qcow2 +40G
    ```
    2. Install qemu
    ```
    @@ -67,8 +67,8 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    ## Example Qemu startup script
    This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data/start_guest.sh`):
    * Command line parameter(optional) to perform qemu **host->guest USB passthrough** based upon usb device string
    * * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * Simple Example telnet console for **guest** (localhost port 4444) and **qemu monitor** (localhost port 4445)
    * tap device cleanup (if not in use)
    @@ -77,6 +77,10 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Adjust `FreeBSD-14.0-BETA3-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    ```shell
    #!/bin/sh
    # Safety check.
    ps -aux | grep -v grep | grep -cq qemu-system && \
    echo "Qemu is already running. Shutdown the guest(s), then retry." && exit 1
    # if arg(1) is set, use it as a string to match on for usb passthrough to guest.
    if [ ! -z ${1} ] ; then
    echo "Attempting to passthrough usb host device based on query string: ${1}"
    usb_map_count=$(usbconfig | grep -cie ${1})
    @@ -91,9 +95,8 @@ if [ ! -z ${1} ] ; then
    echo -n "In qemu monitor, you can inspect with \"info usbhost\" command, "
    echo "or delete the usb device mapping with \"device_del ${1}\""
    fi
    # Cleanup tap0 interfaces that are not in use anymore.
    ifconfig tap0 2>/dev/null | grep -cq -e "Opened by PID" || ifconfig tap0 destroy 2>/dev/null
    ps -aux | grep -v grep | grep -cq qemu-system && \
    echo "Qemu is already running. Shutdown the guest(s), then retry." && exit 1
    echo "Starting Qemu in background..."
    qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -bios edk2-aarch64-code.fd \
    @@ -122,6 +125,12 @@ Connect over telnet protocol from localhost to the **qemu monitor**.
    ```
    telnet localhost 4445
    ```
    #### Example monitor commands
    * `system_reset` - Reboot the vm
    * `system_pwerdown` - Send Powerdown event (may require qemu-guestd)
    * `device_add` usb-host,hostbus=xxx,hostaddr=yyy,id=friendlyname
    * `device_del` <id> (where id=friendlyname)
    https://qemu-project.gitlab.io/qemu/system/monitor.html for complete details.

    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
  25. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 1 addition and 3 deletions.
    4 changes: 1 addition & 3 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,8 +3,6 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    - [Dependancies](#Dependancies)
    - [Getting Started](#Getting-Started)
    - [Example Qemu guest startup script](#Example-qemu-startup-script)
    * [Sub-heading](#sub-heading-2)
    + [Sub-sub-heading](#sub-sub-heading-2)
    <!-- toc -->

    ## Dependancies
    @@ -77,7 +75,7 @@ This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data
    * Error checking to ensure that qemu is not already running.
    * Machine Configuration: CPU = 4, RAM = 4GB
    * Adjust `FreeBSD-14.0-BETA3-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    ```
    ```shell
    #!/bin/sh
    if [ ! -z ${1} ] ; then
    echo "Attempting to passthrough usb host device based on query string: ${1}"
  26. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 8 additions and 2 deletions.
    10 changes: 8 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,11 @@
    # Intro
    This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or newer), and provides an example configuration for running **arm64 (aarch64) FreeBSD guest** on an **amd64 FreeBSD Host**. Concepts can be applied to other architectures as desired, but syntax and capabilities will vary.
    - [Dependancies](#Dependancies)
    - [Getting Started](#Getting-Started)
    - [Example Qemu guest startup script](#Example-qemu-startup-script)
    * [Sub-heading](#sub-heading-2)
    + [Sub-sub-heading](#sub-sub-heading-2)
    <!-- toc -->

    ## Dependancies
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest release as of this writeup is 8.1.0
    @@ -60,8 +66,8 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

    ### (optional) Enable useful example configurations via `/qemu-data/start_guest.sh`
    This example `sh` script provides an arm64/aarch64 qemu guest launch:
    ## Example Qemu startup script
    This example `sh` script provides an arm64/aarch64 qemu guest launch(`/qemu-data/start_guest.sh`):
    * Command line parameter(optional) to perform qemu **host->guest USB passthrough** based upon usb device string
    * * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
  27. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 4 additions and 10 deletions.
    14 changes: 4 additions & 10 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -5,25 +5,19 @@ This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest release as of this writeup is 8.1.0
    * Sufficient disk space (50+GB) on a mounted FreeBSD Host disk (e.g.: `/qemu-data` in this example)
    * Network environment that allows for multiple mac addresses on one switch port (or vswitch port configuration) for bridging mode
    * Configure bridge(as desired) in `/etc/rc.conf` by adding:
    ```
    cloned_interfaces="bridge0"
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    ```
    Modify `vmx0` to match your existing host network interface name. Wireless (802.11) interfaces may not work properly in bridging mode.

    ## Getting Started
    1. Download desired qcow2 image from FreeBSD site, decompress, and resize (to increase by desired size. Example shows adding 40GB):
    ```
    mkdir /qemu-data
    cd /qemu-data
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA1/aarch64/Latest/FreeBSD-14.0-BETA1-arm64-aarch64.qcow2.xz
    fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA1/aarch64/Latest/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2.xz
    unxz FreeBSD-14.0-BETA1-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA1-arm64-aarch64.qcow2 +40G
    ```
    2. Install qemu, novnc, python
    2. Install qemu
    ```
    pkg install qemu novnc python
    pkg install qemu-nox11
    ```
    3. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name)
    ```
    @@ -68,7 +62,7 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \

    ### (optional) Enable useful example configurations via `/qemu-data/start_guest.sh`
    This example `sh` script provides an arm64/aarch64 qemu guest launch:
    * Command line parameter(optional) to perform **host->guest USB passthrough** based upon usb device string
    * Command line parameter(optional) to perform qemu **host->guest USB passthrough** based upon usb device string
    * * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
  28. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 27 additions and 16 deletions.
    43 changes: 27 additions & 16 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,5 +1,5 @@
    # Intro
    This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or newer), and provides an example configuration for running **arm64 (aarch64) FreeBSD guest** on a **amd64 FreeBSD Host**. Concepts can be applied to other architectures as desired, but syntax and capabilities will vary.
    This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or newer), and provides an example configuration for running **arm64 (aarch64) FreeBSD guest** on an **amd64 FreeBSD Host**. Concepts can be applied to other architectures as desired, but syntax and capabilities will vary.

    ## Dependancies
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest release as of this writeup is 8.1.0
    @@ -21,11 +21,11 @@ fetch https://download.freebsd.org/releases/VM-IMAGES/14.0-BETA1/aarch64/Latest/
    unxz FreeBSD-14.0-BETA1-arm64-aarch64.qcow2.xz
    qemu-img resize FreeBSD-14.0-BETA1-arm64-aarch64.qcow2 +40G
    ```
    1. Install qemu, novnc, python
    2. Install qemu, novnc, python
    ```
    pkg install qemu novnc python
    ```
    2. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name)
    3. Configure networking bridge for use by Qemu guests (replace `vmx0` with host network interface name)
    ```
    ifconfig bridge0 create
    ifconfig bridge0 addm vmx0 edge vmx0 up
    @@ -35,24 +35,24 @@ Make permanant in `/etc/rc.conf` by adding:
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    cloned_interfaces="bridge0"
    ```
    3. Create Qemu network ifup/ifdown scripts to handle bridge and tap syntax.
    <br>`vi /etc/qemu-ifup`
    4. Create Qemu network ifup/ifdown scripts to handle bridge and tap syntax.
    `vi /etc/qemu-ifup`
    ```
    #!/bin/sh
    ifconfig bridge0 addm $1 up
    ifconfig $1 up
    ```
    <br>`vi /etc/qemu-ifdown`
    `vi /etc/qemu-ifdown`
    ```
    #!/bin/sh
    ifconfig $1 down
    ifconfig bridge0 deletem $1
    ```
    4. Enable the scripts to be directly executed by root user from qemu using:
    5. Enable the scripts to be directly executed by root user from qemu using:
    ```
    chmod +x /etc/qemu_if*
    ```
    5. Boot Qemu with the pre-installed qcow2 image (text mode)
    6. Boot Qemu with the pre-installed qcow2 image (text mode)
    ```
    qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -bios edk2-aarch64-code.fd -serial telnet:localhost:4444,mux=on,server,wait=off -vnc :0,websocket=on \
    @@ -61,7 +61,22 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0
    ```
    6. (optional) `/qemu/data/start_arm.sh` Example:
    7. Login as `root` with no password, immediately set one using `passwd`.
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

    ### (optional) Enable useful example configurations via `/qemu-data/start_guest.sh`
    This example `sh` script provides an arm64/aarch64 qemu guest launch:
    * Command line parameter(optional) to perform **host->guest USB passthrough** based upon usb device string
    * * Supports any USB device mapping from the Host to Guest based upon detected device seen by `usbconfig list`
    * * Supports USB mapping via nested virtualization/emulation (VMWare VM of FreeBSD amd64 hosting Qemu VM of FreeBSD aarch64)
    * EFI firmware image from qemu-aarch64 `-bios edk2-aarch64-code.fd`
    * Simple Example telnet console for **guest** (localhost port 4444) and **qemu monitor** (localhost port 4445)
    * tap device cleanup (if not in use)
    * Error checking to ensure that qemu is not already running.
    * Machine Configuration: CPU = 4, RAM = 4GB
    * Adjust `FreeBSD-14.0-BETA3-arm64-aarch64.qcow2` disk image name to match applicable qcow2 image desired.
    ```
    #!/bin/sh
    if [ ! -z ${1} ] ; then
    @@ -96,10 +111,6 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    ${usb_qemu_cli} \
    -daemonize
    ```
    7. Login as `root` with no password, immediately set one using `passwd`.
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

    ### Access Qemu guest console
    Connect over telnet protocol from localhost to the **guest console**.
    @@ -117,9 +128,9 @@ telnet localhost 4445
    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
    - [X] Document Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [ ] Document u2f virtual device (emulated and passthrough)
    - [ ] Document Fresh aarch64 install via ISO vs QCOW Image
    - [X] Document USB Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [ ] Document u2f virtual device (emulated and passthrough) - amd64 guest only ? - needs investigation
    - [ ] Document Fresh aarch64 install via ISO vs relase QCOW Image

    ### References
    * https://bugs.launchpad.net/qemu/+bug/1849644
  29. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 38 additions and 22 deletions.
    60 changes: 38 additions & 22 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,15 +1,19 @@
    # Intro
    - [X] Notes from setting up FreeBSD ARM64 (aarch64) for FreeBSD 14-Beta1 in a Qemu runtime
    - [ ] include guest console via vnc, novnc(web) - WIP
    - [X] include guest console via serial(telnet)
    - [X] include resizing of disk image `qemu-img resize`
    This Quickstart receipe for Qemu assumes a recent FreeBSD release (stable/13 or newer), and provides an example configuration for running **arm64 (aarch64) FreeBSD guest** on a **amd64 FreeBSD Host**. Concepts can be applied to other architectures as desired, but syntax and capabilities will vary.

    ## Dependancies
    * Sufficient disk space on a mounted disk (e.g.: `/qemu-data` in this example)
    * Install qemu `pkg install qemu` or `pkg instal qemu-nox11`. Latest release as of this writeup is 8.1.0
    * Sufficient disk space (50+GB) on a mounted FreeBSD Host disk (e.g.: `/qemu-data` in this example)
    * Network environment that allows for multiple mac addresses on one switch port (or vswitch port configuration) for bridging mode
    * Configure bridge(as desired) in `/etc/rc.conf` by adding:
    ```
    cloned_interfaces="bridge0"
    ifconfig_bridge0="addm vmx0 edge vmx0 up"
    ```
    Modify `vmx0` to match your existing host network interface name. Wireless (802.11) interfaces may not work properly in bridging mode.

    # Getting Started
    1. Download qcow2 image from FreeBSD site, decompress, and resize (to increase by desired size. Example shows adding 40GB):
    ## Getting Started
    1. Download desired qcow2 image from FreeBSD site, decompress, and resize (to increase by desired size. Example shows adding 40GB):
    ```
    mkdir /qemu-data
    cd /qemu-data
    @@ -62,15 +66,19 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    #!/bin/sh
    if [ ! -z ${1} ] ; then
    echo "Attempting to passthrough usb host device based on query string: ${1}"
    usb_map_count=`usbconfig | grep -cie ${1}`
    [ ${usb_map_count} -ne 1 ] && echo "Total devices matched: ${usb_map_count} is not equal to 1, please refine." && usbconfig && exit 1
    usb_map=`usbconfig | grep -ie ${1} | grep -o -e [0-9]\.[0-9]`
    usb_map_bus=`echo ${usb_map} | grep -o -e ^[0-9]`
    usb_map_addr=`echo ${usb_map} | grep -o -e [0-9]$`
    usb_host_qemu_cli="-device usb-host,hostbus=${usb_map_bus},hostaddr=${usb_map_addr},id=${1}"
    echo "Mapping usb device `usbconfig | grep -ie ${1}` into the guest."
    usb_map_count=$(usbconfig | grep -cie ${1})
    [ ${usb_map_count} -ne 1 ] && \
    echo "Total devices matched: ${usb_map_count} is not equal to 1, please refine." && \
    usbconfig && exit 1
    usb_map=$(usbconfig | grep -ie ${1} | grep -o -e [0-9]\.[0-9])
    usb_map_bus=$(echo ${usb_map} | grep -o -e ^[0-9])
    usb_map_addr=$(echo ${usb_map} | grep -o -e [0-9]$)
    usb_qemu_cli="-device usb-host,hostbus=${usb_map_bus},hostaddr=${usb_map_addr},id=${1}"
    echo "Mapping usb device $(usbconfig | grep -ie ${1}) into the guest."
    echo -n "In qemu monitor, you can inspect with \"info usbhost\" command, "
    echo "or delete the usb device mapping with \"device_del ${1}\""
    fi
    ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy
    ifconfig tap0 2>/dev/null | grep -cq -e "Opened by PID" || ifconfig tap0 destroy 2>/dev/null
    ps -aux | grep -v grep | grep -cq qemu-system && \
    echo "Qemu is already running. Shutdown the guest(s), then retry." && exit 1
    echo "Starting Qemu in background..."
    @@ -85,23 +93,31 @@ qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0 \
    -usb \
    -device qemu-xhci,id=xhci \
    ${usb_host_qemu_cli} \
    ${usb_qemu_cli} \
    -daemonize
    ```
    7. (optional) Connect over telnet protocol from localhost to the guest console.
    7. Login as `root` with no password, immediately set one using `passwd`.
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things. Package builds for betas may lag ISO releases.

    ### Access Qemu guest console
    Connect over telnet protocol from localhost to the **guest console**.
    *Note: Use Ctrl+] to get a telnet prompt, then quit to exit telnet*
    ```
    telnet localhost 4444
    ```
    7. Login as `root` with no password, immediately set one using `passwd`.
    8. Create users using `adduser`, start sshd using `service sshd enable && service sshd start`
    9. Edit configuration file in `/etc/rc.conf` and give a hostname. verify settings.
    10. Use the new Arm64 (aarch64) version of FreeBSD 14 Beta 1. Since this is now a higher tier variant, there will be pre-built packages for use with `pkg` utility as well as all of the other normal FreeBSD base things.
    ### Access Qemu monitor console
    Connect over telnet protocol from localhost to the **qemu monitor**.
    *Note: Use Ctrl+] to get a telnet prompt, then quit to exit telnet*
    ```
    telnet localhost 4445
    ```

    ### TODO
    - [ ] Determine why attempting to destroy tapX interface using `ifconfig tap0 destroy` format hangs instead of erroring out when there is still a pid associated with the interrface.
    - [X] Write a detection script to check if there is a pid attached, and destroy if not: `ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy`
    - [ ] Document Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [X] Document Yubikey passthrough from ESXi->FreeBSD VM(amd64)->FreeBSD QEMU(aarch64)
    - [ ] Document u2f virtual device (emulated and passthrough)
    - [ ] Document Fresh aarch64 install via ISO vs QCOW Image

  30. daemonhorn revised this gist Sep 23, 2023. 1 changed file with 16 additions and 2 deletions.
    18 changes: 16 additions & 2 deletions FreeBSD Arm64 Qemu.md
    View file
    Original file line number Diff line number Diff line change
    @@ -59,19 +59,33 @@ qemu-system-aarch64 -m 4096M -cpu cortex-a57 -M virt \
    ```
    6. (optional) `/qemu/data/start_arm.sh` Example:
    ```
    #!/bin/csh
    #!/bin/sh
    if [ ! -z ${1} ] ; then
    echo "Attempting to passthrough usb host device based on query string: ${1}"
    usb_map_count=`usbconfig | grep -cie ${1}`
    [ ${usb_map_count} -ne 1 ] && echo "Total devices matched: ${usb_map_count} is not equal to 1, please refine." && usbconfig && exit 1
    usb_map=`usbconfig | grep -ie ${1} | grep -o -e [0-9]\.[0-9]`
    usb_map_bus=`echo ${usb_map} | grep -o -e ^[0-9]`
    usb_map_addr=`echo ${usb_map} | grep -o -e [0-9]$`
    usb_host_qemu_cli="-device usb-host,hostbus=${usb_map_bus},hostaddr=${usb_map_addr},id=${1}"
    echo "Mapping usb device `usbconfig | grep -ie ${1}` into the guest."
    fi
    ifconfig tap0 | grep -cq -e "Opened by PID" || ifconfig tap0 destroy
    ps -aux | grep -v grep | grep -cq qemu-system && \
    echo "Qemu is already running. Shutdown the guest(s), then retry." && exit 1
    echo "Starting Qemu in background..."
    qemu-system-aarch64 -m 4096M -cpu max -smp cpus=4 -M virt \
    -bios edk2-aarch64-code.fd \
    -serial telnet:localhost:4444,mux=on,server,wait=off \
    -monitor telnet:localhost:4445,mux=on,server,wait=off \
    -display none \
    -drive if=none,file=/data/FreeBSD-14.0-BETA1-arm64-aarch64.qcow2,id=hd0 \
    -drive if=none,file=/data/FreeBSD-14.0-BETA3-arm64-aarch64.qcow2,id=hd0 \
    -device virtio-blk-device,drive=hd0 \
    -device virtio-net-device,netdev=net0 \
    -netdev tap,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown,id=net0 \
    -usb \
    -device qemu-xhci,id=xhci \
    ${usb_host_qemu_cli} \
    -daemonize
    ```
    7. (optional) Connect over telnet protocol from localhost to the guest console.