Skip to content

Instantly share code, notes, and snippets.

@roens

roens/RPiOS-on-HA-Yellow.md

Last active June 5, 2023 01:39
Show Gist options
  • Save roens/08276e30a2995129522e9cf09456dd14 to your computer and use it in GitHub Desktop.
Save roens/08276e30a2995129522e9cf09456dd14 to your computer and use it in GitHub Desktop.
Download ZIP
Raspberry Pi OS on Home Assistant Yellow
Raw
RPiOS-on-HA-Yellow.md

So far, this is a WIP

I'll drop WIP status when I've worked out:

  • ✅ Yellow boots Raspberry Pi OS
  • Running in Docker (compose):
    • ✅ Home Assistant
    • ✅ Zigbee2MQTT (Z2M): Can add & communicate with Zigbee devices
    • ❌ Z-Wave JS UI: Can add & communicate with Aeotec Z-Pi 7 radio (I don't yet have any Z-Wave devices)
      • Sticking point so far is working out how to connect it to the proper UART port, which may be conflicting with the built-in Bluetooth radio in the CM4
    • ✅ Mosquitto (MQTT)
    • ✅ Limited communication with MQTT
      • Only HA, Z2M, & Z-Wave containers allowed
      • Zero access from outside the host

Running Raspberry Pi OS (Raspbian) on Home Assistant Yellow

Start with some tooling.

Set up usbboot on laptop

usbboot is an RPi tool for working with a Compute Module 4 (CM4). I mainly use it for mounting the built-in eMMC storage of a CM4 on my laptop over a USB cable.

Install requirements, run one of:

  • brew install libusb pkg-config — macOS
  • apt install git libusb-1.0-0-dev pkg-config — Linux / Cygwin / WSL

Then clone & build it. I found that the binaries in the ./tools dir didn’t get built unless I cloned the entire repo (without using --depth=1).

git clone https://github.com/raspberrypi/usbboot
cd usbboot
make

Among other things, uou can use it to mount the /boot partition of the built-in eMMC storage of your CM4 as a storage device on your laptop with:

./rpiboot

If that seems to hang with Waiting for BCM2835/6/7/2711…, try just power-cycling Yellow.

How to get Serial Console of CM4 on Yellow

It’s great that there are three LEDs on Yellow to provide some degree of feedback. But I find it useful to get into the details as to what’s going on. Make sure these two jumpers are in their “factory default” positions:

  • JP1: UART (as it is labeled on the board)
  • JP2: open (not shorted with the provided jumper)

Then, connect a USB-C cable between Yellow an your laptop. With the jumpers in the positions noted above, whether Yellow is powered or not, your laptop should now see a new /dev/cu.usbserial-NNNN (where NNNN are 4 digits, perhaps 1410 or 1420).

With the USB-UART talking to your laptop, now you can use your favorite tool to connect to it. I tend to use:

screen /dev/cu.usbserial-1420 115200

Home Assistant Yellow

Serial ports:

  • ttyACM0: ?
  • ttyAMA0: “miniUART” … symlinked to serial0 … meant to be built-in Bluetooth
  • ttyAMA1: The Yellow Zigbee/Thread (IEEE 802.15.4) radio
  • ttyAMA2: ?
  • ttyS0: symlinked to serial1 … meant to be GPIO serial port

Preparing Yellow for “not HassOS“

The point of all this is to end up with a Home Assistant Yellow that can run HASS while also affording full access to the host OS running on it. My main reason for this is to be able to run Zigbee2MQTT on my Yellow because many of my Zigbee devices aren’t well supported by ZHA. And I was unable to get the Z2M AddOn to run such that I could access its web frontend. Also, being that I’m migrating from running all this on a full server, in Docker, this is the “easier” (to an extent).

I found this forum posting useful for some of the details in getting Raspbian to boot my Yellow.

Write Yellow Install to USB flash drive with Raspberry Pi Imager

Probably there’s a better path to acquiring the necessary .dtb & .dtbo, but this works. Using Raspberry Pi Imager, click on the big button for “Operating System”, and choose: Other specific-purpose OS -> Home assistants & home automation -> Home Assistant -> Home Assistant OS Installer for Yellow. Then click on the big button for “Storage”, and choose the (should be) one storage device available, then click “Write”.

  • When that finishes, if it’s not still mounted, un/plug it back in to your laptop. Then you can copy the two Yellow-specific /boot files to the MC4 eMMC storage, below.

Write Raspbian to CM4 eMMC

This is like above for the Yellow OS image to USB flash drive but with an adjustment to those jumpers in your Yellow so that the CM4 can be mounted on your laptop. This will use usbboot (setup on your laptop described above).

References:

Mount CM4 eMMC as a USB storage device

Set Yellow from UART to USB mode with the provided jumpers: move JP1 to “USB” (as it is labeled on the board), and JP2 shorted with the provided jumper. Then connect Yellow to your laptop with a USB-C cable and run this to mount the CM4 eMMC on your laptop:

cd usbboot  # where ever you set it up from the section above
sudo ./rpiboot

While that waits, plug power in to your Yellow (either 12V AC adapter or Ethernet connected to a PoE power source). The waiting rpiboot command should see it and do its thing. If not, you can try power cycling your Yellow.

The eMMC storage should now be mounted as a storage device on your laptop.

Write Raspberry Pi OS to CM4 eMMC storage

Again with Raspberry Pi Imager, write the Raspberry Pi OS image to the CM4 eMMC. Click on the big button for “Operating System”, and choose: Raspberry Pi OS (other) -> Raspberry Pi OS Lite (32-Bit). Then click on the big button for “Storage”, and choose the (should be) one storage device available, then click “Write”.

While I happened to choose the 32-bit variation, probably all below will work with the 64-bit image too.

Configure Yellow to boot from NVMe

How to Boot a Pi CM4 from NVMe SSD - {DPHacks}

Use usbboot to point to NVMe partition for /.

cd usbboot  # where ever you set it up from the section above

Change BOOT_ORDER in the recovery boot.conf:

sed -i ‘s/\(BOOT_ORDER=0\)xf25641/\1xf25416/’ recovery/boot.conf

Update pieeprom.bin with the new boot order settings:

cd recovery  # Which is a subdir of usbboot
./update-pieeprom.sh
cd ..

Installing & Configuring things

Install Docker (official)

Install Docker, rather than the docker.io Debian package. Follow the steps outlined in their Documentation, completing with the “Install Docker Engine” section (about a third of the way down the page). This will get you both Docker and Docker Compose.

Install apt packages

These are packages that I found did not come pre-installed with “Raspberry Pi OS Lite” (yeah, just one so far; aside from packages I personally desire, outside of the scope of this document):

sudo apt install \
  unattended-upgrades   # useful

Enable HASS Discoverability

When not using network_mode: host for the HA container, and just exposing its web service on port 8123, we can use avahi to help HA in discovering Zeroconf services (all those wired and WiFi devices it can interact with on your network). Uncomment and alter this line in avahi-daemon.conf:

sudo sed -i ‘s/#\(enable-reflector=\)no/\1yes/’ /etc/avahi/avahi-daemon.conf

# Then restart avahi:
sudo service avahi-daemon restart

The above changes enable:

  • enable-reflector
    • avahi-daemon will reflect incoming mDNS requests to all local network interfaces, effectively allowing clients to browse mDNS/DNS-SD services on all networks connected to the gateway. The gateway is somewhat intelligent and should work with all kinds of mDNS traffic, though some functionality is lost (specifically the unicast reply bit, which is used rarely anyway). Make sure to not run multiple reflectors between the same networks, this might cause them to play Ping Pong with mDNS packets. Defaults to no.

References for avahi:

Docker Compose

Compose makes it easy to .. compose .. a collection of Docker containers which will function together. I prefer storing such configuration and related files in the /opt directory, but you can pick your own if you like. Start by creating a new dir, I've used /opt/home-assistant. In here is where this Compose file and all the other config & data related to the various services will be stored.

While many use Docker-maintained volumes for persistent container storage, I prefer using bind mounted directories from the host, as that allows easy access to all that persistent data from the host system. It also allows for being able to move the dir containing all of that to a new host, to then bring up all the same containers, with all the same persistent data, and not need to get into any Docker internals. If you'd prefer to use Docker-maintained storage, that documentation is here, and switching from this method to that is relatively trivial.

With the Compose file (below) created as /opt/home-assistant/docker-compose.yml, cd /opt/home-assistant, then start with first-time bringup of each service.

Here's the docker-compose.yml file I'm using:

version: '3.7'
services:
  homeassistant:
    image: 'homeassistant/home-assistant'
    container_name: 'home-assistant'
    restart: always
    volumes:
      - ./home-assistant:/config
      - /etc/localtime:/etc/localtime:ro
      - /etc/hosts:/etc/hosts:ro
    ports:
      - '8123:8123'
    env_file: .env
    devices:
      - '/dev/serial1:/dev/serial1'
      - '/dev/bus/usb/001/004:/dev/bt'
    healthcheck:
      test: "wget -qS --spider http://localhost:8123/manifest.json 2>&1 | grep -q 'HTTP/1.1 200 OK'"
      start_period: 20s
    networks:
      - mqtt

  mqtt:
    image: 'eclipse-mosquitto'
    container_name: 'mqtt'
    restart: always
    volumes:
      - ./mosquitto/config:/mosquitto/config
      - ./mosquitto/data:/mosquitto/data
      - ./mosquitto/log:/mosquitto/log
    command: "mosquitto -c /mosquitto-no-auth.conf"
    healthcheck:
      test: "timeout 10 mosquitto_sub -t '$$SYS/#' -C 1 2>&1 | grep -q 'mosquitto version'"
      start_period: 10s
    networks:
      - mqtt

  zigbee2mqtt:
    image: 'koenkk/zigbee2mqtt'
    container_name: 'zigbee2mqtt'
    restart: always
    volumes:
      - ./zigbee2mqtt:/app/data
      - /run/udev:/run/udev:ro
    ports:
      - '8080:8080'
    env_file: .env
    devices:
      - '/dev/ttyUSB0:/dev/ttyACM0'
    depends_on:
      - mqtt
    healthcheck:
      test: "wget -qS --spider http://localhost:8080 2>&1 | grep -q 'HTTP/1.1 200 OK'"
      start_period: 20s
    networks:
      - mqtt

  zwave-js-ui:
    image: 'zwavejs/zwave-js-ui'
    container_name: 'zwave-js-ui'
    restart: unless-stopped
    env_file: .env
    tty: true
    stop_signal: SIGINT
    environment:
      SESSION_SECRET: SOME_SPECIAL_SECRET_OF_YOUR_CHOOSING
      ZWAVEJS_EXTERNAL_CONFIG: /usr/src/app/store/.config-db
    devices:
      - '/dev/serial1:/dev/zwave'
    volumes:
      - ./zwave:/usr/src/app/store
    ports:
      - "8091:8091" # port for web interface
      - "3000:3000" # port for Z-Wave JS websocket server
    networks:
      - mqtt

  esphome:
    image: 'esphome/esphome'
    container_name: 'esphome'
    restart: always
    env_file: .env
    volumes:
      - ./esphome:/config
      - /etc/localtime:/etc/localtime:ro
    ports:
      - '6052:6052'
    networks:
      - hass


  ## Container Maintenance
  # Automatic Image Updates
  watchtower:
    image: 'containrrr/watchtower'
    container_name: 'watchtower'
    restart: always
    environment:
      WATCHTOWER_CLEANUP: 'true'
      WATCHTOWER_INCLUDE_RESTARTING: 'true'
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /etc/localtime:/etc/localtime:ro

  # Stale things cleanup
  docker-gc:
    image: 'spotify/docker-gc'
    container_name: 'docker-gc'
    restart: always
    environment:
      EXCLUDE_FROM_GC: /excludes/exclude-images
      EXCLUDE_CONTAINERS_FROM_GC: /excludes/exclude-containers
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - ./docker-gc:/excludes

networks:
  mqtt:
    driver: bridge

Begin with just starting the two maintenance services:

I'll describe the services in this Compose file below, in order of which would be good to start for the first time:

Watchtower

This is a container which will keep all Docker images updated on the host. It has config but defaults will be fine, so just get it started:

docker-compose up -d watchtower

Docker GC

This is a container which will clean up (delete) old, unused Docker images and containers.

First, create two config files for docker-gc:

docker-gc/exclude-containers:

home-assistant
zigbee2mqtt
zwave-js-ui
mqtt
watchtower
docker-gc

docker-gc/exclude-images:

homeassistant/home-assistant:latest
koenkk/zigbee2mqtt:latest
zwavejs/zwave-js-ui:latest
eclipse-mosquitto:latest
containrrr/watchtower:latest
spotify/docker-gc:latest

docker-compose up -d docker-gc

Zigbee2MQTT (Z2M)

Start the Z2M container to let it generate its default config in the bind mounted directories defined in the Compose file (note the missing -d option, which will run the container in the foreground):

docker-compose up zigbee2mqtt

Once it’s finished starting, hit ctrl-c and let it stop gracefully, then edit zigbee2mqtt/configuration.yaml, changing mqtt.server: mqtt://mqtt (from server: mqtt://localhost).

Then start it again, but as a daemon:

docker-compose up -d zigbee2mqtt

The Z2M “frontend” webui didn’t start working for me till I also set frontend.port: 8080, then rebooted Yellow. But if you're following the steps outlined in this document, you may find it behave more expectedly (not requiring a reboot).

Home Assistant

This is the reason we're in this project. It is the Docker method of installing HA.

docker-compose up -d homeassistant

It should now be reachable at port 8123 of the hostname (or IP address) of your Yellow.

Z-Wave JS UI

I've left this one for last because I'm still working out which serial port my Aeotec Z-Pi 7 is available at.

Z-Wave JS UI defaults to looking for a Z-Wave radio at /dev/zwave. And since we're running it in a container, we can just mount the actual device to there and not need to change that bit of configuration. Much of the other points of config can be adjusted in its web UI, at port 8091, according to the Compose config above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment