Cluster HAT usbboot/rpiboot test image (Updated 2018/03/03).
This is a guide to using my modified Raspbian image with rpiboot (usbboot) for a Cluster HAT with up to 4 Pi Zeros (1.1/1.2/W) which can be modified to work with a ZeroStem or standard USB cable.
This test image currently supports 4 Pi Zeros.
What is usbboot/rpiboot?
The "rpiboot" tool allows you to boot Raspberry Pi Zero (and Pi Zero W, Model A, CM and CM3) without an SD card, "usbboot" is the repository which holds the "rpiboot" tool. The original sources are available at https://github.com/raspberrypi/usbboot my modified sources can be found at https://github.com/burtyb/usbboot.
The standard rpiboot supports booting from a single "boot" directory. My version of rpiboot has been modified to support overlay boot directories based on the USB "path" the device is connected via. This allows the use of a custom configuration file for each slot on the Cluster HAT (or ZeroStem/USB Cable). This is used to give each Pi a custom cmdline.txt which sets the MAC address on both sides of the USB Gadget Ethernet device, mount the right NFS directory, etc.
WARNING: The Pi3b+ kernel has issues with network bridges, VLAN and general network transfers so these images haven't yet been updated to support this model.
ClusterHAT-2017-11-29-lite-2-usbboot.zip (Stretch for Cluster HAT v1.x and v2.x).
This image was released on 2018/03/03 to add support for newer v2.x Cluster HAT which uses a different "on" signal for the USB HUB.
ClusterHAT-2017-09-07-lite-1-usbboot (Stretch for Cluster HAT v1.x and v2.0).
A new Stretch based image released on 2017/11/27 which no longer needs to be upgraded to support the Cluster HAT v2.0 can be downloaded from:
I advise using a 16GB SD card as the image requires 7.5GB, the image will be automatically expanded to fill the SD card on first boot.
ClusterHAT-2017-07-05-lite-2-usbboot (Jessie for Cluster HAT v1.x)
The older ClusterHAT-2017-07-05-lite-2-usbboot.zip Jessie based image can still be downloaded from:
I advise using a 16GB SD card as the image requires 5GB and MUST be expanded with "sudo raspi-config" on the controller before trying to boot any Pi Zero.
Issue with ClusterHAT-2017-07-05-lite-1-usbboot.zip - If you've downloaded the earlier version of this then you may find rpiboot gets stuck after booting the first Pi Zero. To fix this you need to update rpiboot by running the following command on the Controller Pi and then reboot.
sudo sh -c "cd /var/lib/clusterhat/usbboot/;git pull;make"
When using either OLD 2017-07-05 image with a Cluster HAT v2.0 you will also need to upgrade the tools to support the newer HAT - see the forum post for upgrade instructions.
The image should be written to an SD card for your Controller Raspberry Pi by following the standard instructions.
Stretch based images have SSH disabled by default (on Controller/Pi Zeros) and the filesystem will automatically expand.
Older Jessie based image have SSH enabled by default on both the Controller and Pi Zeros, the Controller filesystem must be expanded on first boot using "sudo raspi-config".
The "pi" users password is set to "clusterhat" (without the quotes) you MUST change this as soon as you login AND then expand the Controller filesystem with "sudo raspi-config".
Also please remember to remove SD cards from the Pi Zeros :).
The Pi Zeros have an internal network bridged to "brint" on the Controller, they also have an external network bridged to "brext" on the Controller. The Pi Zeros use VLANs to separate these 2 networks. Untagged traffic is used for internal and VLAN 10 is used for external traffic. It is done this way arround to utilise mounting the NFSROOT filesystem from the kernel command line where configuring VLAN is not possible.
See our original post for more details on this setup.
172.19.180.254 - Is assigned to the Controllers Internal Interface.
172.19.180.1-4 - Are assigned to the Pi Zeros.
These can be used if the Pi Zero doesn't pick up an external IP address correctly or if you want to boot it standalone.
The Controller/Pi Zeros will request the external IP address via DHCP on the Controllers "eth0" device.
The hostnames are "controller.local", "p1.local", "p2.local", "p3.local", "p4.local".
Usage [Cluster HAT]
The image as supplied expects the Cluster HAT to be plugged into the top USB port closest to the Ethernet port on the Controller Pi.
If using a Cluster HAT P1-P4 can be powered on using the standard "clusterhat on" command.
You should be able to watch the Pi Zero request files and boot from by looking at the screen session for "rpiboot".
sudo screen -rx rpiboot # To detach from screen press CTRL+a then CTRL+d
Once connected it will show files being requested as it steps through the boot process and eventually after a few seconds the LED should illuminate on the Pi Zero and it will boot up. It may take more than a minute before the Pi Zero requests the external IP address and starts SSH but you should be able to ping 172.19.180.1/2/3/4 early in the boot process as this is used to mount the root filesystem over NFS.
Pi Zeros can be rebooted as normal and will be booted up again automatically.
Usage [ZeroStem / USB Cable]
This section is a bit long/more complicated than it needs to be right now, will be cleaned up - if you need help post on the forum (If you can post the output of "lsusb -t" before and after you plug in the Pi Zero it should be easy to spot even when using standard Raspbian).
The root file system for the Pi Zeros is stored in /var/lib/clusterhat/nfs/pX where X is 1-4. The base "/boot" filesystem is stored in /var/lib/clusterhat/boot/ and within this directory are symlinks to the specific root filesystems for each USB path.
ls -l /var/lib/clusterhat/boot/1-* lrwxrwxrwx 1 root root 15 Aug 2 09:11 /var/lib/clusterhat/boot/1-1.2.1 -> ../nfs/p4/boot/ lrwxrwxrwx 1 root root 15 Aug 2 09:11 /var/lib/clusterhat/boot/1-1.2.2 -> ../nfs/p3/boot/ lrwxrwxrwx 1 root root 15 Aug 2 09:11 /var/lib/clusterhat/boot/1-1.2.3 -> ../nfs/p2/boot/ lrwxrwxrwx 1 root root 15 Aug 2 09:11 /var/lib/clusterhat/boot/1-1.2.4 -> ../nfs/p1/boot/
These relate to the USB path names so in the example above this relates to.
1-1.2 - Is the top left USB port on the Raspberry Pi 3 which is connected to the USB hub on the Cluster HAT.
1-1.2.4 - Is the first USB port on the Cluster HAT which is used for P1.
1-1.2.3 - Is the second USB port on the Cluster HAT which is used for P2.
1-1.2.2 - Is the third USB port on the Cluster HAT which is used for P3.
1-1.2.1 - Is the fourth USB port on the Cluster HAT which is used for P4.
To use a different USB path topology you'll need to alter these symlinks, first find out the USB path name where you're going to be plugging in the Pi Zero.
So if you're plugging in a Pi Zero with a USB cable or Zero Stem directly into...
Top Left USB port on a Pi 3 the USB pathname would be "1-1.2"
Top Right USB port on a Pi 3 the USB pathname would be "1-1.4"
Bottom Left USB port on a Pi 3 the USB pathname would be "1-1.3"
Bottom Right USB port on a Pi 3 the USB pathname would be "1-1.5"
You can also see the path using the following command.
$ lsusb -t /: Bus 01.Port 1: Dev 1, class="root_hub", Driver=dwc_otg/1p, 480M |__ Port 1: Dev 2, If 0, class="Hub", Driver=hub/5p, 480M |__ Port 1: Dev 3, If 0, class="Vendor" Specific Class, Driver=smsc95xx, 480M |__ Port 2: Dev 4, If 0, class="Hub", Driver=hub/4p, 480M |__ Port 3: Dev 33, If 2, class="Communications", Driver=cdc_acm, 480M |__ Port 3: Dev 33, If 0, class="Communications", Driver=cdc_ether, 480M |__ Port 3: Dev 33, If 3, class="CDC" Data, Driver=cdc_acm, 480M |__ Port 3: Dev 33, If 1, class="CDC" Data, Driver=cdc_ether, 480M |__ Port 1: Dev 47, If 3, class="CDC" Data, Driver=cdc_acm, 480M |__ Port 1: Dev 47, If 1, class="CDC" Data, Driver=cdc_ether, 480M |__ Port 1: Dev 47, If 2, class="Communications", Driver=cdc_acm, 480M |__ Port 1: Dev 47, If 0, class="Communications", Driver=cdc_ether, 480M |__ Port 4: Dev 49, If 1, class="CDC" Data, Driver=cdc_ether, 480M |__ Port 4: Dev 49, If 2, class="Communications", Driver=cdc_acm, 480M |__ Port 4: Dev 49, If 0, class="Communications", Driver=cdc_ether, 480M |__ Port 4: Dev 49, If 3, class="CDC" Data, Driver=cdc_acm, 480M |__ Port 2: Dev 44, If 2, class="Communications", Driver=cdc_acm, 480M |__ Port 2: Dev 44, If 0, class="Communications", Driver=cdc_ether, 480M |__ Port 2: Dev 44, If 3, class="CDC" Data, Driver=cdc_acm, 480M |__ Port 2: Dev 44, If 1, class="CDC" Data, Driver=cdc_ether, 480M
The 1-1.2.4 relates to "Bus 01" - "Port 1" . "Port 2" . "Port 4" (which is Dev 49 in the example above) so by plugging something into the port and running "lsusb -t" before/after you should be able to work out the path name you need.
So for example to change /var/lib/clusterhat/nfs/p1 to be the filesystem for a Pi zero plugged into the bottom right USB port on a Pi 3 you would need to update the symlink to "1-1.5".
sudo rm -f /var/lib/clusterhat/boot/1-1.2.4 sudo ln -s /var/lib/clusterhat/nfs/p1/boot/ /var/lib/clusterhat/boot/1-1.5
If you have any problem working out the path name you should be using please ask on the forum.
The rpiboot process runs in a screen session (under the root user) rather than a systemd service. I do this to make it easier to monitor what's going on when needed without having multiple lines per second end up in the log file. Once things stabalise I expect this to move to a systemd service. This is started in the /etc/rc.local file.
There are still issues with the rpcbind/nfs services not starting on boot so again these are restarted in the /etc/rc.local file which seemed to work OK in my testing.
raspi-config doesn't work as it tries to detect the "/boot" partition is a mountpoint which is isn't as it comes with the root filesystem you can appease it by mounting the "/boot" directory again on the Pi Zero.
sudo mount 172.19.180.254:/var/lib/clusterhat/nfs/p1/boot /boot
Replacing the "p1" as appropriate, you should then be able to run raspi-config to enable the camera/i2c/etc.
If you have any problems, questions or suggestions about the images please use the forum to discuss it.
If you want to make your own image or see how I created this image the script is available here but be aware the script has bugs, it expects to be ran with a ClusterHAT image as the source file on a Raspberry Pi with plenty of storage.