Booting Debian via AoE (ATA over Ethernet) and PXE

AoE is one of the protocols supported by Linux to access storage via network. It uses plain Ethernet for communication and includes a discovery mechanism to find all available targets. I use it to provide disk space to VMs running on different machines.

Next step is to boot via AoE. It is no real problem to use AoE in running systems. However with some help it is even possible to actually boot disk-less machines via AoE. The PXE implementation iPXE provides the AoE support to actually boot from. I will describe the necessary parts.

Setup vblade

The AoE target used it vblade.

vblade needs access to raw sockets. As I prefer to not have run anything as root if it is not necessary, I use filesystem capabilities to allow it access to the network.

setcap cap_net_raw+ep /usr/sbin/vblade

vblade gets the mac address of the initiator, the shelf and slot number, the network device and the block device.

/usr/sbin/vblade -m $mac 0 0 eth0 $dev

Setup a tftp server and iPXE

apt-get install atftpd ipxe
ln -s /usr/lib/ipxe/undionly.kpxe /var/lib/tftpboot

Setup ISC dhcpd

The dhcp server needs to be configured. It needs to hand out two distinct parameter sets. The first is used to chain-load iPXE into the normal PXE stack. The second is for iPXE and sets the root path to the AoE device. They are selected on the iPXE marker in the request.

if exists user-class and option user-class = "iPXE" {
  filename "";
  option root-path "aoe:e0.0";
} else {
  filename "undionly.kpxe";
}

Support AoE via initramfs-tools

The initramfs needs to initialize AoE support. It needs to enable the network device used for communication with the AoE server and wait until it is up. After that it needs to load the aoe module and run aoe-discover. We should have all devices now.

The root device can now be used like any other normal device. After the AoE device is initialized, it can be found via UUID and all the other ways. So no further modifications are necessary over the usage of local disks. The initramfs finds the device as usual and boots from it.

The initramfs support is still a prototyp, but seems to work. For initramfs-tools it needs a hook to include all necessary stuff in the initramfs and a script to actually do the work. Both are shown here.

/etc/initramfs-tools/hooks/aoe:

#!/bin/sh

case $1 in
prereqs)
  echo "udev"
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/aoe-discover

manual_add_modules aoe

/etc/initramfs-tools/scripts/local-top/aoe:

#!/bin/sh

case $1 in
prereqs)
  echo "udev"
  exit 0
  ;;
esac

ifconfig eth0 up
sleep 10
modprobe aoe
aoe-discover
udevadm settle --timeout=30

Problems

Not all parts of this works 100%. Some parts works not for all hardware.

My old notebook is not able to run Linux with this setup. grub loads the kernel via AoE and nothing comes later.
The network may need more time. Especially in large environments with spanning tree enabled, it may need half a minute until any packets will flow.

Some of the problems can be addressed later. Some can't.