Blog

Booting Debian via AoE (ATA over Ethernet) and PXE

Bastian Blank — Sat, 07 Apr 2012 12:35:00 +0200

AoE is one of the protocols supported by Linux to access storage via network. It uses plain Ethernet for communication and includes a discovery mechanism to find all available targets. I use it to provide disk space to VMs running on different machines.

Next step is to boot via AoE. It is no real problem to use AoE in running systems. However with some help it is even possible to actually boot disk-less machines via AoE. The PXE implementation iPXE provides the AoE support to actually boot from. I will describe the necessary parts.

Setup vblade

The AoE target used it vblade.

vblade needs access to raw sockets. As I prefer to not have run anything as root if it is not necessary, I use filesystem capabilities to allow it access to the network.

setcap cap_net_raw+ep /usr/sbin/vblade

vblade gets the mac address of the initiator, the shelf and slot number, the network device and the block device.

/usr/sbin/vblade -m $mac 0 0 eth0 $dev

Setup a tftp server and iPXE

apt-get install atftpd ipxe
ln -s /usr/lib/ipxe/undionly.kpxe /var/lib/tftpboot

Setup ISC dhcpd

The dhcp server needs to be configured. It needs to hand out two distinct parameter sets. The first is used to chain-load iPXE into the normal PXE stack. The second is for iPXE and sets the root path to the AoE device. They are selected on the iPXE marker in the request.

if exists user-class and option user-class = "iPXE" {
  filename "";
  option root-path "aoe:e0.0";
} else {
  filename "undionly.kpxe";
}

Support AoE via initramfs-tools

The initramfs needs to initialize AoE support. It needs to enable the network device used for communication with the AoE server and wait until it is up. After that it needs to load the aoe module and run aoe-discover. We should have all devices now.

The root device can now be used like any other normal device. After the AoE device is initialized, it can be found via UUID and all the other ways. So no further modifications are necessary over the usage of local disks. The initramfs finds the device as usual and boots from it.

The initramfs support is still a prototyp, but seems to work. For initramfs-tools it needs a hook to include all necessary stuff in the initramfs and a script to actually do the work. Both are shown here.

/etc/initramfs-tools/hooks/aoe:

#!/bin/sh

case $1 in
prereqs)
  echo "udev"
  exit 0
  ;;
esac

. /usr/share/initramfs-tools/hook-functions

copy_exec /sbin/aoe-discover

manual_add_modules aoe

/etc/initramfs-tools/scripts/local-top/aoe:

#!/bin/sh

case $1 in
prereqs)
  echo "udev"
  exit 0
  ;;
esac

ifconfig eth0 up
sleep 10
modprobe aoe
aoe-discover
udevadm settle --timeout=30

Problems

Not all parts of this works 100%. Some parts works not for all hardware.

My old notebook is not able to run Linux with this setup. grub loads the kernel via AoE and nothing comes later.
The network may need more time. Especially in large environments with spanning tree enabled, it may need half a minute until any packets will flow.

Some of the problems can be addressed later. Some can't.

Magic Lantern on EOS 500D

Bastian Blank — Tue, 20 Dec 2011 16:45:00 +0100

Magic Lantern is a firmware extension for several video capable Canon DSLR cameras. The (not longer) current release 11.11.11 of the unified branch works on 500D and most newer models except 5D Mk2.

The installation on my 500D was pretty easy. The camera needs the correct firmware installed (1.1.1), otherwise Magic Lantern will refuse to run. It needs one modification to the camera, a debugging setting to load the system from a SD card. The software is then loaded from a specially prepared SD-card every time the camera boots.

Magic Lantern includes a lot of nice extensions for video and liveview mode. My favorites are histogram overlay, edge detection and exposure display. The histogram overlay shows the histogram in different modes (RGB, luma) of the currently displayed view. Edge detection can be used to find the focus via the liveview output. However a lot more features are available.

Linux 3.0 and Xen

Bastian Blank — Thu, 23 Jun 2011 18:35:00 +0200

It took a long time to get all the parts of the Xen support into the Linux kernel. While rudimentary Dom0-support was available since 2.6.38, support for device backends were missing. It was possible to replace this backend with a userspace implementation included in qemu, but I never tested that.

With Linux 3.0, both the traditional block backend and the network backend are available. They are already enabled in the current 3.0-rc3/-rc4 packages in experimental, so the packages can be used as Dom0 and run guests. Right now the backend modules are not loaded, so this still needs some work. Neither the init scripts loads them, because the names where in flux the last time I laid hand on it, nor does the kernel themself expose enough information to load them via udev. I think using udev to load the modules is the way to go.

This step marks the end of a five year journey. Around 2.6.16 the Xen people started to stay really close to Linux upstream. With the 2.6.18 releas this stopped and the tree was pushed in different states into Debian Etch and RHEL 5. After that, Xen upstream ceased work on newer versions completely, only changes to the now old 2.6.18 tree where done. SuSE started a forward port of the old code base to newer kernel versions and Debian Lenny released with such a patched 2.6.26. Around that time, minimal support for DomU on i386 using paravirt showed up and Lenny had two different kernels with Xen support. Since 2.6.28 this support was mature and works rather flawless since. Somehow after that, a new port of the Dom0 support, now using paravirt, showed up. This tree based on 2.6.32 is released with Debian Squeeze. After several more rounds of redefining and polishing it is now mostly merged into the core kernel.

I don't know what the future brings. We have two virtualization systems supported by Linux now. The first is KVM that converts the kernel into a hypervisor and runs systems with help of the hardware virtualization. The later one is Xen that runs under a standalone hypervisor and supports both para- and hardware virtualization. Both works, KVM is easier to use and even works on current System z hardware. It can be used by any user with hopefully enough margin of security between them. Xen's home is more suited for servers, where you don't have users at all. Both have advantages and disadvantages, so everyone have to decide what he needs, there is no "one size fits all".

New software: python-dvdvideo

Bastian Blank — Sun, 29 Aug 2010 19:20:44 +0200

After a long time, I decided to write again. I decided to start with software I wrote for my own usage that could be usefull for other people. I'll start with python-dvdvideo, a DVD-Video reader written in Python 3, and the reference tool dvd-video-backup-image, a generic DVD-Video dumper. Lets see, if this blog will see more postings in the future.

Intention

I started to write this software, because libdvdread was often unable to decipher my newly purchased video DVDs. libdvdread expects a rather valid structure of the filesystem and other metadata on the disk. It will forcefully bail out on several error conditions. So I often ended patching libdvdread to make dvdbackup able to read the new disks.

Usually there are two ways to create backups of such DVDs, as files or complete images. Dumping them as files have large problems if there are certain defects in the filesystem, like some space is referenced in several titlesets. I have a disk that produces 25GiB of output during such a dump. So the less problematic way to do that is to dump the complete image. That is the way I used in the tool I built on top of this.

Parts

The software is devided into several parts. First a small UDF reader. On top of this comes a DVD video reader. It makes use of libdvdcss wrapper. All of this is used to implement a small tool to dump whole images. I will describe this parts here.

UDF reader

The UDF reader implements a minimal set of features. I implemented only the stuff I found as needed and used in the available DVDs. This reader allows to read the lowlevel UDF, used as base of all video DVDs.

DVD video reader

The dvd video reader uses the UDF reader to get the necessary information from the disk. Again this reader is quiet smallish. It only trusts the UDF for the starts of titlesets and expects that anything else is listed in the info files. This allows to read even discs with broken filesystems, which are really common.

libdvdcss wrapper

The libdvdcss wrapper is implemented using ctypes. The ctypes library allows easy access to functions defines in shared object. The library allows calling of the functions and maps arguments and return values to the Python datatypes. This wrapper allows me to read also encrypted DVDs.

Image dumper

This tool allows to dump a encrypted video DVD into a file. It tries to detect encrypted (video/vob files) and unencrypted (info files, otherwise used space) parts of the disk. This way it is able to dump anything, as long as it can read the filesystem and info files. However, some discs contains overlapping areas, which can't be that easily deguised.

The tool includes a small conflict resolver that handles overlapping parts. It uses a set of rules to allow some types to coexist and some to be modified. On of the rules relabels things included in an info files but also a title vob as always unencrypted. With this resolver, most of the problems can be handled and we get a playable result.

License and distribution

This package is licensed GPL 3 or later. It is for new distributed via Alioth.

Conclusion

This tool allows me to dump all video DVDs I got my hands on in the last time. It allows me to watch the videos on my notebook that have no optical disc reader on its own. Maybe someone may need such a tool also.

USA-Urlaub: 5. Tag (Mt. Tabor)

Bastian Blank — Tue, 29 Sep 2009 19:20:44 +0200

Heute habe ich erst mal ein Auto besorgt und dann die neue Unterkunft für die nächsten Tage gesucht.

Irgendwie hatte ich danach nich gerade viel Lust, also bin ich mal auf den nächsten Vulkankrater, den Mount Tabor, gelaufen. Dieser ist ein bewaldeter Hügel mitten in der Stadt und wohl auch ein beliebtes Ziel der Radfahrer.

Harvey W. Scott

An ein paar Stellen sieht man ziemlich weit.

Downtown vom Mount Tabor

Mount Hood

USA-Urlaub: 1. Tag (Konferenz)

Bastian Blank — Tue, 29 Sep 2009 18:30:00 +0200

Die komplett unpassend gestellte inner Uhr hat mich um 0400 aus dem Schlaf gerissen. Meinem Kollegen im Zimmer ging es genau so. Es war also etwas kollektives Dösen angesagt. Der Sonnenaufgang bescherte uns das folgende Bild.

Sonnenaufgang in Portland mit Mount Hood

Bis auf die zu laute Klimaanlage war das dann eine ganz normale Konferenz. Es gab BoFs und Vorträge zu allen möglichen Themen. Der erste BoF war einer der wichtigsten, es ging um Linux Packaging in den Distributionen und wie man das ein wenig besser vernetzen kann.

Abends gab es einen Empfang zusammen mit den Besuchern der LinuxCon in einem Meeresfrüchterestaurant. Das Essen war super und die Unterhaltungen auch. Ich durfte jemandem dann noch erklären, was es mit dem Oktoberfest auf sich hat.

USA-Urlaub: Erste Eindrücke

Bastian Blank — Thu, 24 Sep 2009 20:22:57 +0200

Ich habe die Möglichkeit erhalten, die Linux Plumbers Conf in Portland, Oregon, Vereinigte Staaten zu besuchen. Ich lasse mir dann natürlich nicht die Möglichkeit entgehen, noch etwas Urlaub in dem mir bis jetzt nur aus Erzählungen bekannten Umfeld zu machen.

Mein Flug ging über Atlanta und ein paar Sachen fallen mir dann schon dort auf.

Alles ist groß. Schon der Weg vom Gate zur Passkontrolle sind ein paar hundert Meter. Und bis zum Gate an welchem mein Anschlussflug geht, bin ich noch mal über 10 Minuten unterwegs, inklusive einer Fahrt mit einer automatischen Bahn.
Die Amis scheinen eine Aversion gegen Treppen zu haben. Im ganzen Flughafen Atlanta ist mir gerade mal eine Treppe begegnet. Sonst gab es noch welche an den Notausgängen oder sie waren nur für Mitarbeiter. In Portland gab es dann keine einzige. Es gibt nur Aufzüge und Rolltreppen.

Um 2200 Uhr (0700 Uhr nach der inneren Uhr) war ich dann im Hotel und konnte dann endlich etwas schlafen.

Almighty root

Bastian Blank — Sat, 04 Apr 2009 19:44:14 +0200

I was asked to take a look at a machine where aptitude don't even want to do the upgrade to etch. A first inspection shows some weird repositories in the source.list file and many daemons noone ever should use on that machine. I was able to do the upgrade with apt-get then.

After some time I got asked over modifications in /etc/exports. It basically included the following content:[1]

/     *(rw,async,no_root_squash)
/home *(rw,async,no_root_squash,nohide)
/usr  *(rw,async,no_root_squash,nohide)
/var  *(rw,async,no_root_squash,nohide)

[1]	For those who don't speak NFS: This exports the specified filesystems (`/`, `/home`, `/usr` and `/var`) to everyone, and accepts whatever the client system say.

The machine was taked out of service immediately. We'll have no chance to answer the question whether this was silliness or intend.

Die Strasse, eine Farbwahl

Bastian Blank — Thu, 02 Apr 2009 14:00:14 +0200

Auf anraten der besten Augenoptikerin bin ich inzwischen glücklicher Besitzer einer Sonnenbrille. Ausserdem war heute ein schön warmer und sonniger Tag, genau das richtige Wetter um das Motorrad aus dem Winterschlaf zu holen. Nach etwas gutem Zureden erwachte das Motorrad zu neuem Leben und mit angemessener Schutzkleidung und der abgedunkelten Sicht ging es los.

Nach einer Weile fiel mir ein Auto mit einer komischen Lakierung auf; es sah aus wie ein Effektlack aus Purpur und Türkis, ähnlich der Wertangabe auf den hohen Euroscheinen. Nachdem immer mehr Autos diesen Effekt zeigten wurde ich unsicher und nach der nächsten Kurve fing auch die Strasse an in allen Farben zu leuchten. Mit geöffnetem Visier sah man alles zwar etwas dunkler, aber in der korrekten Farbwahl. Kaum war es wieder geschlossen zeigt sich die Umgebung und auch der Himmel in allen Spektralfarben.

Create version in Genericsetup metadata.xml from setup.py

Bastian Blank — Mon, 09 Mar 2009 00:25:00 +0100

Today I asked if it is possible to do automatic updates of the version in metadata.xml from the (possible mangled) version from setup.py. Nothing popped up, so I wrote an extension of setuptools which does this.

import os.path
from distutils import log
from setuptools.command.egg_info import egg_info as _egg_info

class egg_info(_egg_info):
    def run(self):
        _egg_info.run(self)

        version = self.distribution.metadata.version

        for package in self.distribution.packages:
            path = os.path.join(*(package.split('.') + ['profiles', 'default']))
            if os.path.isdir(path):
                metadata_out = os.path.join(path, 'metadata.xml')
                metadata_in = os.path.join(path, 'metadata.xml.in')
                if os.path.exists(metadata_in):
                    log.info('writing %s', metadata_out)
                    d = open(metadata_in, 'r').read().replace('@VERSION@', version)
                    open(metadata_out, 'w').write(d)

setup(
    cmdclass = {'egg_info': egg_info},
    [...]
)

The version is listed with a placeholder in the input file called metadata.xml.in and is replaced during a normal develop call as done by buildout.

<metadata>
    <version>@VERSION@</version>
</metadata>

UPDATE: This is evil. metadata.xml should list the the config/profile version, not the code version.

Einem geschenkten Gaul schaut man nicht ins Maul

Bastian Blank — Sun, 16 Nov 2008 00:17:51 +0100

Dieses Sprichwort werde ich heute mal großzügig missachten und über das leider relativ nutzlose Geschenk berichten.

Es begab sich, das Pollin Electronic an diesem Wochenend mal wieder die halbjährige Hausmesse stattfinden lies. Als extra Schmankerl sollte es ein "gratis" Mobiltelefon für jeden Besucher geben. Dieses entpuppte sich als ein LG KP100, einem nach dem ersten Eindruck brauchbaren Gerät ohne Kamera und sonstigem Schnickschnack, von Vodafon in Verbindung mit einer CallYa-Karte, welche auch sofort auf den Besucher registriert wurde.

Nach dem Auspacken und etwas mit Strom versorgen sollte es ausprobiert werden. Also die erstbeste - natürlich nicht von Vodafon - SIM-Karte ausgesucht und reingesteckt. Mehr als ein "mit dir mag ich nicht reden" und der Wahl zwischen Notruf und Unlock wollte es sich mit dieser Karte allerdings nicht entlocken lassen; der leidige SIM-Lock hatte also zugeschlagen.

Damit ist der Plan dieses Gerät als Zweittelefon für Umgebungen in denen Kameras unerwünscht sind oder falls gerade mal ausversehen der Akku leer ist zunichte gemacht und ich frage mich was ich mit dem Telefon und der zwangsweise auf mich registrierten Karte anstellen soll, für mich haben diese so keinen Wert und sind ein Fall für den Entsorger.

Laut der Aufschrift hat die Karte ein Guthaben von 10EUR und sollte für 19,95EUR verkauft werden. Da das Telefon gelockt ist muss ich davon ausgehen das es mehr Wert ist als dieser Preis, da die Subventionen ja immer als Grund herhalten mussten.

Aktueller Zustand: Telefon ist gelockt; ich bin sauer. Jemand der sauer ist wird nicht unbedingt gerne Kunde, wenn nicht unbedingt nötig und da das ganze verschenkt wurde gibt es ja nicht gerade den "Zwang" in Form eines Preisschildes das jetzt zu verwenden. Hypothetischer Zustand: Telefon ist nicht gelockt, ich bin nicht sauer. Jemand für den der eine Teil des Geschenks, das Telefon selber, nützlich ist, überlegt sich eher ob der andere Teil, die CallYa-Karte, auch nützlich sein könnte.

Vodafon kostet diese Aktion für jeden nicht gewonnenen Kunden den Einfauspreis für das Telefon und die vielleicht noch mit dem Kartenguthaben erzeugten Kosten. Nur durch das erlangen von Neukunden kann daraus Profit geschlagen werden.

Ich habe den Sinn dieses SIM-Locks nie verstanden. Was soll hier vor wem beschützt werden? Das was sind wohl die Subventionen die in die Telefone fliessen. Beschützt werden müssen diese wohl vor dem Kunden, da dieser sich ein neues Telefon kaufen muss wenn er woanders hingehen will anstatt es einfach mitzunehmen.

Ubuntu, Ubuntu

Bastian Blank — Fri, 10 Oct 2008 15:20:00 +0200

I was forced to try Ubuntu Hardy in the new university pool. The setup includes one Linux server dedicated for the pool, one Windows AD for Kerberos authentication, one Windows fileserver with user data and 20 clients. The clients are new HP machines with a Radeon Xpress 200 card.

First problem was nasty, the X server turned the display black and then crashed, leaving an unusable console behind. Even a blacklist of the radeon module does not work. Somehow the Xorg radeon driver loads the module on its own, ignoring the modprobe blacklist. Only a hard blacklist using install radoen /bin/false in the modprobe config was able to prevent this. Lets hope that the endeavors to remove much priviledges from the X server goes good.

For the homes two setups was tested:

Homes on the Windows server via cifs, mounted via pam_mount using NTLM password.
Homes on the Linux server via nfs version 3.

The first one just produced an error that some Gnome component was unable to lock ~/.ICEauthority. This could be worked around in the Xsession. Otherwise it was usable to slow, which may be a problem with the Windows server.

The later, plus a reinstallation with fglrx, produced first unresponsive OpenOffice.org windows and then a complete unusable Gnome desktop. I would not completely vote against a problem with fglrx or even a broken installation, but the amount of problems exceeded the threshold.

Now we will try Lenny and see if this also happens there. Especially home via nfs is not that uncommon that we can let it broken in a release.

Xen update

Bastian Blank — Sun, 28 Sep 2008 23:08:34 +0200

I found a machine which is not so ancient and did some tests with Xen on it.

Kernels

First was some tests with different Linux kernels and hypervisors (3.2 and 3.3). I have to say the overall compatibility got better. As unpriviledged domain (DomU) only one of the kernels failed, the one from Etch (2.6.18-6-xen-686) on the x86_64 hypervisor because of missing setup code.

For the operation as priviledged domain (Dom0) it looks not so good. The 2.6.18 from Xen 3.1 works mostly, the Lenny-targeted 2.6.26 is a little bit picky about the hardware and seems to work better in the 64bit variant, the 2.6.18 from 3.3 is old but rock-stable.

Stub domain

Xen 3.3 adds the possibility to move a the qemu which provides the emulated hardware for full virtualized domains in its own (paravirtualized) domain. The documentation is not really complete and the whole thing rather fragile. Error messages from the emulation domain are swallowed and depending on the config it also likes to crash.

It wants a new service, a filesystem backend, which is implemented in a root process in the dom0, even if it is not needed for operation. This service is not configurable, exports anything in /exports and allows writing, the code have similar quality then qemu.

Smallest UTF32 to UTF8 converter

Bastian Blank — Sat, 13 Sep 2008 12:30:00 +0200

I found some weird opcodes in the s390 instruction set some time ago. I finally want to use it as I have access to such a machine. The result is rather slim:

.globl cu41
        .type   cu41, @function
cu41:
.L2:
        cu41    %r2,%r4
        ipm     %r1
        srl     %r1,28
        chi     %r1,3
        je      .L2
        ltr     %r1,%r1
        je      .L3
        lcr     %r1,%r1
        lgfr    %r3,%r1
.L3:
        lgr     %r2,%r3
        br      %r14

The cu41 opcode translates UTF32 to UTF8. Each "parameter" is a register pair [1], which describes address and length of the buffer. This opcode will return after a not specified number of translated characters. The rest is boiler plate code for condition code checking. The loop makes sure that anything is converted within the function unless another error occured.

[1]	An even pair. gcc is not yet able to allocate such pairs on its own, so it is hardcoded.

PV-GRUB and partitions

Bastian Blank — Sat, 13 Sep 2008 11:35:00 +0200

Xen 3.3 added a tool called PV-GRUB. It is a GRUB (legacy aka 0.97) built against MiniOS and bootable as a PV kernel. It works fine on a Xen 3.2.1.

It is easy to use, I just set this in my config:

kernel = "/usr/local/lib/xen/boot/pv-grub-x86_64.gz"
extra = "(hd0)/boot/grub/menu.lst"

However it decided to not load my config as defined in the config and simply show me that:

    GNU GRUB  version 0.97  (524288K lower / 0K upper memory)

       [ Minimal BASH-like line editing is supported.   For
         the   first   word,  TAB  lists  possible  command
         completions.  Anywhere else TAB lists the possible
         completions of a device/filename. ]

grubdom>

The reason is quite easy:

grubdom> root (hd0)
 Filesystem type unknown, using whole disk

Some debugging later, the reason is known and fixed.

grubdom> root (hd0)
 Filesystem type is ext2fs, using whole disk

grubdom> kernel /boot/vmlinuz-2.6.27-rc5-amd64

grubdom> initrd /boot/initrd.img-2.6.27-rc5-amd64

grubdom> boot
block error -1 for op 2
close blk: backend at /local/domain/0/backend/vbd/23/51712
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 2.6.27-rc5-amd64 [...]
[    0.000000] Command line:
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] ACPI in unprivileged domain disabled
[...]
[    0.022093] Booting paravirtualized kernel on Xen
[    0.022101] Xen version: 3.2-1