Setting up Ceph the hard way

Bastian Blank - Mon 09 September 2013

Almost all existing documentation tells me how to setup Ceph with one or two layers of abstration. This entry shows how to set it up by hand without needing root permissions.


Ceph consists of two main daemons. One is the monitoring daemon, which monitors the health of the cluster and provides location information. The second is the storage daemon, which maintains the actual storage. Both are needed in a minimal setup.


The monitor daemons are the heart of the cluster. They maintain quorum within the cluster to check if everything can be used. They provide referrals to clients, to allow them to find the data they seek. Without a majority of monitors nothing will work within the cluster.


The storage daemons maintain the actual storage. One daemon maintains one backend storage device.


The default config is understandable, but several things will just not work with it.

Monitor on localhost

By default the monitor daemon will not work on localhost. There is an (undocumented) override to force it to work on localhost:

 mon addr = [::1]:6789

The monitor will be renamed to mon.admin internaly.


Ceph supports IP (IPv6) or legacy-IP (IPv4), but never both. I don't really use legacy-IP any longer, so I have to configure Ceph accordingly:

  ms bind ipv6 = true

One-OSD clusters

For testing purposes I wanted to create a cluster with exactly one OSD. It never got into a clean state. So I asked and found the answer in #ceph:

 osd crush chooseleaf type = 0

Disable authentication

While deprecated, the following seems to work:

 auth supported = none

Complete configuration

 auth supported = none
 log file = $name.log
 run dir = …
 osd pool default size = 1
 osd crush chooseleaf type = 0
 ms bind ipv6 = true

 mon data = …/$name

 mon addr = [::1]:6789

 osd data = …/$name
 osd journal = …/$name/journal
 osd journal size = 100

 host = devel


This is currently based on my updated packages. And they are still pretty unclean from my point of view.


All the documentation tells only about ceph-deploy and ceph-disk. This tools are abstractions that need root to mount stuff and do all the work. Here I show how to do a minimal setup without needing root.

Keyring setup

For some reason even with no authentication the monitor setup wants a keyring. So just set one up:

$ ceph-authtool --create-keyring keyring --gen-key -n mon.
$ ceph-authtool keyring --gen-key -n client.admin

Monitor setup

Monitor setup by hand is easy:

$ mkdir $mon_data
$ ceph-mon -c ceph.conf --mkfs --fsid $(uuidgen) --keyring keyring

After that just start it:

$ ceph-mon -c ceph.conf

OSD setup

First properly add the new OSD to the internal state:

$ ceph -c ceph.conf osd create
$ ceph -c ceph.conf osd crush set osd.0 1.0 root=default

Then setup the OSD itself:

$ mkdir $osd_data
$ ceph-osd -c ceph.conf -i 0 --mkfs --mkkey --keyring keyring

And start it:

$ ceph-osd -c ceph.conf -i 0
starting osd.0 at :/0 osd_data $osd_data $osd_data/journal

Health check

The health check should return ok after some time:

$ ceph -c ceph.conf health