After you’ve followed the familiar Proxmox Ceph Server manual you’ll have a brand new Ceph cluster up and running. Here I’m collecting the basic tasks and commands you’ll need to manage a Ceph cluster.

Ceph structure info

Disk structure

Ceph automatically configures and creates the block device in /dev/rbd/<pool-name>/

root@node1:~# ls -la /dev/rbd/<pool-name>/
total 0
drwxr-xr-x 2 root root 560 mar 26 15:05 .
drwxr-xr-x 3 root root  60 mar 23 13:29 ..
lrwxrwxrwx 1 root root  11 mar 26 12:39 vm-101-disk-1 -> ../../rbd0
lrwxrwxrwx 1 root root  11 mar 26 13:42 vm-102-disk-1 -> ../../rbd1
lrwxrwxrwx 1 root root  11 mar 23 19:34 vm-103-disk-1 -> ../../rbd2
lrwxrwxrwx 1 root root  11 mar 24 08:31 vm-103-disk-2 -> ../../rbd3

It is a normal block device that can be mounted in a regular way

root@node1:~# fdisk -l /dev/rbd0

Disk /dev/rbd0: 20 GiB, 21474836480 bytes, 41943040 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
Disklabel type: dos
Disk identifier: 0x53332713

Device      Boot    Start      End  Sectors   Size Id Type
/dev/rbd0p1 *          63 41110334 41110272  19,6G 83 Linux
/dev/rbd0p2      41110335 41929649   819315 400,1M  5 Extended
/dev/rbd0p5      41110398 41913584   803187 392,2M 82 Linux swap

so

root@node1:~# mount /dev/rbd0p1 /media
root@node1:~# df -h
Filesystem                          Size  Used Avail Use% Mounted on
/dev/rbd0p1                          20G   17G  1,4G  93% /media
root@node1:~# ls -la /media/
total 2,1G
drwxr-xr-x  24 root root 4,0K mar 23 13:43 .
drwxr-xr-x  22 root root 4,0K mar  9 18:06 ..
drwxr-xr-x   2 root root 4,0K apr 10  2012 bin
drwxr-xr-x   3 root root 4,0K ago  8  2012 boot
lrwxrwxrwx   1 root root   11 apr  9  2010 cdrom -> media/cdrom
drwxr-xr-x   2 root root 4,0K set  3  2012 dev
drwxr-xr-x 109 root root  12K mar 23 13:43 etc
drwxr-xr-x  11 root root 4,0K mag 21  2013 home
drwxr-xr-x   2 root root 4,0K ott 26  2009 initrd
...

Benchmarking

BLOCK WRITE 4KB

root@node1:~# /usr/bin/rados -p rbd bench -b 4096 300 write -t 32 \
  --no-cleanup -m 10.0.4.1:6789 -n client.admin --keyring \
  /etc/pve/priv/ceph.client.admin.keyring --auth_supported cephx df

Total time run:         300.015640
Total writes made:      614977
Write size:             4096
Object size:            4096
Bandwidth (MB/sec):     8.0071
Stddev Bandwidth:       3.35268
Max bandwidth (MB/sec): 11.9766
Min bandwidth (MB/sec): 0.03125
Average IOPS:           2049
Stddev IOPS:            858
Max IOPS:               3066
Min IOPS:               8
Average Latency(s):     0.0156087
Stddev Latency(s):      0.0442384
Max latency(s):         1.51347
Min latency(s):         0.00396259