ceph
的块存储创建一个POOL
创建rbd
将RBD映射到主机 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 [[email protected] ~]# rbd -p ceph-demo ls rbd-demo-1.img rbd-demo.img [[email protected] ~]# rbd info ceph-demo/rbd-demo.img rbd image 'rbd-demo.img': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 1686fe84952f block_name_prefix: rbd_data.1686fe84952f format: 2 features: layering, exclusive-lock, object-map, fast-diff, deep-flatten op_features: flags: create_timestamp: Thu Jan 7 15:06:15 2021 access_timestamp: Thu Jan 7 15:06:15 2021 modify_timestamp: Thu Jan 7 15:06:15 2021 [[email protected] ~]# rbd map ceph-demo/rbd-demo.img rbd: sysfs write failed RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable ceph-demo/rbd-demo.img object-map fast-diff deep-flatten". In some cases useful info is found in syslog - try "dmesg | tail". rbd: map failed: (6) No such device or address [[email protected] ~]# rbd -p ceph-demo ls rbd-demo-1.img rbd-demo.img [[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img deep-flatten [[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img fast-diff [[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img object-map [[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img exclusive-lock [[email protected] ~]# rbd info ceph-demo/rbd-demo.img rbd image 'rbd-demo.img': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 16e0aec0c2a8 block_name_prefix: rbd_data.16e0aec0c2a8 format: 2 features: layering op_features: flags: create_timestamp: Thu Jan 7 15:18:34 2021 access_timestamp: Thu Jan 7 15:18:34 2021 modify_timestamp: Thu Jan 7 15:18:34 2021 [[email protected] ~]# rbd map ceph-demo/rbd-demo.img /dev/rbd0 [[email protected] ~]# rbd device list id pool namespace image snap device 0 ceph-demo rbd-demo.img - /dev/rbd0 [[email protected] ~]# fdisk -l /dev/rbd0 Disk /dev/rbd0: 1073 MB, 1073741824 bytes, 2097152 sectors Units = sectors of 1 * 512 = 512 bytes Sector size (logical/physical): 512 bytes / 512 bytes I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes
挂载该设备 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 [[email protected] ~]# mkdir /mnt/ceph-demo [[email protected] ~]# mkfs.ext4 /dev/rbd0 mke2fs 1.42.9 (28-Dec-2013) Discarding device blocks: done Filesystem label= OS type: Linux Block size=4096 (log=2) Fragment size=4096 (log=2) Stride=1024 blocks, Stripe width=1024 blocks 65536 inodes, 262144 blocks 13107 blocks (5.00%) reserved for the super user First data block=0 Maximum filesystem blocks=268435456 8 block groups 32768 blocks per group, 32768 fragments per group 8192 inodes per group Superblock backups stored on blocks: 32768, 98304, 163840, 229376 Allocating group tables: done Writing inode tables: done Creating journal (8192 blocks): done Writing superblocks and filesystem accounting information: done [[email protected] ~]# df -Th /dev/rbd0 Filesystem Type Size Used Avail Use% Mounted on /dev/rbd0 ext4 976M 2.6M 907M 1% /mnt/ceph-demo [[email protected] ~]# cd /mnt/ceph-demo/ [[email protected] ceph-demo]# ls -l total 16 drwx------ 2 root root 16384 Jan 7 15:48 lost+found [[email protected] ceph-demo]# cp /etc/passwd . [[email protected] ceph-demo]# ls lost+found passwd
RBD扩容 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 [[email protected] ~]# rbd info ceph-demo/rbd-demo.img rbd image 'rbd-demo.img': size 1 GiB in 256 objects order 22 (4 MiB objects) snapshot_count: 0 id: 16e0aec0c2a8 block_name_prefix: rbd_data.16e0aec0c2a8 format: 2 features: layering op_features: flags: create_timestamp: Thu Jan 7 15:18:34 2021 access_timestamp: Thu Jan 7 15:18:34 2021 modify_timestamp: Thu Jan 7 15:18:34 2021 [[email protected] ~]# rbd resize ceph-demo/rbd-demo.img -s 5G # 扩容到5G Resizing image: 100% complete...done. [[email protected] ~]# rbd info ceph-demo/rbd-demo.img rbd image 'rbd-demo.img': size 5 GiB in 1280 objects order 22 (4 MiB objects) snapshot_count: 0 id: 16e0aec0c2a8 block_name_prefix: rbd_data.16e0aec0c2a8 format: 2 features: layering op_features: flags: create_timestamp: Thu Jan 7 15:18:34 2021 access_timestamp: Thu Jan 7 15:18:34 2021 modify_timestamp: Thu Jan 7 15:18:34 2021 [[email protected] ~]# df -Th /mnt/ceph-demo # 扩容挂载点 Filesystem Type Size Used Avail Use% Mounted on /dev/rbd0 ext4 976M 2.6M 907M 1% /mnt/ceph-demo [[email protected] ~]# resize2fs /dev/rbd0 resize2fs 1.42.9 (28-Dec-2013) Filesystem at /dev/rbd0 is mounted on /mnt/ceph-demo; on-line resizing required old_desc_blocks = 1, new_desc_blocks = 1 The filesystem on /dev/rbd0 is now 1310720 blocks long. [[email protected] ~]# cd /mnt/ceph-demo/ [[email protected] ceph-demo]# ls lost+found passwd [[email protected] ceph-demo]# cat passwd [[email protected] ~]# df -Th /mnt/ceph-demo Filesystem Type Size Used Avail Use% Mounted on /dev/rbd0 ext4 4.9G 4.0M 4.7G 1% /mnt/ceph-demo
RBD数据迁移 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 模拟ceph--1主机挂机或吊盘 [[email protected] ceph-demo]# umount /mnt/ceph-demo/ # 卸载挂载分区 umount: /mnt/ceph-demo: target is busy. (In some cases useful info about processes that use the device is found by lsof(8) or fuser(1)) [[email protected] ceph-demo]# fuser -mv -k /mnt/ceph-demo/ # 如果报以上错误请用该命令卸载 USER PID ACCESS COMMAND /mnt/ceph-demo: root kernel mount /mnt/ceph-demo root 25679 ..c.. bash [[email protected] ~]# umount /mnt/ceph-demo/ [[email protected] ~]# df -T Filesystem Type 1K-blocks Used Available Use% Mounted on devtmpfs devtmpfs 8189812 0 8189812 0% /dev tmpfs tmpfs 8215516 0 8215516 0% /dev/shm tmpfs tmpfs 8215516 16956 8198560 1% /run tmpfs tmpfs 8215516 0 8215516 0% /sys/fs/cgroup /dev/vda1 ext4 98232872 3026300 90975308 4% / tmpfs tmpfs 8215516 52 8215464 1% /var/lib/ceph/osd/ceph-0 tmpfs tmpfs 1643104 0 1643104 0% /run/user/1000 10.140.11.8:6789:/ ceph 48599040 0 48599040 0% /mnt/cephfs ceph-fuse fuse.ceph-fuse 48599040 0 48599040 0% /mnt/ceph-fuse tmpfs tmpfs 1643104 0 1643104 0% /run/user/0 可以看到ceph-demo已经umount 掉了# 使用ceph--2主机挂载该设备 [[email protected] ceph-demo]# ssh ceph--2 [[email protected] ~]# rbd device list [[email protected] ~]# rbd -p ceph-demo ls rbd-demo.img [[email protected] ~]# rbd map ceph-demo/rbd-demo.img /dev/rbd0 [[email protected] ~]# mkdir ljw [[email protected] ~]# mount /dev/rbd0 /root/ljw/ [[email protected] ljw]# df -Th /root/ljw Filesystem Type Size Used Avail Use% Mounted on /dev/rbd0 ext4 4.9G 4.0M 4.7G 1% /root/ljw
数据验证
参考链接 https://blog.csdn.net/wdz306ling/article/details/87930133