使用ceph提供块存储

ceph的块存储

创建一个POOL

1
2
3
[[email protected] ~]# ceph osd pool create ceph-demo 16 16
pool 'ceph-demo' created
[[email protected] ~]# rbd -p ceph-demo ls

创建rbd

1
2
3
4
5
6
7
[[email protected] ~]# rbd create -p ceph-demo --image rbd-demo.img --size 1G
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo.img
[[email protected] ~]# rbd create ceph-demo/rbd-demo-1.img --size 1G   # 也可以这么创建
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img

将RBD映射到主机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 1686fe84952f
        block_name_prefix: rbd_data.1686fe84952f
        format: 2
        features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
        op_features:
        flags:
        create_timestamp: Thu Jan  7 15:06:15 2021
        access_timestamp: Thu Jan  7 15:06:15 2021
        modify_timestamp: Thu Jan  7 15:06:15 2021
[[email protected] ~]# rbd map  ceph-demo/rbd-demo.img
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable ceph-demo/rbd-demo.img object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img deep-flatten
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img fast-diff
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img object-map
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img exclusive-lock
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 16e0aec0c2a8
        block_name_prefix: rbd_data.16e0aec0c2a8
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Thu Jan  7 15:18:34 2021
        access_timestamp: Thu Jan  7 15:18:34 2021
        modify_timestamp: Thu Jan  7 15:18:34 2021
		
[[email protected] ~]# rbd map  ceph-demo/rbd-demo.img
/dev/rbd0
[[email protected] ~]# rbd device list
id pool      namespace image        snap device    
0  ceph-demo           rbd-demo.img -    /dev/rbd0
[[email protected] ~]# fdisk -l /dev/rbd0

Disk /dev/rbd0: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes

挂载该设备

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[[email protected] ~]# mkdir /mnt/ceph-demo
[[email protected] ~]# mkfs.ext4 /dev/rbd0
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
        32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
[[email protected] ~]# df -Th /dev/rbd0
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/rbd0      ext4  976M  2.6M  907M   1% /mnt/ceph-demo

[[email protected] ~]# cd /mnt/ceph-demo/
[[email protected] ceph-demo]# ls -l
total 16
drwx------ 2 root root 16384 Jan  7 15:48 lost+found
[[email protected] ceph-demo]# cp /etc/passwd .
[[email protected] ceph-demo]# ls
lost+found  passwd

RBD扩容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
        size 1 GiB in 256 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 16e0aec0c2a8
        block_name_prefix: rbd_data.16e0aec0c2a8
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Thu Jan  7 15:18:34 2021
        access_timestamp: Thu Jan  7 15:18:34 2021
        modify_timestamp: Thu Jan  7 15:18:34 2021
[[email protected] ~]# rbd resize ceph-demo/rbd-demo.img -s 5G  # 扩容到5G
Resizing image: 100% complete...done.
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
        size 5 GiB in 1280 objects
        order 22 (4 MiB objects)
        snapshot_count: 0
        id: 16e0aec0c2a8
        block_name_prefix: rbd_data.16e0aec0c2a8
        format: 2
        features: layering
        op_features:
        flags:
        create_timestamp: Thu Jan  7 15:18:34 2021
        access_timestamp: Thu Jan  7 15:18:34 2021
        modify_timestamp: Thu Jan  7 15:18:34 2021
[[email protected] ~]# df -Th /mnt/ceph-demo    # 扩容挂载点
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/rbd0      ext4  976M  2.6M  907M   1% /mnt/ceph-demo
[[email protected] ~]# resize2fs  /dev/rbd0
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/rbd0 is mounted on /mnt/ceph-demo; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/rbd0 is now 1310720 blocks long.
[[email protected] ~]# cd /mnt/ceph-demo/
[[email protected] ceph-demo]# ls
lost+found  passwd
[[email protected] ceph-demo]# cat passwd 

[[email protected] ~]# df -Th /mnt/ceph-demo
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/rbd0      ext4  4.9G  4.0M  4.7G   1% /mnt/ceph-demo

RBD数据迁移

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
模拟ceph--1主机挂机或吊盘
[[email protected] ceph-demo]# umount /mnt/ceph-demo/   # 卸载挂载分区
umount: /mnt/ceph-demo: target is busy.
        (In some cases useful info about processes that use
         the device is found by lsof(8) or fuser(1))
[[email protected] ceph-demo]# fuser -mv -k /mnt/ceph-demo/  # 如果报以上错误请用该命令卸载
                     USER        PID ACCESS COMMAND
/mnt/ceph-demo:      root     kernel mount /mnt/ceph-demo
                     root      25679 ..c.. bash
[[email protected] ~]# umount /mnt/ceph-demo/
[[email protected] ~]# df -T
Filesystem         Type           1K-blocks    Used Available Use% Mounted on
devtmpfs           devtmpfs         8189812       0   8189812   0% /dev
tmpfs              tmpfs            8215516       0   8215516   0% /dev/shm
tmpfs              tmpfs            8215516   16956   8198560   1% /run
tmpfs              tmpfs            8215516       0   8215516   0% /sys/fs/cgroup
/dev/vda1          ext4            98232872 3026300  90975308   4% /
tmpfs              tmpfs            8215516      52   8215464   1% /var/lib/ceph/osd/ceph-0
tmpfs              tmpfs            1643104       0   1643104   0% /run/user/1000
10.140.11.8:6789:/ ceph            48599040       0  48599040   0% /mnt/cephfs
ceph-fuse          fuse.ceph-fuse  48599040       0  48599040   0% /mnt/ceph-fuse
tmpfs              tmpfs            1643104       0   1643104   0% /run/user/0
可以看到ceph-demo已经umount 掉了
# 使用ceph--2主机挂载该设备
[[email protected] ceph-demo]# ssh ceph--2
[[email protected] ~]# rbd device list
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo.img
[[email protected] ~]# rbd map ceph-demo/rbd-demo.img
/dev/rbd0
[[email protected] ~]# mkdir ljw
[[email protected] ~]# mount /dev/rbd0 /root/ljw/
[[email protected] ljw]# df -Th /root/ljw
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/rbd0      ext4  4.9G  4.0M  4.7G   1% /root/ljw

数据验证

1
2
3
4
5
6
7
8
[[email protected] ~]# cd /root/ljw/
[[email protected] ljw]# ls
lost+found  passwd
[[email protected] ljw]# cat passwd
[[email protected] ljw]# cp /etc/fstab .
[[email protected] ljw]# ls 
fstab  lost+found  passwd

参考链接

https://blog.csdn.net/wdz306ling/article/details/87930133

ceph
ceph

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！