使用ceph提供块存储

ceph的块存储

创建一个POOL

1
2
3
[[email protected] ~]# ceph osd pool create ceph-demo 16 16
pool 'ceph-demo' created
[[email protected] ~]# rbd -p ceph-demo ls

创建rbd

1
2
3
4
5
6
7
[[email protected] ~]# rbd create -p ceph-demo --image rbd-demo.img --size 1G
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo.img
[[email protected] ~]# rbd create ceph-demo/rbd-demo-1.img --size 1G # 也可以这么创建
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img

将RBD映射到主机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 1686fe84952f
block_name_prefix: rbd_data.1686fe84952f
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten
op_features:
flags:
create_timestamp: Thu Jan 7 15:06:15 2021
access_timestamp: Thu Jan 7 15:06:15 2021
modify_timestamp: Thu Jan 7 15:06:15 2021
[[email protected] ~]# rbd map ceph-demo/rbd-demo.img
rbd: sysfs write failed
RBD image feature set mismatch. You can disable features unsupported by the kernel with "rbd feature disable ceph-demo/rbd-demo.img object-map fast-diff deep-flatten".
In some cases useful info is found in syslog - try "dmesg | tail".
rbd: map failed: (6) No such device or address
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo-1.img
rbd-demo.img
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img deep-flatten
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img fast-diff
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img object-map
[[email protected] ~]# rbd feature disable ceph-demo/rbd-demo.img exclusive-lock
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 16e0aec0c2a8
block_name_prefix: rbd_data.16e0aec0c2a8
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Jan 7 15:18:34 2021
access_timestamp: Thu Jan 7 15:18:34 2021
modify_timestamp: Thu Jan 7 15:18:34 2021

[[email protected] ~]# rbd map ceph-demo/rbd-demo.img
/dev/rbd0
[[email protected] ~]# rbd device list
id pool namespace image snap device
0 ceph-demo rbd-demo.img - /dev/rbd0
[[email protected] ~]# fdisk -l /dev/rbd0

Disk /dev/rbd0: 1073 MB, 1073741824 bytes, 2097152 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 4194304 bytes / 4194304 bytes

挂载该设备

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
[[email protected] ~]# mkdir /mnt/ceph-demo
[[email protected] ~]# mkfs.ext4 /dev/rbd0
mke2fs 1.42.9 (28-Dec-2013)
Discarding device blocks: done
Filesystem label=
OS type: Linux
Block size=4096 (log=2)
Fragment size=4096 (log=2)
Stride=1024 blocks, Stripe width=1024 blocks
65536 inodes, 262144 blocks
13107 blocks (5.00%) reserved for the super user
First data block=0
Maximum filesystem blocks=268435456
8 block groups
32768 blocks per group, 32768 fragments per group
8192 inodes per group
Superblock backups stored on blocks:
32768, 98304, 163840, 229376

Allocating group tables: done
Writing inode tables: done
Creating journal (8192 blocks): done
Writing superblocks and filesystem accounting information: done
[[email protected] ~]# df -Th /dev/rbd0
Filesystem Type Size Used Avail Use% Mounted on
/dev/rbd0 ext4 976M 2.6M 907M 1% /mnt/ceph-demo

[[email protected] ~]# cd /mnt/ceph-demo/
[[email protected] ceph-demo]# ls -l
total 16
drwx------ 2 root root 16384 Jan 7 15:48 lost+found
[[email protected] ceph-demo]# cp /etc/passwd .
[[email protected] ceph-demo]# ls
lost+found passwd

RBD扩容

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
size 1 GiB in 256 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 16e0aec0c2a8
block_name_prefix: rbd_data.16e0aec0c2a8
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Jan 7 15:18:34 2021
access_timestamp: Thu Jan 7 15:18:34 2021
modify_timestamp: Thu Jan 7 15:18:34 2021
[[email protected] ~]# rbd resize ceph-demo/rbd-demo.img -s 5G # 扩容到5G
Resizing image: 100% complete...done.
[[email protected] ~]# rbd info ceph-demo/rbd-demo.img
rbd image 'rbd-demo.img':
size 5 GiB in 1280 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 16e0aec0c2a8
block_name_prefix: rbd_data.16e0aec0c2a8
format: 2
features: layering
op_features:
flags:
create_timestamp: Thu Jan 7 15:18:34 2021
access_timestamp: Thu Jan 7 15:18:34 2021
modify_timestamp: Thu Jan 7 15:18:34 2021
[[email protected] ~]# df -Th /mnt/ceph-demo # 扩容挂载点
Filesystem Type Size Used Avail Use% Mounted on
/dev/rbd0 ext4 976M 2.6M 907M 1% /mnt/ceph-demo
[[email protected] ~]# resize2fs /dev/rbd0
resize2fs 1.42.9 (28-Dec-2013)
Filesystem at /dev/rbd0 is mounted on /mnt/ceph-demo; on-line resizing required
old_desc_blocks = 1, new_desc_blocks = 1
The filesystem on /dev/rbd0 is now 1310720 blocks long.
[[email protected] ~]# cd /mnt/ceph-demo/
[[email protected] ceph-demo]# ls
lost+found passwd
[[email protected] ceph-demo]# cat passwd

[[email protected] ~]# df -Th /mnt/ceph-demo
Filesystem Type Size Used Avail Use% Mounted on
/dev/rbd0 ext4 4.9G 4.0M 4.7G 1% /mnt/ceph-demo

RBD数据迁移

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
模拟ceph--1主机挂机或吊盘
[[email protected] ceph-demo]# umount /mnt/ceph-demo/ # 卸载挂载分区
umount: /mnt/ceph-demo: target is busy.
(In some cases useful info about processes that use
the device is found by lsof(8) or fuser(1))
[[email protected] ceph-demo]# fuser -mv -k /mnt/ceph-demo/ # 如果报以上错误请用该命令卸载
USER PID ACCESS COMMAND
/mnt/ceph-demo: root kernel mount /mnt/ceph-demo
root 25679 ..c.. bash
[[email protected] ~]# umount /mnt/ceph-demo/
[[email protected] ~]# df -T
Filesystem Type 1K-blocks Used Available Use% Mounted on
devtmpfs devtmpfs 8189812 0 8189812 0% /dev
tmpfs tmpfs 8215516 0 8215516 0% /dev/shm
tmpfs tmpfs 8215516 16956 8198560 1% /run
tmpfs tmpfs 8215516 0 8215516 0% /sys/fs/cgroup
/dev/vda1 ext4 98232872 3026300 90975308 4% /
tmpfs tmpfs 8215516 52 8215464 1% /var/lib/ceph/osd/ceph-0
tmpfs tmpfs 1643104 0 1643104 0% /run/user/1000
10.140.11.8:6789:/ ceph 48599040 0 48599040 0% /mnt/cephfs
ceph-fuse fuse.ceph-fuse 48599040 0 48599040 0% /mnt/ceph-fuse
tmpfs tmpfs 1643104 0 1643104 0% /run/user/0
可以看到ceph-demo已经umount 掉了
# 使用ceph--2主机挂载该设备
[[email protected] ceph-demo]# ssh ceph--2
[[email protected] ~]# rbd device list
[[email protected] ~]# rbd -p ceph-demo ls
rbd-demo.img
[[email protected] ~]# rbd map ceph-demo/rbd-demo.img
/dev/rbd0
[[email protected] ~]# mkdir ljw
[[email protected] ~]# mount /dev/rbd0 /root/ljw/
[[email protected] ljw]# df -Th /root/ljw
Filesystem Type Size Used Avail Use% Mounted on
/dev/rbd0 ext4 4.9G 4.0M 4.7G 1% /root/ljw

数据验证

1
2
3
4
5
6
7
8
[[email protected] ~]# cd /root/ljw/
[[email protected] ljw]# ls
lost+found passwd
[[email protected] ljw]# cat passwd
[[email protected] ljw]# cp /etc/fstab .
[[email protected] ljw]# ls
fstab lost+found passwd

参考链接

https://blog.csdn.net/wdz306ling/article/details/87930133


本博客所有文章除特别声明外,均采用 CC BY-SA 4.0 协议 ,转载请注明出处!