ceph-deploy 安装ceph

ceph 安装

使用ceph-deploy 安装ceph,ceph-deploy现在只能安装nautilus或者之前的版本,octopus和octopus以后的版本不在使用ceph-deploy方式安装,而是建议使用cephadm来安装,后期我会写一个cephadm安装配置ceph集群。

准备机器

下面的机器是我用kvm虚化出来3台,每个机器上有一个数据盘vdb

主机 应用
ceph–1 ceph-deploy ceph-mon ceph-mgr ceph-raw ceph-mds ceph-osd
ceph–2 ceph-mon ceph-mgr ceph-raw ceph-mds ceph-osd
ceph–3 ceph-mon ceph-mgr ceph-raw ceph-mds ceph-osd
1
2
3
4
5
6
7
8
9
10
11
12
13
# 所有机器配置hosts主机名解析
# cat /etc/hosts
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
10.140.11.8 ceph--1
10.140.11.6 ceph--2
10.140.11.24 ceph--3
# lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sr0 11:0 1 478K 0 rom
vda 253:0 0 100G 0 disk
└─vda1 253:1 0 100G 0 part /
vdb 253:16 0 50G 0 disk

deploy节点到ceph主机配置免密登录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
[[email protected] ~]# ssh-keygen 
Generating public/private rsa key pair.
Enter file in which to save the key (/root/.ssh/id_rsa):
Enter passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved in /root/.ssh/id_rsa.
Your public key has been saved in /root/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:g981QUE32gwwVNoVB2aSzyTDEUmg2CQdSTgi05eYUWM [email protected]
The key's randomart image is:
+---[RSA 2048]----+
| [email protected]*Oo.|
| o =.=*o. =*@.o |
| o o..o . +*o |
| . .o |
| . S o |
| . o . . |
| . . |
| |
| |
+----[SHA256]-----+

[[email protected] ~]# ssh-copy-id ceph--1
[[email protected] ~]# ssh-copy-id ceph--2
[[email protected] ~]# ssh-copy-id ceph--3

所有节点配置国内ceph源

1
2
3
4
5
6
7
8
9
10
11
12
# cat /etc/yum.repos.d/ceph.repo
[norch]
name=norch
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/noarch/
enabled=1
gpgcheck=0

[x86_64]
name=x86_64
baseurl=https://mirrors.aliyun.com/ceph/rpm-nautilus/el7/x86_64/
enabled=1
gpgcheck=0

在ceph–1节点配置安装并配置ceph-deploy节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
[[email protected] ~]# mkdir ceph-deploy
[[email protected] ~]# yum -y install python-setuptools ceph-deploy
[[email protected] ceph-deploy]# ceph-deploy new ceph--1
[ceph_deploy.conf][DEBUG ] found configuration file at: /root/.cephdeploy.conf
[ceph_deploy.cli][INFO ] Invoked (2.0.1): /bin/ceph-deploy new ceph--1
[ceph_deploy.cli][INFO ] ceph-deploy options:
[ceph_deploy.cli][INFO ] username : None
[ceph_deploy.cli][INFO ] func : <function new at 0x7f4162516de8>
[ceph_deploy.cli][INFO ] verbose : False
[ceph_deploy.cli][INFO ] overwrite_conf : False
[ceph_deploy.cli][INFO ] quiet : False
[ceph_deploy.cli][INFO ] cd_conf : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f4161c98e18>
[ceph_deploy.cli][INFO ] cluster : ceph
[ceph_deploy.cli][INFO ] ssh_copykey : True
[ceph_deploy.cli][INFO ] mon : ['ceph--1']
[ceph_deploy.cli][INFO ] public_network : None
[ceph_deploy.cli][INFO ] ceph_conf : None
[ceph_deploy.cli][INFO ] cluster_network : None
[ceph_deploy.cli][INFO ] default_release : False
[ceph_deploy.cli][INFO ] fsid : None
[ceph_deploy.new][DEBUG ] Creating new cluster named ceph
[ceph_deploy.new][INFO ] making sure passwordless SSH succeeds
[ceph--1][DEBUG ] connected to host: ceph--1
[ceph--1][DEBUG ] detect platform information from remote host
[ceph--1][DEBUG ] detect machine type
[ceph--1][DEBUG ] find the location of an executable
[ceph--1][INFO ] Running command: /usr/sbin/ip link show
[ceph--1][INFO ] Running command: /usr/sbin/ip addr show
[ceph--1][DEBUG ] IP addresses found: [u'10.140.11.8']
[ceph_deploy.new][DEBUG ] Resolving host ceph--1
[ceph_deploy.new][DEBUG ] Monitor ceph--1 at 10.140.11.8
[ceph_deploy.new][DEBUG ] Monitor initial members are ['ceph--1']
[ceph_deploy.new][DEBUG ] Monitor addrs are ['10.140.11.8']
[ceph_deploy.new][DEBUG ] Creating a random mon key...
[ceph_deploy.new][DEBUG ] Writing monitor keyring to ceph.mon.keyring...
[ceph_deploy.new][DEBUG ] Writing initial config to ceph.conf...

所有节点安装ceph基础软件包

1
# yum -y install ceph ceph-mon ceph-mgr ceph-radosgw ceph-mds ceph-mgr-dashboard

使用ceph-deploy创建mon节点

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
[[email protected] ceph-deploy]# ceph-deploy mon create-initial
[[email protected] ceph-deploy]# ceph-deploy mon add ceph--2 # 添加mon 节点
[[email protected] ceph-deploy]# ceph-deploy mon add ceph--3
[[email protected] ceph-deploy]# ceph-deploy admin ceph--1 ceph--2 ceph--3 # 同步配置文件
[[email protected] ceph-deploy]# ceph -s
cluster:
id: faa2e2c4-98bc-47c4-a5b4-a478721b7ea2
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph--1,ceph--2,ceph--3 (age 49s)
mgr: no daemons active
osd: 0 osds: 0 up, 0 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:

[[email protected] ceph-deploy]# ceph-deploy mgr create ceph--1 ceph--2 ceph--3
[[email protected] ceph-deploy]# ceph -s
cluster:
id: faa2e2c4-98bc-47c4-a5b4-a478721b7ea2
health: HEALTH_WARN
OSD count 0 < osd_pool_default_size 3

services:
mon: 3 daemons, quorum ceph--1,ceph--2,ceph--3 (age 2m)
mgr: ceph--1(active, since 14m), standbys: ceph--2, ceph--3
osd: 0 osds: 0 up, 0 in

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 0 B used, 0 B / 0 B avail
pgs:

创建osd

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
[[email protected] ceph-deploy]# ceph-deploy osd create ceph--1 --data /dev/vdb 
[[email protected] ceph-deploy]# ceph-deploy osd create ceph--2 --data /dev/vdb
[[email protected] ceph-deploy]# ceph-deploy osd create ceph--3 --data /dev/vdb
[[email protected] ceph-deploy]# ceph -s
cluster:
id: faa2e2c4-98bc-47c4-a5b4-a478721b7ea2
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph--1,ceph--2,ceph--3 (age 8m)
mgr: ceph--1(active, since 20m), standbys: ceph--2, ceph--3
osd: 3 osds: 3 up (since 32s), 3 in (since 32s)

data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 147 GiB / 150 GiB avail
pgs:

使用ceph-deploy配置ceph对象存储ceph-raw

1
[[email protected] ceph-deploy]# ceph-deploy rgw create ceph--1 ceph--2 ceph--3

使用ceph-deploy配置ceph文件存储ceph-mds

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
[[email protected] ceph-deploy]# ceph-deploy mds create ceph--1 ceph--2 ceph--3
[[email protected] ceph-deploy]# ceph -s
cluster:
id: faa2e2c4-98bc-47c4-a5b4-a478721b7ea2
health: HEALTH_OK

services:
mon: 3 daemons, quorum ceph--1,ceph--2,ceph--3 (age 17m)
mgr: ceph--1(active, since 29m), standbys: ceph--2, ceph--3
mds: 3 up:standby
osd: 3 osds: 3 up (since 9m), 3 in (since 9m)
rgw: 3 daemons active (ceph--1, ceph--2, ceph--3)

task status:

data:
pools: 4 pools, 128 pgs
objects: 187 objects, 1.2 KiB
usage: 3.0 GiB used, 147 GiB / 150 GiB avail
pgs: 128 active+clean

io:
client: 83 KiB/s rd, 0 B/s wr, 82 op/s rd, 54 op/s wr

开启并设置ceph-dashboard

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
[[email protected] ceph-deploy]# ceph  mgr module enable  dashboard
[[email protected] ceph-deploy]# ceph dashboard create-self-signed-cert
Self-signed certificate created
[[email protected] ceph-deploy]# ceph config set mgr mgr/dashboard/server_addr 10.140.11.8
[root[email protected] ceph-deploy]# ceph config set mgr mgr/dashboard/server_port 8080
[[email protected] ceph-deploy]# ceph config set mgr mgr/dashboard/ssl_server_port 8443
[[email protected] ceph-deploy]# ceph dashboard set-login-credentials admin admin
******************************************************************
*** WARNING: this command is deprecated. ***
*** Please use the ac-user-* related commands to manage users. ***
******************************************************************
Username and password updated
systemctl restart [email protected]
[[email protected] ceph-deploy]# ceph mgr services
{
"dashboard": "https://ceph--1:8443/"
}

请用谷歌浏览器访问https://10.140.11.8:8443/#/dashboard
账号密码admin/admin
dashboard


本博客所有文章除特别声明外,均采用 CC BY-SA 4.0 协议 ,转载请注明出处!