使用ceph提供对象存储

查看ceph集群状态检查rgw服务是否正常

首先先提供一个装有ceph rgw服务的ceph集群，安装方式我就在这里不演示了，详细请见ceph-deploy安装ceph集群这篇文章

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
[[email protected] ~]# ceph -s
  cluster:
    id:     faa2e2c4-98bc-47c4-a5b4-a478721b7ea2
    health: HEALTH_OK

  services:
    mon: 3 daemons, quorum ceph--1,ceph--2,ceph--3 (age 47h)
    mgr: ceph--1(active, since 4d), standbys: ceph--2, ceph--3
    mds: cephfs:1 {0=ceph--2=up:active} 2 up:standby
    osd: 3 osds: 3 up (since 4d), 3 in (since 4d)
    rgw: 3 daemons active (ceph--1, ceph--2, ceph--3)

  task status:
    scrub status:
        mds.ceph--2: idle

  data:
    pools:   7 pools, 208 pgs
    objects: 315 objects, 101 MiB
    usage:   3.3 GiB used, 147 GiB / 150 GiB avail
    pgs:     208 active+clean

  io:
    client:   4.0 KiB/s rd, 0 B/s wr, 3 op/s rd, 2 op/s wr
# radosgw端口默认为7480
[[email protected] ~]# netstat -antupl |grep 7480
tcp        0      0 0.0.0.0:7480            0.0.0.0:*               LISTEN      15194/radosgw
tcp6       0      0 :::7480                 :::*                    LISTEN      15194/radosgw

修改rgw的默认端口

ceph支持修改rgw的默认端口，下面我将rgw默认端口7480 修改为80端口

1
2
3
4
5
6
7
8
9
10
11
12
13
[[email protected] ceph-deploy]# cat ceph.conf 
[client.rgw.ceph--1]
host = ceph--1
rgw frontends = "civetweb port=80"
[client.rgw.ceph--2]
host = ceph--2
rgw frontends = "civetweb port=80"
[client.rgw.ceph--3]
host = ceph--3
rgw frontends = "civetweb port=80"

[[email protected] ceph-deploy]# ceph-deploy  --overwrite-conf  config push ceph--1 ceph--2 ceph--3
# systemctl start [email protected]`hostname -s` # 重启所有主机上的ceph rgw服务

验证是否修改成功

1
2
3
4
5
6
7
8
9
10
11
12
13
14
[[email protected] ceph-deploy]# netstat -antupl |grep radosgw
tcp        0      0 0.0.0.0:80              0.0.0.0:*               LISTEN      35167/radosgw
tcp        0      0 10.140.11.8:33980       10.140.11.24:6800       ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:36764       10.140.11.8:6802        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:44946       10.140.11.8:6800        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:51116       10.140.11.8:3300        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:42700       10.140.11.6:6800        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:44926       10.140.11.8:6800        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:33960       10.140.11.24:6800       ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:42678       10.140.11.6:6800        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:51134       10.140.11.8:3300        ESTABLISHED 35167/radosgw
tcp        0      0 10.140.11.8:36782       10.140.11.8:6802        ESTABLISHED 35167/radosgw
[[email protected] ceph-deploy]# curl http://ceph--1
<?xml version="1.0" encoding="UTF-8"?><ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/"><Owner><ID>anonymous</ID><DisplayName></DisplayName></Owner><Buckets></Buckets></ListAllMyBucketsResult>

使用S3访问CEPH RGW

为S3的访问创建账号

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
[[email protected] ~]# radosgw-admin  user create --uid ceph-s3-user --display-name "Ceph S3 User Dome"
{
    "user_id": "ceph-s3-user",
    "display_name": "Ceph S3 User Dome",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "9HR6Y9PAGBYFKRW5U6XM",   
            "secret_key": "51dSEB2WqZATowS71GPJPo8CoGG2VQZM63ndGfKl"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}

[[email protected] ~]# radosgw-admin  user list
[
    "ceph-s3-user"
]

[[email protected] ~]# radosgw-admin  user info --uid ceph-s3-user


{
    "user_id": "ceph-s3-user",
    "display_name": "Ceph S3 User Dome",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "9HR6Y9PAGBYFKRW5U6XM",
            "secret_key": "51dSEB2WqZATowS71GPJPo8CoGG2VQZM63ndGfKl"
        }
    ],
    "swift_keys": [],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
将access_key和secret_key记录下来

安装配置S3

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
[[email protected] ~]# yum install s3cmd -y
[[email protected] ~]# s3cmd --configure

Enter new values or accept defaults in brackets with Enter.
Refer to user manual for detailed description of all options.

Access key and Secret key are your identifiers for Amazon S3. Leave them empty for using the env variables.
Access Key: 9HR6Y9PAGBYFKRW5U6XM
Secret Key: 51dSEB2WqZATowS71GPJPo8CoGG2VQZM63ndGfKl
Default Region [US]: 

Use "s3.amazonaws.com" for S3 Endpoint and not modify it to the target Amazon S3.
S3 Endpoint [s3.amazonaws.com]: 10.140.11.8:80

Use "%(bucket)s.s3.amazonaws.com" to the target Amazon S3. "%(bucket)s" and "%(location)s" vars can be used
if the target S3 system supports dns based buckets.
DNS-style bucket+hostname:port template for accessing a bucket [%(bucket)s.s3.amazonaws.com]: 10.140.11.8:80/%(bucket)s

Encryption password is used to protect your files from reading
by unauthorized persons while in transfer to S3
Encryption password:
Path to GPG program [/usr/bin/gpg]: 

When using secure HTTPS protocol all communication with Amazon S3
servers is protected from 3rd party eavesdropping. This method is
slower than plain HTTP, and can only be proxied with Python 2.7 or newer
Use HTTPS protocol [Yes]: no

On some networks all internet access must go through a HTTP proxy.
Try setting it here if you can't connect to S3 directly
HTTP Proxy server name:

New settings:
  Access Key: 9HR6Y9PAGBYFKRW5U6XM
  Secret Key: 51dSEB2WqZATowS71GPJPo8CoGG2VQZM63ndGfKl
  Default Region: US
  S3 Endpoint: 10.140.11.8:80
  DNS-style bucket+hostname:port template for accessing a bucket: 10.140.11.8:80/%(bucket)s
  Encryption password:
  Path to GPG program: /usr/bin/gpg
  Use HTTPS protocol: False
  HTTP Proxy server name:
  HTTP Proxy server port: 0

Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)

Now verifying that encryption works...
Not configured. Never mind.

Save settings? [y/N] y
Configuration saved to '/root/.s3cfg'
# 在这个交互配置过程中，只配置了其中access_key和secret_key.

使用S3操作ceph RGW

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[[email protected] ~]# s3cmd mb s3://s3cmd-demo   # 创建bucket
Bucket 's3://s3cmd-demo/' created
[[email protected] ~]# s3cmd ls
2021-01-10 08:00  s3://s3cmd-demo
[[email protected] ~]# s3cmd put /etc/fstab s3://s3cmd-demo/fstab-demo   # 上传文件
upload: '/etc/fstab' -> 's3://s3cmd-demo/fstab-demo'  [1 of 1]
 42 of 42   100% in    2s    20.23 B/s  done
[[email protected] ~]# s3cmd ls s3://s3cmd-demo
2021-01-10 08:09           42  s3://s3cmd-demo/fstab-demo
[[email protected] ~]# s3cmd get s3://s3cmd-demo/fstab-demo lijiawang123 下载文件
download: 's3://s3cmd-demo/fstab-demo' -> 'lijiawang123'  [1 of 1]
 42 of 42   100% in    0s     2.83 KB/s  done
[[email protected] ~]# cat lijiawang123   #验证文件
LABEL=cloudimg-rootfs / ext4 defaults 0 1
[[email protected] ~]# cat /etc/fstab 
LABEL=cloudimg-rootfs / ext4 defaults 0 1

使用swift访问CEPH RGW

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
# 在已创建S3用户数添加swift_keys
[[email protected] ~]# radosgw-admin subuser create --uid ceph-s3-user --subuser=ceph-s3-user:swift  --access=full
{
    "user_id": "ceph-s3-user",
    "display_name": "Ceph S3 User Dome",
    "email": "",
    "suspended": 0,
    "max_buckets": 1000,
    "subusers": [
        {
            "id": "ceph-s3-user:swift",
            "permissions": "full-control"
        }
    ],
    "keys": [
        {
            "user": "ceph-s3-user",
            "access_key": "9HR6Y9PAGBYFKRW5U6XM",
            "secret_key": "51dSEB2WqZATowS71GPJPo8CoGG2VQZM63ndGfKl"
        }
    ],
    "swift_keys": [
        {
            "user": "ceph-s3-user:swift",
            "secret_key": "wLVnoP1hg6M1wMz4H6oBiBMUhFRSx5IgYvtCh6Ed"
        }
    ],
    "caps": [],
    "op_mask": "read, write, delete",
    "default_placement": "",
    "default_storage_class": "",
    "placement_tags": [],
    "bucket_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "user_quota": {
        "enabled": false,
        "check_on_raw": false,
        "max_size": -1,
        "max_size_kb": 0,
        "max_objects": -1
    },
    "temp_url_keys": [],
    "type": "rgw",
    "mfa_ids": []
}
# 验证
[[email protected] ~]# swift -V 1 -A http://10.140.11.8:80/auth -U ceph-s3-user:swift -K wLVnoP1hg6M1wMz4H6oBiBMUhFRSx5IgYvtCh6Ed list
s3cmd-demo
# 设置环境
[[email protected] ~]# cat swift.rc 
export ST_AUTH=http://10.140.11.8:80/auth 
export ST_USER=ceph-s3-user:swift
export ST_KEY=wLVnoP1hg6M1wMz4H6oBiBMUhFRSx5IgYvtCh6Ed

[[email protected] ~]# . swift.rc

[[email protected] ~]# swift  list
s3cmd-demo
[[email protected] ~]# cat swift.rc 
export ST_AUTH=http://10.140.11.8:80/auth 
export ST_USER=ceph-s3-user:swift
export ST_KEY=wLVnoP1hg6M1wMz4H6oBiBMUhFRSx5IgYvtCh6Ed

[[email protected] ~]# . swift.rc 
[[email protected] ~]# swift  list
s3cmd-demo
[[email protected] ~]# swift post swift-demo   # 创建bucket
[[email protected] ~]# swift  list
s3cmd-demo
swift-demo
[[email protected] ~]# swift upload  swift-demo /etc/passwd  # 上传文件
etc/passwd
[[email protected] ~]# swift  list swift-demo
etc/passwd
[[email protected] ~]# swift download swift-demo etc/passwd  # 下载文件
etc/passwd [auth 0.014s, headers 0.017s, total 0.017s, 0.460 MB/s]
[[email protected] ~]# ls etc/passwd 
etc/passwd
ceph
ceph

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！