openstack instance vm_power_state 状态置为 NOSTATE

openstack 上的instance运行正常，但是power_state状态为NOSTATE，导致instance热迁移失败

现象

1、再一次热迁移时候发现迁移时候报以下错误

1
2
# openstack server migrate --live node-53 86487ef4-cc12-4be6-995e-46f5ac093901
Cannot 'os-migrateLive' instance 86487ef4-cc12-4be6-995e-46f5ac093901 while it is in power_state 0 (HTTP 409) (Request-ID: req-6c14e0ee-c3df-42de-873d-9ecc8ad215cc)

2、查看实例，发现虚拟机运行正常，但是 power_state 为 NOSTATE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
# openstack server show 86487ef4-cc12-4be6-995e-46f5ac093901
+-------------------------------------+----------------------------------------------------------------+
| Field                               | Value                                                          |
+-------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                         |
| OS-EXT-AZ:availability_zone         | m_cpu+san                                                      |
| OS-EXT-SRV-ATTR:host                | node-27                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-27                                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-000071aa                                              |
| OS-EXT-STS:power_state              | NOSTATE                                                        |
| OS-EXT-STS:task_state               | None                                                           |
| OS-EXT-STS:vm_state                 | active                                                         |
| OS-SRV-USG:launched_at              | 2018-09-21T05:40:36.000000                                     |
| OS-SRV-USG:terminated_at            | None                                                           |
| accessIPv4                          |                                                                |
| accessIPv6                          |                                                                |
| addresses                           | vlan_10.122.44.0/23=10.122.45.53                               |
| config_drive                        | True                                                           |
| created                             | 2018-09-21T05:40:31Z                                           |
| flavor                              | 16-64-100 (1cbe4ea1-8a67-4027-afd4-8f31a8b94851)               |
| hostId                              | 2730fa3d62e347ddb67e155f6eed973787a868c82316f7e6ba641b10       |
| id                                  | 86487ef4-cc12-4be6-995e-46f5ac093901                           |
| image                               | lenovo-centos-7-release (b9f8f864-4217-4ac6-a116-62ecfa0fc074) |
| key_name                            | None                                                           |
| name                                | SLP3YM7HRCX                                                    |
| progress                            | 100                                                            |
| project_id                          | e992715df18a417997c068e5f9834b0f                               |
| properties                          |                                                                |
| security_groups                     | name='default'                                                 |
| status                              | ACTIVE                                                         |
| updated                             | 2019-10-09T07:58:58Z                                           |
| user_id                             | af518aeb935c4d258f8cb7d302c83797                               |
| volumes_attached                    | id='a821856b-409f-4e6e-becd-eb4b2344c7d8'                      |
+-------------------------------------+----------------------------------------------------------------+

查找原因

列出该虚拟机的历史操作

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
# nova instance-action-list 86487ef4-cc12-4be6-995e-46f5ac093901
+----------------+------------------------------------------+---------+----------------------------+
| Action         | Request_ID                               | Message | Start_Time                 |
+----------------+------------------------------------------+---------+----------------------------+
| create         | req-4d01a466-ce7a-48a9-b575-e129b61bcc30 | -       | 2018-09-21T05:40:30.000000 |
| live-migration | req-2d93bc73-e43f-4ca6-88fc-a6ad1a4021a6 | -       | 2018-09-21T05:44:47.000000 |
| live-migration | req-7fcbfb15-7e3c-471d-8ea5-020d438d23c3 | -       | 2019-03-24T13:00:30.000000 |
| live-migration | req-2b009c4f-caef-4d90-81fd-d480c1b9efad | Error   | 2019-10-09T07:00:13.000000 |
| live-migration | req-b364724b-6b28-4d5f-8f14-05d1c4abeceb | Error   | 2019-10-09T07:07:37.000000 |
| live-migration | req-8669d3b1-e57e-4b35-8d40-a110b64f47c8 | Error   | 2019-10-09T07:29:13.000000 |
| live-migration | req-ecec98ca-3777-4ca3-afcd-83e36381e038 | Error   | 2019-10-09T07:55:12.000000 |
| live-migration | req-6378f839-89c0-43c0-8a65-40e5b775283c | Error   | 2019-10-09T07:58:14.000000 |
| live-migration | req-e966caa7-06e5-4a29-a371-517d98c37121 | Error   | 2019-10-09T07:58:58.000000 |
| live-migration | req-6c14e0ee-c3df-42de-873d-9ecc8ad215cc | Error   | 2019-10-09T08:31:47.000000 |
+----------------+------------------------------------------+---------+----------------------------+

列出该实例的迁移记录

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
# nova  migration-list|grep 86487ef4-cc12-4be6-995e-46f5ac093901
| 84892 | -           | -         | node-27        | node-29            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:07:37.000000 | 2019-10-09T07:07:37.000000 | live-migration |
| 54307 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-03-24T13:00:30.000000 | 2019-03-24T17:10:10.000000 | live-migration |
| 84904 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:29:13.000000 | 2019-10-09T07:29:13.000000 | live-migration |
| 84913 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:55:12.000000 | 2019-10-09T07:55:12.000000 | live-migration |
| 84919 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:58:14.000000 | 2019-10-09T07:58:14.000000 | live-migration |
| 84922 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:58:58.000000 | 2019-10-09T07:58:58.000000 | live-migration |
| 84931 | -           | -         | node-27        | node-53            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T08:31:47.000000 | 2019-10-09T08:31:47.000000 | live-migration |
| 84889 | -           | -         | node-27        | node-89            | -            | error         | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2019-10-09T07:00:13.000000 | 2019-10-09T07:00:13.000000 | live-migration |
| 42241 | -           | -         | node-33        | node-27            | -            | completed     | 86487ef4-cc12-4be6-995e-46f5ac093901 | 189        | 189        | 2018-09-21T05:44:48.000000 | 2018-09-21T05:45:13.000000 | live-migration |
可以看到该实例从node-27节点迁移到node-85、node-53、node-29节点均为迁移成功
查看instance所在的节点
# ssh node-27

root@node-27:~# virsh list --all|grep instance-000071aa 
[email protected]:~# exit
logout
Connection to node-27 closed.
# ssh node-53

[email protected]:~#  virsh list --all|grep instance-000071aa
 91    instance-000071aa              running
[email protected]:~# exit
logout
Connection to node-53 closed.
# openstack server show 86487ef4-cc12-4be6-995e-46f5ac093901
+-------------------------------------+----------------------------------------------------------------+
| Field                               | Value                                                          |
+-------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                         |
| OS-EXT-AZ:availability_zone         | m_cpu+san                                                      |
| OS-EXT-SRV-ATTR:host                | node-27                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-27                                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-000071aa                                              |
| OS-EXT-STS:power_state              | NOSTATE                                                        |
| OS-EXT-STS:task_state               | None                                                           |
| OS-EXT-STS:vm_state                 | active                                                         |
| OS-SRV-USG:launched_at              | 2018-09-21T05:40:36.000000                                     |
| OS-SRV-USG:terminated_at            | None                                                           |
| accessIPv4                          |                                                                |
| accessIPv6                          |                                                                |
| addresses                           | vlan_10.122.44.0/23=10.122.45.53                               |
| config_drive                        | True                                                           |
| created                             | 2018-09-21T05:40:31Z                                           |
| flavor                              | 16-64-100 (1cbe4ea1-8a67-4027-afd4-8f31a8b94851)               |
| hostId                              | 2730fa3d62e347ddb67e155f6eed973787a868c82316f7e6ba641b10       |
| id                                  | 86487ef4-cc12-4be6-995e-46f5ac093901                           |
| image                               | lenovo-centos-7-release (b9f8f864-4217-4ac6-a116-62ecfa0fc074) |
| key_name                            | None                                                           |
| name                                | SLP3YM7HRCX                                                    |
| progress                            | 100                                                            |
| project_id                          | e992715df18a417997c068e5f9834b0f                               |
| properties                          |                                                                |
| security_groups                     | name='default'                                                 |
| status                              | ACTIVE                                                         |
| updated                             | 2019-10-09T08:31:47Z                                           |
| user_id                             | af518aeb935c4d258f8cb7d302c83797                               |
| volumes_attached                    | id='a821856b-409f-4e6e-becd-eb4b2344c7d8'                      |
+-------------------------------------+----------------------------------------------------------------+
root@node-1:~# mysql -unova -pnova
MariaDB [(none)]> select * from nova.instances where uuid='86487ef4-cc12-4be6-995e-46f5ac093901'\G;
*************************** 1. row ***************************
              created_at: 2018-09-21 05:40:31
              updated_at: 2019-10-09 08:31:47
              deleted_at: NULL
                      id: 29098
             internal_id: NULL
                 user_id: af518aeb935c4d258f8cb7d302c83797
              project_id: e992715df18a417997c068e5f9834b0f
               image_ref: b9f8f864-4217-4ac6-a116-62ecfa0fc074
               kernel_id: 
              ramdisk_id: 
            launch_index: 0
                key_name: NULL
                key_data: NULL
             power_state: 0
                vm_state: active
               memory_mb: 65536
                   vcpus: 16
                hostname: slp3ym7hrcx
                    host: node-27
               user_data: NULL
          reservation_id: r-o8dr8ibn
             launched_at: 2018-09-21 05:40:36
           terminated_at: NULL
            display_name: SLP3YM7HRCX
     display_description: SLP3YM7HRCX
       availability_zone: no_san
                  locked: 0
                 os_type: NULL
             launched_on: node-33
        instance_type_id: 189
                 vm_mode: NULL
                    uuid: 86487ef4-cc12-4be6-995e-46f5ac093901
            architecture: NULL
        root_device_name: /dev/vda
            access_ip_v4: NULL
            access_ip_v6: NULL
            config_drive: True
              task_state: NULL
default_ephemeral_device: NULL
     default_swap_device: NULL
                progress: 100
        auto_disk_config: 0
      shutdown_terminate: 0
       disable_terminate: 0
                 root_gb: 100
            ephemeral_gb: 0
               cell_name: NULL
                    node: node-27
                 deleted: 0
               locked_by: NULL
                 cleaned: 1
      ephemeral_key_uuid: NULL
1 row in set (0.00 sec)

ERROR: No query specified

发现实际instance启动在node-53节点上，但是数据库记录在node-27节点上

manager.py 捕获到 InstanceNotFound 设置 vm_power_state=NOSTATE

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
/usr/lib/python2.7/dist-packages/nova/compute/manager.py

    def _query_driver_power_state_and_sync(self, context, db_instance):
        if db_instance.task_state is not None:
            LOG.info(_LI("During sync_power_state the instance has a "
                         "pending task (%(task)s). Skip."),
                     {'task': db_instance.task_state}, instance=db_instance)
            return
        # No pending tasks. Now try to figure out the real vm_power_state.
        try:
            vm_instance = self.driver.get_info(db_instance)
            vm_power_state = vm_instance.state
        except exception.InstanceNotFound:         # 可以看到如果InstanceNotFound
            vm_power_state = power_state.NOSTATE   # power_state设置为NOSTATE
        # Note(maoy): the above get_info call might take a long time,
        # for example, because of a broken libvirt driver.
        try:
            self._sync_instance_power_state(context,
                                            db_instance,
                                            vm_power_state,
                                            use_slave=True)
        except exception.InstanceNotFound:
            # NOTE(hanlind): If the instance gets deleted during sync,
            # silently ignore.
            pass

得到原因

由于之前 migrate ERROR 导致 instance 实际启动的节点和数据库记录的节点不一致，comoute 通过 manager.py 捕获到 InstanceNotFound， 所以把 vm_power_state 状态置为 NOSTATE

解决方案

更新数据库，将数据库中的instance对应的host、node更新成实际instance启动的节点信息。硬重启虚拟机

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
root@node-1:~# mysql -unova -pnova
MariaDB [(none)]> update nova.instances set host='node-53' where uuid='86487ef4-cc12-4be6-995e-46f5ac093901'\G;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

ERROR: No query specified

MariaDB [(none)]> update nova.instances set node='node-53' where uuid='86487ef4-cc12-4be6-995e-46f5ac093901'\G;
Query OK, 1 row affected (0.00 sec)
Rows matched: 1  Changed: 1  Warnings: 0

ERROR: No query specified

MariaDB [(none)]> select * from nova.instances where uuid='86487ef4-cc12-4be6-995e-46f5ac093901'\G;
*************************** 1. row ***************************
              created_at: 2018-09-21 05:40:31
              updated_at: 2019-10-09 08:31:47
              deleted_at: NULL
                      id: 29098
             internal_id: NULL
                 user_id: af518aeb935c4d258f8cb7d302c83797
              project_id: e992715df18a417997c068e5f9834b0f
               image_ref: b9f8f864-4217-4ac6-a116-62ecfa0fc074
               kernel_id: 
              ramdisk_id: 
            launch_index: 0
                key_name: NULL
                key_data: NULL
             power_state: 0
                vm_state: active
               memory_mb: 65536
                   vcpus: 16
                hostname: slp3ym7hrcx
                    host: node-53
               user_data: NULL
          reservation_id: r-o8dr8ibn
             launched_at: 2018-09-21 05:40:36
           terminated_at: NULL
            display_name: SLP3YM7HRCX
     display_description: SLP3YM7HRCX
       availability_zone: no_san
                  locked: 0
                 os_type: NULL
             launched_on: node-33
        instance_type_id: 189
                 vm_mode: NULL
                    uuid: 86487ef4-cc12-4be6-995e-46f5ac093901
            architecture: NULL
        root_device_name: /dev/vda
            access_ip_v4: NULL
            access_ip_v6: NULL
            config_drive: True
              task_state: NULL
default_ephemeral_device: NULL
     default_swap_device: NULL
                progress: 100
        auto_disk_config: 0
      shutdown_terminate: 0
       disable_terminate: 0
                 root_gb: 100
            ephemeral_gb: 0
               cell_name: NULL
                    node: node-53
                 deleted: 0
               locked_by: NULL
                 cleaned: 1
      ephemeral_key_uuid: NULL
1 row in set (0.00 sec)

ERROR: No query specified

MariaDB [(none)]> exit
Bye
# openstack server show 86487ef4-cc12-4be6-995e-46f5ac093901
+-------------------------------------+----------------------------------------------------------------+
| Field                               | Value                                                          |
+-------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                         |
| OS-EXT-AZ:availability_zone         | no_san                                                         |
| OS-EXT-SRV-ATTR:host                | node-53                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-53                                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-000071aa                                              |
| OS-EXT-STS:power_state              | NOSTATE                                                        |
| OS-EXT-STS:task_state               | None                                                           |
| OS-EXT-STS:vm_state                 | active                                                         |
| OS-SRV-USG:launched_at              | 2018-09-21T05:40:36.000000                                     |
| OS-SRV-USG:terminated_at            | None                                                           |
| accessIPv4                          |                                                                |
| accessIPv6                          |                                                                |
| addresses                           | vlan_10.122.44.0/23=10.122.45.53                               |
| config_drive                        | True                                                           |
| created                             | 2018-09-21T05:40:31Z                                           |
| flavor                              | 16-64-100 (1cbe4ea1-8a67-4027-afd4-8f31a8b94851)               |
| hostId                              | f5274ebe433297bc376a4ce09151591735a04d2b6762280866de2729       |
| id                                  | 86487ef4-cc12-4be6-995e-46f5ac093901                           |
| image                               | lenovo-centos-7-release (b9f8f864-4217-4ac6-a116-62ecfa0fc074) |
| key_name                            | None                                                           |
| name                                | SLP3YM7HRCX                                                    |
| progress                            | 100                                                            |
| project_id                          | e992715df18a417997c068e5f9834b0f                               |
| properties                          |                                                                |
| security_groups                     | name='default'                                                 |
| status                              | ACTIVE                                                         |
| updated                             | 2019-10-09T08:31:47Z                                           |
| user_id                             | af518aeb935c4d258f8cb7d302c83797                               |
| volumes_attached                    | id='a821856b-409f-4e6e-becd-eb4b2344c7d8'                      |
+-------------------------------------+----------------------------------------------------------------+



# nova reboot --hard 86487ef4-cc12-4be6-995e-46f5ac093901

# openstack  server show 86487ef4-cc12-4be6-995e-46f5ac093901
+-------------------------------------+----------------------------------------------------------------+
| Field                               | Value                                                          |
+-------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                         |
| OS-EXT-AZ:availability_zone         | no_san                                                         |
| OS-EXT-SRV-ATTR:host                | node-53                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-53                                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-000071aa                                              |
| OS-EXT-STS:power_state              | Running                                                        |
| OS-EXT-STS:task_state               | None                                                           |
| OS-EXT-STS:vm_state                 | active                                                         |
| OS-SRV-USG:launched_at              | 2018-09-21T05:40:36.000000                                     |
| OS-SRV-USG:terminated_at            | None                                                           |
| accessIPv4                          |                                                                |
| accessIPv6                          |                                                                |
| addresses                           | vlan_10.122.44.0/23=10.122.45.53                               |
| config_drive                        | True                                                           |
| created                             | 2018-09-21T05:40:31Z                                           |
| flavor                              | 16-64-100 (1cbe4ea1-8a67-4027-afd4-8f31a8b94851)               |
| hostId                              | 4bfe001f6fa4d443f283121b56139ef8bfbff47be8106387cc19edc6       |
| id                                  | 86487ef4-cc12-4be6-995e-46f5ac093901                           |
| image                               | lenovo-centos-7-release (b9f8f864-4217-4ac6-a116-62ecfa0fc074) |
| key_name                            | None                                                           |
| name                                | SLP3YM7HRCX                                                    |
| progress                            | 0                                                              |
| project_id                          | e992715df18a417997c068e5f9834b0f                               |
| properties                          |                                                                |
| security_groups                     | name='default'                                                 |
| status                              | ACTIVE                                                         |
| updated                             | 2019-10-09T10:04:19Z                                           |
| user_id                             | af518aeb935c4d258f8cb7d302c83797                               |
| volumes_attached                    | id='a821856b-409f-4e6e-becd-eb4b2344c7d8'                      |
|                                     | id='ff7d2dad-1954-4906-ae18-95c76a09b442'                      |
+-------------------------------------+----------------------------------------------------------------+

# openstack server migrate --live node-89 86487ef4-cc12-4be6-995e-46f5ac093901

# openstack  server show 86487ef4-cc12-4be6-995e-46f5ac093901
+-------------------------------------+----------------------------------------------------------------+
| Field                               | Value                                                          |
+-------------------------------------+----------------------------------------------------------------+
| OS-DCF:diskConfig                   | MANUAL                                                         |
| OS-EXT-AZ:availability_zone         | Contingency                                                    |
| OS-EXT-SRV-ATTR:host                | node-89                                                        |
| OS-EXT-SRV-ATTR:hypervisor_hostname | node-89                                                        |
| OS-EXT-SRV-ATTR:instance_name       | instance-000071aa                                              |
| OS-EXT-STS:power_state              | Running                                                        |
| OS-EXT-STS:task_state               | None                                                           |
| OS-EXT-STS:vm_state                 | active                                                         |
| OS-SRV-USG:launched_at              | 2018-09-21T05:40:36.000000                                     |
| OS-SRV-USG:terminated_at            | None                                                           |
| accessIPv4                          |                                                                |
| accessIPv6                          |                                                                |
| addresses                           | vlan_10.122.44.0/23=10.122.45.53                               |
| config_drive                        | True                                                           |
| created                             | 2018-09-21T05:40:31Z                                           |
| flavor                              | 16-64-100 (1cbe4ea1-8a67-4027-afd4-8f31a8b94851)               |
| hostId                              | 4bfe001f6fa4d443f283121b56139ef8bfbff47be8106387cc19edc6       |
| id                                  | 86487ef4-cc12-4be6-995e-46f5ac093901                           |
| image                               | lenovo-centos-7-release (b9f8f864-4217-4ac6-a116-62ecfa0fc074) |
| key_name                            | None                                                           |
| name                                | SLP3YM7HRCX                                                    |
| progress                            | 0                                                              |
| project_id                          | e992715df18a417997c068e5f9834b0f                               |
| properties                          |                                                                |
| security_groups                     | name='default'                                                 |
| status                              | ACTIVE                                                         |
| updated                             | 2019-10-09T10:04:19Z                                           |
| user_id                             | af518aeb935c4d258f8cb7d302c83797                               |
| volumes_attached                    | id='a821856b-409f-4e6e-becd-eb4b2344c7d8'                      |
|                                     | id='ff7d2dad-1954-4906-ae18-95c76a09b442'                      |
+-------------------------------------+----------------------------------------------------------------+
openstack
openstack

本博客所有文章除特别声明外，均采用 CC BY-SA 4.0 协议 ，转载请注明出处！