ZStack 3.10.0>Product Manuals>Maintenance Manual>Physical Infrastructure Maintenance

Physical Infrastructure Maintenance

This chapter provides some methods for physical infrastructure maintenance. In a production environment, you need to perform maintenance operations by referring to the procedures described in this document. Otherwise, uncontrollable risks will be caused.

Power Maintenance in Equipment Room

Planned Shutdown

To perform planned shutdown in an equipment room, follow these steps:

Stop all running businesses.
Place all primary storages into maintenance mode. After all primary storages enter maintenance mode, make sure that all VM instances are shut down.
Place all hosts into maintenance mode.
Stop the management node (MN). The methods to stop the MN in different scenarios are as follows:
- In a single MN scenario, run the zstack-ctl stop command to stop the MN.
- In a multi-MN (host) HA scenario, run the zsha2 stop-node command in each MN to stop the MNs and shut down the zsha2 service.
Verify that no I/O operation is performed in the primary storages of the Cloud, and run the poweroff command on each host to normally shut down all hosts.
Disable the storages, such as the NFS, Ceph, SAN, and SMP storages.
Disable the switches and other hardware facilities.

Planned Power-on

To perform planned power-on in an equipment room, follow these steps:

Power on the switches.
Power on the storages, such as the NFS, Ceph, SAN, and SMP storages.
Power on the server.
Check the service status of the MN and make sure that the MN service is started successfully. The methods to check the MN status in different scenarios are as follows:
- In a single MN scenario, run the zstack-ctl status command to view the MN status.
- In a multi-MN (host) HA scenario, run the zsha2 status command in one of the MNs to view the HA status and the MN status.
Log in to the ZStack Cloud, enable all primary storages and hosts and make sure that all hosts and primary storages are in the connected state.
Start the VM instances and resume the businesses.

Recovery from Unexpected Power Outage

The steps to recover from unexpected power outage in an equipment room are the same as those in planned power-on.

Power on the switches.
Power on the storages, such as the NFS, Ceph, SAN, and SMP storages.
Power on the server.
Check the service status of the MN and make sure that the MN service is started successfully. The methods to check the MN status in different scenarios are as follows:
- In a single MN scenario, run the zstack-ctl status command to view the MN status.
- In a multi-MN (host) HA scenario, run the zsha2 status command in one of the MNs to view the HA status and the MN status.
Log in to the ZStack Cloud, enable all primary storages and hosts and make sure that all hosts and primary storages are in the connected state.
Start the VM instances and resume the businesses.

Equipment Room Relocation

When you relocate an equipment room, perform maintenance by following these steps:

Power off all power supplies by referring to Planned Shutdown.
Mark the access ports of all switch wiring and hosts.
Pack the server, move the server to the new equipment room, and connect the server.
Power on all power supplies by referring to Planned Power-on.
Check the recovery status of the network, Cloud, and VM instances.

Switch Maintenance

Management Network Switch Maintenance

The maintenance on management network switches might affect the proper running of the business. Please exercise caution. If this kind of maintenance is necessary, we recommend that you perform this operation during off-peak hours. The maintenance procedure is as follows:

Turn off VM high availability in the global settings of the Cloud.
Stop the management node (MN). The methods to stop the MN in different scenarios are as follows:
- In a single MN scenario, run the zstack-ctl stop command to stop the MN.
- In a multi-MN (host) HA scenario, run the zsha2 stop-node command in each MN to stop the MN and the zsha2 service.
Adjust or restart related management switches.
Check the IP connectivity of all hardware resources (such as the host, primary storage, and backup storage) at the access end of the switch to ensure that all the management networks in the access end can communicate with each other properly.
Note: If a node fails to connect to the MN, troubleshot this issue before you proceed with subsequent operations.
Start and check the MN service status to ensure that the MN service starts successfully. The methods to check the MN startup in different scenarios are as follows:
- In a single MN scenario, run the zstack-ctl start command to start the MN service, and run the zstack-ctl status command to view the MN status.
- In a multi-MN (host) HA scenario, run the zsha2 start-node command in each MN to start the MN service, and run the zsha2 status command to view the HA status and the MN status.
Make sure that all hosts and primary storages are in the connected state, and VM high availability is enabled in the global settings of the Cloud.

Business Network Switch Maintenance

We recommend that you perform maintenance on business network switches during off-peak hours to avoid affecting important services. The maintenance procedure is as follows:

Place the hosts accessed to the switches into maintenance mode.
Adjust or restart related service switches.
Start a host that is in maintenance mode and check the connectivity. For example, you can use this host to create a test VM instance and check the connectivity between this VM instance and other VM instances. If the connectivity succeeds, the switch interface configuration corresponding to this host is correct and available. If the network is unreachable, continue to check the connectivity.
Repeat Step 3 to check the connectivity in turn to ensure that all business networks are available.

Storage Network Switch Maintenance

The maintenance on storage network switches directly affects all the business. You must stop all business before you perform the maintenance. The maintenance procedure is as follows:

Place all primary storages associated with the storage switches on the Cloud into maintenance mode.
Log in to the UI of Ceph Enterprise and choose Settings > Disable data recovery > Ban.
Adjust or restart related storage switches.
Make sure that the storage networks of all storage nodes are interconnected and reachable.
Log in to the UI of Ceph Enterprise and choose Settings > Enable Data Recovery > Enable.
Enable the primary storages that are in the maintenance mode and make sure that the primary storages are in the connected state.
Start the stopped VM instances.

Next Chapter:FAQ