Physical Infrastructure Maintenance

This chapter provides operational guidelines for physical infrastructure maintenance. For operations in production environment, you must follow the standardized procedures documented here to avoid unmanageable risks.


Power Maintenance in Equipment Room

Planned Shutdown

To perform planned shutdown in an equipment room, follow these steps:
  1. Stop all running businesses and power off all VM instances.
  2. Make all primary storages enter maintenance mode.
  3. Make all hosts enter maintenance mode.
  4. Stop the management node (MN). Methods of stopping MNs vary depending on deployment scenarios:
    • In a single-MN scenario, run the zstack-ctl stop command to stop the MN.
    • In a dual-MN HA scenario, run the zsha2 stop-node command on each MN to stop MNS and the zsha2 service.
  5. Verify that no I/O operation is performed in the primary storages of the Cloud, and run the poweroff command on each host to normally shut down all hosts.
  6. Disable the storages such as NFS, Ceph, SAN, or SMP.
  7. Power off the switches and other hardware facilities

Planned Power-On

To perform planned power-on in an equipment room, follow these steps:
  1. Power on the switches.
  2. Enable the storages such as NFS, Ceph, SAN, or SMP.
  3. Power on the servers.
  4. Check the status of the MN to make sure that the MN service starts successfully. Methods of starting MN service vary depending on deployment scenarios:
    • In a single-MN scenario, run the zstack-ctl status command to view its status.
    • In a dual-MN HA scenario, on either MN, run the zsha2 status command to view the HA mode and the MN status.
  5. Log in to ZStack Cloud and enable all primary storages and hosts. Make sure that all hosts and primary storages are in the connected state.
  6. Enable the VM instances and resume the businesses.

Recovery from Unexpected Power Outage

The steps to recover from unexpected power outage in an equipment room are the same as those in Planned Power-On:
  1. Power on the switches.
  2. Enable the storages such as NFS, Ceph, SAN, or SMP.
  3. Power on the servers.
  4. Check the status of the MN to make sure that the MN service starts successfully. Methods of starting MN service vary depending on deployment scenarios:
    • In a single-MN scenario, run the zstack-ctl status command to view its status.
    • In a dual-MN HA scenario, on either MN, run the zsha2 status command to view the HA mode and the MN status.
  5. Log in to ZStack Cloud and enable all primary storages and hosts. Make sure that all hosts and primary storages are in the connected state.
  6. Enable the VM instances and resume the businesses.

Equipment Room Relocation

To move the equipment room to another location, please follow the steps below:
  1. Power off all involved facilities under normal conditions in sequence by referring to Planned Shutdown.
  2. Mark the access ports of all switches and hosts.
  3. Pack up the servers. Move the servers to the new equipment room and reconnect them.
  4. Power on all involved facilities in sequence by referring to Planned Power-On.
  5. Check the recovery status of the network, Cloud, and VM instances.

Switch Maintenance

Management Network Switch Maintenance

The maintenance of management network switches may affect business continuity. Please exercise caution. If this kind of maintenance is necessary, we recommend you schedule this operating during off-peak business hours. The maintenance procedures are as follows:
  1. Disable HA Policy in the Platform Setting.
  2. Stop management nodes (MNs). Methods of stopping MNs vary depending on deployment scenarios:
    • In a single-MN scenario, run the zstack-ctl stop command to stop the MN.
    • In a dual-MN HA scenario, run the zsha2 stop-node command on each MN to stop MNS and the zsha2 service.
  3. Adjust or reboot the related management switches.
  4. Verify the IP connectivity of all hardware resources at the access end of the switch (including host, primary storage, and image storage) to ensure that all the management networks communicate with each other properly.
    Note: If a node fails to connect the MN, troubleshoot the issue before you proceed with subsequent operations.
  5. Enable the MN and check its status to ensure that the MN service starts successfully. Methods of starting MN service vary depending on deployment scenarios:
    • In a single-MN scenario, run the zstack-ctl start command to start the MN service and run the command zstack-ctl status to view its status.
    • In a dual-MN HA scenario, run the zsha2 start-node command on each MN respectively to start MN services. On either MN, run the zsha2 status command to view the HA mode and the MN status.
  6. Ensure that all hosts and primary storages are in the connected status, and HA Policy is enabled in the Platform Setting.

Business Network Switch Maintenance

We recommend you schedule the maintenance of business network switches during off-peak business hours so as to avoid affecting business continuity. The maintenance procedures are as follows:
  1. Make the host connected to the switch enter maintenance mode.
  2. Reboot the related switch.
  3. Enable any host in the maintenance mode and check its connectivity. For example, supposing you create a test VM instance and then check its connectivity with other business VMs. If the connectivity succeeds, the switch interface configurations corresponding to the host is correct and available. If the connectivity fails, you have to continue to check the connectivity.
  4. Repeat Step 3 and check the connectivity in turn to ensure that all business networks are available.

Storage Network Switch Maintenance

The maintenance of storage network switches will directly affect all business, so you must stop all business before you perform the maintenance. The maintenance procedures are as follows:
  1. Make all primary storages associated with the storage switch on the Cloud enter maintenance mode.
  2. Log in to the user interface (UI) of Ceph Enterprise and choose Setting > Disable Data Recovery > Ban.
  3. Adjust or reboot the related storage switches.
  4. Ensure that the storage networks of all storage nodes are interconnected and accessible.
  5. Log in to the UI of Ceph Enterprise and choose Setting > Enable Data Recovery > Enable.
  6. Enable the primary storages that are in the maintenance mode and make sure that the primary storages are in the connected state.
  7. Start the stopped VM instances.

Archives

Download Document Archives

Download

Already filled the basic info?Click here.

Enter at least 2 characters.
Invalid mobile number.
Enter at least 4 characters.
Invalid email address.
Wrong code. Try again. Send Code Resend Code (60s)

An email with a verification code will be sent to you. Make sure the address you provided is valid and correct.

同意 不同意

I have read and concur with the Site TermsPrivacy PolicyRules and Conventions on User Management of ZStack Cloud

Download

Not filled the basic info yet? Click here.

Invalid email address or mobile number.
同意 不同意

I have read and concur with the Site TermsPrivacy PolicyRules and Conventions on User Management of ZStack Cloud

Email Us

contact@zstack.io
ZStack Training and Certification
Enter at least 2 characters.
Invalid mobile number.
Enter at least 4 characters.
Invalid email address.
Wrong code. Try again. Send Code Resend Code (60s)

同意 不同意

I have read and concur with the Site TermsPrivacy PolicyRules and Conventions on User Management of ZStack Cloud

Email Us

contact@zstack.io
Request Trial
Enter at least 2 characters.
Invalid mobile number.
Enter at least 4 characters.
Invalid email address.
Wrong code. Try again. Send Code Resend Code (60s)

同意 不同意

I have read and concur with the Site TermsPrivacy PolicyRules and Conventions on User Management of ZStack Cloud

Email Us

contact@zstack.io

The download link is sent to your email address.

If you don't see it, check your spam folder, subscription folder, or AD folder. After receiving the email, click the URL to download the documentation.

The download link is sent to your email address.

If you don't see it, check your spam folder, subscription folder, or AD folder.
Or click on the URL below. (For Internet Explorer, right-click the URL and save it.)

Thank you for using ZStack products and services.

Submit successfully.

We'll connect soon.

Thank you for using ZStack products and services.