I once had to upgrade a Cisco UCS Manager and the process went very smoothly, without any downtime and I documented all the procedures.
For those who do not know this system here is a brief description:
Cisco UCS Manager supports the entire Cisco UCS server and Cisco HyperFlex Series hyperconverged infrastructure portfolios. It enables server, fabric, and storage provisioning as well as, device discovery, inventory, configuration, diagnostics, monitoring, fault detection, auditing, and statistics collection. You can extend the benefits of Cisco UCS Manager globally across an enterprise to thousands of servers in multiple domains with Cisco UCS Central Software.
More information:
In this post I will show you how to Upgrade UCS 2.2(3c) to version 2.2(8f), but it also applies to other versions. I'm going to show you how to Upgrade the infrastructure (UCS) software, Fabric Interconnects and I/O modules firmware in a two-stage operation.
So, let's start!
Stage 1 (Checks and Downloads)
- Disable Call Home;
- Backup all Settings;
- Check status HA Fabric Interconnects;
- Check I / O Modules;
- Check Status Servers and Adapters;
- Download Software Firmware Bundles;
- Download Firmware for Interconnet Fabric.
Note: I have noticed that only the Java version 7 update 40 was compatible with this version of UCS Manager. Older versions of UCS are very sensitive to new versions of Java (due to certificates and security). So you have to use version 7.
Before Upgrade my UCS manager looks like this:
Disable Call Home features. Don’t forget to save the changes.
Go to Admin --> Communication Management --> Call Home, and turn off call home. You don’t want Cisco calling you thinking UCSM is on fire, when you’re doing an upgrade, right?
Check alerts and verify the system is healthy before proceeding. Fix anything that’s potentially a problem. Check if backup configuration exists. If not, take a backup of your UCS configuration.
View the installed firmware versions and check for compatibility issues:
Check Primary and Subordinate Interconnects
- Overall Status: Operable
- High Availability Details – Ready: Yes
Check State of both IO Modules
- Overall Status: Operable
Check Stats of the Servers Adapters
- Operability: Operable
Check size of local storage on both interconnects
Download firmware from Cisco and Upload the Infrastructure Bundle into UCSM, so it’s ready to deploy.
Stage 2 (Upgrades)
- Upgrade and Activation;
- Upgrade UCS Manager Software;
- Upgrade Firmware I/O Modules;
- Upgrade Firmware Subordinate Fabric Interconnect;
- Upgrade Firmware Primary Fabric Interconnect;
- Enable Call Home.
Determine the primary and subordinate Fabric Interconnect. I like to SSH into UCSM and run the following command to give me that info plus general cluster health status before proceeding:
UCS# show cluster extended-state
Record this information, will will need to know what FI is subordinate and primary.
Upgrade UCS Manager Software. Go to Equipment --> Firmware Management --> Firmware AutoInstall --> Activate Firmware. Activate the new version of UCS Manager. Verify it completes and no unexpected errors result. It is possible that sometimes errors are expected, and it’s OK to proceed. In this case I'm going to Upgrade to 2.2(8f).
Change Startup version to new firmware uploaded and click Ok. During the process you will loose access to the console and you will be prompted to re-login. Wait a few minutes before attempting to do so. After re-login check the new version installed.
Update the firmware on the I/O Modules by going to Equipment --> Chassis --> Chassis number --> IO Modules --> IO Module you want to upgrade --> General --> Update firmware. Repeat on the second IO Module. You can track the progress on the Update Status portion of the general page. After that, activate the I/O Modules firmware on the General tab.
Clear the checkbox for "Set Startup Version Only" to have the code change take effect immediately. If you leave this option enabled, you’ll need to reboot the IOM Module yourself. I recommend clearing it, and let UCSM reboot it for you.
You may also receive the error: "Failed start activation. Manual upgrade/activation is disallowed because the Default Infrastructure Policy Startup Version is set."
To resolve this, you must clear the startup version of the default infrastructure pack before directly upgrading or activating:
UCS# scope org
UCS /org # scope fw-infra-pack default
UCS /org/fw-infra-pack # set infra-bundle-version ""
UCS /org/fw-infra-pack* # commit-buffer
Upgrade Fabric Interconnects. First upgrade the subordinate Fabric going to Equipment --> Installed Firmware. Right click the subordinate FI, select Activate Firmware, and select the new firmware package. Verify when the FI comes back up and it's running the proper new version and that your network and storage redundancy are working properly.
You will receive a warning message that the fabric will reboot. After the fabric upgrade it will start auto-upgrading the correspondent I/O Module if you haven’t did that already.
In the end, the activate status should be ready. Failover fabric primary to subordinate by connecting to UCSM via SSH, and run the following commands
UCS# connect local-mgmtIn this case the subordinate is A and we are going to promote him to primary. You will loose access to the console. Wait a few minutes before re-connect.
UCS# cluster lead a
Repeat the process for the other interconnect fabric and active the firmware on the formerly primary FI, which is now the subordinate by repeating the above. Verify it’s running the proper new version and your network and storage redundancy are working properly.
At the end ,validate all the firmware versions and acknowledge the new fabric ports as server if new chassis are connected. Validate network connectivity and storage multipathing.
The final step is to Turn back On CallHome. As good practice take a backup of the final configuration.
That's how you do a manual Cisco UCS Manager Firmware Upgrade.
Note: Thanks to Bruno Monteiro for some of the screenshots and guidelines.
See you next time!
No comments:
Post a Comment