We are going to upgrade a handful of ESX 4.1 hosts to 5.5 or 6, if the hardware supports. It is going to be a two step process, the first of which will be upgrade to 5.1 using the ISO. We need console access to do this. These are X series IBM servers. A part of our pre-upgrade checklist is to ensure we have IMM access.
We have one server where we cannot reach the IMM remotely. We have seen this before and managed to solve it by resetting the IMM via LanOverUsb or cold restart of the server- graceful shutdown of ESX(i), followed by unplug, plug both power supplies but not this time.
Our network guy validated the switch configuration. We re-seated the cable. The MGMT NIC had green light, was blinking and our network guy saw they were exchanging packets. We even ran a cable between the IMM port and a laptop.
As the host is classic ESX 4.1, we used the asu utility from the host to check the configuration. The IP, Mask and GW are correct. The autoneg and duplexing too matches the switch port.
[firstname.lastname@example.org ~]# asu show IBM Advanced Settings Utility version 9.21.78C Licensed Materials - Property of IBM (C) Copyright IBM Corp. 2007-2012 All Rights Reserved Unable to locate a script required to set up LAN-over-USB device, tried location cdc_interface.sh Successfully discovered the IMM via SLP. Discovered IMM at IP address 169.254.95.118 Connected to IMM at IP address 169.254.95.118 IMM.SSH_SERVER_KEY=Installed IMM.SSL_HTTPS_SERVER_CERT=Private Key and Cert/CSR not available. IMM.SSL_HTTPS_SERVER_CSR=Private Key and Cert/CSR not available. IMM.SSL_LDAP_CLIENT_CERT=Private Key and Cert/CSR not available. IMM.SSL_LDAP_CLIENT_CSR=Private Key and Cert/CSR not available. IMM.SSL_SERVER_DIRECTOR_CERT=Private Key and Cert/CSR not available. ...
At this point, the only sensible thing before calling in support sounds like reset to factory default. The host is housed in a small facility that does not have kvm or crash kart. I thought of taking my desk monitor and keyboard but this isn’t my home lab. I am 99.9% certain I will not break anything but I choose to be extra cautious.
There’s got to be another way to reset the IMM to default settings. The asu utility has resetrsa which does not work on IMM. Then I realize I can ssh to the IMM using the LanOverUsb interface from the Service Console.
Once inside the IMM cli, I used clearcfg command to reset the IMM to factory default configuration.
Next I restored backup configuration using the kcs interface.
asu restore backup_file --kcs
I saved the IMM configuration before the reset using
asu save backup_file --group all
I expected the IMM to start working right after the restore but it did not. As a last resort, I rebooted it.
asu rebootimm --kcs
Following the reboot, I was able to ping the IMM IP address. The configuration backup and restore did not include the login credentials. Therefore, I ssh’d to the IMM as USERID/PASSW0RD, factory default credentials and created a login profile.
users -2 -n root -p password -a super
We are back on track!