Cannot connect to CIMC

We cannot reach the CIMC of a C series Cisco server. It was working perfectly before. We couldn’t ping it. Soft reboot, physical power cycle did not help. There are no errors on the switch ports. The ports are the server and switch do not have link. A colleague was able to reach the CIMC by directly connecting his laptop to the CIMC NIC port. This proves the CIMC is working fine. He set the CIMC NIC to auto negotiate which fixed the issue. We are still at lost why it worked before as there has been no change in the switch.

Auto-negotiate is our standard configuration. This server being one of the first we deployed, we might have missed this step.

This is our standard CIMC network configuration.

scope cimc
scope network
set mode dedicated
set redundancy none
set auto-negotiate yes

How to find the serial number and server model in Linux and ESXi

RHEL 6 on IBM X Series

[root@ibm ~]#  dmidecode | grep -A4  'System Information'
System Information
        Manufacturer: IBM
        Product Name: System x3650 M4 : -[7915AC1]-
        Version: 0B
        Serial Number: XXXXXX

Continue reading

Megaraid, media errors

Application(Hadoop) logs I/O errors:

2016-02-15 02:48:04,911 WARN org.apache.hadoop.hdfs.server.common.Storage: Failed to add storage for block pool: \
BP-   2136893094-Server_IP-1400619662809 : BlockPoolSliceStorage.recoverTransitionRead: attempt to load an used \
block  storage: /path
ExitCodeException exitCode=1: du: cannot access `/path': Input/output error
du: cannot access `/path': Input/output error
du: cannot access `/path': Input/output error

SCSI reports “Medium Error” in /var/log/messages:

Feb 15 02:47:04 hostame kernel: EXT4-fs error (device sdi): __ext4_get_inode_loc: unable to read inode \
block -  inode=50331696, block=201326626
Feb 15 02:47:33 hostame kernel: sd 0:2:8:0: [sdi] Unhandled sense code
Feb 15 02:47:33 hostame kernel: sd 0:2:8:0: [sdi] Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE
Feb 15 02:47:33 hostame kernel: sd 0:2:8:0: [sdi] Sense Key : Medium Error [current]
Feb 15 02:47:33 hostame kernel: sd 0:2:8:0: [sdi] Add. Sense: No additional sense information
Feb 15 02:47:33 hostame kernel: sd 0:2:8:0: [sdi] CDB: Read(10): 28 00 60 00 01 10 00 00 08 00

Continue reading

Reading plan for 2016

This is more like a wish list than real plan but I got to write it down somewhere to increase the chance to really reading them.

  • Mastering VMware vSphere 6
  • Cisco Unified Computing System (UCS) (Data Center): A Complete Reference Guide to the Cisco Data Center Virtualization Server Architecture (Networking Technology)

  • Networking for VMware Administrators

  • Red Hat RHCSA/RHCE 7 Cert Guide: Red Hat Enterprise Linux 7 (EX200 and EX300)

  • The Phoenix Project: A Novel About IT, DevOps, and Helping Your Business Win

  • The Visible Ops Handbook: Implementing ITIL in 4 Practical and Auditable Steps

* Data Center Virtualization Fundamentals by Gustavo A. A. Santana

vNIC, vmnic naming mismatch in UCS and ESXi

I cannot get an ESXi host management network up in UCS B series blade. My friend has trouble kickstarting Linux on a different B series blade.

Check the vNIC ordering in UCS. The interface ordering in the OS usually match the vNIC ordering in UCS but not always. For instance it should be
vmnic0 = vNIC0
eth0 = vNIC0

Depending on how it was setup this is not always the case. Hence match the mac address in OS and UCS to verify. It saved my day.