C-Series Integration with UCSM - Dual Adapters

When integrating a C-series with UCSM, it is possible under some circumstances to see dual adapter cards, even though the rack mount server only has 1 adapter card installed in it.  A closer look at the adapter card will reveal that one of them has a bogus serial number.  You will also notice that a service profile will not associate to the rack mount server.  It will be stuck in the following state in the FSM of the server:

The defect that identifies this issue is: CSCtl41022.  This issue is resolved in 1.2(2i) and above. The C-series server will need to be decommissioned from UCSM in order to upgrade the firmware on it.

On a side note, the following guide outlines integrating a C-Series server with UCSM:
http://www.cisco.com/en/US/partner/docs/unified_computing/ucs/c/hw/C250M1/install/ucsm-integration.html

Set UCS SPAN Destination Interface Speed

If the sniffer connected to the traffic monitoring port on the Fabric Interconnect (port 1-8 on a 6120) is running at 1Gbps, the Status of the Port will be failed with the additional information as “SFP Validation Failed” even with a valid SFP in the port. The speed of the port needs to be set at 1 Gbps for it to be in UP state as it does not auto-negotiate. The GUI option for changing speed of a monitoring port to 1 Gbps is not present (Defect ID: CSCti86217) so as a workaround, set the as an Uplink port, set the admin speed to 1 Gbps (under the LAN tab) and then unconfigure the port. The admin speed sticks as 1 Gbps and then the port can be set as a Monitor port and will come up at 1Gbps. 

In my experience, setting the port's admin speed to 1 Gbps from 'Show Interface' in the equipment tab is not enough to force the port to preserve the 1 Gbps speed.

Blinking Cursor in VM of UC on UCS Apps

Currently, most OVA templates will try to boot off the hard drive which does not have any functional OS.  A blinking cursor is seen on the machine console.

You can work around this issue by manually configuring the boot order in the virtual machine's BIOS.  You can force the VM to boot into the BIOS on start-up in the following location within the VM's settings:

The defect, CSCtn40643, addresses this issue.

How to Analyze C-Series Tech-Support Bundle

1. Keep the name of the techsupport as <filename>.tar.gz

2. The techsupport bundle contains the following folders
            a)mnt
            b)tmp
            c)var
            d)obfl
            e)usr

3. OBFL (On Board Failure Logging) contains the logs of the CIMC , IPMI, SEL etc. It's circular and writes itself when the max number of events are reached.
            This folder should contains obfl files of 76KB size. New file is created once 76KB size is reached. The time stamp will tell the latest created file.
            Many types of failures should be looked into these files.
            IOCTL failures, network packet drop, Tulip driver issue, Process or memory issue.
            Use these keywords mainly error, fail, fault, disturbingly, critical, to search for such instances.

4. usr folder contains another folder IPMI. The IPMI folder contains the SDR file NVRAM_SDR00. This File is used decode the SEL.

5. tmp folder contains two files: one with same name as the bundle and another with filename_bios_post_results
            The first file will give most of the information about the system, firmware etc
            Get the BMC and BIOS versions. Board serial number and other information, CPU information, DIMM population and type information, card and adapter details, boot order information (SM BIOS TABLE) , Network settings, Memory and CPU consumption information.
            It also contains ifconfig information that will indicate what MAC addresses are active and what are the ips picked. The sensor listing with list the reading, fan speed and indicate in case any of these   sensors are in Critical or NR state. It also gives LED information.
            The BIOS file indicates the BIOS post completion information. It requires deeper knowledge of these codes to interpret this information.

6. var - The var folder contains multiple folders and files. Core folder contains any core dump file generated. It is a very complex process to analyze these files.
            IPMI folder contains the sensor alarm information and will tell what sensors are in alarmed state.
            Log folder contains log files like critical, peci, messages and avct_server. These can be opened with notepad.
            The avct_server file contains the VKVM related logs. search for fail, error, fault, to get the error messages logged related to VKVM. Messages and Critical files contains SEL, IPMI and CIMC logs and will give fairly good amount of error messages here.
            nuova folder contains information related to Rank margin test and memory status.
            The service folder contains the process status. and run contains information on pid.

7. mnt - This folder contains SEL file which can be used to decode SEL in association with SDR file.avct_ems_cfg contains some of the information on persistent data.cert contains certificate information.
            Links for sensor information

C-Series Tech-Support from CLI

Tech-support data can be exported from the GUI under the Admin tab -> Utilities, but it can also be collected via the CLI.

The following commands can be used from an SSH session into the server's CIMC to configure and start the collection of tech-support data:

scope cimc
scope tech-support
set tftp-ip x.x.x.x
set path supportfile
commit
start

You can view progress of the tech-support collection/upload by using the 'show detail' command repeatedly.

Reference: http://www.cisco.com/en/US/products/ps10493/products_configuration_example09186a0080b10d5e.shtml

How to Read UCS B-Series Tech-Support Files - UCSM Detail

I just posted a document on the Cisco Support Community Forums detailing the output of the .tar file generated after running 'show tech-support ucsm detail'.  Hopefully this will give you the power to dig further into your UCS environment when an issue arises.

 The document can be found at the following link:
https://supportforums.cisco.com/docs/DOC-17071

Low Voltage CMOS Battery

The following procedure can be used in order to confirm a voltage problem alert being reported on a B-series server's CMOS battery.

The fault that will be sent is the following:
 code="F0424"
 descr="Possible loss of CMOS settings: CMOS battery voltage on server 1/1 is lower-critical"

In order to further troubleshoot this issue, the following commands can be used in order to collect
information about the CMOS battery's sensor:

 connect cimc <chassis>/<server>
 sensors

The sensor you should look for is: P3V_BAT_SCALED.
If the sensor reading is below 3.0V, a bad battery is typically the suspect.

The replacement part ID for the CMOS battery is: N20-MBLIBATT=

Circuit Trace in UCS

You may be wondering how does a packet travel from my adapter through the UCS system to the upstream switch?  The following commands can help you trace out the data path of a packet.

'show service-profile circuit server x/y' will show the VIFs associated to a particular blade:
Server: 1/1
    Fabric ID: A
        VIF        vNIC       Link State Overall Status Admin Pin  Oper Pin   Transport
        ---------- ---------- ---------- -------------- ---------- ---------- ---------
                41            Unknown    Unknown        0/0        0/0        Unknown
              9158            Up         Active         0/0        0/0        Ether
               964 eth0       Up         Active         0/0        0/3        Ether
               966 fc0        Up         Active         0/0        0/0        Fc
    Fabric ID: B
        VIF        vNIC       Link State Overall Status Admin Pin  Oper Pin   Transport
        ---------- ---------- ---------- -------------- ---------- ---------- ---------
                42            Unknown    Unknown        0/0        0/0        Unknown
              9159            Up         Active         0/0        0/0        Ether
               965 eth1       Up         Active         0/0        0/3        Ether
               967 fc1        Up         Active         0/0        0/0        Fc

Using these numbers, we can search for the active border interfaces on the system in order to determine which uplink an interface is using to reach the outside world:

'connect nxos'
'show pinning border-interfaces active'

--------------------+---------+----------------------------------------
Border Interface     Status    SIFs                                   
--------------------+---------+----------------------------------------
Po3                  Active    veth961 veth965 veth973 veth977 veth981
                               veth985 veth989 veth993 veth997  
                               veth1001 veth1005 veth1009 veth1013
                               veth1017 veth1021 veth1025 veth1029
                               veth1033 veth1037 veth1041 veth1045
                               veth1049 veth1053 veth1057 veth1061
                               veth1065 veth1069 veth1073 veth1081
                               veth1085                         

Total Interfaces : 1

We can see from the output that all traffic for virtual interfaces connected to FI B are being sent out port-channel 3.  We can take this even further and see which uplink in the port-channel has been hashed to carry the uplink traffic for a particular source.

The following command shows the hashing algorithm that will determine what uplink is used:

'show port-channel load-balance'
Port Channel Load-Balancing Configuration:
System: source-dest-ip

Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-dest-mac
IP: source-dest-ip source-dest-mac

With this in mind, the following command can be used to determine the uplink.  The source IP and destination MAC addresses are needed to complete this command:
show port-channel load-balance forwarding-path interface port-channel 3 <src ip> <dst-ip> <src-mac> <dst-mac>

In a later post, I will document how to determine the IOM backplane ports and fabric ports that a particular server is using to reach the upstream network.