8.30.2011

Serial over LAN Configuration on UCS Blades

Serial over LAN (SoL) is a mechanism that enables the input and output of the serial port of a managed system to be redirected via an SSH session over IP. SoL provides a means of reaching the host console via CIMC.

The SoL session will display line-oriented information such as boot messages, and character-oriented screen menus such as BIOS setup menus. If the server boots an operating system or application with a bitmap-oriented display, such as Windows, the SoL session will no longer display. If the server boots a command-line-oriented operating system (OS), such as Linux, you may need to perform additional configuration of the OS in order to properly display in an SoL session.
In the SoL session, your keystrokes are transmitted to the console except for the function key F2. To send an F2 to the console, press the Escape key, then press 2.


In order to configure SoL on your UCS blades, you would first need to create a Serial over LAN policy that enables the feature and provides you with the ability to set the baud rate:



Once this is complete, the policy can be applied to your blade's service profile.  Applying this policy to your service profile will not require a blade reboot.



It is also recommended that an IPMI policy be assigned to the service profile as well.  



Additionally, the BIOS of the blade needed to be configured in order to enable the serial interface for console redirection.  This canbe done via a BIOS policy, or directly in the BIOS of the blade by going to Server Management -> Console Redirection.  Ensure that the baud rate matches what was set inside of your SoL policy.






At this point, you can SSH into the IP of the KVM for the UCS blade.  This is found under the Admin tab -> Communication Services -> Management IP Pool.  The username and password that is used to log into the session should be the same as that, which was created previously in the IPMI profile.  You should now be able to log into the console as a user other than root.  Note: If you launch the SSH to CIMC for SoL, you will receive an error message if you are on a Mac:



In Windows XP, I received this same error message, but in order to resolve it, needed to associate my SSH client in UCSM under Options -> External Applications:




After this, I was able to successfully have an SSH session launches to the KVM IP of the specified blade:




Note: I made the following modifications to my RedHat host.  
Added the following to /etc/inittab:
       se:2345:respawn:/sbin/agetty 9600 ttyS0
Added the following to /etc/securetty:
       ttyS0

UCS Supported/Recommended Firmware

The following table shows the supported/recommended firmware for UCS blades as of today, August 30th, 2011:





More information on supported versions for adapters and other components can be found here: http://www.cisco.com/en/US/docs/unified_computing/ucs/release/notes/OL_24086.html

Re-number UCS Chassis

A UCS chassis can be renumbered when it is recommissioned into UCSM.  This procedure can be done from the CLI or from the GUI.  Note: This feature is available in 1.3(1t) and later, and 1.4(2x) and later.

From the CLI:
recommission chassis vendor-name model-name serial-num [chassis-num]   

From the GUI:
Step 1   In the Navigation pane, click the Equipment tab.
Step 2   On the Equipment tab, expand Equipment > Chassis.
Step 3   Verify that the Chassis node does not include the following:
    •    The chassis you want to renumber
    •    A chassis with the number you want to use
If either of these chassis are listed in the Chassis node, decommission those chassis. You must wait until the decommission FSM is complete and the chassis are not listed in the Chassis node before continuing. This might take several minutes.
Step 4   On the Equipment tab, click the Chassis node.
Step 5   In the Work pane, click the Decommissioned tab.
Step 6   For the chassis that you want to renumber, do the following:
    a.    Right-click the chassis and choose Re-commission Chassis.
    b.    In the Chassis ID field of the Re-commission Chassis dialog box, type or use the arrows to choose the ID that you want to assign to the chassis
    c.    Click OK
Step 7   If Cisco UCS Manager GUI displays a confirmation dialog box, click Yes.



Sources:
http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/cli/config/guide/1.4/UCSM_CLI_Configuration_Guide_1_4_chapter36.html#task_D9A1BBEB813A40DEB845D1C2163EAF3A

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/1.4/UCSM_GUI_Configuration_Guide_1_4_chapter38.html#task_7CE3D4DF1CEF42A2A7BE144FEDF8DAB5

Uplink Port Needed for Appliance Port

For reference, posting a link from the Unified Computing Blog regarding the need for an uplink if you are going to be using appliance ports: http://www.unifiedcomputingblog.com/?p=252

8.22.2011

Set UCS SPAN Destination Interface Speed



If the sniffer connected to the traffic monitoring port on the Fabric Interconnect (port 1-8 on a 6120) is running at 1Gbps, the Status of the Port will be failed with the additional information as “SFP Validation Failed” even with a valid SFP in the port. The speed of the port needs to be set at 1 Gbps for it to be in UP state as it does not auto-negotiate. The GUI option for changing speed of a monitoring port to 1 Gbps is not present (Defect ID: CSCti86217) so as a workaround, set the as an Uplink port, set the admin speed to 1 Gbps (under the LAN tab) and then unconfigure the port. The admin speed sticks as 1 Gbps and then the port can be set as a Monitor port and will come up at 1Gbps. 






In my experience, setting the port's admin speed to 1 Gbps from 'Show Interface' in the equipment tab is not enough to force the port to preserve the 1 Gbps speed.

8.11.2011

UCS Database Browser - Visore

The database browser, Visore, can be viewed by browsing to the following link: http(s)://<ip of FI>/visore.html

Nothing can be changed from Visore, but different aspects of the database can be viewed for configuration verification.

Blinking Cursor in VM of UC on UCS Apps

Currently, most OVA templates will try to boot off the hard drive which does not have any functional OS.  A blinking cursor is seen on the machine console.

You can work around this issue by manually configuring the boot order in the virtual machine's BIOS.  You can force the VM to boot into the BIOS on start-up in the following location within the VM's settings:


The defect, CSCtn40643, addresses this issue.

Black KVM Screen on C-Series Server

After an upgrade from 1.2(1b) to 1.3(2d), some C210 servers may experience a black KVM screen after the memory test/Cisco splash screen.  POST of PCI devices/onboard NICs will not be viewable.  You can break into these options by connecting up a physical monitor to the server.

In order to workaround this issue, the PC's Java cache must be cleared.
Steps to clear the Java cache can be found at the following link: http://www.java.com/en/download/help/plugin_cache.xml

The defect that addresses this issue is: CSCtl93037 - vKVM is blank after updating the CIMC FW from AP to BP.

Temporarily Add System VLAN to VEM

You can temporarily add system vlans with vemset if someone forgot and rebooted an ESXi host:
vemset system-vlan <vlan-id> ltl <ltl-num>


Example: If someone forgot to add management as a system vlan on their vEthernet port profile, and is now locked out of the host.

Check which vmk port you want:
~ # esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type   
vmk0       130                 IPv4      14.17.125.14                            255.255.255.0   14.17.125.255   00:50:56:7c:24:a9 1500    65535     true    STATIC 

Find the ltl:
~ # vemcmd show port
  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port
   17     Eth4/1     UP   UP    F/B*    305     0  vmnic0
   18     Eth4/2     UP   UP    F/B*    305     0  vmnic1
   49      Veth3     UP   UP    FWD       0     0  vmk0

Find the vlan:
~ # vemcmd show port vlans
                        Native  VLAN   Allowed
  LTL   VSM Port  Mode  VLAN    State  Vlans
   17     Eth4/1   T        1   FWD    64,125,129,225,325
   18     Eth4/2   T        1   FWD    64,125,129,225,325
   49      Veth3   A      125   FWD    125

Make it a system VLAN:
~ # vemset system-vlan 125 ltl 49

8.03.2011

How to Analyze C-Series Tech-Support Bundle

1. Keep the name of the techsupport as <filename>.tar.gz

2. The techsupport bundle contains the following folders
            a)mnt
            b)tmp
            c)var
            d)obfl
            e)usr

3. OBFL (On Board Failure Logging) contains the logs of the CIMC , IPMI, SEL etc. It's circular and writes itself when the max number of events are reached.
            This folder should contains obfl files of 76KB size. New file is created once 76KB size is reached. The time stamp will tell the latest created file.
            Many types of failures should be looked into these files.
            IOCTL failures, network packet drop, Tulip driver issue, Process or memory issue.
            Use these keywords mainly error, fail, fault, disturbingly, critical, to search for such instances.

4. usr folder contains another folder IPMI. The IPMI folder contains the SDR file NVRAM_SDR00. This File is used decode the SEL.

5. tmp folder contains two files: one with same name as the bundle and another with filename_bios_post_results
            The first file will give most of the information about the system, firmware etc
            Get the BMC and BIOS versions. Board serial number and other information, CPU information, DIMM population and type information, card and adapter details, boot order information (SM BIOS TABLE) , Network settings, Memory and CPU consumption information.
            It also contains ifconfig information that will indicate what MAC addresses are active and what are the ips picked. The sensor listing with list the reading, fan speed and indicate in case any of these   sensors are in Critical or NR state. It also gives LED information.
            The BIOS file indicates the BIOS post completion information. It requires deeper knowledge of these codes to interpret this information.

6. var - The var folder contains multiple folders and files. Core folder contains any core dump file generated. It is a very complex process to analyze these files.
            IPMI folder contains the sensor alarm information and will tell what sensors are in alarmed state.
            Log folder contains log files like critical, peci, messages and avct_server. These can be opened with notepad.
            The avct_server file contains the VKVM related logs. search for fail, error, fault, to get the error messages logged related to VKVM. Messages and Critical files contains SEL, IPMI and CIMC logs and will give fairly good amount of error messages here.
            nuova folder contains information related to Rank margin test and memory status.
            The service folder contains the process status. and run contains information on pid.

7. mnt - This folder contains SEL file which can be used to decode SEL in association with SDR file.avct_ems_cfg contains some of the information on persistent data.cert contains certificate information.
            Links for sensor information