9.27.2011

UCS KVM .jnlp Files Not Opening Properly After Java Upgrade on Mac

Was Java recently upgraded on your Mac and now the KVM .jnlp files lost their association with Java Web Start?  Don't worry, it happened to me too.  In order to resolve this issue, you will just need to reassociate the files with the Java Web Start.app file.  I used the one located in /System/Library/CoreServices.



Make sure to select the "Always Open With" checkbox

9.22.2011

Repeated Characters When Typing in VM Console

Sometimes when typing in a VM's console, you may experience repeats of the same character.  This could make it near impossible to log into a VM through its console if there is an issue with it's networking.

The workaround is to configure the typematicMinDelay parameter to a value of 200000.
This is documented in the following KB article:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=196

Another way to set this display is by right clicking on the VM and selecting Edit Settings.  From here you can select General in the Options tab, and then click on the Configuration Parameters button.  (Note: This must be performed when the VM is powered off).  In the new window that pops up, you can choose to add a new row with the typematicMinDelay parameter and its associated value:


9.16.2011

Manual Install of 1000v VEM on ESXi 5.0

The command to manually install the VEM file on ESXi 5.0 is a bit different than on previous versions.
In order to manually install the VEM, you will need to copy the correct .vib file over to the host.  Currently the version number is: 4.2.1.1.4.1.0-3.0.4.  I have not figured out why yet, but the file needs to be installed from the /var/log/vmware directory!  When trying to install it from another directory, such as /tmp, I received an error message.

The command to perform the install is: esxcli software vib install -v cross_cisco-vem-v131-4.2.1.1.4.1.0-3.0.4.vib --no-sig-check



9.15.2011

F11 on a Mac for ESX Installs

To save everyone time out there, I am pretty sure that Fn+Ctrl+option+F11 in Mac OS Snow Leopard would pass F11 through to the ESX installation menu.  This does not appear to be the case in Lion.  After hours of struggling to find the correct key sequence this evening, your best bet is to go into the System Preferences and do the following:

1. Go into the Mission Control settings and set 'Show Desktop' to be any other F- key besides F11.
2. Go into the keyboard settings and check the box that says 'Use all F1, F2, etc. keys as standard function keys.

After this, you will be able to just press only the F11 key in order to pass it through to the ESX install.

If you are running VMware Fusion and have a Windows virtual machine on your Mac, there is also an option at the top, Virtual Machine -> Send Key -> F11, to send the appropriate function key to the installation menu.

9.12.2011

ESXi Caches Old MAC address from UCS Service Profile

Per CSCte08176:

VMware's ESXi hypervisor caches the MAC address of it's physical NIC after it put's it in promiscuous mode.  If the administrator subsequently changes the mac-address value defined in the associated service profile for that blade, the NIC will have a new MAC address, while the older MAC is not removed or updated from the ESXi hypervisor.  This potentially leads to a MAC address conflict if the initial MAC address is re-assigned to a different blade (via service profile association).

One way to confirm this problem is by collecting "esxcfg-nics -l" and "esxcfg-vmknic -l", and comparing the mac-address output in both. The first one should correctly display the mac-address assigned in the service-profile while the second will display the cached mac-address.

VMware KB article: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1031111

As a workaround, administrators can do a "esxcfg-vmknic -d" to delete the virtual nic which is using the cached mac address. On recreation of that virtual nic it will have a VMW OUI MAC address.

9.02.2011

C-Series Integration with UCSM - Dual Adapters

When integrating a C-series with UCSM, it is possible under some circumstances to see dual adapter cards, even though the rack mount server only has 1 adapter card installed in it.  A closer look at the adapter card will reveal that one of them has a bogus serial number.  You will also notice that a service profile will not associate to the rack mount server.  It will be stuck in the following state in the FSM of the server:



The defect that identifies this issue is: CSCtl41022.  This issue is resolved in 1.2(2i) and above. The C-series server will need to be decommissioned from UCSM in order to upgrade the firmware on it.

On a side note, the following guide outlines integrating a C-Series server with UCSM:
http://www.cisco.com/en/US/partner/docs/unified_computing/ucs/c/hw/C250M1/install/ucsm-integration.html

SFTP Files off of FI

At first glance, it seems that in 1.4(3m), SFTP file transfers don't ask for a password from the FI to a remote server, but it does... just not after it asks for your username.  SFTP transfers also now provide a sliding timeline so you can see the percentage of file transfer that has been completed.

UCS-B(local-mgmt)# copy workspace:/techsupport/20110828134224_UCS_UCSM.tar sftp:
Server name/IP: 14.0.25.91
Remote username: jen
Remote filepath: /ucsm.tar
jen@14.0.25.91's password:
Hello, I'm freeFTPd 1.0Connected to 14.0.25.91.
sftp> put  /workspace/techsupport/20110828134224_UCS_UCSM.tar  "/ucsm.tar"
Uploading /workspace/techsupport/20110828134224_UCS_UCSM.tar to /ucsm.tar
/workspace/techsupport/20110828134224_UCS 100%   14MB   1.4MB/s   00:10   
sftp> quit
UCS-B(local-mgmt)#

8.30.2011

Serial over LAN Configuration on UCS Blades

Serial over LAN (SoL) is a mechanism that enables the input and output of the serial port of a managed system to be redirected via an SSH session over IP. SoL provides a means of reaching the host console via CIMC.

The SoL session will display line-oriented information such as boot messages, and character-oriented screen menus such as BIOS setup menus. If the server boots an operating system or application with a bitmap-oriented display, such as Windows, the SoL session will no longer display. If the server boots a command-line-oriented operating system (OS), such as Linux, you may need to perform additional configuration of the OS in order to properly display in an SoL session.
In the SoL session, your keystrokes are transmitted to the console except for the function key F2. To send an F2 to the console, press the Escape key, then press 2.


In order to configure SoL on your UCS blades, you would first need to create a Serial over LAN policy that enables the feature and provides you with the ability to set the baud rate:



Once this is complete, the policy can be applied to your blade's service profile.  Applying this policy to your service profile will not require a blade reboot.



It is also recommended that an IPMI policy be assigned to the service profile as well.  



Additionally, the BIOS of the blade needed to be configured in order to enable the serial interface for console redirection.  This canbe done via a BIOS policy, or directly in the BIOS of the blade by going to Server Management -> Console Redirection.  Ensure that the baud rate matches what was set inside of your SoL policy.






At this point, you can SSH into the IP of the KVM for the UCS blade.  This is found under the Admin tab -> Communication Services -> Management IP Pool.  The username and password that is used to log into the session should be the same as that, which was created previously in the IPMI profile.  You should now be able to log into the console as a user other than root.  Note: If you launch the SSH to CIMC for SoL, you will receive an error message if you are on a Mac:



In Windows XP, I received this same error message, but in order to resolve it, needed to associate my SSH client in UCSM under Options -> External Applications:




After this, I was able to successfully have an SSH session launches to the KVM IP of the specified blade:




Note: I made the following modifications to my RedHat host.  
Added the following to /etc/inittab:
       se:2345:respawn:/sbin/agetty 9600 ttyS0
Added the following to /etc/securetty:
       ttyS0

UCS Supported/Recommended Firmware

The following table shows the supported/recommended firmware for UCS blades as of today, August 30th, 2011:





More information on supported versions for adapters and other components can be found here: http://www.cisco.com/en/US/docs/unified_computing/ucs/release/notes/OL_24086.html

Re-number UCS Chassis

A UCS chassis can be renumbered when it is recommissioned into UCSM.  This procedure can be done from the CLI or from the GUI.  Note: This feature is available in 1.3(1t) and later, and 1.4(2x) and later.

From the CLI:
recommission chassis vendor-name model-name serial-num [chassis-num]   

From the GUI:
Step 1   In the Navigation pane, click the Equipment tab.
Step 2   On the Equipment tab, expand Equipment > Chassis.
Step 3   Verify that the Chassis node does not include the following:
    •    The chassis you want to renumber
    •    A chassis with the number you want to use
If either of these chassis are listed in the Chassis node, decommission those chassis. You must wait until the decommission FSM is complete and the chassis are not listed in the Chassis node before continuing. This might take several minutes.
Step 4   On the Equipment tab, click the Chassis node.
Step 5   In the Work pane, click the Decommissioned tab.
Step 6   For the chassis that you want to renumber, do the following:
    a.    Right-click the chassis and choose Re-commission Chassis.
    b.    In the Chassis ID field of the Re-commission Chassis dialog box, type or use the arrows to choose the ID that you want to assign to the chassis
    c.    Click OK
Step 7   If Cisco UCS Manager GUI displays a confirmation dialog box, click Yes.



Sources:
http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/cli/config/guide/1.4/UCSM_CLI_Configuration_Guide_1_4_chapter36.html#task_D9A1BBEB813A40DEB845D1C2163EAF3A

http://www.cisco.com/en/US/docs/unified_computing/ucs/sw/gui/config/guide/1.4/UCSM_GUI_Configuration_Guide_1_4_chapter38.html#task_7CE3D4DF1CEF42A2A7BE144FEDF8DAB5

Uplink Port Needed for Appliance Port

For reference, posting a link from the Unified Computing Blog regarding the need for an uplink if you are going to be using appliance ports: http://www.unifiedcomputingblog.com/?p=252

8.22.2011

Set UCS SPAN Destination Interface Speed



If the sniffer connected to the traffic monitoring port on the Fabric Interconnect (port 1-8 on a 6120) is running at 1Gbps, the Status of the Port will be failed with the additional information as “SFP Validation Failed” even with a valid SFP in the port. The speed of the port needs to be set at 1 Gbps for it to be in UP state as it does not auto-negotiate. The GUI option for changing speed of a monitoring port to 1 Gbps is not present (Defect ID: CSCti86217) so as a workaround, set the as an Uplink port, set the admin speed to 1 Gbps (under the LAN tab) and then unconfigure the port. The admin speed sticks as 1 Gbps and then the port can be set as a Monitor port and will come up at 1Gbps. 






In my experience, setting the port's admin speed to 1 Gbps from 'Show Interface' in the equipment tab is not enough to force the port to preserve the 1 Gbps speed.

8.11.2011

UCS Database Browser - Visore

The database browser, Visore, can be viewed by browsing to the following link: http(s)://<ip of FI>/visore.html

Nothing can be changed from Visore, but different aspects of the database can be viewed for configuration verification.

Blinking Cursor in VM of UC on UCS Apps

Currently, most OVA templates will try to boot off the hard drive which does not have any functional OS.  A blinking cursor is seen on the machine console.

You can work around this issue by manually configuring the boot order in the virtual machine's BIOS.  You can force the VM to boot into the BIOS on start-up in the following location within the VM's settings:


The defect, CSCtn40643, addresses this issue.

Black KVM Screen on C-Series Server

After an upgrade from 1.2(1b) to 1.3(2d), some C210 servers may experience a black KVM screen after the memory test/Cisco splash screen.  POST of PCI devices/onboard NICs will not be viewable.  You can break into these options by connecting up a physical monitor to the server.

In order to workaround this issue, the PC's Java cache must be cleared.
Steps to clear the Java cache can be found at the following link: http://www.java.com/en/download/help/plugin_cache.xml

The defect that addresses this issue is: CSCtl93037 - vKVM is blank after updating the CIMC FW from AP to BP.

Temporarily Add System VLAN to VEM

You can temporarily add system vlans with vemset if someone forgot and rebooted an ESXi host:
vemset system-vlan <vlan-id> ltl <ltl-num>


Example: If someone forgot to add management as a system vlan on their vEthernet port profile, and is now locked out of the host.

Check which vmk port you want:
~ # esxcfg-vmknic -l
Interface  Port Group/DVPort   IP Family IP Address                              Netmask         Broadcast       MAC Address       MTU     TSO MSS   Enabled Type   
vmk0       130                 IPv4      14.17.125.14                            255.255.255.0   14.17.125.255   00:50:56:7c:24:a9 1500    65535     true    STATIC 

Find the ltl:
~ # vemcmd show port
  LTL   VSM Port  Admin Link  State  PC-LTL  SGID  Vem Port
   17     Eth4/1     UP   UP    F/B*    305     0  vmnic0
   18     Eth4/2     UP   UP    F/B*    305     0  vmnic1
   49      Veth3     UP   UP    FWD       0     0  vmk0

Find the vlan:
~ # vemcmd show port vlans
                        Native  VLAN   Allowed
  LTL   VSM Port  Mode  VLAN    State  Vlans
   17     Eth4/1   T        1   FWD    64,125,129,225,325
   18     Eth4/2   T        1   FWD    64,125,129,225,325
   49      Veth3   A      125   FWD    125

Make it a system VLAN:
~ # vemset system-vlan 125 ltl 49

8.03.2011

How to Analyze C-Series Tech-Support Bundle

1. Keep the name of the techsupport as <filename>.tar.gz

2. The techsupport bundle contains the following folders
            a)mnt
            b)tmp
            c)var
            d)obfl
            e)usr

3. OBFL (On Board Failure Logging) contains the logs of the CIMC , IPMI, SEL etc. It's circular and writes itself when the max number of events are reached.
            This folder should contains obfl files of 76KB size. New file is created once 76KB size is reached. The time stamp will tell the latest created file.
            Many types of failures should be looked into these files.
            IOCTL failures, network packet drop, Tulip driver issue, Process or memory issue.
            Use these keywords mainly error, fail, fault, disturbingly, critical, to search for such instances.

4. usr folder contains another folder IPMI. The IPMI folder contains the SDR file NVRAM_SDR00. This File is used decode the SEL.

5. tmp folder contains two files: one with same name as the bundle and another with filename_bios_post_results
            The first file will give most of the information about the system, firmware etc
            Get the BMC and BIOS versions. Board serial number and other information, CPU information, DIMM population and type information, card and adapter details, boot order information (SM BIOS TABLE) , Network settings, Memory and CPU consumption information.
            It also contains ifconfig information that will indicate what MAC addresses are active and what are the ips picked. The sensor listing with list the reading, fan speed and indicate in case any of these   sensors are in Critical or NR state. It also gives LED information.
            The BIOS file indicates the BIOS post completion information. It requires deeper knowledge of these codes to interpret this information.

6. var - The var folder contains multiple folders and files. Core folder contains any core dump file generated. It is a very complex process to analyze these files.
            IPMI folder contains the sensor alarm information and will tell what sensors are in alarmed state.
            Log folder contains log files like critical, peci, messages and avct_server. These can be opened with notepad.
            The avct_server file contains the VKVM related logs. search for fail, error, fault, to get the error messages logged related to VKVM. Messages and Critical files contains SEL, IPMI and CIMC logs and will give fairly good amount of error messages here.
            nuova folder contains information related to Rank margin test and memory status.
            The service folder contains the process status. and run contains information on pid.

7. mnt - This folder contains SEL file which can be used to decode SEL in association with SDR file.avct_ems_cfg contains some of the information on persistent data.cert contains certificate information.
            Links for sensor information

7.27.2011

C-Series Tech-Support from CLI

Tech-support data can be exported from the GUI under the Admin tab -> Utilities, but it can also be collected via the CLI.

The following commands can be used from an SSH session into the server's CIMC to configure and start the collection of tech-support data:

scope cimc
scope tech-support
set tftp-ip x.x.x.x
set path supportfile
commit
start

You can view progress of the tech-support collection/upload by using the 'show detail' command repeatedly.

Reference: http://www.cisco.com/en/US/products/ps10493/products_configuration_example09186a0080b10d5e.shtml

6.23.2011

How to Read UCS B-Series Tech-Support Files - UCSM Detail

I just posted a document on the Cisco Support Community Forums detailing the output of the .tar file generated after running 'show tech-support ucsm detail'.  Hopefully this will give you the power to dig further into your UCS environment when an issue arises.

 The document can be found at the following link:
https://supportforums.cisco.com/docs/DOC-17071

3.29.2011

Change vCenter Data Center Name for 1000v

If you have changed your vCenter Data Center name and are in need of changing it in your Nexus 1000v switch, the following procedure can be followed:

From the 1000v, disconnect the VSM from vCenter:
JenN1K# conf t
JenN1K(config)# svs connection vc
JenN1K(config-svs-conn)# no connect

Change the name in the 1000v's running configuration to reflect the Data Center name change:
JenN1K(config-svs-conn)# vmware dvs datacenter-name <New name>

Reconnect the svs connection:
JenN1K(config-svs-conn)# connect

Ensure the configuration is saved in the event of a reboot or switchover:
JenN1K# copy running-config startup-config

Low Voltage CMOS Battery

The following procedure can be used in order to confirm a voltage problem alert being reported on a B-series server's CMOS battery.

The fault that will be sent is the following:
 code="F0424"
 descr="Possible loss of CMOS settings: CMOS battery voltage on server 1/1 is lower-critical"

In order to further troubleshoot this issue, the following commands can be used in order to collect
information about the CMOS battery's sensor:

 connect cimc <chassis>/<server>
 sensors

The sensor you should look for is: P3V_BAT_SCALED.
If the sensor reading is below 3.0V, a bad battery is typically the suspect.

The replacement part ID for the CMOS battery is: N20-MBLIBATT=

3.28.2011

Circuit Trace in UCS

You may be wondering how does a packet travel from my adapter through the UCS system to the upstream switch?  The following commands can help you trace out the data path of a packet.

'show service-profile circuit server x/y' will show the VIFs associated to a particular blade:
Server: 1/1
    Fabric ID: A
        VIF        vNIC       Link State Overall Status Admin Pin  Oper Pin   Transport
        ---------- ---------- ---------- -------------- ---------- ---------- ---------
                41            Unknown    Unknown        0/0        0/0        Unknown
              9158            Up         Active         0/0        0/0        Ether
               964 eth0       Up         Active         0/0        0/3        Ether
               966 fc0        Up         Active         0/0        0/0        Fc
    Fabric ID: B
        VIF        vNIC       Link State Overall Status Admin Pin  Oper Pin   Transport
        ---------- ---------- ---------- -------------- ---------- ---------- ---------
                42            Unknown    Unknown        0/0        0/0        Unknown
              9159            Up         Active         0/0        0/0        Ether
               965 eth1       Up         Active         0/0        0/3        Ether
               967 fc1        Up         Active         0/0        0/0        Fc

Using these numbers, we can search for the active border interfaces on the system in order to determine which uplink an interface is using to reach the outside world:

'connect nxos'
'show pinning border-interfaces active'

--------------------+---------+----------------------------------------
Border Interface     Status    SIFs                                   
--------------------+---------+----------------------------------------
Po3                  Active    veth961 veth965 veth973 veth977 veth981
                               veth985 veth989 veth993 veth997  
                               veth1001 veth1005 veth1009 veth1013
                               veth1017 veth1021 veth1025 veth1029
                               veth1033 veth1037 veth1041 veth1045
                               veth1049 veth1053 veth1057 veth1061
                               veth1065 veth1069 veth1073 veth1081
                               veth1085                         

Total Interfaces : 1

We can see from the output that all traffic for virtual interfaces connected to FI B are being sent out port-channel 3.  We can take this even further and see which uplink in the port-channel has been hashed to carry the uplink traffic for a particular source.

The following command shows the hashing algorithm that will determine what uplink is used:

'show port-channel load-balance'
Port Channel Load-Balancing Configuration:
System: source-dest-ip

Port Channel Load-Balancing Addresses Used Per-Protocol:
Non-IP: source-dest-mac
IP: source-dest-ip source-dest-mac

With this in mind, the following command can be used to determine the uplink.  The source IP and destination MAC addresses are needed to complete this command:
show port-channel load-balance forwarding-path interface port-channel 3 <src ip> <dst-ip> <src-mac> <dst-mac>

In a later post, I will document how to determine the IOM backplane ports and fabric ports that a particular server is using to reach the upstream network.

3.27.2011

The Beginning...

Hey all.  I have started this blog this morning as means for me to post information about general UCS best practices, troubleshooting tips, and any neat tricks that I think UCS administrators should know.  This blog can also be used to ask questions or post comments about a specific UCS component, topology or anything else that may be UCS-related. 

I am still working to come up with a catchy and creative name for this one-stop shop, so if you have any suggestions, let me know.