How to Analyze C-Series Tech-Support Bundle

1. Keep the name of the techsupport as <filename>.tar.gz

2. The techsupport bundle contains the following folders
            a)mnt
            b)tmp
            c)var
            d)obfl
            e)usr

3. OBFL (On Board Failure Logging) contains the logs of the CIMC , IPMI, SEL etc. It's circular and writes itself when the max number of events are reached.
            This folder should contains obfl files of 76KB size. New file is created once 76KB size is reached. The time stamp will tell the latest created file.
            Many types of failures should be looked into these files.
            IOCTL failures, network packet drop, Tulip driver issue, Process or memory issue.
            Use these keywords mainly error, fail, fault, disturbingly, critical, to search for such instances.

4. usr folder contains another folder IPMI. The IPMI folder contains the SDR file NVRAM_SDR00. This File is used decode the SEL.

5. tmp folder contains two files: one with same name as the bundle and another with filename_bios_post_results
            The first file will give most of the information about the system, firmware etc
            Get the BMC and BIOS versions. Board serial number and other information, CPU information, DIMM population and type information, card and adapter details, boot order information (SM BIOS TABLE) , Network settings, Memory and CPU consumption information.
            It also contains ifconfig information that will indicate what MAC addresses are active and what are the ips picked. The sensor listing with list the reading, fan speed and indicate in case any of these   sensors are in Critical or NR state. It also gives LED information.
            The BIOS file indicates the BIOS post completion information. It requires deeper knowledge of these codes to interpret this information.

6. var - The var folder contains multiple folders and files. Core folder contains any core dump file generated. It is a very complex process to analyze these files.
            IPMI folder contains the sensor alarm information and will tell what sensors are in alarmed state.
            Log folder contains log files like critical, peci, messages and avct_server. These can be opened with notepad.
            The avct_server file contains the VKVM related logs. search for fail, error, fault, to get the error messages logged related to VKVM. Messages and Critical files contains SEL, IPMI and CIMC logs and will give fairly good amount of error messages here.
            nuova folder contains information related to Rank margin test and memory status.
            The service folder contains the process status. and run contains information on pid.

7. mnt - This folder contains SEL file which can be used to decode SEL in association with SDR file.avct_ems_cfg contains some of the information on persistent data.cert contains certificate information.
            Links for sensor information