Troubleshooting the Outdated VM Manager Data and No VM Manager Data statuses

Background

Reasons behind the data retrieval failure

Before you begin troubleshooting

Starting from application update 9.2.12, you do not need to go to the computer where VM Manager tool is installed, to get the data that are needed for troubleshooting purposes. When cannot retrieve VM manager data from a computer, go to the Computer Support Data panel and download the logs from the related computer to investigate the issue. For more information about how to download the log package, see: Collecting logs for troubleshooting purposes.

Troubleshooting

Identify if the computer runs on a public cloud and then additionally identify it as runing on a public cloud

For detailed information, refer to Identifying computers on public clouds.

Check whether all VM Managers are defined in BigFix Inventory

1. Log in to BigFix Inventory.
2. Go to Management > VM Managers, and check whether all the VM Manager connections are defined. If not, see: Adding VM Managers in central mode or Adding VM Managers in distributed mode.
   Note: Make sure that the VM Manager URL is in the correct format. The format differs between technologies, and versions.

Check whether the status of all VM Manager connections shows as OK

If any of the VM Manager connections is in status other than OK, refer to: VM manager statuses.

Check whether VM Manager Tool is properly configured.

1. The VM Manager Tool is installed at:

   BESClient/LMT/VMMAN

   C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN

   1. To ensure that the tool was installed successfully, log in to the BigFix console.

   2. In the navigation panel, click Actions and select Install VM Manager Tool.

   3. Open to the Computers tab, and check whether the status is set to Completed. If the installation failed, check which line of the action script caused the failure. Double-click the listed computer, and analyze the View Action Info to find the relevant line. If you cannot find the Install VM Manager Tool action, see: Troubleshooting: Enabling the VM Managers panel.

2. Run the following command and check whether the VM Manager Tool is working:

   BESClient/LMT/VMMAN/vmman.sh -run

   BESClient\LMT\VMMAN\vmman.bat -run

3. Test the connection to VM Managers by running the following command.

   BESClient/LMT/VMMAN/vmman.sh -testconnection

   BESClient\LMT\VMMAN\vmman.bat -testconnection

4. Check the statuses of the VM Managers by running the following command.

   BESClient/LMT/VMMAN/vmman.sh -status

   BESClient\LMT\VMMAN\vmman.bat -status

   For more information, see: Running VM Manager tool and VM Manager tool command-line options.

In case of the central and distributed VM Manager Tool, check whether VM Manager Tool results are generated and properly communicated to the BigFix Inventory server

1. Check whether the files with scan results are packed for upload.
   1. Go to Computer Support Data panel and collect the logs from the computer that you need to investigate. For more information about how to download the log package, see: Collecting logs for troubleshooting purposes.
   2. Go to the following directory.
      • \sha1_files\logcollector_0_computer_id_logCollectorAgent.tar\computer_id_logCollectorAgent\var\opt\BESClient\LMT\VMMAN\computer_id
      • \sha1_files\logcollector_0_computer_id_logCollectorAgent\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\computer_id_vmman.zip
2. If the result files are not packed for upload, check whether the upload is scheduled.
   1. Log in to the BigFix console.
   2. Click Actions in the navigation tree, and select Schedule VM Manager Tool Scan Results Upload.
   3. Check the details and make sure that the state is marked as Open. If not, see: Uploading collected data.
   4. If the action failed, check which line of the action script caused failure. Go to the Computer tab, double-click the listed computer, and analyze the View Action Info to find the relevant line. You can also try forcing the upload of collected data.
3. Check whether the scan results exist in the BES server sha1 directory in the log package that you downloaded from the affected computer.
   • /sha1/vmman_scan_*_computer_id_*_vmman.tar.gz
   • \sha1\vmman_scan_*_computer_id_*_vmman.zip

   If the Schedule VM Manager Tool Scan Results Upload action was completed successfully, but the files are still not uploaded, go to the log package that was downloaded from this computer and in the sha1 directory check the Index.txt file. If the file contains the following error MaxArchiveSize: Exceeded, see: Configuring VM manager for subcapacity reporting.

   Note: Starting from application update 9.2.14, you can check whether the value of _BESClient_ArchiveManager_MaxArchiveSize parameter is exceeded on the Computer Support Data panel. For more information, see: Checking whether the maximum archive size is exceeded.
4. Check whether file with scan results is available in the BigFix database, go to the log package that was downloaded from Computer Support Data panel and check whether the BIGFIX_Uploads and BIGFIX_uploads_availability files contain the data. If you cannot find the results, restart the FillDB service on the BigFix server.
   • Run the following command: /etc/init.d/besfilldb restart.
   • Find the FillDB service on the list of services, and restart it.
   It takes several minutes for the database to update after the restart.
   • If the issue occurs more than once, upgrade the BigFix server. For more information, see: Software/Hardware data may not appear in the ILMT/BFI UI after successful ETL import because of the BigFix FillDB APAR IV83671.
5. Check whether you can find the result data in DatasourceFile in the log package that was collected from this computer.
   • If the result file exists in the BigFix database, but it is not found in DatasourceFile file, contact the IBM® support.
6. Check whether you can find the results in adm.last_imported_scan file in the log package from this computer.
   • If the results data exists in all databases (listed in point 5 and 6), but cannot be found in adm.last_imported_scan file, it means that the import of data fails.
     • Check the BigFix Inventory import logs, which are located in the following directory and troubleshoot the issue.

       Installation_directory/wlp/usr/servers/server1/logs/imports

       You can find also the last import log attached to the log package from this computer.

Check whether the VM Manager Tool collects valid and complete results.

Continue with this solution if the result file was imported without errors, and at least one of your VM Manager connection shows Outdated VM Manager Data or No VM Manager Data status, perform the following steps:

1. Collect UUIDs of the affected computers.
   1. To troubleshoot for Outdated VM Manager Data status, perform the following steps.
      1. Log in to BigFix Inventory, Computer Support Data panel, and collect logs from the affected computer.
      2. Open a file that matches the following pattern and has the highest timestamp.

         /sha1_files/cit_capacity_*_<computer_ID>_tlm_hw.tar.gz/<computer_ID>_tlm_hw.tar/tlm_hw_<date_time>_<timestamp>.xml

         \sha1_files\cit_capacity_*_<computer_ID>_tlm_hw.zip\tlm_hw_<date_time>_<timestamp>.xml

      3. Get the UUID of the computer.

         <?xml version="1.0" encoding="UTF-8"?>
             <Hardware>
             (...)
                 <ComponentID version="1">
                     <Manufacturer>VMware, Inc.</Manufacturer>
                     <Product>VMware Virtual Platform</Product>
                     <Version>None</Version>
                     <SerialNumber>VMware-12 34 56 78 12 34 12 34-12 34 12 34 56 78 90 12</SerialNumber>
                     <Type></Type>
                     <UUID>12345678-1234-1234-1234-123456789012</UUID>
                 </ComponentID>
             (...)
                 <VirtualMachineGuest version="1">
                     <UUID>VMware-12 34 56 78 12 34 12 34-12 34 12 34 56 78 90 12</UUID>
                     <HypervisorType>VMware</HypervisorType>
                 </VirtualMachineGuest>
             (...)
             </Hardware>

      4. Remove any prefixes or spaces in the obtained UUID. Ensure that the UUID conforms to the following pattern: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. For example, 12345678-1234-1234-1234-123456789012.
   2. To troubleshoot for No VM Manager Data status, perform the following steps:
      1. Log in to BigFix Inventory.
      2. Log in to BigFix Inventory, and view the Capacity Scan Health widget. It shows the number of endpoints with the No VM Manager Data status.
      3. Click the No VM Manager Data to see the list of computers with no VM Manager data on the Hardware Inventory report.
         
         
      4. Export the report to CSV file to view the information about the server ID.
      5. Open the CSV file and choose one of the obtained UUIDs. Remove the TLM_VM_ prefix from the UUID.
2. Obtain files with VM Manager Tool scan results.
   1. Go to the VM Manager Tool installation directory and run the following command:

      /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/debugData.zip \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\debugData.zip

   2. Go to the debugData.zip file generated in the VM Manager Tool installation directory.
3. Check whether the UUID that you obtained in step 1 exists in the file with scan results that is located in the upload subdirectory of the debugData.zip package.

   The UUID you obtained in step 1 can differ from the UUID that is returned by the VM Manager Tool. Both values consist of 32 identical characters. However, the first 16 characters are arranged in different order. Only the final 16 characters are the same. For example, the value in the Server ID column can be TLM_VM_12345678-1234-1234-1234-123456789012, but the value that is retrieved by the VM Manager Tool is 78563412-3412-3412-1234-123456789012.

   This issue was fixed in application update 9.2.3. To solve the problem, upgrade BigFix Inventory to the latest version. After the upgrade, upload the new capacity data to produce new scan results. To do that, run a single capacity scan and force upload of its results. For more information, see: Initiating the capacity scan on all computers. The capacity data is updated in up to two days, when the capacity scans and the VM manager data are imported.

4. If the UUID of the affected computer does not exist in the scan results files, check the following possible causes:
   1. The UUID is duplicate.
      1. To verify whether the UUID is duplicated, search all trace.log and config_file.log files for the following information.

         Duplicates of UUIDs are found on VM Manager, 
         URL: https://srvvcspr01/sdk/vimService.wsdl.
         Discarded UUIDs for guests: 4227e866-7121-d731-235a-343077d7ee93.

         The trace.log and config_file.log files are stored in the following location:

         /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/logs

         \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\logs

         Note: To improve readability of logs, the config file in config directory is used to record all communication with a hypervisor. Each config file has a separate log and a separate name, such as <config_name.log>. For example, config file, vmmconf_8762375104473669393.properties has a log file named vmmconf_8762375104473669393.log. To avoid the confusion with log file names, only one log file is created per config file. Also, for every hypervisor, the config file has one log file.
      2. If some UUIDs are duplicated, go to the virtual machine that has the duplicated UUID, and change it so that it becomes unique.
   2. The user who is connecting with the VM manager on VMware does not have a permission to collect data about the virtual machine. Check the VM manager settings, and permissions. For more information, see: Verifying permissions for VMware communication.
   3. The VM manager is not valid for the virtual machine.
5. If the UUID of the affected computer exists in the scan results file, but the computer still has the Outdated VM Manager Data or No VM Manager Data status, perform the following steps.
   1. Check whether debug data is collected for the affected computer.
      • For local or disconnected VM Manager Tool, go to the VM Manager Tool installation directory and run the following command:
        • ./vmman.sh -retrievedebugdata
        • vmman.bat -retrievedebugdata
        Then, go to the debugData.zip file generated in the VM Manager Tool installation directory.
      • For central or distributed VM Manager Tool, open the log package, and go to the debugData.zip file that is stored in the following location.
        • /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/debugData.zip
        • \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\debugData.zip
   2. Go to the debug/<VM_manager_connection_ID> subdirectory of the debugData.zip file.
   3. Check whether the login.xml file is correct, and error-free. The login.xml file can contain the authentication error but the VM manager connection test was successful, the user login, or password probably contains one of the following characters: $ or \.

      <faultcode>ServerFaultCode</faultcode>
      <faultstring>Cannot complete login due to an incorrect user name
      or password.</faultstring>
      <detail>
      <InvaildLoginFault xmlns="urn:vim25" xsi:type="InvalidLogin"/>

      To solve the problem, upgrade the VM Manager Tool to the latest version. For more information, see: Checking the VM Manager tool version and Updating VM Manager Tool.
      Note: After you upgrade the VM Manager Tool, download fresh log package and check the debug data.
   4. Open the retrieveProperties.xml file. The file contains information about all virtual machines, including VMs with duplicated UUIDs, and the <HostCpuPackage> data. If the <HostCpuPackage> tag is not in the file, the user that connects to the VM manager is underprivileged. Ensure that the user has sufficient permissions. For more information, see: Verifying permissions for VMware communication and How to set the correct permissions for LMT - VMware communication.