Troubleshooting the Outdated VM Manager Data and No VM Manager Data statuses

Background

You must grab mustgather with refreshed supplementary data or the logcollector from the fixlet.

Reasons behind the data retrieval failure

Before you begin troubleshooting

Starting from application update 9.2.12, you do not need to go to the computer where VM Manager tool is installed, to get the data that are needed for troubleshooting purposes. When cannot retrieve VM manager data from a computer, go to the Computer Support Data panel and download the logs from the related computer to investigate the issue. For more information about how to download the log package, see: Collecting logs for troubleshooting purposes.

Troubleshooting

Identify if the computer runs on a public cloud and then additionally identify it as runing on a public cloud

For detailed information, refer to Identifying computers on public clouds.

Check whether all VM Managers are defined in BigFix Inventory

1. Log in to BigFix Inventory.
2. Go to Management > VM Managers, and check whether all the VM Manager connections are defined. If not, see: Adding VM Managers in central mode or Adding VM Managers in distributed mode.
   Note: Make sure that the VM Manager URL is in the correct format. The format differs between technologies, and versions.

Check whether the status of all VM Manager connections shows as OK

If any of the VM Manager connections is in status other than OK, refer to: VM manager statuses.

Check whether VM Manager Tool is properly configured.

1. The VM Manager Tool is installed at:

   BESClient/LMT/VMMAN

   C:\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN

   1. To ensure that the tool was installed successfully, log in to the BigFix console.

   2. In the navigation panel, click Actions and select Install VM Manager Tool.

   3. Open to the Computers tab, and check whether the status is set to Completed. If the installation failed, check which line of the action script caused the failure. Double-click the listed computer, and analyze the View Action Info to find the relevant line. If you cannot find the Install VM Manager Tool action, see: Troubleshooting: Enabling the VM Managers panel.

2. Run the following command and check whether the VM Manager Tool is working:

   BESClient/LMT/VMMAN/vmman.sh -run

   BESClient\LMT\VMMAN\vmman.bat -run

3. Test the connection to VM Managers by running the following command.

   BESClient/LMT/VMMAN/vmman.sh -testconnection

   BESClient\LMT\VMMAN\vmman.bat -testconnection

4. Check the statuses of the VM Managers by running the following command.

   BESClient/LMT/VMMAN/vmman.sh -status

   BESClient\LMT\VMMAN\vmman.bat -status

   For more information, see: Running VM Manager tool and VM Manager tool command-line options.

In case of the central and distributed VM Manager Tool, check whether VM Manager Tool results are generated and properly communicated to the BigFix Inventory server

1. Check whether the files with scan results are packed for upload.
   1. Go to Computer Support Data panel and collect the logs from the computer that you need to investigate. For more information about how to download the log package, see: Collecting logs for troubleshooting purposes.
   2. Go to the following directory.
      • \sha1_files\logcollector_0_computer_id_logCollectorAgent.tar\computer_id_logCollectorAgent\var\opt\BESClient\LMT\VMMAN\computer_id
      • \sha1_files\logcollector_0_computer_id_logCollectorAgent\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\computer_id_vmman.zip
2. If the result files are not packed for upload, check whether the upload is scheduled.
   1. Log in to the BigFix console.
   2. Click Actions in the navigation tree, and select Schedule VM Manager Tool Scan Results Upload.
   3. Check the details and make sure that the state is marked as Open. If not, see: Uploading collected data.
   4. If the action failed, check which line of the action script caused failure. Go to the Computer tab, double-click the listed computer, and analyze the View Action Info to find the relevant line. You can also try forcing the upload of collected data.
3. Check whether the scan results exist in the BES server sha1 directory in the log package that you downloaded from the affected computer.
   • /sha1/vmman_scan_*_computer_id_*_vmman.tar.gz
   • \sha1\vmman_scan_*_computer_id_*_vmman.zip

   If the Schedule VM Manager Tool Scan Results Upload action was completed successfully, but the files are still not uploaded, go to the log package that was downloaded from this computer and in the sha1 directory check the Index.txt file. If the file contains the following error MaxArchiveSize: Exceeded, see: Configuring VM manager for subcapacity reporting.

   Note: Starting from application update 9.2.14, you can check whether the value of _BESClient_ArchiveManager_MaxArchiveSize parameter is exceeded on the Computer Support Data panel. For more information, see: Checking whether the maximum archive size is exceeded.
4. Check whether file with scan results is available in the BigFix database, go to the log package that was downloaded from Computer Support Data panel and check whether the BIGFIX_Uploads and BIGFIX_uploads_availability files contain the data. If you cannot find the results, restart the FillDB service on the BigFix server.
   • Run the following command: /etc/init.d/besfilldb restart.
   • Find the FillDB service on the list of services, and restart it.
   It takes several minutes for the database to update after the restart.
   • If the issue occurs more than once, upgrade the BigFix server. For more information, see: Software/Hardware data may not appear in the ILMT/BFI UI after successful ETL import because of the BigFix FillDB APAR IV83671.
5. Check whether you can find the result data in DatasourceFile in the log package that was collected from this computer.
   • If the result file exists in the BigFix database, but it is not found in DatasourceFile file, contact the IBM® support.
6. Check whether you can find the results in adm.last_imported_scan file in the log package from this computer.
   • If the results data exists in all databases (listed in point 5 and 6), but cannot be found in adm.last_imported_scan file, it means that the import of data fails.
     • Check the BigFix Inventory import logs, which are located in the following directory and troubleshoot the issue.

       Installation_directory/wlp/usr/servers/server1/logs/imports

       You can find also the last import log attached to the log package from this computer.

Check whether the VM Manager Tool collects valid and complete results.

Continue with this solution if the result file was imported without errors, and at least one of your VM Manager connection shows Outdated VM Manager Data or No VM Manager Data status, perform the following steps:

1. Collect UUIDs of the affected computers.
   1. To troubleshoot for Outdated VM Manager Data status, perform the following steps.
      1. Log in to BigFix Inventory, Computer Support Data panel, and collect logs from the affected computer.
      2. Open a file that matches the following pattern and has the highest timestamp.

         /sha1_files/cit_capacity_*_<computer_ID>_tlm_hw.tar.gz/<computer_ID>_tlm_hw.tar/tlm_hw_<date_time>_<timestamp>.xml

         \sha1_files\cit_capacity_*_<computer_ID>_tlm_hw.zip\tlm_hw_<date_time>_<timestamp>.xml

      3. Get the UUID of the computer.

         <?xml version="1.0" encoding="UTF-8"?>
             <Hardware>
             (...)
                 <ComponentID version="1">
                     <Manufacturer>VMware, Inc.</Manufacturer>
                     <Product>VMware Virtual Platform</Product>
                     <Version>None</Version>
                     <SerialNumber>VMware-12 34 56 78 12 34 12 34-12 34 12 34 56 78 90 12</SerialNumber>
                     <Type></Type>
                     <UUID>12345678-1234-1234-1234-123456789012</UUID>
                 </ComponentID>
             (...)
                 <VirtualMachineGuest version="1">
                     <UUID>VMware-12 34 56 78 12 34 12 34-12 34 12 34 56 78 90 12</UUID>
                     <HypervisorType>VMware</HypervisorType>
                 </VirtualMachineGuest>
             (...)
             </Hardware>

      4. Remove any prefixes or spaces in the obtained UUID. Ensure that the UUID conforms to the following pattern: xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx. For example, 12345678-1234-1234-1234-123456789012.
   2. To troubleshoot for No VM Manager Data status, perform the following steps:
      1. Log in to BigFix Inventory.
      2. Log in to BigFix Inventory, and view the Capacity Scan Health widget. It shows the number of endpoints with the No VM Manager Data status.
      3. Click the No VM Manager Data to see the list of computers with no VM Manager data on the Hardware Inventory report.
         
         
      4. Export the report to CSV file to view the information about the server ID.
      5. Open the CSV file and choose one of the obtained UUIDs. Remove the TLM_VM_ prefix from the UUID.
2. Obtain files with VM Manager Tool scan results.
   1. Go to the VM Manager Tool installation directory and run the following command:

      /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/debugData.zip \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\debugData.zip

   2. Go to the debugData.zip file generated in the VM Manager Tool installation directory.
3. Check whether the UUID that you obtained in step 1 exists in the file with scan results that is located in the upload subdirectory of the debugData.zip package.

   The UUID you obtained in step 1 can differ from the UUID that is returned by the VM Manager Tool. Both values consist of 32 identical characters. However, the first 16 characters are arranged in different order. Only the final 16 characters are the same. For example, the value in the Server ID column can be TLM_VM_12345678-1234-1234-1234-123456789012, but the value that is retrieved by the VM Manager Tool is 78563412-3412-3412-1234-123456789012.

   This issue was fixed in application update 9.2.3. To solve the problem, upgrade BigFix Inventory to the latest version. After the upgrade, upload the new capacity data to produce new scan results. To do that, run a single capacity scan and force upload of its results. For more information, see: Initiating the capacity scan on all computers. The capacity data is updated in up to two days, when the capacity scans and the VM manager data are imported.

4. If the UUID of the affected computer does not exist in the scan results files, check the following possible causes:
   1. The UUID is duplicate.
      1. To verify whether the UUID is duplicated, search all trace.log and config_file.log files for the following information.

         Duplicates of UUIDs are found on VM Manager, 
         URL: https://srvvcspr01/sdk/vimService.wsdl.
         Discarded UUIDs for guests: 4227e866-7121-d731-235a-343077d7ee93.

         The trace.log and config_file.log files are stored in the following location:

         /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/logs

         \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\logs

         Note: To improve readability of logs, the config file in config directory is used to record all communication with a hypervisor. Each config file has a separate log and a separate name, such as <config_name.log>. For example, config file, vmmconf_8762375104473669393.properties has a log file named vmmconf_8762375104473669393.log. To avoid the confusion with log file names, only one log file is created per config file. Also, for every hypervisor, the config file has one log file.
      2. If some UUIDs are duplicated, go to the virtual machine that has the duplicated UUID, and change it so that it becomes unique.
   2. The user who is connecting with the VM manager on VMware does not have a permission to collect data about the virtual machine. Check the VM manager settings, and permissions. For more information, see: Verifying permissions for VMware communication.
   3. The VM manager is not valid for the virtual machine.
5. If the UUID of the affected computer exists in the scan results file, but the computer still has the Outdated VM Manager Data or No VM Manager Data status, perform the following steps.
   1. Check whether debug data is collected for the affected computer.
      • For local or disconnected VM Manager Tool, go to the VM Manager Tool installation directory and run the following command:
        • ./vmman.sh -retrievedebugdata
        • vmman.bat -retrievedebugdata
        Then, go to the debugData.zip file generated in the VM Manager Tool installation directory.
      • For central or distributed VM Manager Tool, open the log package, and go to the debugData.zip file that is stored in the following location.
        • /sha1_files/logcollector_*_<computer_id>_logCollectorAgent.tar.gz/<computer_id>_logCollectorAgent.tar/var/opt/BESClient/LMT/VMMAN/debugData.zip
        • \sha1_files\logcollector_*_<computer_id>_logCollectorAgent.zip\Program Files (x86)\BigFix Enterprise\BES Client\LMT\VMMAN\debugData.zip
   2. Go to the debug/<VM_manager_connection_ID> subdirectory of the debugData.zip file.
   3. Check whether the login.xml file is correct, and error-free. The login.xml file can contain the authentication error but the VM manager connection test was successful, the user login, or password probably contains one of the following characters: $ or \.

      <faultcode>ServerFaultCode</faultcode>
      <faultstring>Cannot complete login due to an incorrect user name
      or password.</faultstring>
      <detail>
      <InvaildLoginFault xmlns="urn:vim25" xsi:type="InvalidLogin"/>

      To solve the problem, upgrade the VM Manager Tool to the latest version. For more information, see: Checking the VM Manager tool version and Updating VM Manager Tool.
      Note: After you upgrade the VM Manager Tool, download fresh log package and check the debug data.
   4. Open the retrieveProperties.xml file. The file contains information about all virtual machines, including VMs with duplicated UUIDs, and the <HostCpuPackage> data. If the <HostCpuPackage> tag is not in the file, the user that connects to the VM manager is underprivileged. Ensure that the user has sufficient permissions. For more information, see: Verifying permissions for VMware communication and How to set the correct permissions for LMT - VMware communication.

Troubleshooting VM Manager Tools panel operations

The following troubleshooting guidance applies to actions that are triggered from the VM Manager Tools panel, such as installation, uninstallation, update, and configuration of VM Manager tool installations. For a description of action types and statuses, see: VM Manager Tools action statuses.

An action stays in Pending status

The action was triggered but has not completed. The fixlet action is waiting to be processed or is currently running on the target computer.

Action:

1. Log in to the BigFix console.
2. Click Actions in the navigation tree and locate the action that corresponds to the VM Manager tool operation.
3. Check the action status and the target computer status on the Computers tab.
4. Verify that the target computer is reachable and that the BigFix client is running.
5. If the action remains Pending for an extended period, check whether the BigFix client on the target computer is able to receive and execute actions.

An action becomes Failed

The action did not complete successfully. Finished actions are stopped on the BigFix side.

Action: Investigate the failure in the BigFix console by reviewing the action information for the corresponding fixlet.

1. Log in to the BigFix console.
2. Click Actions in the navigation tree and locate the failed action.
3. Open the Computers tab, double-click the target computer, and analyze View Action Info to identify which line of the action script caused the failure.
4. Correct the underlying issue and retry the action from the VM Manager Tools panel.

An action becomes Not Relevant

The fixlet that corresponds to the action is not relevant on the target computer. This means that the fixlet relevance conditions were not met. For example, the target computer does not meet the operating system or architecture requirements, or the VM Manager tool is already installed at the expected version.

Action:

1. Log in to the BigFix console.
2. Locate the fixlet (for example, Install VM Manager Tool) and check its relevance on the target computer.
3. Verify that the target computer meets the installation requirements, including a supported 64-bit operating system.

An action becomes Expired after 12 hours

The action neither succeeded nor failed within 12 hours. The action was stopped and marked as Expired.

Action:

1. Log in to the BigFix console and review the action information to determine why the action did not complete.
2. Verify that the target computer is reachable and that the BigFix client is running and able to process actions.
3. Check whether network or firewall issues prevented the action from completing.
4. Retry the action from the VM Manager Tools panel if appropriate.

Update returns "Update: Not Relevant"

The update fixlet is not relevant on the target computer. This status appears when the newest version of the VM Manager tool is already installed or when the fixlet relevance conditions are not met on the target computer.

Action:

1. Check the currently installed version of the VM Manager tool by using the VM Manager Information analysis. For more information, see: Checking the VM Manager tool version.
2. If the installed version is already the latest, no action is required.
3. If the version is outdated but the fixlet is not relevant, verify that the target computer meets the installation requirements and check the fixlet relevance in the BigFix console.

Configuration appears overwritten after the first distributed edit or add action

When you add or edit a VM manager on a distributed VM Manager tool for the first time from the user interface, the VM Manager tool configuration and all of its VM managers are sent together. This first configure action can overwrite custom application configuration that was previously set directly on the endpoint.

Action:

1. Before making the first change from the user interface, note any custom parameter values that were configured directly on the endpoint in the vmmmainconf.properties file.
2. After the configure action completes, verify the VM Manager tool configuration on the endpoint and reapply any custom values if necessary.
3. For subsequent changes, the configuration is sent incrementally and does not overwrite the entire configuration.

Note: Configuration changes that are made directly on endpoints are not synchronized back to BigFix Inventory. The VM Manager Tools panel reflects only the configuration that was set through the application.

VM Managers disappear after deleting a VMMT from the VM Manager Tools panel

After a VM Manager tool is deleted from the VM Manager Tools panel, its associated VM managers are hidden from the VM Managers report.

Action:

• This is expected behavior. Deleting a VM Manager tool from the VM Manager Tools panel removes the installation and hides all linked VM manager connections.
• If you need the VM manager connections again, reinstall the VM Manager tool on the same or a different computer and re-create the connections.

Important: Reinstalling a VM Manager tool on the same computer preserves the existing VM Manager tool application configuration and VM manager connections. Deleting a VM Manager tool does not preserve them.

Endpoint-side manual changes are not visible in BigFix Inventory

Configuration changes that are made directly on the endpoint, for example by editing the vmmmainconf.properties file, are not synchronized back to BigFix Inventory.

Action:

• Use the VM Manager Tools panel or the Advanced Server Settings panel to make configuration changes that must be visible in BigFix Inventory.
• If you must edit the configuration file directly on the endpoint, be aware that the VM Manager Tools panel and the sam.vm_manager_tool_configs table reflect only the configuration that was set through the application. For more information, see: Advanced server settings.