Troubleshooting Hyper-V connection issues

Glossary

The connection between BigFix Inventory and Hyper-V is established through the Windows Management Instrumentation (WMI) queries. These queries use either PowerShell over the DCE-RPC mechanism, or WinRM that uses SOAP, or XML over HTTP protocol. The BigFix Inventory client communicates with the Hyper-V host through the VM Manager interface, VM Manager Tool. The following list describes the basic components of the virtual environment:

Server: The Hyper-V host where the WinRM service is located.
Client: BigFix Inventory host that can be either a Windows or Unix computer.

The client authentication on both Windows and Unix systems uses the same authentication sequence: NTLM, NTLMV2, or HTTP Basic. The Windows domain members undergo Kerberos network authentication when you use PowerShell as the communication interface.

Troubleshooting multiple Hyper-V VM Manager Connections

When there are many Hyper-V VM Manager connections that are configured with WinRM interface, VM Manager Tool might not be able to correctly establish the connections simultaneously in multiple threads. In this case, you must set the vmm_thread_pool_size parameter to 1 to lower the number of connection threads. Additionally consider adding more VM Manager Tools to distribute the load between them.

Troubleshooting Hyper-V with the server diagnostic script

To detect potential problems with adding, or using the Hyper-V VM Manager, use the diagnostic script that checks the Hyper-V and BigFix Inventory servers.
- NTLM_debug.jar is a script that is dedicated for non-Windows systems. To download the script, see: Troubleshooting of No VM Manager Data status. This script only checks the authentication schema, and tries to send out a SOAP, or XML queries to Hyper-V host. Therefore, it is recommended to additionally use a Windows script to troubleshoot the server from a Windows computer.
- Hyperv_precheck.wsf is a Windows script that uses WinRM commands. To download the script, see: Troubleshooting of No VM Manager Data status. You need to set up WinRM on Hyper-V hosts to use the script. For more information, see: Configuring WinRM on Hyper-V hosts.
  The script checks:
  - Whether the Virtual Machine Management Service (VMMS) is running.
  - Whether the WinRM service is running.
  - Whether the listeners for WinRM interface are defined.
  - Whether the Certificate Thumbprint for HTTPS communication is in place.
  - Whether the Negotiate authentication scheme is set as a valid protocol.
  - Whether AllowUnencrypted is set for HTTP protocol. However, it does not check whether the HTTP Basic authentication is enabled in WinRM communication.
  - The remote connectivity and server certificates by checking the trusted hosts in the WinRM configuration.
  - The Local Security Policy settings that are used for NTLM, or NTLMV2 authentication.
  Ensure that you are an administrator with sufficient permissions to run WinRM commands and run the script.
```
cscript hyperv_precheck.wsf /r:<transport>://<Hyper-V_server_name>:<port>/wsman
 /u:<user_id> /p:<password> /s:<server> [/v:true] [/d:true]
```
  Where:
  
  <transport>
  
  Is either HTTP or HTTPS.
  
  <Hyper-V_server_name>
  
  Is the host name of the Hyper-V server. If you are using HTTPS, the host name must match the CN in the certificate.
  
  <port>
  
  Is the port number on which the Windows Remote Management client for the HTTP or HTTPS transport listens.
  
  <user_id>
  
  Is the user ID that is used to connect to the Hyper-V server.
  
  <password>
  
  Is the password that is used to connect to the Hyper-V server.
  
  <server>
  
  Is either the Hyper-V or BigFix Inventory server where the script is run.
  
  /v
  
  Verbose mode.
  
  /d
  
  Debug mode.
  1. Run the script on the Hyper-V server. Use the /s:hyperv command.
  2. Run the script on the BigFix Inventory server. Use the /s:LMT command.
  3. If any of the checks fail, rerun the commands in the verbose mode to obtain detailed information about errors. Use the /v:true /d:true command.

Manual Hyper-V troubleshooting

If the hyperv_precheck.wsf script was not able to diagnose the Hyper-V issues with satisfactory results, proceed with the manual troubleshooting.

Verify whether the WinRm service is running. Run the following command in command prompt.
```
sc query WinRM
```
The query results should include the following information.
```
SERVICE_NAME: WinRM
STATE       : 4  RUNNING
```
Note: The Windows User Account Control (UAC) can affect user access to the WinRM service. If you use the Negotiate authentication scheme, only the Administrator account can access the service. To allow all accounts to access the service, set the following registry value.
```
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\
LocalAccountTokenFilterPolicy to '1'
```
Verify the WinRM configuration properties by using command prompt.
1. Run the following commands on the Hyper-V server:
  - winrm get winrm/config/service
  - winrm enumerate winrm/config/listener
2. Run the following commands on the client:
  - winrm get winrm/config/client
  Note: If Hyper-V and BigFix Inventory are installed on the same host, you can use a single command to retrieve the required information: winrm get winrm/config.
Set up a simple test case by changing some of the WinRM settings.
1. On the BigFix Inventory server, add an asterisk to the Trustedhosts list.
```
winrm set winrm/config/client @{TrustedHosts="*"}
```
  TrustedHosts="*" forces the client to abandon authentication of the remote end. However, the remote end still requires the client authentication. Typically, TrustedHost is set on the client to the Hyper-V server name.
2. Allow Basic authentication and unencrypted traffic on a Hyper-V server.
```
winrm set winrm/config/service/auth @{Basic="true"}
winrm set winrm/config/service @{AllowUnencrypted="true"}
```
  Basic="true" enables HTTP Basic authentication through user and password in clear text. AllowUnencrypted= true allows the transfer of the unencrypted data through HTTP between the server and the client.
3. Check the WinRM configuration setup.
```
'winrm get winrm/config/client':
Client:
        NetworkDelayms = 5000
        URLPrefix = wsman
        AllowUnencrypted = true
        Auth
            Basic = true
            Digest = true
            Kerberos = true
            Negotiate = true
            Certificate = true
            CredSSP = false
        DefaultPorts
            HTTP = 5985
            HTTPS = 5986
        TrustedHosts = *
 
'winrm get winrm/config/service' on the Hyper-V server:
Service:
    RootSDDL = O:NSG:BAD:P(A;;GA;;;BA)(A;;GA;;;S-1-5-21-3273298017-2363932476
    -3643925056-1633)S:P(AU;FA;GA;;;WD)(AU;SA;GWGX;;;WD)
    MaxConcurrentOperations = 4294967295
    MaxConcurrentOperationsPerUser = 15
    EnumerationTimeoutms = 600000
    MaxConnections = 15
    MaxPacketRetrievalTimeSeconds = 120
    AllowUnencrypted = true
    Auth
        Basic = true
        Kerberos = true
        Negotiate = true
        Certificate = true
        CredSSP = false
        CbtHardeningLevel = Relaxed
    DefaultPorts
        HTTP = 5985
        HTTPS = 5986
    IPv4Filter = *
    IPv6Filter = *
    EnableCompatibilityHttpListener = true
    EnableCompatibilityHttpsListener = true
    CertificateThumbprint
```
  If the previous change to Basic = true resolved the authentication issue, apparently the client and the server are not able to negotiate the authentication protocol. The Basic authentication scheme is not recommended, unless WinRM is set up with HTTPS. It might cause security exposure by sending a user name, a password and the message body in clear text. Microsoft uses three protocols during the Negotiate scheme: Kerberos, NTLMV2, and NTLM. Usually, the client and the server choose the best authentication mechanisms they agree upon. BigFix Inventory does not allow the Kerberos protocol. To check whether the client and the server allow the NTLMV2 or NTLM protocol, use the following registry query.
```
reg query HKEY_LOCAL_MACHINE\System\CurrentControlSet\control\LSA\MSV1_0
```
  The following parameters under this registry key control the authentication schema behavior:
  - NtlmMinClientSec
  - NtlmMinServerSec
  To configure these parameters, see: How to disable LM authentication on Windows NT. Alternatively, the parameters can be modified through the appropriate Group Policy: Computer Configuration > Windows Settings > Local Policies > Security Options:LAN Manager authentication level
To manually identify the appropriate authentication mechanism for WinRM service on the remote host, use the following commands:
- Run the command from Windows PowerShell.
```
Test-WSMan -ComputerName http://<Hyper-V_server_name>:<port> 
-Auth <authentification_scheme> -Cred <user_id>
```
  Where:
  
  <Hyper-V_server_name>
  
  Is the host name of the Hyper-V server. If you are using HTTPS, the host name must match the CN in the certificate.
  
  <port>
  
  Is the port number on which the Windows Remote Management client for the HTTP or HTTPS transport listens.
  
  <authentification_scheme>
  
  Is the authentication scheme: Basic or Negotiate
  
  <user_id>
  
  Is the user ID to Windows PowerShell.
- Run the command from the command prompt.
```
winrm identify -r:http://<Hyper-V_server_name>:<port> 
-auth <authentification_scheme> -u:<user_id> -p:<password>
```
  Where:
- <Hyper-V_server_name>
  
  Is the host name of the Hyper-V server. If you are using HTTPS, the host name must match the CN in the certificate.
  
  <port>
  
  Is the port number on which the Windows Remote Management client for the HTTP or HTTPS transport listens.
  
  <authentification_scheme>
  
  Is the authentication scheme: Basic or Negotiate.
  
  <user_id>
  
  Is the user ID to WinRM.
  
  <password>
  
  Is the password to WinRM.

Event tracing

If BigFix Inventory cannot connect to the Hyper-V host, use the custom Windows Event Tracing. When the settings of the Client and the server are conflicting, the Security event log generates the following error message:

Unknown user name
or bad password

. You cannot determine that the actual issue is the unsupported authentication protocols. The Windows event tracing obtains the diagnostic data about the WMI and WinRM queries.

To start Event Tracing on the Hyper-V server, use the following commands:
- logman.exe start winrmtrace -p Microsoft-Windows-Winrm -o winrmtrace.etl -ets
- logman.exe start wmitrace -p Microsoft-Windows-WMI-Activity -o wmitrace.etl -ets
Run the VM Manager connection test.
- vmman.bat -testconnection
- vmman.sh -testconnection

The formatted WinRM Event Trace Log file, winrmtrace.etl, contains the information about the user authentication issue.

<Task>User authentication </Task>
<Message>Sending HTTP 401 response to the client and disconnect the connection after
 sending the response</Message>

Additionally, the file consists of the SOAP, or XML queries that are sent from the Client and provide essential data that is useful during troubleshooting the Hyper-V problems.

Troubleshooting Hyper-V connection status with Wireshark

Note:

To analyze the network traffic stream on Unix with no GUI environment, use tcpdump. To dump all the traffic to the external file use

tcpdump -vvv -XX
-w tcpdump.out

. The tcpdump.out file can be viewed with Wireshark.

Successful connection test between BigFix Inventory and Hyper-V with PowerShell interface.

Parameters:

vmman.bat -testconnection
vmman.sh -testconnection

vmm_communication_interface=POWERSHELL

PowerShell uses the NTLM authentication protocol.
The screen shows the Wireshark console with the Hyper-V connection details.

It does not connect to the WinRM ports: 5985 and 80, but it uses a DCE-RPC mechanism to connect to the Endpoint Mapper service via port 135.

Select the RemoteCreateInstance response entry.
To check whether the authentication finished successfully, expand the parameters, and check the value of HResult. If authentication was success, HResult is marked as S_OK.

PowerShell v2 by default uses NTLMv1 for negotiation. To update NTLM, use the following registry setting:

HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\LSA\LMCompatibilityLevel

For more information, see: How to enable NTLM 2 authentication.

Unsuccessful connection test between BigFix Inventory and Hyper-V with NTLM interface.

Parameters:

```
vmm_communication_interface=NTLM
```
```
vmm_url=http://.../wsman
```
The default port: 80

The connection test failed, and the trace.log contains the following error messages.

(...)com.ibm.license.mgmt.vmmanager.hyperv.net.HttpConnector::send::
An error occured while trying to send request to http://.../wsman

(...)com.ibm.license.mgmt.vmmanager.hyperv.net.HttpConnector::send:java.net.
ConnectException: Connection refused: connect

While the errors from the trace.log file do not provide enough details, the Wireshark capture proves that the issue is related to TCP. Use the following command to check for listener on port 80.

winrm enumerate winrm/config/listener

If no listeners are detected, run the following command to set up listeners.

winrm set winrm/config/service @{EnableCompatibilityHttpListener="true"}

Note: The setup of listeners on port 80 can collide with the other HHTP services on this computer. To avoid the issues, specify a dedicated WinRM port, such as 5985 or 5986, on the vmm_url connection string.

Unsuccessful connection test between BigFix Inventory and Hyper-V with NTLM interface. HTTP Error 401.

The connection test failed, and the trace.log contains the following error messages.
```
(...)com.ibm.license.mgmt.vmmanager.hyperv.net.HttpConnector::retrieve::
Response Code is: 401
```
```
(...)com.ibm.license.mgmt.vmmanager.hyperv.net.HttpConnector::retrieve::
The following response code was returned while connecting to VM Manager 
http://.../wsman: responseCode = 401
```
The following Wireshark capture lists the NTLM negotiation.

The first four rows show the successful NTLM negotiation. The fourth row ends the so-called four-way NTLM handshake. The last line detects HTTP Error 401: Unauthorized, and expands into the following HTTP response.

The WWW-Authenticate: Negotiate response indicates that the server is ready to use the NTLM, NTLMV2, or HTTP Basic authentication scheme. However, the server already successfully responded to the negotiation sequence. Ensure that the WinRM AllowUnencrypted = true.
Note: VM Managers do not provide the HTTP body encryption. WinRM accepts only the HTTP body that is encrypted with Negotiate, or Kerberos protocol.
The Wireshark capture shows the unsuccessful NTLM negotiation handshake.

The four-way NTLM handshake ends with a 401 error. The most common reason behind this error is the lack of sufficient access rights. To check whether the accounts assigned to the Administrators group are allowed to access WinRM, configure the following Windows registry entry.
```
HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\
System\LocalAccountTokenFilterPolicy
```

Unsuccessful connection test between BigFix Inventory and Hyper-V caused by the bug related to vmm_login domain prefix interpretation.

When you add a VM Manager to BigFix Inventory, make sure that you provide the administrator user name in one of the following formats:

user_name@domain, for example: test@cluster.com
user_name\domain, for example: test\cluster.com

Starting from application update 9.2.16, you can also provide the administrator user name in one of the following formats:

domain@user_name, for example: cluster.com@test
domain\user_name, for example: cluster.com\test

Note: Make sure that you upgrade the VM Manager Tool to version 9.2.16, not only the BigFix Inventory server.

If you do not follow this pattern, and provide the user name in the incorrect format, an error is generated during the VM Manager authentication.
The screen shows the Wireshark capture that contains the domain authentication error.

The screen shows the Wireshark capture that contains the domain authentication error.

When the error occurs, the user name is assigned to the incorrect field.