Troubleshooting

When problems occur, you can determine what went wrong by viewing messages in the appropriate log files that provide information about how to correct errors.

Log files

Enhanced logging with clearer error reporting and error handling to improve troubleshooting.

SCCPlugin.log

Lists the results of the downloads related to the execution of the SCC download plug-in. The amount of information depends on the logging level.

The log can be found in the following locations:

On Windows systems: %PROGRAM FILES%\BigFix Enterprise\BES Server\DownloadPlugins\SCCProtocol
On Linux systems: /var/opt/BESServer/DownloadPlugins/SCCProtocol

The following log files can be found in the client folder in the directory /var/opt/BESClient/EDRDeployData.

EDR_DeploymentResults.txt: Lists the results of the EDR deployment and the Zypper output. The log file indicates if the normal Zypper process is used for either a standard repository or SMT.

register-repo.log: Lists the results of the repository registration action of the SLE Custom Repository Management dashboard.

register-SMT.log: Lists the results of the SMT registration action of the SLE Custom Repository Management dashboard.

unregister-repo.log: Lists the results of the unregister repository action of the SLE Custom Repository Management dashboard.

unregister-SMT.log: Lists the results of the unregister SMT action of the SLE Custom Repository Management dashboard.

snapper_rollback.log: Lists Btrfs snapshot rollback feature that is available from the SLE Btrfs Snapshot Management dashboard.

Download plug-in logging levels

The logging level determines the amount of detail that the SCC download plug-in writes to the log files. Set the logging level in the %PROGRAM FILES%\BigFix Enterprise\BES Server\DownloadPlugins\SCCProtocol\plugin.ini file.

Note: Logging level values are case-sensitive.

The following logging levels are listed in order of increasing amount of information logged:

ERROR: Contains errors related to the execution of the download plug-in, which might indicate an impending fatal error.
WARNING: Contains information about failed downloads, and reasons for failure.
INFO: Contains general information outlining the progress and successful downloads, with minimal tracing information.
DEBUG: Contains fine-grained information used for troubleshooting issues. This is the most verbose level available.

Note: Setting the logging level to DEBUG increases the amount of information to log, which might have an impact on performance. You must only increase the logging level to DEBUG when investigating an issue.

Understanding dependency failures

Some dependency requirements cannot be determined by Fixlet relevance. In some cases, multiple levels of dependencies or conflicting third-party packages can prevent the installation of a Fixlet content.

You can manually review these details by using the EDR_DeploymentResults.txt file, which is written locally on an endpoint in the client folder by the EDR Plug-in. You can also review the analysis Endpoint Dependency Resolution - Deployment Results in the Patching Support site.

The EDR_DeploymentResults.txt file is a text file that contains several lines for each Fixlet that runs on the system. Each line contains a date or time stamp and the associated Fixlet ID number. The log shows the type and status of the installation, whether or not it was a test run and if it was successful or failed.

If the deployment was successful, the log lists out the packages that were successfully installed or upgraded during the action.

If the deployment failed, the log shows the error output from the EDR Plug-in.

Common <type> tag errors found in the EDR_DeploymentResults.txt file

The most common <type> tag errors that are found in the EDR_DeploymentResults.txt file are as follows:

No solution (noSolution): This error means that the requested target packages cannot be installed on the system, because there are conflicts between the target packages and the existing packages on the system.

Incomplete baseline (incompleteBaseline)

This error might appear if third-party packages are fulfilling dependencies and are not recognized by the resolver.

If the <type>incompleteBaseline</type> tag is found in the EDR_DeploymentResults.txt file and <name>name of RPM</name> tag is NOT in the /var/opt/BESClient/EDR_UnsupportedPackages.txt file, then you must install the RPM separately.

However, if the RPM is in the EDR_DeploymentResults.txt file, you must downgrade to a version that is supported. You can use the <sanityCheckError> tag from the EDR_DeploymentResults.txt file to identify what version of RPM is supported. You can only proceed with patching the system when the supported RPM version is installed.

Forbidden package list error (packageInForbiddenList): This error is caused by a package on the target, which is listed in the forbidden package list. You must remove the package from the forbidden package list to successfully run the Fixlet.

Empty solution (EmptySolution): This error means that the resolver cannot find a solution with the given set of information. For example, the packages that are installed on the system or the packages that it is trying to install. To resolve this error, it is often better to have the system as standard as possible. Installing additional third-party software and removing packages from the base operating system can make this more difficult for the resolver to resolve all dependencies. It is suggested to move toward using the Native Tools site instead.

Bash script error

If you get an error similar to the following error message:

Hard failure exit code 'execute prefetch plug-in' "/bin/bash" 
"{parameter "sitefolder"}/ResolveDependencies.sh"..." 
(action 159317) Exited with exit code of 2

You must complete the following steps:

Open the mentioned bash script and add the following after line 2 of the script:
```
set -x
logpath=</path/of/your/choice>
exec >$logpath 2>&1
```
Deploy the action immediately after you update the script.
Note: The client might override the file, so do not to wait too long between updating the script and deploying the action.

This procedure creates the file that is mentioned in the log path, with a line-by-line detailed output for the script.

Issues when the custom repository is not a mirror of the vendor

When you use a custom repository that is not a mirror of the vendor site, it is possible that the default gpgcheck is being done as part of the installation. The GPG signature files might not be included in the repository. The files are not checked for authenticity and might cause the installation to fail.

To resolve this issue, ensure that when you register the endpoints in the SLE Custom Repository Management dashboard, you add gpgcheck=0 to Additional Fields.

Cannot connect to the mirror server

One of the possible reasons for not being able to connect to the mirror server could be due to an authentication error. Check the SCCPlugin.log found in %PROGRAM FILES%\BigFix Enterprise\BES Server\DownloadPlugins\SCCProtocol on Windows systems, and /var/opt/BESServer/DownloadPlugins/SCCProtocol on Linux systems.

The log typically contains error code 401 when issue with authentication occur, as described in the sample log:

Searching package details...
URL: https://scc.suse.com/connect/organizations/repositories?page=1 | 
STATUS: 401 Incorrect Login Credentials for 
https://scc.suse.com/connect/organizations/repositories?page=1
Could not find token for authentication to mirror URL.

To resolve this issue, reconfigure the download plug-in and ensure that the mirror server credentials (organization credentials) is correct. You can get the mirror credentials for your organization from SUSE Customer Center or Novell Customer Center. The credentials are identical. For more information, see Mirroring Credentials.

Novell account lockout

Account lockouts are common but temporary. Contact Novell Support if you get locked out of your account.

One possible reason for an account lock out is due to invalid credentials. Ensure that you use the mirror server configuration from Novell when you register or configure the download plug-in. The mirror credentials refer to the Organization Credentials listed in the in the SUSE Customer Center or Novell Customer Center. For more details, see Mirroring Credentials.

Error deploying Fixlets from custom sites

The Fixlet site name is hardcoded in the relevance of the Fixlets because the relevance can only accept one value. When you deploy Fixlets from a custom site, the Fixlets would fail because they are still referencing to the original Fixlet site. To resolve this issue, ensure that your endpoints are subscribed to the original Fixlet site so that they can grab all the relevant site files.

If you do not want to stay subscribed to the original Fixlet site, but be able to deploy custom Fixlets successfully, complete the following steps:

Make a custom copy of the necessary site files.
Host the site files either in your own custom site or online.
Modify the custom Fixlet appropriately.

Package installation using a custom repository failed

If you get the error message: Install Failure: zypper -n install - Error:

Consider the following steps to troubleshoot the issue:

Ensure that the custom repository is enabled. To check, the Disable custom repository support - SUSE Linux Enterprise task (ID#16) in the Patching Support site must be relevant to the endpoint.
The endpoint must has the required repositories. To check, run zypper repos.
Ensure that the latest site is gathered.
Run a Fixlet from a site and collect the following logs when the action is completed:
- The Client log is located at /var/opt/BESClient/__BESData/__Global/Logs.
- The EDR_DeploymentResults.txt log is located at /var/opt/BESClient/EDRDeployData.
If the EDR_DeploymentResults.txt log file shows that the zypper command interprets each string after InstallPackages.sh -f as a package to install, check whether the bc package is installed on the endpoint. To check, run any of the following commands:
```
which bc
```
or
```
rpm -q bc
```

Missing prerequisite packages

Patching issues might occur when the endpoints are not installed with the necessary utility. Ensure that the zlib and zypper packages are installed on the endpoints.

Possible issues with the SCC Download Plug-in

Insufficient Permissions to Write to Cache

Symptoms: When the SCC Download Plug-in attempts to cache a package, you get the message in the download plug-in log (by default, it is located at <BES Server Directory>\DownloadPlugins\SCCProtocol):

ERROR : Insufficient
permissions to write cache. Please check your plugin cache directory permissions.

Causes: The BigFix Server account does not have permission to write to the directory where the download plug-in is trying to cache the repository metadata.

Resolving the problem: Check the download plug-in cache directory permissions. Make sure that the BigFix Server user ('Administrator' on Windows, 'root' on Linux) has permission to write to cache directory.

Cache Drive Does not Exist

Symptoms: When the SCC Download Plug-in attempts to cache a package, you get the message in the download plug-in log: ERROR : Cache device not ready.

Causes: The download plug-in cache directory is pointing to a drive that does not exist on the BigFix server.

Resolving the problem: Update the plugin.ini file to point to the correct cache path.

Cache Drive is Full

Symptoms: When the SCC Download Plug-in attempts to cache a package, you get the message in the download plug-in log: ERROR : Insufficient space left on cache device.

Causes: The drive where the cache is being written to is full.

Resolving the problem: Increase the free disk space on the cache device.

Package is not Found

Symptoms: The download plug-in log outputs the message

ERROR CODE 252: Package is
not found in the reposource.

Causes: There can be numerous possible causes for such an error, here are a few:

Novell might have removed the package from their repository.
The package has been superseded.
There are metadata issues on the endpoint.
A cross dependency exists between third-party packages.

Resolving the problem: Retrieve the EDR logs (/var/opt/BESClient/EDRDeployData) and client logs (/var/opt/BESClient/__BESData/__Global/Logs) from the endpoints, and contact HCL BigFix Support.

Download errors due to network configuration

Downloads may fail due to network configuration, such as proxy or firewall, preventing traffic through the URLs that are accessed by the SCC download plug-in or the SCC download cacher.

If you encounter such issues, check the following configuration:

The BigFix server whitelist, <BigFix server installation>\Mirror Server\Config\DownloadWhitelist.txt must contain the following entry:
```
SCCProtocol://.*
```
The following URLs must be accessible from the BigFix server, and from the endpoints running the SCC download cacher:
- https://scc.suse.com:443
- https://updates.suse.com:443
- http://sync.bigfix.com:80

Prefetch plug-in error

If you have taken an action on a Fixlet that failed on a line with execute prefetch plug-in in the Actionscript. Subsequent calls to the same prefetch plug-in from any Actionscript might fail on the endpoint. The script might have been blacklisted, causing the prefetch plug-in error.

To verify, check the client log. You will see either of the following messages for the Fixlet action executing the prefetch plug-in:

execute prefetch plug-in' didn't complete within 300 seconds. Black listing plug-ins 
matching the sha1 hash of 'name of 'bash' until agent is restarted.

Execute prefetch plug-in attempting to reuse plug-in which took too long earlier.

To resolve this issue, do the following actions:

Restart the BigFix client to clear the blacklist.
Set the _BESClient_ActionManager_PrefetchPlugInTimeoutSeconds client configuration setting with sufficient time for the patch to install and resolve dependencies. This client setting indicates how long the client should wait before blacklisting the script. You can use the Change Timeout for Prefetch Plugins task, available from the Patching Support site, to set the setting to 30 minutes (1800 seconds).

Note: The _BESClient_ActionManager_PrefetchPlugInTimeoutSeconds setting varies based on the endpoint and the Fixlet being installed. To get the desired value, take the slowest endpoint and increase the setting to a high number, such as 3,000 seconds, then run a large Fixlet and see how long it takes. You can then take that number and multiple it by two. Alternatively, set the client setting to 600 seconds and adjust it accordingly if the suggested value does not work for you.

Error when /var is mounted as noexec

All available Fixlets use an executable that by default runs directly from the /var directory, a partition on the endpoint. The Fixlets will not work when /var is set with the noexec option, regardless of whether the SCC Download Plug-in or Custom Repository solution is used. Therefore, ensure that the /var directory is not set with the noexec option by doing the following steps:

Check the client log to see if the prefetch plug-in returned the exit code 126.

Run mount as the root user to check the mount option that is currently used:

[root@host ~]# mount
/dev/mapper/vg_data-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw) 
/dev/sda1 on /boot type ext4 (rw,nodev) 
/dev/mapper/vg_data-lv_var on /var type ext4 (rw,noexec,nosuid,nodev) 
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

If /var is set to noexec, you must take one of the following actions:

Remove the noexec mount option.
Move /var/opt/BESClient to a different partition, which is not noexec, and create a symbolic link to it in its place.
Run the Set the path for _BESClient_LinuxPatch_executable_directory Fixlet and specify an alternative directory to run the executable for patching. The directory path must be a valid, absolute path name. It can contain only alphanumeric characters, forward slashes, and underscores.

Null error when configuring BigFix Patch download plug-ins

When the BigFix server and the BigFix client on the BigFix server do not have the same version, users might encounter a null error. The error occurs because BigFix server 8.x and 9.x versions handle encryption differently. The version of the client on the BigFix server is used to determine the BigFix server version and it is assumed that the version is the same for the BigFix server and the client on the BigFix server.

Ensure that the version of the BigFix server and BigFix client on the BigFix server match to avoid null errors when configuring the download plug-in. At a minimum, the version must be on the same major version level, for example 8.x or 9.x.

EDR_DeploymentResults.txt reports package cannot be found

This is caused due to an outdated cache, and it needs the latest metadata. The cache gets cleared on its own after a certain period of time, but sometimes new content appears before the cache is erased automatically. You can manually clear the cache to see if the issue gets fixed.

The cache is located in the SCC download plugin folder. If you are using version 1.1.0.0 or higher of the SCC download plugin and would like to clear the cache on every execution, you can modify the plugin.ini file by adding the following lines: [Experimental] alwaysClearCache = 1