Troubleshooting

When problems occur when patching CentOS endpoints, review the log files to determine what went wrong and how to correct the errors.

Log files

Enhanced logging with clearer error reporting and error handling to improve troubleshooting.

CentOSPluginR2.log

Lists the results of the downloads related to the execution of the CentOS Download Plug-in R2. The amount of information depends on the logging level.

The log can be found in the following locations:

On Windows systems: %PROGRAM FILES%\BigFix Enterprise\BES Server\DownloadPlugins\CentOSR2Protocol
On Linux systems: /var/opt/BESServer/DownloadPlugins/CentOSR2Protocol

The following log files can be found in the client folder in the directory /var/opt/BESClient/EDRDeployData.

EDR_DeploymentResults.txt: Lists the results of the EDR deployment and the YUM output.

register-repo.log: Lists the results of the repository registration action of the CentOS Custom Repository Management dashboard.

unregister-repo.log: Lists the results of the unregister repository action of the CentOS Custom Repository Management dashboard.
yum_history.log: Lists the results from the YUM Transaction History dashboard.

Other useful log files:

yum.log: This is the official log that YUM generates by default in /var/log/yum.log. It lists all the YUM-related operations and transactions.

Download plug-in logging levels

The logging level determines the amount of detail that the CentOS Download Plug-in R2 writes to the log files. Set the logging level in the %PROGRAM FILES%\BigFix Enterprise\BES Server\DownloadPlugins\CentOSR2Protocol\plugin.ini file.

Note: Logging level values are case-sensitive.

The following logging levels are listed in order of increasing amount of information logged:

ERROR: Contains errors related to the execution of the download plug-in, which might indicate an impending fatal error.
WARNING: Contains information about failed downloads, and reasons for failure.
INFO: Contains general information outlining the progress and successful downloads, with minimal tracing information.
DEBUG: Contains fine-grained information used for troubleshooting issues. This is the most verbose level available.

Note: Setting the logging level to DEBUG increases the amount of information to log, which might have an impact on performance. You must only increase the logging level to DEBUG when investigating an issue.

Enable the client debug log

Use the Configure the client setting to enable the Debug Log for Linux Patching Fixlet (ID #57) from the Patching Support site.

Prefetch plug-in error

If you have taken an action on a Fixlet that failed on a line with execute prefetch plug-in in the Actionscript. Subsequent calls to the same prefetch plug-in from any Actionscript might fail on the endpoint. The script might have been blacklisted, causing the prefetch plug-in error.

To verify, check the client log. You will see either of the following messages for the Fixlet action executing the prefetch plug-in:

execute prefetch plug-in' didn't complete within 300 seconds. Black listing plug-ins 
matching the sha1 hash of 'name of 'bash' until agent is restarted.

Execute prefetch plug-in attempting to reuse plug-in which took too long earlier.

To resolve this issue, do the following actions:

Restart the BigFix client to clear the blacklist.
Set the _BESClient_ActionManager_PrefetchPlugInTimeoutSeconds client configuration setting with sufficient time for the patch to install and resolve dependencies. This client setting indicates how long the client should wait before blacklisting the script. You can use the Change Timeout for Prefetch Plugins task, available from the Patching Support site, to set the setting to 30 minutes (1800 seconds).

Note: The _BESClient_ActionManager_PrefetchPlugInTimeoutSeconds setting varies based on the endpoint and the Fixlet being installed. To get the desired value, take the slowest endpoint and increase the setting to a high number, such as 3,000 seconds, then run a large Fixlet and see how long it takes. You can then take that number and multiple it by two. Alternatively, set the client setting to 600 seconds and adjust it accordingly if the suggested value does not work for you.

Error when /var is mounted as noexec

All available Fixlets use an executable that by default runs directly from the /var directory, a partition on the endpoint. The Fixlets will not work when /var is set with the noexec option, regardless of whether the CentOS Download Plug-in R2 or Custom Repository solution is used. Therefore, ensure that the /var directory is not set with the noexec option by doing the following steps:

Check the client log to see if the prefetch plug-in returned the exit code 126. For example:

Run mount as the root user to check the mount option that is currently used:

[root@host ~]# mount
/dev/mapper/vg_data-lv_root on / type ext4 (rw)
proc on /proc type proc (rw)
sysfs on /sys type sysfs (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620)
tmpfs on /dev/shm type tmpfs (rw) 
/dev/sda1 on /boot type ext4 (rw,nodev) 
/dev/mapper/vg_data-lv_var on /var type ext4 (rw,noexec,nosuid,nodev) 
none on /proc/sys/fs/binfmt_misc type binfmt_misc (rw)

If /var is set to noexec, you must take one of the following actions:

Remove the noexec mount option.
Move /var/opt/BESClient to a different partition, which is not noexec, and create a symbolic link to it in its place.
Run the Set the path for _BESClient_LinuxPatch_executable_directory Fixlet and specify an alternative directory to run the executable for patching. The directory path must be a valid, absolute path name. It can contain only alphanumeric characters, forward slashes, and underscores.

Missing GPG key

Deploy the Import RPM-GPG-KEY-centos-release task (ID# 301), available in the Fixlet site, before deploying any of the patch Fixlets to avoid issues during deployment.

Repository metadata is too large

The repository metadata that is used in patching is provided by the vendor and can be large. If the /var directory does not have enough space to store the metadata, set an alternative directory with sufficient space to store the metadata on the endpoint. You can use the Set the path for _BESClient_LinuxPatch_metadata_directory to set the directory where the repository metadata will be created.

Null error when configuring BigFix Patch download plug-ins

When the BigFix server and the BigFix client on the BigFix server do not have the same version, users might encounter a null error. The error occurs because BigFix server 8.x and 9.x versions handle encryption differently. The version of the client on the BigFix server is used to determine the BigFix server version and it is assumed that the version is the same for the BigFix server and the client on the BigFix server.

Ensure that the version of the BigFix server and BigFix client on the BigFix server match to avoid null errors when configuring the download plug-in. At a minimum, the version must be on the same major version level, for example 8.x or 9.x.