Managing Downloads

BigFix uses several methods to ensure that downloads are efficient and make the best use of available bandwidth. Among other techniques, caching is used extensively by all the BigFix elements, including servers, relays, and clients.

When an action on a client runs a download file command, the existence of the file is checked first in the client local cache. If the client cannot find it locally, it requests the file from its parent, typically a relay. In turn, the relay checks its own cache. If it finds the file, it immediately sends it down to the requesting client. Otherwise, it passes the request up to its parent, which might be another relay and the process continues. Ultimately, a server retrieves the file from an internal server or the Internet, caches it, and then passes it back down the chain. After receiving the file, each relay in the chain caches it, and continues to forward it down to the original client, which also caches it.

If the agent runs the download now command while performing the action, the file is requested and collected from the URL specified in the action script.

Each cache retains the file until it runs out of space. At that point, the cache is purged of the least-recently used (LRU) files to provide more space. You can view the relay cache size and other relay information by activating the Analysis ID# 227 BES Relay Cache Information available from the BES Support Site. The default cache size is 1 GB, but you can change it by using the Task ID# 148 BES Relay/Server Setting: Download Cache Size, also from the BES Support site.

There might be situations that require files to be manually downloaded and cached, typically because such files are not publicly available, in which case you must download the files directly from the source. Review the Fixlet Description tab for more information about specific manual cache requirements. You can pre-populate the download cache by copying files to the download cache location __Download. You can also delete these files manually.

The caches are stored as sub-folders of the program folder, which is created by default at %PROGRAM FILES%\BigFix Enterprise on Windows systems, and /var/opt/BES Server on Linux systems. The server download cache is BES Server\wwwrootbes\bfmirror\downloads\sha1, and the client download cache is found at BES Client\__BESData\__Global\__Cache\Downloads.

As well as the download cache, relays maintain an action cache (also 1 GB) holding all the files needed for each Action, and clients maintain a Utility cache.

For information about troubleshooting relays, including bandwidth and downloading, see Relay Health.

The client collects the file by requesting it from the url listed in the action script in one of the following ways:

When the complete set of downloads can be computed by parsing the action script, the complete set of downloads is computed by the server. The agent can ask the relay with a single request if the prefetch downloads are available for a specific action. In this request, the agent sends up the action ID, and the server response indicates all the files are available, or they are not. If these are all available, the agent starts requesting the files by their ordinal number (1 indicates the first file in the script, 2 indicates the second file in the script, etc.). If the files are not available, the relay informs the agent they are not and begins the process of fetching them, and the agent notifies that it is waiting for downloads to be available and put itself into a pending downloads state for that action for 10 minutes, at which time it asks the relay again, if the downloads are available for the specific action.
When the downloads for an action become available on a relay, a notification is sent to the children of the relay, which uses the notification to accelerate requesting the downloads again. If notification messages are blocked for any reason, the agents 10 minute 'ask the relay again' behavior will eventually result in the agent detecting that the downloads are available, and begin to collect them. Child relays are also notified by their parent when the downloads based on the action ID and the ordinal numbers become available. They use this notification to accelerate their own request for the downloads again.
For downloads where any of the download url, size, and hash value are listed in the action script such that only the agents can compute them, the agents query their parent relay using an itemized downloads available request. The request contains a list of download items the particular agent needs. The relay and client behave the same way as described above, delaying subsequent requests, waiting for notifications

Resuming a download

If the download fails for connection problems, the download process is resumed as follow:

If the client is downloading from a BigFix Relay or Server, the download can be restarted at 10,000 byte chunks. This means that, when the client process is restarted, it verifies the 10,000 byte blocks already received, and then it resumes the download after the last verified block.
If the client is running a direct download from another server's URL, when the client process restarts, the download starts again from the beginning.

Downloading directly from the Internet site

In addition to the existing client settings for Download Direct, starting from Version 10 Patch 1, you can configure your clients to download specific resources directly from the site where they are located, to mitigate the network impact and bandwidth requirements for relays serving VPN-based clients.

You can specify that all resource requests to a specific set of domains must be downloaded directly from the Internet and not from the relay. Use the client setting named _BESClient_Download_Direct_Domainlist to specify the list of domains for which the direct download is desired.

Note: If the PeerNest is enabled, the behavior remains unchanged. The resource is still requested from the peer URL.

Note: This setting should be used as an alternative to using _BESClient_Download_Direct. The _BESClient_Download_Direct setting, in fact, will force all resources to be downloaded directly from the Internet, regardless of the domain specified.

If the direct download from the Internet fails, you can specify that the clients attempt to download the file from the BigFix Relay or Server. Use the client setting named _BESClient_Download_DirectRecovery to enable this behavior.

Note: The setting has effect only if the client settings _BESClient_Download_Direct or _BESClient_Download_Direct_Domainlist are enabled.

For more details about these settings, see Download.

Enable Direct Download based on network

Starting from Version 10 Patch 7, a new feature enables you to allow the Direct Download only for BigFix Clients connected to a specific subnet.

You can specify the list of subnets that allow the Direct Download with the new setting _BESClient_Download_Direct_SubnetList. The setting accepts only subnets specified in CIDR notation format, for example: 192.1.77.0/25;192.1.0.0/16.

In case of computers with multiple network interfaces, the subnet considered when checking the allowed list is the subnet of the IP Address connected to the BigFix Relay.

In case of direct downloads in progress, if the Client reregisters using a new IP address that does not belong to any of the subnets in the list, then the Client interrupts the ongoing download.

When the Client interrupts the direct download in progress, the following error is logged:

The direct download (Action <action_id>) was canceled after Relay Select: 
the address connected to the relay is changed.

Note: The _BESClient_Download_Direct_SubnetList setting provides added value to all situations in which a Direct Download is expected, for instance when the _BESClient_Download_Direct setting is enabled or the URL belongs to the _BESClient_Download_Direct_Domainlist setting.

For more details about this setting, see Download.

Restart download after Relay switch

Starting from Version 10 Patch 7, a new feature allows you to interrupt the download in progress on a Relay switch. By default, if a BigFix Client moves to a new relay while a download operation is in progress (from the former relay), the file download continues from the former relay, if that is still reachable by the Client.

Only if the former relay is no longer reachable, the download fails, and a new download is attempted from the new Relay.

Enabling the new setting named _BESClient_Download_ResetOnRelaySwitch allows you to stop the download from the former relay, even if it is still reachable and, then, restart the download from the new Relay.

In the BigFix Client log file, after the relay switch, when the Client interrupts the download in progress from the former Relay, the following error is logged:

The download from Relay (Action <action_id>) was canceled after Relay Select: 
the relay is changed.

For more details about this setting, see Download.

Automatic URL redirection from HTTP to HTTPS

In addition to the existing configuration settings for Download, starting from Version 10 Patch 8, the URL redirection from HTTP to HTTPS will be handled both on the Bigfix server/relay and on the BigFix client (direct download).

To support the download from an HTTPS URL, it will be necessary to provide appropriate trusted certificates to verify the remote server identity.

In case of a download from an HTTPS URL, the certificate of the remote server will be validated using the CA bundle distributed through the BES Support site. The CA bundle is the file that contains root and intermediate certificates of trusted authorities.

Before this new feature, the CA bundle was pre-installed on the BigFix server and already used for gathering purposes. With this new feature, the CA bundle will be distributed and kept up-to-date through the BES Support site for download purposes.

For more details about these settings, see Download.

For more details about customizing HTTPS for downloads, see Customizing HTTPS for downloads.

Relay Drive Space Protection From Downloads

Starting from Version 11, an optimization for the BigFix Relay was introduced.

To prevent the BigFix Relay ActiveDownloads folder from filling up, a new setting named _BESRelay_Download_ActiveDownloadsMaxSizeMB was created. This setting represents the maximum allowed size of the folder contents. For more information about this setting, see Download.

The ActiveDownloads folder is used by the BigFix Relay to store the contents it is downloading, before caching them for the other BigFix Clients.

Introducing this cap means that the BigFix Relay does not allow to download files that exceed the limit you set. Before start downloading a file, the BigFix Relay checks the ActiveDownloads free space, and, depending on the setting value, allows the download only if the file size does not exceed the remaining space.