Working with PeerNest

The BigFix Client includes a new feature named PeerNest, that allows to share binary files among Clients located in the same subnet. The feature is available starting from BigFix Version 9.5 Patch 11.

A practical use case is a branch office connected to the data center through a slow link: with earlier BigFix versions, the suggested configuration required a Relay to be installed in the branch office in order to download and cache large payloads.

With PeerNest, the BigFix Clients can share downloaded binaries and therefore reduce the number of communications going outside of the branch office even if a Relay is not installed locally. In this way, multiple Clients generate on the Relay the download load of a single Client, because only one Client downloads from the Relay and then shares the download with the peers.

Note: Up to BigFix 10.0.7, the PeerNest feature does not work if it is enabled on BigFix clients residing on a subnet hosting also clients belonging to a different BigFix deployment. Starting from BigFix 10.0.7, peer requests coming from clients belonging to different deployments are ignored and so this limitation is no longer applicable.

Use of PeerNest can help reducing the number of Relays in some complex BigFix deployment scenarios, therefore reducing infrastructural costs.

An introduction video can be found here: https://www.youtube.com/watch?v=tXRX3zlw1aQ.

Enabling PeerNest

To enable the PeerNest feature, set to 1 the following configuration setting on the Client:

_BESClient_PeerNest_Enabled = 1

The Client enables all the PeerNest feature in order to locally optimize the download of binaries.

This configuration setting requires a restart of the Client to be effective.

Note: If you require BigFix to optimize the download of the binaries required to execute actions, ensure that the hash of the file is specified inside the prefetch statement.

PeerNest configuration settings

For details about all available settings, see Peer to peer mode.

PeerNest in depth

When multiple Clients are executing actions requiring the prefetch of a binary file, they check with their peers if the file is already cached in the subnet. If the binary was not cached then the Clients can elect one of them as responsible for downloading from the Relay (Figure 1): the peer with the highest priority, among the peers requiring the file, will manage the download; if all peers have the same priority, the computer with the lowest ID will download the file from the Relay. By doing so, for a given action, all files will likely be downloaded from the Relay by the same computer, and so the downloads will be serialized (one file at a time), thus ensuring minimum bandwidth occupation in the link between Relay and computer.

The following figure shows in details the master election process:

A will be elected as master as it has the highest priority among the peers requiring the file (A, B, E).
C and D do not require the file, so even if D has higher priority than A, D is not involved in the master election process.

Figure 1: Master election

As soon as the elected master (A) starts downloading the file from the Relay (Figure 2) it notifies the other peers that there is a download in progress, so they wait for it without taking further actions. The master sends periodic notifications about the download.

Figure 2: Download in progress

If the peers did not receive the download in progress message (download failed, Client down, network issue), a new election process is started over and another peer becomes master.

When the elected master finishes the download, it moves the file into the PeerNest cache and notifies the other peers about its availability; the peers interested in the file then start downloading it from master and share it, using the same mechanism (Figure 3). In this way, multiple Clients generate on the Relay the download load of a single Client, because only one Client downloads from the Relay and then shares the download with the peers.

The following figure shows how the peers interested in the file (B, E) start downloading it directly from peer A, instead of downloading it from the Relay. When their download completes, the file is cached to be available for future usage: so the Clients A, B and E will share the downloaded file with C and D.

Figure 3: File sharing

Clients with peer enabled will start up an HTTP Server listening on port 52311 for peer transfers. Each Client can serve at the same time a maximum of _BESClient_PeerNest_MaxActiveFileDownloads other peers (default value is 5).

The Client priority comes also into play when there are 2 or more peers available to share the same file. The Client that wants to download the file creates a memory list of peers serving the file; it will pick up the peer randomly with weighted probability, based on priority: for instance, if the memory list is made up of two peers, say C1 with priority W, and C2 with priority 2W, picking C2 will be twice as likely as picking C1. In this case, the legacy retry-behavior applies – ruled by the _BESClient_Download_RetryMinutes and _BESClient_Download_RetryLimit Client settings – with the following addition:

The peer from which a failed download attempt has occurred gets removed from the memory peer list (unless that peer is the only one in memory), and so the next attempt will be done with another peer.
If the retry-count reaches the limit, the Client will go to download the file directly from the Relay.

If the Client fails to download the file because the peer that it has tried to connect to is already serving 5 downloads (default case), the retry-count is not increased and the peer is not removed from the memory peer list.

PeerNest behavior for files bigger than the PeerCache size

Starting from Version 10 Patch 2, a new feature allows time optimization for actions that require prefetching of large files. This feature ensures that the peers elected for downloads are those with real ability to store files rather than simply those with highest priority.

In fact, prior to Version 10.0.2, in a PeerNest configuration, when BigFix Clients are performing actions requiring the prefetch of a binary file, they check if the file is already cached in the subnet. If the binary file is not cached yet, the BigFix Clients can elect one of them as responsible for downloading from the BigFix Relay and then sharing the file with the peers. The elected peer is generally the one with the highest priority assigned. If the elected peer does not have enough cache to store the payload, the distribution fails, and all the other peers wait needlessly.

Starting from Version 10 Patch 2, with the _BESClient_PeerNest_UseNoSpaceDownload setting, you can configure the PeerNest to elect for downloads only the peers with a cache limit higher than the payload size. The same option forces the peers with high priority but not enough cache limit in passive mode. In this way, you avoid that the peers have a useless waiting and the file is downloaded sequentially. With this option, you also provide the peers with the ability to download the file directly from the BigFix Relay without waiting for the other peers that might eventually fail.

Excluding a list of subnets

Starting from Version 10 Patch 4, a new feature allows you to disable the PeerNest mechanism for BigFix Clients connected to a specific subnet.

The devices that use the PeerNest mechanism, send and receive messages using the UDP communication protocol. When you have many devices connected on the same subnet, they share network traffic among themselves. This might cause an excessive consumption of network resources. According to your specific needs, you might want to limit the UDP traffic on some specific network subnets. For example, if you have BigFix Clients running in a VPN infrastructure.

Starting from Version 10 Patch 4, with the _BESClient_PeerNest_ExcludedSubnetList setting, that contains a list of subnets (for example: 192.1.77.0/25;192.1.77.129/25), you can exclude from the PeerNest mechanism the Clients that belong to one of the subnets specified in the list. When a Client with PeerNest enabled (_BESClient_PeerNest_Enabled = 1) registers with the BigFix Relay, the new feature identifies the subnet of the Client and checks if its subnet belongs to this list. If yes, then the PeerNest mechanism is disabled for the Client.

Moreover, when you need to change the Client subnet, for example when you connect to a VPN, if the new subnet is already included into the Exclusion Subnet List, the P2P mechanism is automatically disabled like shown in the following figure.

Adding/modifying/deleting the value of the new setting requires a restart of the BigFix Client.

Best Practices

A good practice is to assign higher priorities to computers with a better link to the Relay and enough resources to serve the other peers of the subnet (and possibly a steady power supply).

PeerNest requires an increased disk storage space for caching files. The default PeerNest cache size is 2GB, which is enough for many scenarios; it must be increased in case of transfer of large files (patch management, software distribution, etc.) in order to fit the cache. The PeerNest cache is intended as a temporary storage, so usage and lifetime can be fine tuned with the following parameters: _BESClient_PeerNest_DownloadsCacheLimitMB, _BESClient_PeerNest_MinimumDiskFreeMB, _BESClient_PeerNest_MinimumCacheDays, _BESClient_PeerNest_MaximumCacheDays.

PeerNest requires UDP (port 52311) communication to be enabled, in order to allow the BigFix Clients to communicate with each other.

It also requires TCP (port 52311) to allow the BigFix Clients to download files from a peer, and the subnet supporting multicasting.

It is recommended to set PeerNest in passive mode (using the _BESClient_PeerNest_IsPassive configuration setting) on Clients that cannot open this port or do not want to use additional disk space for caching. Passive Clients will only download from the other peers but will not share content. The following figure shows an example of configuration.

Bandwidth Throttling

Clients with peer enabled will start up an HTTP Server; the other peers can connect to it for downloading.

Each Client serving files to the other peers can control the amount of bandwidth allocated for that purpose. The _BESClient_HTTPServer_ThrottleKBPS setting defines the total number of kilobytes that the Client gives to all of the peers combined per second (0 means no limit): if its value is 1000 KB/sec and there are 10 peers downloading simultaneously, the Client will send data to each at 100 KB/sec (for a total of 1000 KB/sec).

Troubleshooting scenario 1

On BigFix Clients hosted on an Operating System that has the Internet Protocol version 6 (IPv6) disabled or not configured:

If you want to use the PeerNest feature, you must:

Set on these Clients the _BESClient_Comm_IPCommunicationsMode configuration setting as follows:
```
_BESClient_Comm_IPCommunicationsMode = OnlyIpv4
```
Restart the Clients for the changes to take effect.

Troubleshooting scenario 2

On BigFix Clients that have an active polling set using the _BESClient_Comm_CommandPollEnable and _BESClient_Comm_CommandPollIntervalSeconds configuration settings:

If you want to use the PeerNest feature, you must not configure these Clients to be "passive" PeerNest agents. Do not enable on them the _BESClient_PeerNest_IsPassive configuration setting. Otherwise, depending on the timing of the polling, multiple Clients in a subnet can download the same binary, without sharing it.