Enabling FIPS compliance on an automated server installation

About this task

Enable During Server Installation
To enable FIPS compliance on the BigFix® Remote Control Server, run the Remote Control Server Installer and select Enable FIPS and Enable NIST SP800-131A in the Web server parameters panel.
Enable Manually
To enable FIPS compliance can be configured manually on the BigFix® Remote Control Server instead of running the Remote Control Server Installer by following this procedure:

Procedure

  1. Edit the java.security file that is found at the following directory.
    Windows® systems
    %TRC_SERVER_PATH%\java\jre\lib\security\java.security

    Where %TRC_SERVER_PATH% is the path for the installation directory for the BigFix® Remote Control Server.

    Linux® / UNIX® systems
    $TRC_SERVER_PATH/java/jre/lib/security/java.security

    Where $TRC_SERVER_PATH is the path for the installation directory for the BigFix® Remote Control Server.

  2. Modify the security.provider.x= list so the following entry is the first one in the list:

    security.provider.1=com.ibm.crypto.FIPS.provider.IBMJCEFIPS

    Fix the number sequence of the other items in this list so that all items are numbered in sequence.

    For example:
    • The full list after the changes when performed on a Remote Control server build 10.0.0.0808 or later is as follows: 
      security.provider.1=com.ibm.crypto.plus.provider.IBMJCEPlusFIPS
security.provider.2=com.ibm.jsse2.IBMJSSEProvider2
security.provider.3=com.ibm.crypto.plus.provider.IBMJCEPlus
security.provider.4=com.ibm.crypto.provider.IBMJCE
security.provider.5=com.ibm.security.jgss.IBMJGSSProvider
security.provider.6=com.ibm.security.cert.IBMCertPath
security.provider.7=com.ibm.security.sasl.IBMSASL
security.provider.8=com.ibm.xml.crypto.IBMXMLCryptoProvider
security.provider.9=com.ibm.xml.enc.IBMXMLEncProvider
security.provider.10=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
security.provider.11=sun.security.provider.Sun
    • The full list after the changes when performed on a Remote Control server build 10.0.0.0807 or earlier is as follows: 
      security.provider.1=com.ibm.fips.jsse.IBMJSSEFIPS
security.provider.2=com.ibm.crypto.fips.provider.IBMJCEFIPS
security.provider.3=com.ibm.jsse2.IBMJSSEProvider2
security.provider.4=com.ibm.crypto.provider.IBMJCE
security.provider.5=com.ibm.security.jgss.IBMJGSSProvider
security.provider.6=com.ibm.security.cert.IBMCertPath
security.provider.7=com.ibm.security.sasl.IBMSASL
security.provider.8=com.ibm.xml.crypto.IBMXMLCryptoProvider
security.provider.9=com.ibm.xml.enc.IBMXMLEncProvider
security.provider.10=com.ibm.security.jgss.mech.spnego.IBMSPNEGO
security.provider.11=sun.security.provider.Sun
  3. Add the following lines: 
    ssl.SocketFactory.provider=com.ibm.jsse2.SSLSocketFactoryImpl
ssl.ServerSocketFactory.provider=com.ibm.jsse2.SSLServerSocketFactoryImpl
  4. Save the file.
  5. Edit the jvm.options that are found in the following directory.
    Windows® systems
    %TRC_SERVER_PATH% \wlp\usr\servers\trcserver\jvm.options

    Where %TRC_SERVER_PATH% is the path for the installation directory for the BigFix® Remote Control Server.

    Linux® / UNIX® systems
    $TRC_SERVER_PATH/wlp/usr/servers/trcserver/jvm.options

    Where $TRC_SERVER_PATH is the path for the installation directory for the BigFix® Remote Control Server.

  6. Add the following lines:
    • Remote Control server build 10.0.0.0808 or later
      -Dcom.ibm.jsse2.usefipsprovider=true
-Dcom.ibm.jsse2.sp800-131=strict
-Dcom.ibm.jsse2.overrideDefaultTLS=true 
-Dcom.ibm.jsse2.usefipsProviderName=IBMJCEPlusFIPS
    • Remote Control server build 10.0.0.0807 or earlier 
      -Dcom.ibm.jsse2.usefipsprovider=true
-Dcom.ibm.jsse2.sp800-131=strict
-Dcom.ibm.jsse2.overrideDefaultTLS=true
  7. Save the file.
  8. Log on to the BigFix® Remote Control Server with a valid admin ID and password.
  9. Click Admin > Edit properties files
  10. In the common.properties file set FIPS.compliance to true.
  11. Click Submit.
  12. Click Admin > Reset Application. Restart the server service.
  13. Restart the server service.

Results

Check to see whether the BigFix® Remote Control Server is configured for FIPS by completing the following step.

  • Click Admin > View Current Server Status.

The following fields show that FIPS compliance is enabled.

  • Enabled FIPS mode: The value of this field is determined by the FIPS.compliance property in the common.properties file.
  • JVM configured for FIPS: The value of this field is determined by the configuration of the JVM and the security providers that are listed in the java.security file.
Troubleshooting:
  • Browser or Controller connection with the Remote Control server may fail

    If the server is configured to operate in FIPS mode with the following exception in the messages.log file java.lang.NullPointerException com.ibm.ws.channel.ssl.internal.SSLConnectionLink 238, Browser or Controller connection with the Remote Control server might fail.

    This is a side effect of the adoption of IBM Java 8.0.6.26. If the issue persists, do the following:
    1. Stop the Remote Control Server.
    2. Open the ..\TRC\java\jre\lib\security\java.security file and add the RSAPSS value as the last entry of the jdk.tls.disabledAlgorithms.

      The updated property list must look as follows:

      jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, DESede, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, DES_CBC, RSAPSS
    3. Start the Remote Control Server.
  • The Join Broker Session operation may fail
    When the secondary controller connects to the primary controller and the environment is configured to operate in FIPS mode, the Join Broker Session operation may fail. The primary controller may show an exception like the foll following exception in the messages.log file:
    SEVERE - The connection was refused with pkt type [260]

    This is a side effect of the adoption of IBM Java 8.0.6.26. If the issue persists, do the following:

    Open the ..\Controller\jre\lib\security\java.security file and add the RSAPSS value as the last entry of the jdk.tls.disabledAlgorithms. The updated property list should look as follows: 
    jdk.tls.disabledAlgorithms=SSLv3, RC4, DES, MD5withRSA, DH keySize < 1024, DESede, \ EC keySize < 224, 3DES_EDE_CBC, anon, NULL, DES_CBC, RSAPSS

  • The playback of a session recording via the Server Web Interface may fail to start with no error message shown if the Server is Configured to operate in FIPS mode

    This is a side effect of the adoption of IBM Java 8.0.6.26. If the issue persists, do the following:
    1. Save the TRCPlayer.trcjws file as provided by the server.
    2. Edit the File and remove the line <argument>--forcefips</argument>
    3. Save the file and execute the TRCPlayer.trcjws file by clicking on it.