Troubleshooting the Domino OIDC provider

Here are some tips for configuring and maintaining your Domino OIDC provider.

Verifying configuration through statistics

The server console command sh stats Security.OIDC.Provider can be used to show the OIDC Provider's stats. In addition to the typical stats showing successful and failed requests, there is a set of statistics intended to help track down misconfigured clients and applications. If a new application is failing to work properly and you see stats under "Security.OIDC.Provider.Auth.Missing.*", "Security.OIDC.Provider.Auth.Invalid.*", "Security.OIDC.Provider.Token.Code.Missing.*", or "Security.OIDC.Provider.Token.Code.Invalid.*" you can use the stat to determine which idpcat field or client configuration setting should be checked.

If all clients are correctly configured, "Security.OIDC.Provider.Token.Code.Missing.PKCE" and "Security.OIDC.Provider.Token.Code.Invalid.PKCE" might be signs of a PKCE downgrade attack and should be investigated.

Enabling tracing

There are several notes.ini variables that can be enabled to help diagnose issues with the Domino OIDC Provider or client applications. In general, setting a variable to "1" should only result in tracing on errors, and higher settings log more and more information. We recommend against leaving tracing enabled above a level of "1" for long periods of time as it will have a negative performance impact.

DEBUG_OIDC_PROVIDER=(0,1,2,3,4,5) Set to "1" to detect errors; set to "3" to diagnose issues.
DEBUG_OIDC_CONFIG=(0,1,2,3,4) Traces loading configuration from idpcat. "1" will trace normal functioning, not just errors. "3" traces every field read.
DEBUG_OIDCP_SESSIONS=(0,1,2,3,4,5) Traces user login sessions and authorization grants. Setting to "1" will trace normal functioning, not just errors.
DEBUG_OIDC_VAULT=(0,1,2,3,4,5) Useful for tracing Notes Federated Login with OIDC and Nomad Federated Login with OIDC.

Configuration updates

Updates made to Registered OAuth Client documents in idpcat will be processed dynamically by OIDC Provider servers without the need to restart the Domino server or the HTTP server task. This includes registering new clients. We recommend editing idpcat.nsf on the primary server and then replicating idpcat.nsf from the primary server to each of the secondary servers. Since Registered OAuth Clients are indexed by client_id, changing the client_id for an existing client is equivalent to creating a new client. If a client is removed from idpcat or their client_id is changed, the old client_id will still be seen via "oidcp show clients" until the server is restarted.

Key rotation initiated using the oidcp keys new server console command on the primary server will automatically replicate key updates from the primary to the secondary OIDC Provider servers, as described in Key Rotation. If a secondary server is down when key rotation occurs, manual replication of idpcat.nsf might be needed.

Changes to OIDC Provider documents are not processed dynamically and will require a server restart to process. See the following section for details.

Adding a new Domino server to an existing OIDC provider

Since OIDC Provider documents are rarely changed after their initial setup and not updated dynamically, there is a recommended procedure for making changes to an established provider.

Before you begin: Run the oidcp show clients server console command on the primary server and note the current configuration.

  1. Update the configuration:
    1. On the primary server, open names.nsf and edit the Internet Site document for the OIDC provider to be extended and add the new Domino server to the Domino servers that host this site field. Save the document.
    2. On the primary server, open idpcat.nsf and open the document for the OIDC provider to be extended. Click the Edit button, the Refresh Internet Site Servers button, and then the Save & Close button.
    3. On the new secondary server, run the tell http quit server console command to prevent the server from attempting to prematurely process the updates.
    4. Restart the primary Domino server to update the Domino OIDC provider with its new configuration.
    5. Run the oidcp show clients server console command on the primary server and observe the changed configuration. Ensure that this is correct before continuing.
  2. Update the OIDC provider's keys:
    1. Run the oidcp show keys server console command on the primary server and pick a key of type "Ae". Remember its value from the "Alg" column (ES256, RS256, and so on). We will roll over this key in order to re-encrypt the keys saved in idpcat for the new set of servers.
    2. Run the oidcp keys new <alg> server console command on the primary server to create a new pending key. The process of creating a new pending key will rewrite all of the provider's keys to idpcat, encrypt them for the current set of Domino servers (which now includes the new server), and then replicate those changes to the secondary servers.
    3. Run the oidcp show keys server console command again to confirm that a new pending key with the correct algorithm is present.
  3. Launch the new provider server: Run load http on the new secondary server to read the new OIDC provider configuration and keys and launch the Domino OIDC provider on that server.
  4. Trust but verify:
    1. Run oidcp show clients on the secondary server and confirm that the information shown on the secondary server matches the information shown on the primary server. Note that if the primary is hosting more providers than the secondary, the secondary will only show a subset of the information from the primary.
    2. Run oidcp show keys on the secondary server and confirm that the same keys are present on both servers.

Adding custom claims

Some applications or Relying Parties might expect tokens to contain certain claims that Domino does not include by default. Administrators can add custom claims with fixed string values that will be added to all of the access and ID tokens issued by a given Domino server by using notes.ini variables as follows:

  • OIDC_PROVIDER_CUSTOM_CLAIMS=2
  • OIDC_PROVIDER_CUSTOM_CLAIM_NAME_1=tid
  • OIDC_PROVIDER_CUSTOM_CLAIM_VALUE_1=1234
  • OIDC_PROVIDER_CUSTOM_CLAIM_NAME_2=products
  • OIDC_PROVIDER_CUSTOM_CLAIM_VALUE_2=none

This example would add two custom claims, "tid":"1234" and "products":"none" to the access and ID tokens issued by the Domino server. If multiple Domino servers are hosting an OIDC Provider, the notes.ini variables should be set on all of them.

Excessively long lists of custom claims will be truncated. Creating custom fixed-value claims with the same names as claims that Domino adds to the token is not recommended, not supported, and will have adverse effects.

Notes, Reminders, and Limitations

  • Anonymous access must be disabled on the internet site serving the OIDC provider.
  • We highly recommend setting the notes.ini variable DominoSessionCookieUniqueNames=1 to prevent clustered Domino servers hosting the same website from over-writing each other's session cookies.
  • The Authorization Code flow with PKCE is the only authentication flow recommended by current security best practices, and as such it is the only flow supported by the Domino OIDC provider.
  • Beware of whitespace - configuring via cut and paste on some operating systems may result in leading or trailing whitespace in configuration fields. Since the OAuth and OIDC specs require exact byte-by-byte matches for security reasons, leading or trailing whitespace and capitalization issues may result in unexpected failures.
  • Removing a Domino server from an existing Domino OIDC provider can be done as follows:
    1. Do steps 1a through 2c in the preceding "Adding a new Domino server to an existing OIDC provider" section on this page.
    2. Manually replicate idpcat.nsf from the primary server to the server to be removed.
    3. Restart the server to be removed.

      The removal can be verified with the oidcp show clients server console command on the former secondary server.

  • Multiple OIDC providers can be managed through a single idpcat.nsf database.
  • If a single Domino server is hosting more than one OIDC rovider, it must be the primary server for all of its providers or a secondary server for each of its providers. Individual Domino servers may not be the primary server for one provider and a secondary server for another.
  • The Domino OIDC provider does not support redirect URIs that contain the "?" or "&" characters. Domino-based clients should enable Simplify redirect URL in their Trusted OIDC Provider documents to use /auth/protocol/oidc instead of /names.nsf?OIDCLogin for their redirect URL.
  • The Domino OIDC provider supports client_ids and client secrets up to 255 characters long composed of alphanumeric characters, plus "-" dash, "." dot, and "_" underscore.
  • In a clustered environment, token revocation is local to each Domino server in 14.5.
  • The Domino OIDC provider is only supported on 64-bit Windows and Linux servers.
  • If the oidcp show clients server console command outputs No OIDC Provider is configured for this web site, your OIDC provider's configuration was not loaded from the idpcat database. Some things to check:
    1. Internet Sites need to be enabled in the server document to be used. Under the Basics tab, set Load Internet configurations from Server\Internet Sites documents to Enabled.
    2. The idpcat database must be named idpcat.nsf and live in the server's data directory.
    3. Don't forget to restart the Domino server after changing Internet Site documents or OIDC Provider documents in idpcat.nsf.