Securing SMTP sessions using the STARTTLS extension

SMTP sessions conducted over a standard TCP/IP channel are vulnerable to eavesdropping because the unencoded transmission can be easily intercepted. To protect SMTP communications, servers can use transport-layer security (TLS), more commonly known as TLS encryption, to provide privacy and authentication.

Some servers support TLS for SMTP communications by sending and receiving SMTP traffic through the TLS port (port 465 by default) only. However, because this requires that both the sending and receiving servers support SMTP over TLS, this solution isn't always practical.

To provide TLS security for SMTP transfers over TCP/IP, Domino® supports the use of negotiated TLS. In a negotiated TLS scheme, the sending and receiving hosts each use the SMTP STARTTLS extension, defined in RFC 2487, to signal their readiness to negotiate an TLS connection. The receiving server displays the STARTTLS keyword in response to the sending server's EHLO command. The sending server issues the STARTTLS command to request the creation of a secure connection. After the initial TLS handshake completes successfully, the two parties proceed to set up an TLS channel between them. Both the sending and receiving server must possess TLS certificates.

Supporting STARTTLS for outbound SMTP sessions

A Domino® server configured to use negotiated TLS for outbound mail connects to the receiving server's SMTP TCP/IP port (port 25 by default). If the initial SMTP response from the receiving server indicates that it supports the STARTTLS extension, Domino® issues the STARTTLS command to request the use of TLS to encrypt the rest of the session.

If the receiving server did not advertise support for STARTTLS in response to the Domino® server's EHLO command, the sending Domino® server continues with an unencrypted SMTP TCP/IP session.

To enable outbound STARTTLS support, set the SMTP outbound TCP/IP port status to: Negotiated TLS.

Supporting STARTTLS for inbound SMTP sessions

You can configure Domino® to support the STARTTLS command for inbound SMTP transactions. When a Domino® SMTP server is set to use negotiated TLS for inbound sessions, the server advertises support for STARTTLS in response to EHLO commands the TCP/IP port receives from connecting hosts. A connecting host can then issue the STARTTLS command to request an encrypted session.

If Domino® is configured to require STARTTLS for SMTP sessions over TCP/IP and a connecting host cannot meet this demand, no mail is sent over the connection.

To enable inbound STARTTLS support:

  • Enable the SMTP listener task.
  • Enable the SMTP inbound TCP/IP port.
  • Enable the STARTTLS ESMTP extension. This causes Domino® to advertise STARTTLS as one of its supported extensions in the ESMTP EHLO greeting response.
  • (Optional) Enable name-and-password authentication for the TLS port. Although SMTP sessions that use negotiated TLS are conducted over the Domino® TCP/IP port, Domino® uses the authentication options you set for the server's TLS port to determine how to handle name-and-password arguments.

Requiring name and password authentication for SMTP STARTTLS sessions

Enabling ESMTP support for negotiated TLS allows a server to accept requests to use TLS over TCP/IP from remote servers that connect anonymously. However, not all inbound connections are anonymous. A connecting SMTP server may be configured to send Domino® a name and password by means of the ESMTP AUTH command.

To support connections from SMTP clients that send a name and password during a negotiated TLS session, set the value of the Name & password field for the SMTP inbound TLS port to Yes. You do not have to enable the TLS port. If the TLS port does not support name-and-password authentication, the Domino® SMTP server rejects the AUTH command from the remote server and returns an error indicating that the command is not implemented.

Even though Domino® receives the AUTH command over the TCP/IP port, Domino® uses the TLS name-and-password authentication settings to determine whether to accept the AUTH request because it receives the command in the context of an TLS session. The Name & password authentication setting for the TCP/IP port is ignored.