CRL and OCSP mustgather

Things to check first

No client certificate provided

Verify that the client is providing an SSL certificate on the failing request by adding %{SSL_CLIENT_DN}e to your LogFormat directives.

OCSP revocation is undetermined?

One reason for an undetermined revocation status over OCSP, on a seemingly valid OCSP response, is the complicated trust model of who may sign the OCSP response for a given certificate.

  • If the signer of the OCSP response is the same certificate as the issuer of the certificate being checked, no extra setup/checking is performed.

  • If the signer of the OCSP response is issued by the same issuer as the certificate being checked, no no extra setup/checking is performed.

  • In other cases, such as when the signer of the OCSP response is unrelated or only indirectly related to the certificate chain of the certificate being checked, the signer of the OCSP response must be present in the *.kdb (not just the root CA that signed the OCSP responder)

CRL uploaded under wrong baseDN

Prior to GSKit version 7.0.3.17, the only place a CRL is searched for is on the configured SSLCRLHostname with a "base" equal to the DN of the issuer of the certificate being checked.

LDIF created from DER encoded CRL

By default openssl will create PEM (base64 encoded text with header and footer) CRLS, but if you feed these into LDIF you end up with your data being base64 encoded twice. If you're creating an LDIF to load your LDAP system, make sure the CRL is in the binary DER format.

An example LDIF:

dn: cn=testCA,o=myorg,c=US
objectclass: cRLDistributionPoint
objectclass: pkiCA
objectclass: entrustNamedObject
caCertificate;binary:< file:///tmp/CA.der
certificateRevocationList;binary:< file:///tmp/crl.der

Where /tmp/crl.der is a DER encoded CRL and can be processed by commands such as openssl crl -inform DER -in /tmp/crl.der -text

LDAP user not authorized to download CRL

  • If you've statically configured and loaded a CRL repository

    • Verify a valid SSLCRLUserID value if the LDAP server doesn't allow anonymous binds.

    • Verify the crl option was used on the sslstash call

    • Verify the stashed password along with the userid specified in SSLCRLUserID can perform the following ldapsearch and get a non-empty temporary file created: ldapsearch -t -D cn=YOURUSERID -L -w YOURPASSWORD -b "cn=YOURISSUERINFO" "(certificateRevocationList=*)" certificateRevocationList\;binary::

    • Verify the sslstash file was not transferred from another system in non binary fashion

  • If the CRL is published in a certificate under a CRLDistributionPoint extension:

    • The server listed must support anonymous queries, SSLCRLUserID and the stashed password are NOT used to query a CRL listed in a client certificate

IHS cannot access the CRL distribution point

If the signing certificate specifies a CRLDistributionPoint extension and SSLCRLHostName is configured, then IHS will always attempt to contact the CRL distribution point specified in the signing certificate, and if it cannot access it, will report that it is unable to verify that the certificate is not revoked, even if it can access the SSLCRLHostname LDAP server.

If the customer configuration is such that IHS will not be able to contact the CRL distribution point, we recommend instead using an internal OCSP server:

  1. Remove SSLCRLHostname, SSLCRLPort, SSLCRLUserID from configuration. This will stop IHS from trying to access the CRL distribution point configured in the signing certificate.

  2. Set up an OCSP server that IHS can access, and load it with the certificate revocation information. (IHS does not recommend any particular OCSP server implementation.)

  3. Configure IHS to use OCSP to validate certificates by setting SSLOCSPResponderURL to the URL used to access the OCSP Server.

  4. Important: do NOT set SSLOCSPEnable. SSLOCSPEnable might cause IHS to try again to access certificate revocation information on the internet.

  5. Set up a periodic process to retrieve current CRL information and import it to the OCSP server.

Miscellaneous Configuration Issues

  • Verify The SSLClientAuth directive specifies the crl option.

  • Verify the 0/none option for SSLClientAuth is not specified.

SSLOCSPResponderURL does not point to a valid responder

Confirm with some other piece of software that your responder is functioning correctly

OCSP responder is ignored

Make sure GSKit 7.0.4.11 or higher is installed, with some levels after 7.0.3.27 but before 7.0.4.11 OCSP is not processed correctly.

Confirm that CRL is not expired (Next Update: field)

$ openssl crl -in crl.pem -text|grep "Next Update"

Next Update: Jun 23 10:58:45 2009 GMT

Crashes with CRL enabled bfeore 7.0.4.21

GSKit before 7.0.4.21 can crash with CRL support enabled. Usually affected systems will crash 100% of the time during the first CRL lookup, potentially after the CRL hostname has been changed.

Memory leak with CRL enabled before 7.0.4.11

GSKit before 7.0.4.11 suffers from a memory leak during CRL processing

CRL/OCSP FAQs

CRLDistributionPoint support

For GSKit version 7.0.3.17 and later, If an issuing certificate contains the CRLDistributionPoint extension (CDP) it will be searched for a CRL before the server specified by SSLCRLHostname. The userid and password specified via SSLCRLUserID and SSLCRLStashFile are not used when connecting to servers specified this way in the certificate chain.

If the certificate contains a CRLDistributionPoint extension (CDP) that information is given precedence over the statically configured LDAP server (SSLCRLHostname). The order in which the information is used is described in the FAQ topic: 'What is the precedence order for using the various mechanisms for obtaining certificate revocation information?'

If there is no statically configured LDAP server with CRL information, SSLCRLHostname can be specified with the special value URI. No other SSLCRL* directives need to be specified.

Checking certificate for CRL Distribution Point (CDP) extensions

You can use a tool like ikeyman or openssl to display the CDP or Issuer name from the signer certificate:

openssl x509 -in signer.crt -noout -text

Is a statically configured LDAP server with CRL info really necessary?

It was originally necessary. However, it is now possible to specify the special value of "URI" as the parameter for SSLCRLHostname to skip checking of the statically configured LDAP server, but still check CRL Distribution Point in the certificate when present.

CRL via HTTP CDP size limits

CRL over HTTP is limited to 200KB by default. The size can be overridden, in bytes, by environment variable GSK_HTTP_CDP_MAX_RESPONSE_SIZE

Does GSKit cache CRL's?

GSKit doesn't cache CRL's by default. The internal GSKit CRL cache can be activated by setting native environment varibales in $IHSROOT/bin/envvars such as export GSK_CRL_CACHE_SIZE=32 (number of entries).

The CRL cache only caches statically configured LDAP CRL's and HTTP forms of CRLDistributionPoint (the latter in V8 and later). The CRL cache is deprecated in GSKit v8 and has never been exposed by IHS directly.

CRL's can be very large.

What happens if GSKit cannot download a CRL or get a valid OCSP response?

If cached revocation data is available, GSKit will use it. If the certificate was revoked using the cached data, the handshake will fail. If it was not revoked in the cached data, the handhsake will succeed and the IHS directive SSLUnknownRevocationStatus comes into play. GSKit caches CRL data for the smaller of 12 hours or the "next update time" encoded in the CRL itself.

Does GSKit check for an updated CRL before the "next update time" has been reached if it has a CRL cached in memory?

Yes, GSKit also expires cached CRL's every 12 hours, in addition to the "next update time" encoded in the CRL itself

What is the precedence order for using the various mechanisms for obtaining certificate revocation information?

The revocation information will be obtained with the following precedence (assuming each is defined and enabled), processing stops at the first source which has a non-expired response:

  1. OCSP responder

    • Enabled and defined by SSLOCSPResponderURL directive

  2. OCSP responder

    • Enabled with SSLOCSPEnable.

    • Defined in Authority Information Access (AIA) certificate extension.

  3. CLRDistributionPoint LDAP X.500 name

    • Enabled as a side effect of SSLCRLHostname, SSLCRLPort.

    • Defined in the CRL DistributionPoint (CDP) certificate extension. Not used when SSLCRLHostname = "URI"

    • If an LDAP server is configured, and processing gets this far, and the LDAP server is down, the handshake will fail.

  4. CLRDistributionPoint LDAP URI

    • Enabled as a side effect of SSLCRLHostname, SSLCRLPort.

    • Defined in the CRLDistributionPoint (CDP) certificate extension.

    • (For GSKit 7.0.3.31 and later, file:// and http:// CDP's are also followed)

  5. CRL database on LDAP Server

    • Enabled and defined by SSLCRLHostname, SSLCRLPort, SSLCRLUserID, and SSLCRLStashFile directives. Not used when SSLCRLHostname = "URI"
      Path in LDAP determined by the DN of the certificate issuer certificate.

    • If an LDAP server is configured, and processing gets this far, and the LDAP server is down, the handshake will fail.

When does GSKit contact the LDAP server (specified by SSLCRLHostname) when CRL is configured?

  • The host specified by SSLCRLHostname is checked for connectivity at startup, once per SSL virtualhost it is configured in.

The following server contacts occur if there is no CRLDistributionPoint defined in the cert, or if a CRLDistributionPoint is defined but can't be accessed, or it returns out-of-date information. They do not occur if a CRLDistributionPoint is defined in the issuer cert and GSKit recieves unexpired data from it:

  • Once per issuer, per SSL virtualhost, per IHS child process during an SSL handshake to retrieve and cache the CRL.

  • Whenever a handshake is performed and the latest CRL previously fetched (cached) for this issuer, in this SSL virtualhost, and in this process has reached its internal "next update time" or the cached CRL is 12 hours or more old.

Can the CRL cache be viewed or queried?

No, it exists only in memory and is local to each IHS child process.

Can the CRL cache be updated when IHS isn't running, or at startup?

No, only when a certificate is being checked for an actual handshake.

Can the CRL cache be cleared?

Since the cache lives independently in memory in each IHS child process, a restart of IHS or graceful restart of IHS will clear the CRL cache.

Why does my testing contradict the behavior described here?

When IHS is configured to consult LDAP for a CRL, it will also follow CRLDistributionPoint URL's in the certificate chain itself, so this other source of revocation info is being considered (before CRL in the hard-coded LDAP)

Should I use CRL or OCSP?

OCSP is strongly recommended over CRL, as the loading and parsing of large CRL's can become a source of delay. It is normally administratively easier to frontend your legacy CRL with an OCSP server rather than an LDAP server. OCSP was introduced as the successor to CRL-over-LDAP.

Is there an OCSP cache?

IBM HTTP Server does not use an OCSP cache

MustGather

  • System and web server information

  • GSKit Trace gathered from IHS startup through a CRL validation failure. Should be recreated immediately after IHS startup, to avoid GSKit trace wrap overwriting any traffic before the failure that might have been cached by GSKit. Then copy the GSKit trace to a new file immediately after recreating the problem.

  • IP trace between IHS and LDAP during a CRL validation failure

  • Source CRL, any intermediate files used in creating LDIF or loading LDAP, description of process used to load LDAP.

  • KeyFile for IHS system along with the matching .sth file and KDB password

  • Full certificate chain for Client Certificate in question, along with Signer Certificate.

  • ErrorLog during failing CRL connection where LDAP_DEBUG=65535 has been set in the environment, LogLevel has been set to debug, and SSLTrace is enabled

  • Determine the baseDN for your CRL search (Issuer Name or explicit CRL Distribution Point extension in signer cert, or observe via IP trace) and send the temporary output files created by the following command ldapsearch -t -D cn=YOURUSERID -L -w YOURPASSWORD -b "cn=YOURISSUERINFO" "(certificateRevocationList=*)" certificateRevocationList\;binary::

  • Add the following string to your LogFormat directives to capture the user-agent and CN of the client certificate:

     SSL_CLIENT_DN=\"%{SSL_CLIENT_DN}e\" UA\"%{User-Agent}i\"

  • If OCSP is being used, an IP trace of the communication between IHS and the responder

  • The recommended minimum level of GSKit for CRL processing is 7.0.4.27