LDAP mustgather

Known issues to check for first

LDAP SSL failures after upgrading underlying Z hardware to z14

PI90141 is required to fix startup hangs on z14 systems. In some cases it has also resolved LDAP SSL compatibility issues on z14 too.

LDAP SSL failures after moving to 9.0/8.5.5.14

By default, the LDAP client in IHS 9.0 and 8.5.5.14 (and later, on AIX, Linux, and Solaris) will try to negotiate TLSv1.2 with the server. Some servers might be intolerant of this TLS1.2 handshake. TLS1.2 on the outbound LDAP connection can be disabled by setting the following environment variables in $IHSROOT/bin/envvars:

#Omit TLSv1.2
export LDAP_OPT_SECURITY_PROTOCOL=SSLV3,TLS10,TLS11

Handshake failures on zOS

On zOS, the LDAP library does not use TLSv12 or later by default. You can opt-in to using TLSv12 with a mixture of LDAP and System SSL environment variables. For mod_ibm_ssl, the System SSL environment variables will be overridden by the server configuration for non-LDAP SSL traffic. Append the following to $IHSROOT/bin/envvars:

export GSK_PROTOCOL_TLSV1_2=1
export LDAP_SSL_CIPHER_FORMAT=CHAR4
# https://www.ibm.com/docs/en/zos/2.5.0?topic=programming-cipher-suite-definitions#csdcwh__tttcsd
# IHS defaults as of 9.0.5.16, translated to the format required by GSK_V3_CIPHER_SPECS_EXPANDED
export GSK_V3_CIPHER_SPECS_EXPANDED="C02CC02BC024C023C00AC009C030C02FC028C027C014C013"

# Change the value on the next line to 1 to enable the TLSv13 snippet below.
USE_TLSV13=0

# Only change the following to fine-tune TLSv13 related settings
if [ "$USE_TLSV13" -eq 1 ]; then
  export GSK_PROTOCOL_TLSV1_3=1

  # https://www.ibm.com/docs/en/zos/2.4.0?topic=programming-cipher-suite-definitions#csdcwh__telcsd
  export GSK_V3_CIPHER_SPECS_EXPANDED="13011302${GSK_V3_CIPHER_SPECS_EXPANDED}"
  # https://www.ibm.com/docs/en/zos/2.4.0?topic=programming-cipher-suite-definitions#csdcwh__tttcsd
  export GSK_CLIENT_TLS_KEY_SHARES="00230029"
  # https://www.ibm.com/docs/en/zos/2.4.0?topic=programming-cipher-suite-definitions#csdcwh__tttcsd
  export GSK_CLIENT_ECURVE_LIST="00230024002500290030"
  # https://www.ibm.com/docs/en/zos/2.4.0?topic=programming-cipher-suite-definitions#csdcwh__sign2and3
  # Note: leading 0201 is SHA1-RSA
  export GSK_TLS_SIG_ALG_PAIRS="02010601050104010301080608050804"
fi

"Decoding Error" during authorization on z/OS

On z/OS V2R1 and earlier, some search/compare operations can return a "Decoding Error" if the server returns multiple responses to the same LDAP query. This does not occur on z/OS V2R2 or later.

Delays with require ldap-group and large groups

Searching for nested groups in large groups is inefficient due to the common structure of nested groups in LDAP (both users and subgroups are 'members' of the parent group). If your target group only has direct members, set AuthLDAPMaxSubGroupDepth to zero.

mod_ibm_ldap BasicIfNoCert prompts for basic auth with 401 even with a cert present

In 6.1 and earlier, it may be necessary to change the order of the LoadModule for mod_ibm_ldap relative to the LoadModule for mod_auth. Usually, having mod_ibm_ldap earlier is what's desired (in which case it can check the certificate before mod_auth triggers the 401 response).

z/OS LDAP server-specific issues

When an LDAP server on z/OS is configured to use the SDBM (or RACF) backend, the LDAP functionality is greatly diminished. This limits the ability of IBM HTTP Server to use such an LDAP server for access control.

  • mod_ibm_ldap can successfully authenticate, but authorization will generally not work (checking of groups or attributes)

  • mod_ldap/mod_authnz_ldap can perform both authentication and basic authorization (see APAR PK81733).

Performance issues (excessive traffic to the ldap server, slow ldap response, entries not being cached)

The mod_ldap module has a number of directives that can be used for performance tuning. The mod_ldap module and its directives are described at: http://publib.boulder.ibm.com/httpserv/manual70/mod/mod_ldap.html.

If you are seeing symptoms such as the following, then you may need to tune your ldap cache:

  • excessive traffic to the ldap server

  • slow ldap response

  • expected entries missing from the cache

  • "util_ldap.c: Cache insertion failure" messages in the error.log with LogLevel debug

If the system supports shared memory, then you should try increasing the size of the shared memory cache. Without configuration the default varies depending on the maintenance level. Prior to PI93624, the default was 100KB. After PI93624, and in IHS 9.0, the default is 4MB. You can increase the size using the LDAPSharedCacheSize directive. For example, to increase the size from the to 10M, use the following in your confiuration file (the size is in bytes):

LDAPSharedCacheSize 10000000

The size of the cache should be set based on the number of entries you expect to be added to the cache. The more logins that are cached, the larger you will want to make the size.
To help make this determination, you can enable the ldap-status handler as instructed by the 'Monitoring the Cache' section of the mod_ldap documentation page.
Once enabled, the cache status can then be viewed by using a browser to access http://<servername>/cache-info
You should do some initial monitoring to determine how many entries are being cached (number of rows in the output) and adjust the cache size as needed. A suggested cache size for x number entries can't be provided, but with some initial monitoring, you should be able to evaluate and determine an appropriate cache size for your expected usage patterns. Some situations may need a vastly bigger cache than the default - perhaps 1M or 10M in size, but that has to be evaluated per situation by the customer. Even those are rather small in terms of memory usage.

If you start experiencing ldap performance issues at a later date, then you should probably do some new monitoring to determine if the number of entries being cached has increased since your initial assessment. If so, its probably time to increase the cache again.

LDAP FAQs

  • Does LDAP authentication work with Active Directory?

    Yes, but for reasonable performance either the "global catalog" port must be used or Active Directory must be front-ended by the Active Directory Application Mode daemon

  • Can IHS require a user to be a member of two groups?

    The two LDAP modules behave differently, and neither is configurable.

    • mod_ibm_ldap only grants access when all LDAPRequire directives are satisfied.

    • mod_ldap/mod_authnz_ldap, as well as most standard Apache HTTP Server modules, grants access when any Require directive is satisfied.

  • Can I change the SSL protocols used by ldaps:// (LDAP over SSL)?

    In the following cases, the protocol can be controlled as described in #tls12.

    • IHS 9.0, any platform.

    • IHS 8.5.5.14 and later on Linux, AIX, and Solaris. See PI96321.

    For other combinations, there is no way to configure SSL protocols nor to enable TLS 1.2.

    • In IHS 8.0, 8.5, and 8.5.5 (Prior to 8.5.5.14, or on Windows and HP-UX): SSLv3, TLSv1.0, and TLSv1.1 are used.

    • In IHS 7.0, only SSLv3 and TLSv1.0 are used.

Documents to gather

Collecting data for problems with the IBM HTTP Server for LDAP authentication problems. Gathering this MustGather information before calling IBM support will help you understand the problem and save time analyzing the data.

There are two possible modules that IHS might be using for LDAP authentication.

If using IHS before version 7.0 on non-z/OS platforms, you are using mod_ibm_ldap. If using IHS on z/OS, or version 7.0 or later, you might be using mod_ibm_ldap, or mod_ldap and mod_authnz_ldap; check the IHS configuration file to see which modules are loaded.

The following list of files are needed. Include the SSL information if the HTTP request is being received over SSL, or the LDAP server is being accessed over SSL.

  1. Stop IBM HTTP Server.

  2. Clear all logs in the install_root/logs directory.

  3. Edit the httpd.conf file. Change Loglevel to debug.

  4. Enable LDAP tracing:

    • If using mod_ldap instead of mod_ibm_ldap, you can skip defining LDAP_TRACE_FILE; it is ignored and all LDAP trace goes to the IHS error log.)

    • For Windows:

      • Create a system variable called: LDAP_TRACE_FILE with a value of e.g. c:\ldaptrace.log

      • Create a system variable called: LDAP_DEBUG with a value to 65535.

    • For UNIX:

      • Edit install_root/bin/envvars with a text editor to add these lines:

        LDAP_TRACE_FILE=/path/ldaptrace.log
export LDAP_TRACE_FILE
LDAP_DEBUG=65535
export LDAP_DEBUG

    • For zOS, review and uncomment the example LDAP* environment variables in bin/envvars. The output always goes to an external file.

  5. If using SSL, enable SSL traces:

    • To enable mod_ibm_ssl trace, add this line to the bottom of the httpd.conf file: SSLTrace

    • To enable IBM Global Security Kit (GSKit) trace:

      • Windows: Create system variable: GSK_TRACE_FILE and set it to a value of e.g. c:\gsk_trace.log

      • Unix: Edit install_root/bin/envvars with a text editor to add these lines:

                GSK_TRACE_FILE=/path/gskit.log
        export GSK_TRACE_FILE

  6. Start IBM HTTP Server.

  7. If possible, setup a binary, unlimited capture length packet capture between IHS and the LDAP server.

  8. Recreate the problem, noting the time/URL and expected/observed result.

  9. Capture the following: netstat -na \> netstat.out

  10. Stop IBM HTTP Server.

  11. Collect the following data files:

    • The output of the DescribeConfig must-gather tool

    • IHS Configuration files - httpd.conf - any additional IHS configuration files loaded by the Include directive or LDAPConfigFile directives.

    • IHS log files

      • error_log

      • access_log

      • Any additional ErrorLog or CustomLog

    • netstat.out

    • If using mod_ibm_ldap: - ldaptrace.log - ldap.prop (value of LDAPConfigFile)

    • If using SSL: - IBM Global Security Kit (GSKit) trace (gskit.log)

    • Include the date and time of failure along with the browser version and the full URL that resulted in the LDAP failure.

    • A binary, unformatted packet capture of the traffic between IHS and LDAP if available.

  12. Collect the following additional information

    • If the failure is authentication, show the target users LDAP information with an ldapsearch command line client. Show the command and output.

    • If the failure is authentication, show the target user and required groups LDAP information with an ldapsearch command line client. Show the command and output.

  13. Follow instructions to send diagnostic information to IBM support.