FAQ: Edge Caching Proxy

Caching Proxy MustGather

Enabling MustGather

Configuring IBM Caching Proxy to write trace can be done with the TraceModule and TraceLog config directives. Note that the command line arguments -debug and -mtv must also be specified when starting IBM Caching Proxy.

It is also necessary to redirect stdout and stderr to a file so that configuration details are also collected. This can be done using a redirect:

ibmproxy -mtv -debug > /opt/ibm/edge/cp/server_root/logs/ibmproxy.log 2>&1

To trace all modules, the "all" argument can be specified for TraceModule:

TraceLog /opt/ibm/edge/cp/server_root/logs/trace
TraceModule All

If TraceModule all is too verbose, tracing can be enabled on specific modules. For example, the list of modules typically used to debug a caching problem is:

TraceLog /path/to/trace/file
TraceModule htcaio
TraceModule htcbio
TraceModule htccompact
TraceModule htcfs
TraceModule htcfsck
TraceModule htcfsops
TraceModule htcgc
TraceModule htcgceval
TraceModule htchash
TraceModule htcheap
TraceModule htchio
TraceModule htcio
TraceModule htciter
TraceModule htclock
TraceModule htclsort
TraceModule htcwhdr
TraceModule htcwrtr

List of Traceable Modules

aa auth bag base cfgdll
cgi config content dirbrw dll
dns error fileio format ftp
gcother gopher hash heap htcaio
htcbio htccompact htcfs htcfsck htcfsops
htcgc htcgceval htchash htcheap htchio
htcio htciter htclock htclsort htcwhdr
htcwrtr http icapi if javelinbase
javelinpics lex list log mempool
netmonitor nls oru perfmon pics
proxy proxycache proxychain proxyinit workqueue
rcacfg rcaclnt rcacomm rcahash rcalatency
rcaoop rcaserv rchunk request selftest
snmp socket socks ssi ssl
stack status stringlib syscall tcp
time timer threadpool user url
vary workthread workthread-callback connection-pool rtsp
persist res scm javelin proxygroup
rcagroup gcgroup      

PAC-LDAP MustGather

Additional MustGather should be provided in order to troubleshoot the PAC-LDAP authorization module: PacErr_Client.log and PacErr_Server.log. To increase the detail of the trace, two enviornment variables should be set prior to starting the IBM Caching Proxy:

PAC_DEBUG_LEVEL=64
LDAP_DEBUG=65535
LDAP_DEBUG_FILE=/opt/ibm/edge/cp/server_root/logs/ldapclient_trace.log

export PAC_DEBUG_LEVEL
export LDAP_DEBUG
export LDAP_DEBUG_FILE

Error messages

"Error encounted while loading DLL or function"

On Linux, you're likely missing the 32-bit compat-libstdc++-33 operating system package.

If you have compat-libstdc++-33 installed and are still seeing this error, please make sure that all directories in the path /opt/ibm/edge/cp/lib are world readable and executable. This is necessary since Caching Proxy switches the user/group of the process before trying to load the shared library libhttpdapi.so.

You can verify the permissions for every directory by running the following command. The permissions for your directory should look similar to the following:

$ namei -om /opt/ibm/edge/cp/lib/libhttpdapi.so
f: /opt/ibm/edge/cp/lib/libhttpdapi.so
 dr-xr-xr-x root root /
 drwxr-xr-x root root opt
 drwxr-xr-x root root ibm
 drwxr-xr-x root root edge
 drwxr-xr-x root bin  cp
 drwxrwxr-x root bin  lib
 -rwxr-xr-x root bin  libhttpdapi.so

SSL questions

What are the default SSL/TLS protocols?

The SSL/TLS protocols enabled by default change several times in Caching Proxy 8.5.5. The changes are summarized in the following table and are described in more detail after the table.

Caching Proxy 8.5.5 Default Protocols

Versions (inclusive)

Protocols Enabled by Default

  • 8.5.5.0 - 8.5.5.3: SSLv3, TLSv1.1

  • 8.5.5.4: SSLv3

  • 8.5.5.5 - 8.5.5.11: None

  • 8.5.5.12 - current: TLSv1.0, TLSv1.1

Detailed changes to Default Protocols

  • 8.5.5.0 - 8.5.5.3: Original SSL configuration

    Caching Proxy enables SSLv3 and disables TLSv1.0 by default. IBM Global Security Kit (GSKit) enables TLSv1.1 by default, but is not disabled by Caching Proxy.

  • 8.5.5.4: PI23780 disables all TLS protocols by default

    PI23780 adds two directives to configure the TLS version and its ciphers. The directives are TLSVersion and TLSCipherSpecs, respectively. All TLS protocols are disabled by default and can be enabled with the TLSV1Enable directive.

  • 8.5.5.5 - 8.5.5.11: PI28050 disables SSLv2 and SSLv3 to avoid the POODLE Attack

    This APAR effectively causes all SSL/TLS protocols to be disabled from 8.5.5.5 to 8.5.5.11. In addition to TLS protocols already being disabled by default, PI28050 adds the TLSOnly directive which disables SSLv2 and SSLv3 by default.

  • 8.5.5.12+: PI75968 enables TLSv1.0 and TLSv1.1 by default

    The TLSv1.0 and TLSv1.1 protocols are now enabled by default in order to have SSL working out-of-the-box when SSLEnable is ON. TLSv1.2 was not enabled by default since it is possible that it would cause some certificates to no longer be accepted.

    PI75968 changes TLSV1Enable from OFF to ON_TLSV10_TLSV11 by default. It also added functionality to the TLSVersion directive to allow multiple TLS protocols to be specified. The default value for TLSVersion, which controls which TLS protocols are enabled by default, also changed from ALL to TLSV10 TLSV11. To allow older configurations to work, TLSVersion will still enable all TLS protocols if TLSV1Enable is set to ON.

Caching Proxy 9.0 Default Protocols

Versions (inclusive) Protocols Enabled by Default
9.0.0.0 - 9.0.0.4 None
9.0.0.5 and above TLSv1.0, TLSv1.1

How do I setup TLSv12 / TLSv1.2 / TLS 1.2?

This is a basic configuration you can append to the sample ibmproxy.conf:

# Basic SSL config
SSLEnable ON
KeyRing .../key.kdb
KeyRingStash .../key.sth
# Must be present to cache requests that show up over SSL
SSLCaching ON

# Before PI75968 only:
# V3CipherSpecs Must be specified. Can be "".
V3CipherSpecs "352F"
TLSV1Enable ON
# end pre-PI75968

# After PI75968, opt in to TLSV1.2
TLSVersion ALL

# Use PI90112 defaults prior to PI90112:
TLSV11CipherSpecs 2F35
TLSV12CipherSpecs 9C9D3C3D2F35

For more details, consult this technote.

How do I enable ECDHE ciphers?

ECDHE / DHE cipher suites are supported in IBM Caching Proxies fixpacks 8.5.5.15 and 9.0.0.10 and later with PH01138 using the following directive:

TLSExtendedCipherSpecs <TLSV10 | TLSV11 | TLSV12> cipher-long-name...

Multiple ciphers (up to 8) may be set per TLSExtendedCipherSpecs. If more than 8 are needed, multiple TLSExtendedCipherSpecs directives can be used. A list of supported TLSv1.2 cipher specifications can be found here.

Avoid using the older directives TLSV12CipherSpecs and TLSv11CipherSpecs with TLSExtendedCipherSpecs. Any cipher specifications set with the older directives will be ignored for any TLS protocol configured with TLSExtendedCipherSpecs.

Example: Using ECDHE Ciphers

# Enable ECDHE RSA ciphers
TLSExtendedCipherSpecs TLSV12 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
TLSExtendedCipherSpecs TLSV12 TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
TLSExtendedCipherSpecs TLSV12 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384
TLSExtendedCipherSpecs TLSV12 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256
# Support older clients
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_128_GCM_SHA256
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_256_GCM_SHA384
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_128_CBC_SHA256
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_256_CBC_SHA256
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_128_CBC_SHA
TLSExtendedCipherSpecs TLSV12 TLS_RSA_WITH_AES_256_CBC_SHA

What ciphers are supported in Caching Proxy?

Any cipher with a two characters short name can be configured using the following directives:

  • TLSV12CipherSpecs

  • TLSV11CipherSpecs

Ciphers with a 4 character short name must be set using TLSExtendedCipherSpecs using their long name. See How do I enable ECDHE ciphers? for an example usage.

TLSv1.2

Short Name Long Name Default? Notes
9C TLS_RSA_WITH_AES_128_GCM_SHA256 Yes -
9D TLS_RSA_WITH_AES_256_GCM_SHA384 Yes -
3C TLS_RSA_WITH_AES_128_CBC_SHA256 Yes -
3D TLS_RSA_WITH_AES_256_CBC_SHA256 Yes -
2F TLS_RSA_WITH_AES_128_CBC_SHA Yes -
35 TLS_RSA_WITH_AES_256_CBC_SHA Yes -
C030 TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 No -
C02F TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 No -
C028 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 No -
C027 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 No -
C014 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA No -
C013 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA No -
C009 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA No Requires an ECDSA personal certificate.
C00A TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA No Requires an ECDSA personal certificate.
C023 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 No Requires an ECDSA personal certificate.
C024 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 No Requires an ECDSA personal certificate.
C02C TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 No Requires an ECDSA personal certificate.
C02B TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 No Requires an ECDSA personal certificate.
History of TLSv1.2 Default Ciphers
APAR8.08.59.0Default TLSV1.2 Ciphers
PI90112 [8.0.0.15, ...) [8.5.5.14, ...) [9.0.0.7, ...) 9C9D3C3D2F35
PI74512 [8.0.0.14, 8.0.0.15) [8.5.5.12, 8.5.5.14) [9.0.0.4, 9.0.0.7) 9C9D3C3D2F350A
Initial [8.0.0.0, 8.0.0.14) [8.5.5.0, 8.5.5.12) [9.0.0.0, 9.0.0.4) 9C9D3C3D2F350A3B02

TLSv1.1

The supported cipher short names (two-character codes) are 2F350A0504090201.

The current default ciphers for TLSv1.1 are:

2F - TLS_RSA_WITH_AES_128_CBC_SHA
35 - TLS_RSA_WITH_AES_256_CBC_SHA
# Note: ciphers below are disabled by default after PI90112.
0A - TLS_RSA_WITH_3DES_EDE_CBC_SHA
05 - TLS_RSA_WITH_RC4_128_SHA
04 - TLS_RSA_WITH_RC4_128_MD5
09 - TLS_RSA_WITH_DES_CBC_SHA
02 - TLS_RSA_WITH_NULL_SHA
01 - TLS_RSA_WITH_NULL_MD5
History of TLSv1.1 Default Ciphers
APAR 8.0 8.5.5 9.0 Default TLSv1.1 Ciphers
PI90112[8.0.0.15, ...) [8.5.5.14 , ...) [9.0.0.7, ...) 2F35
PI74512[8.0.0.14, 8.0.0.15) [8.5.5.12, 8.5.5.14) [9.0.0.4, 9.0.0.7) 2F350A
Initial[8.0.0.0, 8.0.0.14) [8.5.5.0, 8.5.5.12) [9.0.0.0, 9.0.0.4) 2F350A0504090201

SSLv2, SSLv3, and TLSv1.0

See the knowledge center.

How do I redirect clients from HTTP to HTTPS?

The Redirect directive can be used to redirect clients from HTTP to HTTPS. Since this mapping rule should be applied first, the Redirect directive should be above other mapping rules (e.g. Pass directives).

HTTP to HTTPS Redirect Example

# Redirect HTTP to HTTPS
Redirect /* https://server-name/* server-name:80

TLS Protocols are not enabled with TLSV1Enable set to 'on'

After PI75968 and before PI90112, the argument for TLSV1Enable is case-sensitive and must be set to ON and not on.

PI90112 fixed the case-sensitivity issue so TLSV1Enable now accepts lowercase arguments also.

Proxy Questions

Protecting a Forward Proxy

A forward proxy can be protected with basic authentication using the following configuration:

Protection PROXY-PROT {
      ServerID     CP_Proxy_Auth
      AuthType     Basic
      PasswdFile   /opt/ibm/edge/cp/server_root/protect/passdb
      GetMask      All@(*)
      PutMask      All@(*)
      PostMask     All@(*)
      Mask         All@(*)
}

Protect * PROXY-PROT
Proxy http:*

# Enable SSL tunneling to allow SSL requests to be handled in a forward proxy configuration.
SSLTunneling On
Enable CONNECT OutgoingPorts 443

passdb was created using the htadm provided in the sbin directory of the CP root:

# Assumes you are in CP root
$ sbin/htadm -create server_root/protect/passdb
$ sbin/htadm -adduser server_root/protect/passdb your-user-name

Note about Protections

For protection to work properly, the Protect directive must be placed before any Pass, Exec, or Proxy directives in the configuration file.

Protecting Specific Proxied Requests

Protection can also be enabled for specific sites (e.g. www.google.com and www.bing.com). The following example demonstrates how use different protections depending on the proxied request. In particular, proxy requests to both www.google.com and www.amazon.com use the protections defined in PROXY-PROT-A, while proxy requests to www.bing.com would use PROXY-PROT-B.

Protection PROXY-PROT-A {
      ServerId     CP_Proxy_Auth_A
      AuthType     Basic
      GetMask      All@(*)
      PutMask      All@(*)
      PostMask     All@(*)
      Mask         All@(*)
      PasswdFile   /opt/ibm/edge/cp/server_root/protect/passdb_a
}

Protection PROXY-PROT-B {
      ServerId     CP_Proxy_Auth_B
      AuthType     Basic
      GetMask      All@(*)
      PutMask      All@(*)
      PostMask     All@(*)
      Mask         All@(*)
      PasswdFile   /opt/ibm/edge/cp/server_root/protect/passdb_b
}

Protect www.google.com:443*  PROXY-PROT-A
Protect www.amazon.com:443*  PROXY-PROT-A
Protect www.bing.com:443*    PROXY-PROT-B

Proxy http:*

# Enable SSL tunneling to allow SSL requests to be handled in a forward proxy configuration.
SSLTunneling On
Enable CONNECT OutgoingPorts 443

LDAP questions

LDAP: error occurred during SSL initialization, looking for credentials in: pac_keyfile.kdb

This error can happen for a variety of reasons. The obvious reason for this error is that the pac_keyfile.kdb is not found in the expected path, which is cp-root/server_root/pac/creds/pac_keyfile.kdb.

The error can also happen if the IBM Tivoli LDAP library can't load the IBM Global Security Kit (GSKit) library. A potential cause of this is if LDAP library wants to load GSK 7 but the system has GSK 8. In such a case, this error can be resolved by setting the GSKIT_CLIENT_VERSION environment variable to the GSK version desired (for example: GSKIT_CLIENT_VERSION=8).

To diagnose any other issues, the LDAP client traces should be collected and reviewed. Please see the PAC-LDAP MustGather section for information on how to collect the LDAP client traces.

Start / stop questions

Cannot open pid file for writing

The PidFile directive ships commented out by default, but points to a directory that is not writable by the nobody/nogroup user Caching Proxy changes to before writing out its pid file. Changing the PidFile to point to the server_root/logs/ directory instead is a writable alternative.

cpwizard doesn't work

Some causes on Windows for cpwizard to fail:

  • Make sure you didn't install 8.5.0.0 by mistake, PM76719 is needed on Windows.

  • If your installation path has spaces, make a copy of cpwizard.bat that looks like this instead:

    java -cp "%~dp0\cpwizard.jar" com.ibm.websphere.edge.wizard.CPWizard

  • If you installed to Program Files, you'll need to run cpwizard as an Administrator

Miscellaneous directive related questions

LimitRequestFied/LimitRequestFiedSize oddities{#LimitRequestFieldSize} -->

When LimitRequestFiedSize is breached, the request does not result in a failure. Instead, a long header is truncated and remaining data is treated as a subsequent header. If it is not syntactically valid, it is ignored, but will be passed down to the next server if the request is being proxied.

LimitRequestField does not seem capable of increasing the limit.

How do I use Custom Error Pages?

The default error pages should not be modified directly as they are serviceable. This means that any changes to the error pages will be lost when upgrading Caching Proxy.

Instead, the error pages should be copied to a new directory outside of the root directory for Caching Proxy and ibmproxy.conf should be updated to use the new file(s).

Example: Using a Custom badrequest Error Page

To use a custom 400 Bad Request, the following changes to ibmproxy.conf should be made:

# Update /errorpages/badrequest.htmls to /myerrorpages/badrequest.htmls
ErrorPage badrequest            /myerrorpages/badrequest.htmls

# Add a new mapping for myerrorpages to the location the error pages were copied to
Pass            /myerrorpages/*   /opt/ibm/edge/errorpages/*

How do I set headers in Caching Proxy?

Caching Proxy does not have the capability to set custom request or response headers.

How do I configure HTTP Strict Transport Security?

HSTS can only be configured on the backend since Caching Proxy is unable to set response headers. Setting HSTS for static content served by Caching Proxy is not possible.

Installation Issues

Caching Proxy is overwriting files in /opt/ibm/edge/cp after fixpack maintenance

Most files in /opt/ibm/edge/cp are serviceable, meaning that they should not be modified because they will be overwritten when upgrading Caching Proxy to a later fixpack.

If you need to make changes to a file, first copy the file to a location outside of the root Caching Proxy directory (i.e. /opt/ibm/edge/cp) and modify ibmproxy.conf to use the file. The changes can then be made to the file and would be preserved after applying fixpack maintenance.

Caching Proxy fails to install due to IBM Global Security Kit (GSKit) error on Linux on s390x

When installing Caching Proxy 8.5.5.4 and below, the install will fail if the system has a newer version of GSK installed already.

The suggested solution is to install a later 8.5.5 fixpack that contains the patch to allow later versions of IBM Global Security Kit (GSKit). A workaround, however, is for the user to remove the later version of IBM Global Security Kit (GSKit) and use the IBM Global Security Kit (GSKit) library provided by CP. See the example below for instructions on how to uninstall IBM Global Security Kit (GSKit).

Example: Uninstalling IBM Global Security Kit (GSKit)

# Check if IBM Global Security Kit (GSKit) is installed
$ rpm -qa | grep gsk

gskcrypt31-8.0.50.87s390x
gskssl31-8.0.50.87.s390x

# Remove/Erase IBM Global Security Kit (GSKit)
$ rpm -evv gskcrypt31-8.0.50.87s390x gskssl31-8.0.50.87.s390x

Startup Issues

Caching Proxy fails at startup referencing Symbol ReadFile (number ...) is not exported from dependent module /unix.

This startup error occurs on AIX systems where "lsdev -Cc iocp" returns "Defined" instead of "Available".

To resolve the problem:

  1. Run smitty iocp

    • Select Change/Show Characteristics of I/O Completion Ports.

    • Change the state of IOCP from Defined to Available.

  2. Restart the machine.

  3. Run lsdev again to confirm the updated status of IOCP.

Caching Proxy does not start automatically on AIX

The init script on AIX is pointing to an invalid file:

/etc/rc.ibmproxy -> /opt/ibm/edge/cp/etc/en_US/ibmproxy/rc.ibmproxy

This can be fixed by relinking to the actual file:

ln -sf /opt/ibm/edge/cp/etc/en_US/rc.ibmproxy /etc/rc.ibmproxy

Note that before PI95286, the symlink will be broken after applying fixpack maintenance for Caching Proxy. Users upgrading to a fixpack of Caching Proxy that does not contain PI95286 on AIX will have to relink the init script after the upgrade is complete.

ibmproxy on AIX unable to load /usr/lib/libicui18n.a after updating AIX

ibmproxy may fail to start after updating AIX due to AIX overwriting the links to /usr/lib/libicui18n.a that point to libicu that Caching Proxy provides.

Users affected by this issue will see the following error being reported after Caching Proxy fails to start up:

0509-150   Dependent module /usr/lib/libicui18n.a could not be loaded.
0509-022   Cannot load module /usr/lib/libicui18n.a.

A few solutions are available, but they are not ideal. The problem is due to a combination of a few design issues:

  1. sbin/ibmproxy is not a script that we can modify to configure LIBPATH.

  2. Caching Proxy uses global libraries and relies on /usr/lib.

Solution 1: Modify the LIBPATH in Caching Proxy init script

If you are using the Caching Proxy init script to start Caching Proxy, this method is suggested.

Note

It is suggested that users do not modify the init script provided by Caching Proxy directly as they are serviceable and will be replaced after fixpack maintenance.

Users should copy the init script to a location outside of /opt/ibm/edge/cp and relink /etc/rc.ibmproxy to point to it.

Procedure

The following changes should be made to /etc/rc.ibmproxy:

  1. Find the line where startsrc is called and add LIBPATH to it:

# Original line
startsrc -s $cmd -a "$3" -e "LC_ALL=$LANG"

# Line after adding LIBPATH
startsrc -s $cmd -a "$3" -e "LC_ALL=$LANG LIBPATH=$LIBPATH"

  1. Find the line where LIBPATH is modified near the end of the file and add Caching Proxy's ICU library to the beginning:

# Original LIBPATH line
LIBPATH=/usr/jdk_base/lib:/usr/jdk_base/lib/aix/native_threads:/opt/ibm/
edge/lb/servers/lib:$LIBPATH

# Line after adding Caching Proxy's ICU lib to the beginning of path:
LIBPATH=/opt/ibm/edge/icu/lib:/usr/jdk_base/lib:/usr/jdk_base/lib/aix/na
tive_threads:/opt/ibm/edge/lb/servers/lib:$LIBPATH

Solution 2: Create a script to run ibmproxy

Another solution is to create a script to run ibmproxy and use that to configure the environment for Caching Proxy before starting it.

Example ibmproxyctl Script

#!/bin/sh

LC_ALL=$LANG
LIBPATH=/opt/ibm/edge/icu/lib/:/usr/lib:/lib:/opt/ibm/edge/lb/servers/lib
export LC_ALL LIBPATH

/usr/sbin/ibmproxy $@

Procedure

  1. Create the ibmproxyctl file in a directory outside of /opt/ibm/edge/cp so that it is not deleted during fixpack maintenance (e.g. /opt/ibm/edge/mycpfiles).

  2. Make ibmproxyctl executable:

chmod +x /opt/ibm/edge/mycpfiles/ibmproxyctl

  1. Create a link to ibmproxyctl in /usr/sbin:

ln -s /opt/ibm/edge/mycpfiles/ibmproxyctl /usr/sbin/ibmproxyctl

  1. Remove the old ibmproxy subsystem:

/usr/bin/rmssys -s ibmproxy

  1. Recreate the ibmproxy subsystem using the new script:

/usr/bin/mkssys -s ibmproxy -p /usr/sbin/ibmproxyctl -u 0 -G tcpip

Starting and Stopping Caching Proxy using ibmproxyctl

Caching Proxy can be started using the command:

startsrc -s ibmproxy

and stopped using:

stopsrc -s ibmproxy