FAQ: Cryptographic accelerator

Things to check first if crypto hardware is not working

• Luna/Gemalto: Ensure Apache = 1; is set in the Misc section of Chrystoki.conf (or similar Luna configuration file)

• LoadFile / LD_PRELOAD missing on Linux (SSL0154E)

• httpd.conf-configured "User" is not a member of PKCS11 group on Linux (or pkcsslotd not started)

• PKCS11Direct was used on V8 and later

• Token is locked or needs password change

• On z/Linux, make sure /var/lock/opencryptok is group-writable, and the group is set to pkcs11. Contact your distribution vendor if the permissions are inconsistent.

• It has been reported that the crypto library provided with the IBM 4764, which is not certified with GSKit v8, can hang or crash when used with 64-bit IHS. Using a 32-bit IHS (where available) may provide relief for this "other configuration".

• Crashes when using crypto hardware may be resolved by installing GSKit 8.0.50.18 or newer. A fix was added into that version of GSKit to resolve an occurring hang/crash situation when attempting to use certificates from a crypto card. This GSKit can be obtained by installing the PI13422 interim fix.

IBM 4765 support/details

As of this writing, The IBM 4765 is supported by virtue of being listed in this document: https://www.ibm.com/support/pages/gskit-version-8s-support-pkcs11-device-integration-and-cpu-acceleration.

Requirements

• To satisfy the assymetric offload only requirement when using this card with IHS, use the following stanza in the same context as SSLPKCSDriver:

  # Disable symmetric cipher offload for 4765
  SSLAttributeSet 417 550
  # Disable digest offload for 4765
  SSLAttributeSet 417 552
  

• At least APAR IV75204/IV76220/IV76279 and IV76225/IV75996/IV76280 is needed to use this adapter with IHS. The following filesets are required: security.acf security.pkcs11 security.pkcs11.tools

• Summary of other requirements:

   Install CCA Support Progam and load programs in coprocessor
   Activate the "Y4" adapter, create a token and set a security officer PIN with p11admin
   Use IHSROOT/bin/gskcapicmd to create a certificate or CSR on the token
   Use IHSROOT/bin/gskcapicmd to create a secondary KDB to contain CA certs.
   Use IHSROOT/bin/sslstash to stash the user PIN for the PKCS11 token
   Configure SSLPKCSDriver /usr/lib/pkcs11/ibm_pkcs11_64.so, SSLServerCert with the token name:label, and SSLStashFile.
  

• How do I see if a 4765 is being used for offload?

  Review the "Requests" field in the output of cryptstat Crypt0 to confirm RSA is being offloaded. If no offload is occuring, one item to check is that the Y4 adapter shows as "active" rather than "available" in p11admin.

• How do I use a SHA2 certificate with a 4765?

  Ikeyman cannot create SHA2 based certificates or CSR's while using a 4765, because the 4765 does not support some SHA2 algorithms. You must use gskcapicmd from GSKit 8.0.50.44 or later.

  Some example command-line invocations follow:

  IHSROOT=/opt/IHS
  TOKENLABEL=token1
  PKCS11_PASS=1234
  SECONDARY_PASS=WebAS
  
  # List tokens
  $IHSROOT/bin/gskcapicmd -keydb -list -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so -pw $PKCS11_PASS
  
  # List certs
  $IHSROOT/bin/gskcapicmd -cert -list -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so -tokenlabel "$TOKENLABEL" -pw $PKCS11_PASS
  
  # Add PEM-encoded CA certificate to secondary KDB. Obtained from the CA.
  $IHSROOT/bin/gskcapicmd -cert -add -db $IHSROOT/conf/secondary.kdb -pw $SECONDARY_PASS \
       -label sha2-ca -file /tmp/cacert.pem -trust enable
  
  # Create certificate request (CSR)
  $IHSROOT/bin/gskcapicmd -certreq -create -crypto  /usr/lib/pkcs11/ibm_pkcs11_64.so -pw $PKCS11_PASS -dn "cn=`hostname`"  \
       -size 2048 -sigalg SHA256WithRSA -secondarydb $IHSROOT/conf/secondary.kdb -secondarydbpw $SECONDARY_PASS            \
       -tokenlabel "$TOKENLABEL" -label sha2 -file /tmp/certreq.cer
  
  # Receive PEM-encoded signedcertificate from CA
  
  $IHSROOT/bin/gskcapicmd -cert -receive -crypto /usr/lib/pkcs11/ibm_pkcs11_64.so                                          \
        -tokenlabel "$TOKENLABEL" -pw $PKCS11_PASS -secondarydb $IHSROOT/conf/secondary.kdb -secondarydbpw $SECONDARY_PASS \
         -file /tmp/certreq.cer.cer
  

Ikeyman crypto related FAQs

Why isn't "PKCS11Config" an option in Ikeyman?

• Check the *.cfg path specified in java.security to ensure the file is present and readable, as an invalid path will cause the PKCS11Config setting to be suppressed. The added entry looks like this:

  security.provider.13=com.ibm.crypto.pkcs11impl.provider.IBMPKCS11Impl /opt/crypto/pkcs11.cfg

• In the "pkcs11.cfg" referenced from java.security, An invalid token name will NOT suppress "PKCS11Config", but it also won't work later.

• In the "pkcs11.cfg", if the path to the PKCS11 library is invalid, the "PKCS11Config" will be suppressed.

• In the "pkcs11.cfg", if the PKCS11 library is not loadable by the OS (dependencies, wrong architecture compared to JRE), the "PKCS11Config" will be suppressed.

Why doesn't IHS 7.0 or later recognize the certificates stored on my cryptographic device?

The "PKCS11Config" method of configuring a PKCS11 token must be used. "PKCS11Direct" will not work at runtime, reporting:

SSL0227E: SSL Handshake Failed, Specified label could not be found in the key file

If your problem is with the runtime recognizing your token (e.g. SSL0154E), add the proper LoadFile or LD_PRELOAD entry as suggested here.

Why does ikeyman try to open a directory as a DLL via PKCS11Config?

Sometimes an error message will be shown in the Ikeyman stack trace window that references your parent directory instead of the libpath= in your *.cfg file.

com.ibm.gsk.ikeyman.error.KeyManagerException: com.ibm.pkcs11.PKCS11Exception: Invalid dll name /opt/IBM

This is a bug in Ikeyman 8.0.333 and is fixed in Ikeyman 8.0.334. If IHS maintenance does not resolve the issue, contact support for a test fix of Ikeyman 8.0.334.

CoolThreads/Niagra/UltraSparc offload in T1/T2 servers?

This information/configuration is deprecated and preserved for historical purposes only. Modern SPARC systems offer CPU instructions that can be used to accelerate SSL without complicated PKCS11 configuration. See #solcpuins.

Information on configuring this platforms crypto acceleration with IHS used to be available at the following URLs, but have not been available for quite some time.

• http://wikis.sun.com/display/BluePrints/Accelerating+IBM+HTTP+Server+Cryptographic+Operations+Using+Sun+Servers+with+CoolThreads+Technology

• http://wikis.sun.com/download/attachments/106727484/821-0040.pdf

The instructions contained detailed steps for using Solaris crypto tools to create a token and certificate. Details about configuring the platform are not available from IBM support.

Configuring RSA offload via the PKCS#11 interface is critical to achieve good SSL performance on this platform prior to T4/T5/M6/M7 hardware levels

Under some configurations, IBM HTTP Server may see a dramatic SSL handshake performance degradationwhen using the RSA offload capabilities of the Niagra platform. Customers encountering slow PKCS#11 performance should work with oracle to ensure they have all applicable fixes for concurrent access to the PKCS#11 driver (and to confirm that hardware offload is being used).

When the software token described in the whitepaper above is used, the Niagra PKCS#11 driver will also default to doing many other cryptographic operations in software instead of using the coprocessors. Information on disabling these mechanisms in the software-based driver are discussed in section 4 http://www.oracle.com/technetwork/systems/articles/web-server-t1-jsp-141041.html

Some users have reported importing of existing keys is difficult in this environment. We recommend using Ikeyman to convert any pre-existing keys to PKCS12 format, then using the native Solaris tools (pk12util, pktool) to perform any interaction with the cryptographic token. One such report, accompanied with a recipe for using pre-existing (non self-signed) keys is documented in this thread: http://www.ibm.com/developerworks/forums/thread.jspa?threadID=364420&tstart=0

A number of users have reported that while pktool appears to work for importing an existing certificate, they don't actually work unless pk12util is used.

Does IHS work with the KSSL SSL proxy in Solaris?

IHS has not been tested with KSSL. KSSL is a kernel-level SSL proxy on Solaris that can exploit Niagra cryptographic accelerators. If you configure KSSL, you don't configure IHS for SSL at all and IHS is not aware of any SSL parameters being used.

Questions about configuring KSSL should be directed to Oracle. If you're following instructions for Apache, convert your CMS KeyFile (*.kdb) into PKCS12 using the GSKit tools then extract the private key and certificate files from the PKCS12.

Does IHS work use the crypto acceleration instructions in T4/T5/M6/M7 servers?

Yes, in 8.5.5.9 and later (or earlier 8.5.5.x w/ cumulative GSKit APARS PI54962/PI60207). No configuration is required in IHS or the operating system to take advantage of the added CPU insructions.

What cryptographic hardware is supported with IBM HTTP Server?

• For IBM HTTP Server 8.0 and later, the supported hardware is listed here

• z/Linux cryptogaphic accelerators have been reported to work in both ICA and CCA mode, although IHS and GSkit are agnostic to the underlying technology used after calling into the configured SSLPKCSDriver

If an adapter is not on the list, then the Tivoli GSKit team is not aware of successful testing with the corresponding release of GSKit. The IHS team does not test or certify cryptographic hardware, and cannot assist with the setup or configuration of cryptographic hardware.

Why does my PKCS11 token routinely get erased?

Consult your enterprise Linux vendor for an updated opencryptoki library to fix this truncation of the soft-token.

How can I tell if my PKCS11 token is ready to be used by IBM HTTP Server?

Detecting an improperly initialized cryptographic token

Note: all examples assume the first token is being used, so each pkcsconf invocation contains "-c 0".
('pkcsconf' is not available on all unix systems. For AIX, refer to the AIX crypto tools section. and use the referenced documentation to determine equivalent commands that can be used.)

Run the following command in a terminal:

$ pkcsconf -t -c 0| grep Flags

If the output matches any of the following patterns, your PKCS11 token is not usable by IHS because it has not been configured properly.

• The 'n' digits below are placeholders that show the column of interest (5th/6th digit counting from the right).

• No leading digits need to be present when a user or security officer PIN error is indicated by the 5th or 6th digit; more significant digits may be present.

output	explanation	action (see list below)
Flags: 0xnCnnnn	USER_PIN_LOCKED and USER_PIN_TO_BE_CHANGED	3,4
Flags: 0xn8nnnn	USER_PIN_TO_BE_CHANGED	4
Flags: 0xn4nnnn	USER_PIN_LOCKED	3,4
Flags: 0xCnnnnn	SO_PIN_LOCKED and SO_PIN_TO_BE_CHANGED	1,2,3,4
Flags: 0x8nnnnn	SO_PIN_TO_BE_CHANGED	2
Flags: 0x4nnnnn	SO_PIN_LOCKED	1,2,3,4

For more information about the Flag values see this file

Solutions (make sure these complete succesfully)

1. To correct SO_PIN_LOCKED, contact a system administrator and have them run: (Note: This may destroy corresponding personal certificates) pkcsconf -I -c 0

2. To correct SO_PIN_TO_BE_CHANGED, change (set) the Security Officer PIN: pkcsconf -P -c 0

3. To correct USER_PIN_LOCKED, contact a system administrator or initialize the user PIN (may destroy corresponding personal certificates): pkcsconf -u -c 0

4. To correct USER_PIN_TO_BE_CHANGED, change (set) the user PIN: pkcsconf -p -c 0 Note: Solution 2 uses a upper case 'P' and solution 4 uses an lower case 'p'.

An example of a properly configured token is:

Flags: 0x44D

Examples of improperly configured token is:

• Flags: 0x8044D

• Flags: 0xC044D

• Flags: 0xA044D

What cryptographic operations does IHS offload when configured with SSLPKCSDriver?

IHS offloads RSA cryptographic operations (assymetric operations) when configured with SSLPKCSDriver, if supported by the cryptographic hardware.

IHS 6.0.2.39, 6.1.0.29, 7.0.0.0 and later can be configured to offload symmetric crypto operations (e.g. DES, RC4, AES), if the cryptographic hardware supports it, with the following configuration in the SSLEnabled virtual host:

# Symmetric offload
SSLAttributeSet 417 549

However, many crypto cards cannot do this concurrently and can fail mysteriously at runtime. This is no longer recommended.

How can I tell if offload in IHS or the Plugin is working?

To confirm offload in your IHS/PLG environment:

* restart IHS (clear Plugin connection pool)

* z/Linux: note the value in /proc/driver/z90crypt

    Per-device successfully completed request counts
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 0000006B 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

  (On Solaris, use  kstat -n ncp0 | grep rsa. On AIX, check cryptstat Crypt0)

* Send a request through IHS with SSL  that will hit your Application Server
  (curl -vk https://localhost/context-root/foo.jsp)

* z/Linux: The number in /proc/driver/z90crypt should now jump by 1 if either IHS or
  Plugin are using z90crypt acceleration or by 2 if both are.

    Per-device successfully completed request counts
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 0000006C 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
        00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000

  (On Solaris, use  kstat -n ncp0 | grep rsa. On AIX, check cryptstat Crypt0)

* If the number is not right, make sure there were no SSL initialization errors
  during Plugin startup.  Optionally disable the HTTP transport in
  plugin-cfg.xml to make sure SSL is occuring.

* To test subsequently with backend connections, you must restart IHS so you
  can be sure the Plugin does not have a cached SSL connection (No handshake
  required, so no increase in RSA acceleration count).

Errors on z/Linux using cryptographic accelerator

Possible symptoms include:

• Java crash in ikeyman on z/Linux when PKCS11 token is being manipulated

• Error Message: "Cryptographic token initialization failed. Cryptographic token support will not be available."

• Inability to use crypto offload at runtime

• Parent process crashes

• SSL0154E at startup

Configuration of this environment is quite complicated. It is documented in some details in whitepapers such as IBM WebSphere Application Server Version 8 for Linux on IBM System z – SSL Setup and Performance Study

Cause #1: Mandatory environment setup for z/Linux crypto offload

Warning: This section of the doc has not been confirmed in many years.

On z/Linux, the PKCS11 driver that IHS must load to communicate with the z/Linux crypto device depends directly on OpenSSL. Some low level parts of the security library used by IHS, Tivoli Global Security Kit (GSKit) are derived from low-level parts of OpenSSL. Unfortunately, loading them both on the same process is not straithforward or reliable on Linux despite GSKit taking great efforts to localize/namespace the derived bits.

Due to this conflict over two "closely related" dependencies in the same process, it has been necessary to force the operating systems version of libcrypto to be loaded first.

For command line tools, this is done by specified with LD_PRELOAD in the shell:

LD_PRELOAD=/usr/lib/libcrypto.so:/usr/lib64/libcrypto.so
export LD_PRELOAD

For the IHS runtime, use the LoadFile directive to point to the 32-bit or 64-bit system copy of libcrypto. For example, LoadFile /usr/lib64/libcrypto.so

This library conflict can result in obscure certificate management command line tool messages:

  CTGSK3026W The key file "pkcs11" does not exist or cannot be read.
  CTGSK2137W The label does not exist on the PKCS#11 device.

Cause #2: Invalid PIN on crypto token

During configuration (for example, when using the pkcsconf tool on Linux), some error messages may not have been noticed and the "user PIN" may not have been set after being initialized. The PIN may also have been locked due to too many failing password attempts.

If ikeyman crashes due to this issue, the top of the javacore backtrace will look contain this method: com.ibm.gsk.ikeyman.basic.CryptographicToken.c_BuildKeyLabelList(Native Method)

If the token had been properly setup once in the past, then the PIN became locked/expired/ after key management was complete, IHS will likely crash in the parent process with an error resembling this: [notice] seg fault or similar nasty error detected in the parent process

Solution

The user PIN must be changed/set (pkcsconf -p) after being initialized (pkcsconf -u). If the user PIN is locked due too many incorrect passwords, the token must be re-initialized (pkcsconf -I)

Cause #3: Crypto offload fails after SLES11 SP1?

Maintenance from Novell is required to use crypto offload on this OS after SP1. Contact Novell for a fix for bugzilla #643843 and #582774

Cause #4: pkcsslotd not configured/started

The "User" directive must be a member of the pkcs11 group, and the pkcsslotd daemon must be started.

Parent process crashes on z/Linux with crypto enabled

This is the result of a crypto token being uninitlized, or locked. See this topic.

z/Linux (ICA) crypto offload further info

Configuring WebSphere V7.0 and IBM HTTP Server V7.0 to use Cryptographic Hardware for SSL Acceleration on Linux on IBM System z has some z/Linux info for IHS and WebSphere Application Server, with Ikeyman v7 oriented instructions.

2006 redbook on using z/Linux crytographic hardware (Ikeyman v7)

"Using IKEYMAN Version 8 to store keys on a PKCS11 device" has PKCS11 information for Ikeyman v8 and later (as used by default with IHS v7 and later).

Mandatory configuration changes for GSkit 7.0.4.x on z/Linux

AIX crypto tools

The 'pksconf' utility referred to at various places within this document is not available on AIX systems.
However, there are PKCS11 crypto utilities (p11km & p11admin) that are available (believed to have been introduced in AIX 6.1, technology level 4).

You can refer to the Tools subsection of this document: [Public Key Cryptography Standards #11] (https://www-304.ibm.com/support/knowledgecenter/ssw_aix_71/com.ibm.aix.security/pkcs_over.htm) in the AIX 7.1 Knowledge Center for detailed info.

Here is a summary description of the tools:

• p11admin

  The 'PKCS11 administration tool' is the tool for managing security officer, performance tuning, and cryptographic hardware assistance features on the AIX OS. This tool allows an administrator or security officer to manage the tokens AIX Cryptographic Services control. You can use this tool to manage PKCS11 tokens and slots, reset user passwords, confirm object deletions, specify object trust, activate hardware assistance and tune performance.

• p11km

  The PKCS #11 Key Management tool is the centralized tool for managing keys, certificates, and PKCS #11 data on the AIX operating system.

  Additionally, ikeyman and gskcapicmd can create keys and certificates via generic PKCS11 interfaces.

PKCS11 Mustgather : Supplemental data to collect for cryptographic token related errors

If a crypto accelerator is being used, collect the following doc:

• Some evidence using tools provided by the crypto offload vendor that the PKCS11 library is ready to be used.

  • Linux only: Output of 'pkcsconf -itsm' as root (to display the PKCS11, token and slot info plus the mechanism list).
    (or the equivalent output using whatever PKCS11 crypto utilities are available for your system.

  • Linux only: Demonstrate that "pkcsslotd" is running and the configured IHS 'User' is a member of the "pkcs11" group (as well root).

  • AIX only, refer to the AIX crypto tools section and provide details of the PKCS11 environment.

• Description of the problem along with what has recently changed.

• IHS configuration, logs, and GSKit trace (gsktrace_log).

• Listing of available tokens and certificates produced by bin/gskcapicmd -keydb -list -crypto ... and bin/gskcapicmd -cert -list -crypto ... -tokenlabel ...

  • The ellipsis above should be replaced with the PKCS11 driver path and token label that are used on your system.

• For IHS v8 and later, collect the java.security file and the "pkcs11.cfg" that was added as per the infocenter.

• Describe Any LoadFile or LD_PRELOAD added setup in the environment prior to launching IHS.