Certificate Management mustgather¶
Other useful information (in other documents)¶
Diagnostic steps to take before reproducing¶
Install the unrestricted JCE policy files (link)
If IHS 7.0 apply WASSDK updates to update Ikeyman
If IHS 8.0 or later, apply latest recommended IHS maintenance.
Known issues and their solutions¶
$IHSROOT/bin/gskcapicmd -cert -validate fails¶
The validation mechanism provided in gskcapicmd (and Ikeyman) are meant to assist in certificate deployment problems and troubleshooting. The result of the validation will not always agree with how IHS or clients will validate a certificate.
Validation via $IHSROOT/bin/gskcapicmd could fail for a number of reasons that would not fail under the actual IHS runtime. If validation succeeds you can be reasonably confident that the certificate chain is complete and will be usable by the IHS runtime barring any other runtime configurations like OCSP and CRL checking.
Append the -fips false parameter to have gskcapicmd validate in a way
much closer to how IHS operates by default.
IHS allows two validation modes at runtime, PKIX (RFC5280) and native (legacy). The command-line validator only validates against the stricter PKIX mode, unless
-fips falseis added.The validator built into Ikeyman and $IHSROOT/bin/gskcmd appear to be more tolerant, but does not match the exact validation of the IHS runtime.
IHS 9.0.0.0 and 9.0.0.1 runtimes on distributed allow PKIX only by default, but can be configured to allow native validation with the directive
SSLAllowLegacyCerts ON. 9.0.0.2 and later default toSSLAllowLegacyCerts ON.
"The password is invalid or the keystore has been corrupted" when attempting to access the keystore.¶
Problem: The JDKs provided in 8.0.0.9 and 8.5.5.3 has a defect in the IBMCMSProvider 2.50 that fails to open very old (WAS 3.5 era) keystores. Usually, a bad password or corruption message will be issued.
Solution: Apply one of the following WASSDK APARs to upgrade the
IBMCMSProvider to 2.52 or later and resolve the issue.
PI28437: SHIP JAVA apar IV66110 for WAS V8553 using
Java 6/26
PI28436: SHIP JAVA apar IV66110 for WAS V8553 using
Java 7
PI28435: SHIP JAVA apar IV66110 for WAS V8553 using
Java 7.1
Errors with Ikeyman on Sun/HP JDKs such as "java.io.IOException: invalid subject" or "java.lang.IllegalArgumentException: improperly specified input name" or 'java.io.IOException: Invalid keyword "POSTALCODE"'¶
This is a defect in Ikeyman prior to mid-2015. If you experience this
issue, use gskcapicmd as an interim workaround.
This will be resolved by iKeyman 8.0.408 and CMS provider 2.54, which
will be included in Java versions 8.0.1.0, 7.1.3.0, 7.0.9.0, 6.1.8.4,
6.0.16.4
These are expected to ship with WAS/IHS fixpacks 8.5.5.6, 8.0.0.11,
7.0.0.39(?)
Certificate management error messages¶
"The database cannot be created or opened because permission was denied¶
by the operating system"
If this message is seen while opening a PKCS11 cryptographic token:
Make sure the IBM Global Security Kit (GSKit) version is at least 7.0.4.14.
For IHS 6.1 and earlier, ensure the native IBM Global Security Kit (GSKit) Ikeyman is being used: removing gskikm.jar.
Otherwise, verify the KDB is writable by the user running ikeyman.
"A duplicate certificate already exists in the database"¶
The certificate authority being added to the KDB may already exist with some other label
Ikeyman may be misreporting an obscure encoding error
"Invalid SAN data: DNSName Components must begin with a letter"¶
Java based certificate management tools, like Ikeyman, cannot manipulate certificates with a Subject Alt Name that begins with anything other than a letter. You must use the native command-line certificate management tools (bin/gsk7capicmd or bin/gskcapicmd).
Command-line certificate management problems¶
gsk7cmd -certreq -list doesn't show some certificate requests¶
If a certificate request is recreated via the `-certreq -recreate`
action in gsk7cmd or gsk7capicmd, it will not appear in the list of
certificate requests.
Solution: The certificate remains in the Personal Certificates list
only, and Ikeyman will still allow you to perform a receive
operation on the renewal.
gsk7capicmd -cert -receive returns GSKKM_ERR_REQKEY_FOR_CERT_NULL¶
gsk7capicmd before version 7.0.4.20 cannot receive a certificate
that was issued as a result of a a "renew" or "recreate request".
Solution: Use an identical command line but substitute `gsk7cmd` for
`gsk7capicmd`
gsk7capicmd before version 7.0.4.37 cannot receive a certificate
that was issued as a result of a a "renew" or "recreate request"
*AND* was issued by a different CA then the original.
Solution: Use Ikeyman or ikeycmd from Java 6 or later (do not remove
gskikm.jar).
`gskcmd returns 'Too many values for parameter '-dn' were¶
specified:'`
This is a JRE defect. Use gskcapicmd instead, or edit the "ikeycmd"
script in your bundled JRE and add double-quotes around the last two
characters in the script -- `$@`.
All the signer certificates must exist in the key database¶
To successfully use a certificate, your KeyFile must be able to validate the new certificate all the way up to a trusted root in the signer certificate section of the KDB. In the past, Ikeyman would return an error and prevent the new certificate from being received or imported, but current releases do not block the operation. Instead, a newly received certificate can be explicitly validated using a button in the GUI.
If your certificate was issued by a certificate authority that is not among the default trusted certificate authorities automatically included in new KDB files by ikeyman, you must add the certificate of the issuer (the signer) to your KDB before receiving your certificate
Your certificate authority will often make the root CA / Signer certificates for your new certificates available as *.arm or *.crt downloads.
Adding the root signer certificate on the command line¶
The example below shows commands to create a new certificate signing request (CSR), add a certificate authority, receives an issued certificate, then validates the newly added certificate:
cd /opt/IHS
bin/gskcapicmd -certreq -create -db conf/key.kdb -stashed -file /tmp/my-cert.csr -size 2048 -sigalg SHA512 -dn "cn=example.com" -label my-new-cert -san_dnsname example.com,www.example.com
# Send /tmp/mycert.csr to CA and receive /tmp/my-cert.arm and /tmp/digicert.arm
bin/gskcapicmd -cert -add -db conf/key.kdb -stashed -label digicert -file /tmp/digicert.arm
bin/gskcapicmd -cert -receive -db conf/key.kdb -stashed -file /tmp/my-cert.arm -default_cert enable
bin/gskcapicmd -cert -validate -db conf/key.kdb -stashed -label my-new-cert
Adding the signer certificate in Ikeyman¶
Download the Root CA / Signer Certificate from your certificate authority
Open your KDB, click "Signer Certificate", then "add".
Enter a label and click "OK"
Open the signer cert you just created, and check that the "Set the certificate as a trusted root" is checked
If ikeyman still issues the same error message when you try to receive your certificate, use the following procedure to verify that the signer certificate is the same as the one that actually signed your certificate
openssl x509 -text -in certificate_from_certificateauthority.crt|grep Issuer:openssl x509 -text -in signer_certificate.cer |grep Subject:
If the output of the two commands isn't the same distinguished name, you are either using the wrong signer certificate or your certificate is signed using an intermediate certificate.
Intermediate Certificates ¶
Some certificate authorities issue certificates that are signed by an intermediate issuer, and not one of the default trusted root CA certificates that are pre-loaded into your KDB. Your certificate authority should provide any intermediate certificates required to build the trust chain and you must add them to your KDB before receiving your signed certificate.
Solution: To add the correct intermediate certificate(s), download all intermediate certificates from your certificate authority. Perform the steps outlined here for each certificate starting from the root CA and ending with the signer certificate that issued your certificate.
The public key of $label is the same as the key of $DN¶
When import or save-as from PKCS12, it may sometimes be necessary to change the pre-calculated label for personal certificates if their "PKCS12 friendly name" appears as a very long DN string.
Ikeyman error messages¶
Ikeyman: An error occurred while inserting keys to the database¶
Solution: This can occur when importing from a PKCS12 or CMS key file, onto a CMS Cryptographic Token. It is resolved by upgrading IBM Global Security Kit (GSKit) to at least 7.0.3.27 and removing gskikm.jar for IHS 6.1 and earlier.
If upgrading and removing gskikm.jar does not resolve the issue, adding the private key and signer certificates in separate steps can sometimes workaround the issue:
Verify your unrestricted JCE policy files are installed
Make a copy of the PKCS12 file, privkey.p12, and open the copy in ikeyman.
Select "Signer Certificates" and "extract" each signer certificate necessary for your personal certificate into a file.
Remove each signer certificate from the PKCS12 file.
Close the PKCS12 file.
Open the CMS Cryptographic Token in ikeyman and choose the applicable secondary CMS key database.
Select "Signer Certificates" and "add" each extracted signer certificate.
Select "Personal Certificates" and "import" from your privkey.p12
Ikeyman: "The certificate request created for the certificate is not in the key database"¶
This is caused when the certificate being received doesn't match the certificate request in the database, or there is no certificate request in the database, or the certificate is in the wrong format. So, the problem can be with the database or with the certificate itself. You must receive the new certificate into the same database used to generate the certificate request.
Steps you can take:
Make sure the database contains a certificate request.
If not, you are attempting to recieve the certificate into the wrong database.Show the details of the certificate using the following command:
openssl x509 -in serverid.arm -text -noout
{where 'serverid.arm' is the name of the certificate file}If the above command produced an error message, then the cert file may be in the wrong format (ex: a PKCS7 (p7b) file).
Refer to: Import, or receive of a personal certificate failsCheck the values in the cert (displayed by the openssl command above) to make sure it appears to be the correct certificate.
For example, check theValidity 'Not Before'and'Not After'dates to make sure this is a new certificate and not the old certificate file.
If the values aren't correct, then you have the wrong certificate file. Make sure you downloaded the correct certificate file from the CA, and haven't mixed it up with some pre-existing cert file that you already had.Show the details of the certificate request as follows:
Extract the certificate request from the database that you are attempting to receive the certificate into:Open the database in iKeyman
Display the list of Personal Certificate Requests
Select the cert request you believe was used to request the certificate you are trying to receive
Click the '
Extract...' button and specify a target location and filename (ex:certreq.arm) for the certificate request file
Run the following command to display the details in the extracted certificate request file
openssl req -text -noout -verify -in certreq.arm
{where 'certreq.arm' is the name of the file created by the extract performed above. Don't use an existing certreq.arm file since there is no way to be certain it came from the database you are currently using.}Compare the '
Modulus' and 'Exponent' values displayed for the certificate and the certificate request.
These values should be the same in each.If the '
Modulus' and 'Exponent' values do not match between the certificate and the certificate request, then you either have an incorrect certificate file or are using the wrong database (or at least the original certificate request is no longer in the database, which it makes it a 'wrong' database from a cert receive perspective.)Make sure you downloaded the correct 'new' cert file from the CA, and are attempting to receive that same cert file you downloaded, and haven't mixed it up with another cert file that you already had.
Locate the correct database with the original certificate request. Verify it by using the comparison procedure described above.
NOTE: Certificates can only be received into the KDB where the request was originally created. Attempting to recreate a certifciate request in a KDB by duplicating the parameters of a previously submitted certificate request will not allow you to receive the certificate into the KDB. The database being used must contain the actual original certificate request.
If you cannot locate a database with a certifcate request that matches the certifcate, then you will need to create a new certifcate request and obtain a new certificate.
Certificates in the key database should have unique labels. If there is a problem receiving a certificate into the key database due to duplicate labels (case-insensitive), it is recommended that the non-CSR certificate be renamed using
gskcapicmd -cert -rename -label LABEL -new_label NEW_LABEL -db KEYDB_FILE. To prevent duplicate label issues, it is recommended that the JDK be updated to one of the following releases (or later): 6.0.16.25, 6.1.8.25, 7.0.9.40, 7.1.3.40, 8.0.3.0.This issue is related to a defect in iKeyman (IV80403) that allowed certificates in the same key database to have the same labels. This can cause a certificate request to not be found if it has the same label as an existing certificate in the key database. When viewing the key database using iKeyman (or gskcmd), the certificate will show under the "Certificates" group instead of the "Certificate Requests" group. The certificates under the correct groups when using gskcapicmd.
Ikeyman widgets perform unexpectedly¶
It has been reported that ikeyman sometimes performs erratically when it is displayed on the PC X11 Server distributed by Exceed. The behavior is not seen when used with XFree86/Xorg X11 servers running natively or under Cygwin, and is also not seen when being run in a VNC server running on the IHS system.
Reported Symptom: When selecting a country name from the selection box, the select box may be reset to default (US)
Solution: Display ikeyman on a different X11 server, or contact X11 server vendor.
Ikeyman: Wrong version reported in "About" dialogue¶
In IHS 7.0 and earlier, the bundled java's gskikm.jar provides a different level of Ikeyman than what's bundled with IBM Global Security Kit (GSKit). This can cause differences in the version numbers reported between the java-based certificate management tools and the native IBM Global Security Kit (GSKit) and IHS runtimes.
In IHS 6.1 and earlier, a manual post-install configuration step is required to setup IHS to use the proper (up to date) certificate management implementation. See #GSKIKM for instructions.
In IHS 7.0, leaving gskikm.jar in place causes a more up to date Ikeyman and gsk7cmd to be used (v8). To revert to the legacy Ikeyman 7.0 with IHS 7.0, see #GSKIKM.
Ikeyman 8.0 in IHS 7.0¶
Ikeyman 8.0 is used in IHS 7.0 when
$IHSROOT/java/jre/lib/ext/gskikm.jaris present.When IHS 7.0 has been setup to use Ikeyman 8.0 as described above,
$IHSROOT/bin/gsk7cmdalso uses the updated codebase.With IHS 7.0, only Ikeyman 8.0 is supported on Solaris or HP-UX.
Authoritative documentation for Ikeyman 8.0 is available here
Information on the PKCS11 configuration used by Ikeyman v8 in IHS 7.0 is available here
removing gskikm.jar to use IBM Global Security Kit (GSKit)-provided Ikeyman¶
A file provided by the IBM JRE, gskikm.jar, can override the version of Ikeyman, which would otherwise be dictated by the level of IBM Global Security Kit (GSKit) maintenance applied alongside IHS.
Solution: Ensure the JVM being used to run Ikeyman does not have a file named gskikm.jar under $JAVA_HOME/jre/lib/ext/. If this file exists move it into a different directory (changing the filename alone is not sufficient).
For IHS 6.0.x, this is in the _jvm subdirectory within the IHS installation directory.
For IHS 6.1.x and later, this is in the java subdirectory within the IHS installation directory, and the file should always be moved.
For IHS 7.0, the file must not be moved on Solaris and HP-UX, but may optionally be removed to use the legacy v7 certificate management on other operating systems.
Ikeyman: Can't create CMS key database, error loading native CMS library¶
IHS 7.0 before 7.0.0.5 can report this error when installed from the 64-bit supplement CDs, because of a mismatch between the architecture of the bundled Java and the architecture of the bundled IBM Global Security Kit (GSKit).
This is resolved automatically at version 7.0.0.5 and later.
IHS 6.1.0.11 and earlier, installed from 64-bit WebSphere images (including images for 64-bit Linux and Windows), cannot create CMS key databases because the bundled 64-bit JRE cannot properly load the 32-bit IBM Global Security Kit (GSKit) CMS native library.
This problem is corrected in IHS 6.1.0.13 and later, where both a 32-bit and 64-bit IBM Global Security Kit (GSKit) are installed on 64-bit platforms. The updated ikeyman script will choose the appropriate version of IBM Global Security Kit (GSKit) to run based on the architecture of the JRE found at runtime.
If the following message in the console accompanies the CMS native library error while runnning gsk7cmd or ikeyman:
"The library initialization routine was not successfully called."Update the JRE bundled with IHS to the latest level (Using the latest WASSDK fixpack)
Remove gskikm.jar after the update,
mv IHSROOT/java/jre/lib/ext/gskikm.jar IHSROOT/lib
This symptom has been observed in IHS 7.0 with GSkit 7.0.4.27 on AIX.
IHS 6.1 and earlier: Ensure the JVM being used to run Ikeyman does not have a file named gskikm.jar under $JAVA_HOME/jre/lib/ext/. See removing gskikm.jar.
All releases prior to 7.0: Validate your global IBM Global Security Kit (GSKit) installation
IHS 2.0.42 on PPC Linux: Ensure /usr/lib/libgsk6cms.so exists and is a valid symlink
IHS 2.0.42: Ensure /usr/lib/libgsk5cms.so exists and is a valid symlink
IHS 1.3.28/2.0.47 on IA32 Linux: Ensure /usr/lib/libgsk7cms_gcc295.so exists and is a valid symlink
IHS 1.3.28, 2.0.47, 6.0, 6.1: Ensure /usr/lib/libgsk7cms.so exists and is a valid symlink
Linux: Make sure RPM compat-libstdc++ is installed (ldd `which gsk7ver` should not be missing any libraries)
Solution: install compat-libstdc++ which provides libstdc++.so.5
New private key only has 1023 bits instead of 1024 (or 2047 bits instead of 2048)¶
A defect in older levels of Java causes ikeyman to create new certificates with a 1023 bit private key instead of a 1024 private bit key.
If the procedure in the following technote has been followed to enable
2048 bit keys for v6.0, then this same problem can cause the key to be
2047 bit insetad of 2048:
The iKeyman utility within IBM HTTP Server V6.0 does not provide the
option to create a certificate request (CSR) 2048 key
size
Solutions:
For levels prior to IHS 6.0, the system JRE (based on JAVA_HOME environment variable) is used for key management. Upgrade to the latest service release of IBM Java 1.4.2 or 1.5
{Note: IHS 6.0 uses a bundled Java that cannot be upgraded}Install IHS 6.1 or later on any supported OS + a recent WASSDK fixpack (to update the java), and use that system for key management.
Use gsk7capicmd to create the certificate request since it doesn't have a Java dependency.
Error received during import/receive: "Error processing X509 certificate" ¶
This error is typically seen in Ikeyman when one or more X509 extensions present in the certificate are malformed. Some common errors follow.
If your certificate is part of an existing key database, you can extract the certificate with the following command. Otherwise, you can analyze your certificate directly.
gsk7cmd -cert -extract -label label -db key.kdb -pw abc123 -format ascii -target cert.arm
Causes follow.
Stray newlines in CA certificate file¶
View the certificate you're trying to add in a text editor. Some
versions of Java will reject your certificate if there is whitespace
or newlines before or after the "-----BEGIN CERTIFICATE-----" or
"-----END CERTIFICATE-----" separators.
In ikmtrace.log (or debugTrace\*)(`ikeyman -x` output), the Java
exception looks like:
\<KeyStoreManager::addCACertificate 9999-4,
CertificateException=java.security.cert.CertificateException: Unable
to initialize, java.io.EOFException
IssuerAltName/SubjectAltName is present but value is null¶
Use the openssl command line tool to view the extension in question:
`openssl x509 -in cert.arm -noout -text|grep -C1 EMPTY`
```
X509v3 Issuer Alternative Name:
<EMPTY>
```
Solution: CA needs to recreate compliant certificate
Certificate contains malformed T.61 string¶
A frequent problem with T.61 strings is the encoding of anything but
the most basic of non US-ASCII characters, including symbols like a
dollar sign ('$') or hash mark ('\#') or anything with the "high
order bit" set:
Use openssl command line tool to view the ASN.1 structures:
`openssl asn1parse -in cert.arm | grep
T61`
```
503:d=5 hl=2 l= 23 prim: T61STRING :MyCA $50 Warranty
```
In this second example, the cert has some non US-ASCII bytes in a
certificate field, any unprintable output in the first command, or
any output at all in the second, is a sign to try the workaround
below:
`$ openssl asn1parse -in clientcert.arm | egrep 'T61' `
```
223:d=5 hl=2 l= 14 prim: T61STRING :Testbruger ���
```
`$ openssl asn1parse -in clientcert.arm | egrep 'T61'|od -t x1|egrep
'[89abcdef][0-9a-zA-Z]' `
0000060 54 65 73 74 62 72 75 67 65 72 20 e6 f8 e5 0a
Solution: CA needs to recreate compliant certificate; note that
RFC3280 deprecates the use of T61String in new certificates.
Workaround: Set environment variable GSK\_T61\_AS\_LATIN1=YES in
\<ihsinst\>/bin/envvars
Invalid NameConstraints extension¶
The NameConstraints extension must not be empty, or Java will reject
the certificate.
- Find the offset of the
extension
openssl asn1parse -in CA.cer | grep :2.5.29.30 | awk -F: '{print $1}'
If there's no output, the certificate doesn't have this bug.
- Find the value of the extension using the offset from the first
step (a decimal number)
```
openssl asn1parse -in CA.cer -strparse number
```
- If there's no output, the certificate is invalid and must be
fixed by the customer.
Import, or receive of a personal certificate fails ¶
The *.cer sent by your certificate authority is normally a single X509 certificate, but some issuers provide what amounts to an entire keyring embedded in the *.cer and ikeyman cannot use this directly.
This format is supported differently by different IBM tools over time.
If the the file you're given is clearly named with a "p7b" extension or if
If openssl x509 -in certificate.cer -text produces an error
message but openssl pkcs7 -print_certs < certificate.cer does
not, then you have a PKCS7 file.
Even when a CA stores your issued certificate and some signers in a P7B file, they may still require you obtain the root certificate from their website.
PKCS7B limitations¶
gskcmd -cert -receiveadds all signers, including the root, and receives any personal certificates that matches an outstanding certificate request in the KDB.Ikeyman behaves the same way as gskcmd. If you trust the contents of a PKCS7B file, this is the fastest way to receive a certificate and all its signers.
If the root trust anchor is not present, you'll still have to obtain it and add it to the key store separately.
gskcapicmd -cert -receivewill not store any signing certificates, it only recieves the issued personal certificategskcapicmd
-cert -addcan be used to add the signers individually after extracting them from the PKCS7B file. See the next section.gskcapicmd
-cert -addin IBM Global Security Kit (GSKit) 8.0.55.25 and earlier will add all signer certificates, including the root, and receive the issued personal certificate.If the root trust anchor is not present, you'll still have to obtain it and add it to the key store separately.
gskcapicmd
-cert -addafter IBM Global Security Kit (GSKit) 8.0.55.25 behavior depends on whether a personal cert matching a certificate request is found in the KDB.If no matching certificate is found, the signers including the root are added to the KDB.
If a matching certificate is found, the personal certificate is received and the non-root signers are added to the KDB.
gskcapicmd
-cert -addcan be used to add the root signer individually after extracting them from the PKCS7B file. See the next section.
Splitting a PKCS7B file¶
The output of openssl pkcs7 -print_certs < certificate.p7b
command lists a series of certificates. To serialize each certificate
to a file, the following command creates p7cert*.pem files in your working directory:
rm -f p7cert*.pem
openssl pkcs7 -in from-my-ca.p7b -print_certs|awk 'split_after==1{n++;split_after=0} \
/-----END CERTIFICATE-----/ {split_after=1} \
{if(length($0) > 0) print > sprintf("p7cert%d.pem", n)}'
You can view details of each certificate with a command similar to the following:
$ for cert in p7cert*.pem; do echo "$cert: " && openssl x509 -in $cert -text|egrep -i "(Subject|Issuer):"; done
p7cert0.pem:
Issuer: C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
Subject: C = US, ST = New Jersey, L = Somewhere, O = "Example Org", CN = www.example.com
p7cert1.pem:
Issuer: C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2009 Entrust, Inc. - for authorized use only", CN = Entrust Root Certification Authority - G2
Subject: C = US, O = "Entrust, Inc.", OU = See www.entrust.net/legal-terms, OU = "(c) 2012 Entrust, Inc. - for authorized use only", CN = Entrust Certification Authority - L1K
And add or receive the certificates and validate a certificate by label:
$ bin/gskcapicmd -cert -add -db conf/key.kdb -stashed -file p7cert1.pem-label intermediate
# root.pem obtained separately
$ bin/gskcapicmd -cert -add -db conf/key.kdb -stashed -file root.pem -label root
# The cert we have a CSR for
$ bin/gskcapicmd -cert -receive -db conf/key.kdb -stashed -file p7cert0.pem
# Review certs
$ bin/gskcapicmd -cert -list personal -db conf/key.kdb -stashed
*- mycert2021
# The issued certificate will usually be first, aka p7cert0.pem:
$ bin/gskcapicmd -cert -validate -db conf/key.kdb -stashed -file p7cert0.pem -label mycert2022
Empty User Notice extension¶
If the User Notice field of the Certificate Policies extension is an empty sequence, the level of Java in the base IHS 6.1.0.0 install will throw an exception during certificate management.
$ openssl x509 -in failing.arm -text | grep -C1 "User Notice:"
CPS: 1.2.4.5.5
User Notice:
^^^^^^^^^^^^^^^ whitespace
$ dumpasn1 failing.der | grep -A2 unotice
686 8: OBJECT IDENTIFIER unotice (1 3 6 1 5 5 7 2 2)
696 0: SEQUENCE {}
^^^^^^^^^^^ empty sequence
Solution: Upgrading the bundled JRE to latest available SDK maintenance will fix the problem.
6.1 release
Download the "Java SDK" fix pack for the current fix pack release (example name:6.1.0-WS-WASSDK-AixPPC32-FP0000017) and use the Update Installer to apply it to the IBM HTTP Server installation just as you would for an IBM HTTP Server fix pack.6.0 release
The bundled Java for this IBM HTTP Server release cannot be updated. Set JAVA_HOME to point to a recent WebSphere-provided level of the JDK and rungsk7ikmdirectly.
DSA key used in server certificate¶
When software other than ikeyman is used to create a certificate or certificate request, the servers certificate may use a DSA key instead of an RSA key. DSA keys are not supported as server certificates.
When such a certificate is selected for an SSL handshake, IHS issues the following message:
SSL0210E: SSL Handshake Failed, ERROR validating ASN fields in certificate.
openssl x509 -in cert.arm -text|grep "Public Key Algorithm"
Public Key Algorithm: dsaEncryption
^^^
To check In ikeyman, select the detailed view of the personal certificate and find the "Subject Public Key Information" field. A value of OID 1.2.840.10040.4.1 signifies a DSA public key.
Solution: Create a new certificate request, either using ikeyman or forcing the use of an RSA key.
Issuer does not exist in secondary key database while using cryptographic accelerator¶
When cryptographic hardware is in use, iKeyman will sometimes report "Error processing X509 certificate" instead of "All the signer certificates must exist in the key database" when the issuer of a certificate you're receiving is not present in your secondary key file.
Verify the issuer of the certificate you're trying to receive exists in the "Signer Certificates" section in iKeyman
Ikeyman: The specified database has been corrupted (Strong Encryption).¶
Often PKCS12 files (or other Key Databases) use strong encryption that is not available in the default JCE policy files provided by java.
To test if the cryptography level in your PKCS12 file exceeds the JCE defaults, use the keytool command supplied in your JRE:
keytool -list -v -keystore /tmp/your.p12 -storetype pkcs12 -storepass password
... Unsupported keysize or algorithm parameters
Solution: Install the appropriate JCE policy files for your JRE:
Java 5 on all platforms, or Java 1.4.2 on AIX, Linux, Windows IBM
unrestricted JCE policy
files
Java 1.4.2 on HPUX, Solaris: Sun unrestricted JCE policy files
Ikeyman: The specified database has been corrupted
Outside of PCKS12 keystores, the most common cause of this message is using an incorrect password. If IHS can service SSL requests (even if clients terminate the handshake due to an expired individual certificate) with the KDB, that means the KDB is not corrupt and the password being provided does not matched the stashed password.
Ikeyman: "Validation failed: Unsupported format or algorithm."¶
Ikeyman versions prior to version 8.0.399 may report this error message
when receiving a cert into the database.
In the instance where this was experienced, the cert was a SAN cert. The
error displayed, but the cert was received anyway.
Solution: Obtain Ikeyman 8.0.399 or newer. Since Ikeyman ships with Java, you will need to update the IHS Java. This can be done by installing a WAS SDK iFix over your IHS install. For IHS 8.0 and 8.5.5, the PI27099 iFix (available approximately Nov. 2014) will have the newer Ikeyman. For IHS 7.0, it is currently unknown which SDK iFix will contain the newer Ikeyman, but it is possible that the PI27101 iFix (available approximately Nov. 2014) will have it.
gsk7cmd reports: "Exception in thread "main" java.lang.NoClassDefFoundError: com/ibm/asn1/ASN1Exception"¶
Solution: Rename $JAVA_HOME/lib/ext/gskikm.jar to
$JAVA_HOME/lib/ext/gskikm.bak as described
here
erroneously reported "A duplicate certificate already exists in the database"¶
If you cannot add a CA certificate using ikeyman, and you're sure it's
not actually a duplicate, try adding the CA with gsk7capicmd to check
if a different error code is issued.
gsk7capicmd -cert -add -db key.kdb -pw YOURPASSWORD -file your_issuer.arm -label myca
If the error reported by gsk7capicmd is
GSKKM_ERR_VALIDATION_KEY_SIGNATURE, and you're using IBM Global Security Kit (GSKit) 7.0.4.1
or higher, see DER encoding error
below.
Certificate validation failures with IBM Global Security Kit (GSKit) 7.0.4.1 and higher¶
TLS certificates use an encoding called DER (Distinguished Encoding Rules). A certificates signature is calculated over this DER encoding. A common problem for cert validation involves two related but different causes:
A certificate is issued that clearly contains an invalid DER encoding. The certificate MUST be viewed before it has ever entered a IBM Global Security Kit (GSKit) KDB to determine this.
A certificate is issued that contains a valid DER encoding but the signature matches an invalid DER encoding (BER).
The most pervasive invalid DER encoding involves "default values" for certain sequences within the certificate, such as the "critical:FALSE" (critical: 0) flag on most certificate extensions. These should never appear in the DER encoding of a certificate, and if they appear in .cer/.crt/*.pem files provided by the CA then it is a clear red flag that the wrong representation may have been signed;
ASN.1 encoding problem deep dive¶
Below is the definition of each TLS certificate extension.
Extension ::= SEQUENCE {
extnId EXTENSION.&id ({ExtensionSet}),
critical BOOLEAN DEFAULT FALSE,
extnValue OCTET STRING
-- contains a DER encoding of a value of type &ExtnType
-- for the extension object identified by extnId
}
In the example below of an invalid DER encoding, the beginning "SEQUENCE" marks a new certificateExtenson whose first field is the id of a particular extension (formatted for us by openssl below as "X509v3 Basic Constraints") and whose second field is a boolean with a default of false. A proper DER encoding, and thus a proper signature value, would not include the "BOOLEAN" line with a value of 0 below:
Note: You must use as input the certificate originally issued by the CA, not one that has been previously imported into an IBM keystore. In some versions of the IBM tools, the DER encoding will be canonicalized during the add/receive operation.
$ openssl asn1parse -in /tmp/pmrs/ok-ca.cer|grep -B1 'BOOLEAN :0'
507:d=4 hl=2 l= 15 cons: SEQUENCE
509:d=5 hl=2 l= 3 prim: OBJECT :X509v3 Basic Constraints
514:d=5 hl=2 l= 1 prim: BOOLEAN :0 <--- criticality field, should not be present, or part of signature, in DER form
517:d=5 hl=2 l= 5 prim: OCTET STRING [HEX DUMP]:30030101FF
Each extension must be checked against its definition in the OID database, as there are also extension-specific fields which may contain default-valued fields that must NOT be encoded with their default values.
Solution¶
Contact your certificate authority and provide them the info above to re-issue your certificate (or CA, depending on which is invalid). No workaround or circumvention is possible as improper DER encoding causes invalid cryptographically secure signatures which various software will need to check.
Cannot import certificate encrypted with a password that differs from keystore password¶
Some non-IBM Global Security Kit (GSKit) certificate management utilities allow you to create private keys with one password and store them in a keystore with a different password. In IBM Global Security Kit (GSKit) utilities, it is assumed the private key and keystore password are the same.
If you try to import a personal certificate of this type, IBM Global Security Kit (GSKit) will report that the private key is corrupted or unsupported, because it tries to decrypt it with the keystore password.
The only known workaround is to use whatever native tool created the keystore and change the passwords.
Duplicate certificate label for personal certificates¶
In IBM HTTP Server v7 or later, the Ikeyman v8 bundled with java lists
some personal certificates twice. This will be fixed in future JSSE
maintenance. In the interim, backing up then removing the key.rdb
request database will remove the GUI abnormality.
There is no functional problem when the personal certificate label is duplicated.
Cannot renew Verisign certificate¶
When issuing a certificate, Verisign may add custom text to the Distinguished Name of the requested certificate in several Organizational Unit (OU) fields. When you renew this certificate via ikeyman, the Certificate Signing Request (CSR) will include the additional OU fields originally added by Verisign.
Verisign may reject the CSR because it already explicitly contains the Verisign OU.
Solution: Create a new certificate signing request instead of clicking "renew" in Ikeyman.
Adding certificates before they're valid¶
If an end-entity or issuer certificate is created with its beginning validity date in the future, it cannot be added to a KeyFile via ikeyman.
If the validity date is a short amount of time in the future due to differences in system time, as opposed to being intentionally post-dated, wait until the time on the ikeyman system "catches up" to the time on the system where the certificate was issued.
If the certificate's validity date is far in the future, consider re-issueing the certificate with the current date.
If the certificate cannot be re-issued with the current date within the validity range, wait until the certificate is valid to add it to your KeyFile.
If you cannot wait until the validity date to add the certificate to your KeyFile, copy the KeyFile to a workstation where the system date can be set to a date within the certificate's validity range and add the certificate on that system. When the issuer or end-entity certificate becomes valid, IHS will not begin to use it until IHS has been restarted within the validity range.
Signers deleted while changing password¶
During a password change, Ikeyman 8.0.373 and 8.0.375 can delete signer certs. It is resolved by JDK APAR IV42756 which includes Ikeyman 8.0.378.
IKeyman and gsk7cmd shows error code 23 when processing a kdb file (non-UTF8 chars not supported)¶
If you get this error, then it is possible that a certificate contains
non-UTF8 characters in the label. IKeyman and gsk7cmd (the Java-based
tools) do not support non-UTF8 characters in certificate labels.
In one particular situation, a certificate was determined to have
non-UTF8 quotation marks ("") in the label, which resulted in a
NullPointerException and the error code 23.
The fix is to correct the label names to contain ONLY UTF-8 characters (removing the non-UTF8 quotation marks in the example cited above).
An improvement to IKeyman is planned for IKeyman 8.0.383 & CMSPROVIDER 2.46 (or higher) to display a better error message than just error code 23, but it still won't support the non-UTF8 characters.
Note: gsk7capicmd and gsk8capicmd ('C' based tools) are both able to successfully list certs containing these non-UTF8 characters in the labels, but the labels are still considered to be 'defective' and the non-UTF8 characters should be removed. Future versions of gskcapicmd will be updated to show a warning on labels that contain non-UTF8 characters, but will otherwise still process them as before. The documentation for gskcapicmd will also be updated to state that only the UTF-8 character set is supported for keystore labels, and that use of characters outside of UTF-8 is unsupported and the behavior is undefined.
Failure using Verisign certificates with IE under TLS1.2¶
If a new certificate is enabled, or TLS1.2 is enabled for the first time, and Internet Explorer clients abruptly close the connection, check the certificate chain on the server to make sure no md2 or md5 signature algorithms are used between the servers certificate and the root CA.
One such signer certificate, "Class 3 Public Primary Certification Authority", was re-issued with a SHA signature, and must be updated from Verisigns website in the IHS keystore.
Ikeyman 8.0.375 and later contains the SHA version of this certificate as an option in the "populate keystore" dialog. The earliest IKeyman version to contain the SHA version of this certificate that is available with an IHS fixpack install is 8.0.378 and gets installed with the Java that is installed with the IHS 8.0.0.7 and 8.5.5.1 fixpacks.
Hangs or delays using key management tools and VMware¶
On systems running inside of VMware, ikeyman and related tools can encounter a shortage of random data and appear to hang after many certificate operations.
Solution: Perform complicated key management tasks on a native platform or retry with the following VMware configuration option set to 'false'
monitor_control.virtual_rdtsc
Mustgather: Gathering documentation for IHS support when problem is not a known issue¶
Ikeyman problems¶
IHS 6.1 and later support a "-x" flag to ikeyman to collect traces. IHS 6.1 writes them to your working directory, and IHS 7.0 and later write them to /tmp.
Restart Ikeyman with
<ihsroot>/bin/ikeyman -xRecreate the problem and record the exact steps you take as well as the difference in expected/observed results into a text file (steps.txt). Append the KDB password to the list of steps.
IHS 6.1 and earlier: Collect $PWD/ikm* $PWD/jnitrace*
IHS 7.0 and later : Collect /tmp/ikeyman*, /tmp/ikm* (Ikeyman v7 only), /tmp/jnitrace*, and /tmp/debugTrace* (Replaces /tmp/ikm* in Ikeyman v8).
If you also get unexpected output with
gsk7capicmd, set the native environment variable GSKCAPICMD_TRACE_FILE=/tmp/gskcapicmd.trc and reproduce your issue.Send the following to IBM support:
The version-specific Ikeyman logs described above
If applicable, the output of gsk7capicmd (gskcapicmd) as well as its resulting trace file.
KDB file in use along with accompanying .sth/.crl/.rdb files.
steps.txt, which contains the steps and expected/observed results as well as the KDB password.
httpd.conf
Any intermediate certificate provided by Certificate Authority
Details of cryptographic token configuration described above (pkcsconf output), when appropriate.