# FAQ: WAS Plug-in
## Interactions with Apache
### Does the WAS Webserver Plug-in support Apache 2.4?
Yes, the WAS Webserver Plug-in supports Apache 2.4 in 8.5.5.2/8.0.09 and
later
(PI06036)
### How can I log my WebSphere-based authentication in the IHS access log?
If WebSphere is configured to use HTTP Basic Authentication, IHS can
only log the userid and password together in base64 encoded form. This
is accomplished by adding `%{Authorization}i` to your `LogFormat`
directive.
If WebSphere is configured to use form-based authentication, IHS cannot
log a username. As an exception, if the application code itself sets a
cookie or HTTP header based on the logged in userid, this cookie or
header can be logged by IHS.
Example `LogFormat` additions for logging of incoming cookies or
response headers (full information is available in the documentation for
[mod\_log\_config](http://publib.boulder.ibm.com/httpserv/manual60/mod/mod_log_config.html#logformat)).
- `%{cookiename}C` (note that this cookie would only be available on
requests that come after the login, not during the login
itself)
- `%{headername}o`
### What about mod\_dir, mod\_rewrite, and the WebSphere plug-in? (IHS 2.0 and above)
What is the the priority of the following directives?
- mod_dir(dir_module)
- mod_rewrite(rewrite_module)
- WebSpherePlugin(ibm_app_server_http_module)
mod\_dir only handles objects which can be served by IHS as static
files, so it cannot be used to redirect requests to WebSphere. mod\_dir
has the lowest priority of these modules, and the priority cannot be
changed. It will only try to handle a request if the request was for a
directory and no other module has decided to serve the request.
With IHS 2.0, mod\_rewrite always takes precedence over the WebSphere
plug-in and, with the proper configuration, mod\_rewrite can first
rewrite URLs and then the WebSphere plug-in can see the rewritten URL
and decide whether or not to serve it.
Example: Customer wants to use mod\_rewrite to change URL /home to
/servlet/home/, and customer has configured the WebSphere plug-in to
handle /servlet/\*.
Here is a mod\_rewrite directive to map /home to /servlet/home, and at
the same time pass it through to the WebSphere plug-in to allow it to
see the rewritten URL. The PT flag on the RewriteRule is what allows the
WebSphere plug-in to process the rewritten URL.
```
RewriteRule ^/home /servlet/home [PT]
```
Note: In IHS 1.3, the actual order of the LoadModule or AddModule
directives also makes a difference. The LoadModule or AddModule for
mod\_rewrite needs to come after the WebSphere plug-in is activated to
allow mod\_rewrite to rewrite URLs and then have the WebSphere plug-in
process the rewritten URL. This is not the case with IHS 2.0, where
mod\_rewrite always takes
precedence.
### How does mod\_cache interact with the WebSphere Plugin?
mod\_cache can cache content generated by the WebSphere Plugin if it has
the appropriate HTTP headers in the response, however this cache does
not interact with the Plugin ESI cache. When mod\_cache is cacheing
content generated by the WebSphere Plugin you will not see evidence of
the WebSphere Plugin being called for the cached
request.
### How can I change requests to affect how the plugin handles them?
Changing or hideing a URL from the WAS Plug-in is documented extensively
[in this page](plugin_alter_uri.html).
To hide a URL from the WAS Plug-in, set the per-request environment
variable "skipwas" to any value:
```apache
SetEnvIf Request_URI ^/+app1/+BlockMe skipwas=1
```
### Can the http_plugin.log be rotated?
Yes, by writing logs to an external program such as the rotatelogs binary provided by
IHS/Apache.
Further details in APAR PI16910: http://www-01.ibm.com/support/docview.wss?uid=swg1PI16910
Example syntax:
```
```
Note: The rotatelogs program does not directly support archive/deletion of old logs. The intent of rotatelogs is to allow external mechanisms
to archive/delete logs that are not actively in use by the server. The "logrotated" package on linux is an example of taking further action
on rotated logfiles. Simple archive/compress/delete can be accomplished by passing a custom script to rotatelogs `-p` option.
## Header related questions
### Requests handled by the plugin/WAS have the wrong Content-Type
The solution to this is almost always to configure WebSphere Application
Server to provide the correct Content-Type with the response, not to try
to "fix" it in IHS or the plugin.
If WAS is providing no Content-type in the response, then IHS will add
the [DefaultType](http://publib.boulder.ibm.com/httpserv/manual70/mod/core.html#defaulttype)
usually `text/plain`. In IHS 9.0 or later, DefaultType is not configured by default.
Directives such as AddType or ForceType have no effect on responses from WAS that contain
a Content-Type header. These directives all run before the WAS Plugin and their result
is over-written.
`Header set Content-Type` or `Header edit Content-Type` can change the type successfully.
### Why does the WAS Plugin send a different encoding then the incoming request used?
By default, the WAS Plugin uses the URL that has been decoded by Apache
and split into various sub-components. It can then re-encode the
components in unexpected ways. After PM31189, you can have the Plugin
start with the URl the way the client encoded it and do no further
decoding or encoding. The apache environment variable
`websphere-nocanon` turns this feature on.
If the character you're having trouble with is '/' (%2f), you'll also
need the IHS portion of PM31189 to pass through %2f un-decoded with
`AllowEncodedSlashes NoDecode`.
## Retries, timeouts, markdowns, etc
### When will the WAS Plug-in retry a request?
A number of conditions *prevent* a request from being retried by the WAS
Plug-in.
- If the retry is due to `ServerIOTimeout` being exceeded, the value
of `ServerIOTimeoutRetry` limits the number of retries.
- If the HTTP request method is `POST`, and the request body does not
fit in the `PostBufferSize`.
Note: `PostBufferSize` defaults to 64k before 8.5.0.0, zero is the
recommended value.
- If the application server responds with a non-503 HTTP response.
Provided none of the above conditions are met, the following conditions
result in a retry:
- If the WAS plugin fails to obtain a TCP connection, or fails to
complete an SSL handshake, the request is retried on another server.
Note: Not a retry in the strictest sense, since no HTTP request is
written in this case. A failure here always results in a markdown.
- If the plugin fails to read (EOF or reset connection) from an
existing (pooled) connection, the request will be retried on the
same server.
- If the plugin waits longer than `ServerIOTimeout` seconds, for I/O
with the application server to complete, the request is retried.
- The server will be marked down if the `ServerIOTimeout` setting is a
negative value.
- If the request had affinity, and the server is not marked down as a
result of the timeout or other activity in the webserver process,
the retry will occur on the same server.
- If the application server responds with a 503 HTTP status code, the
request is retried. See the affinity related caveats above in the
`ServerIOTimeout` bullet. `MarkBusyDown` determines if the server
returning the 503 is marked down.
- The markdown of the server in this case depends on the custom
property `MarkBusyDown`
### What happens during an unplanned application server outtage?
If an application server terminates unexpectedly, several things unfold.
This is largely WebSphere edition independent.
- The application servers operating system closes all open sockets.
- WebServer threads waiting for the response in the WAS Plug-in are
notified of EOF or ECONNRESET.
- If the error occurred on a new connection to the application server,
it will be marked down in the current webserver process. This server
will not be retried until a configurable interval expires
(`RetryInterval`).
- If the error occurred on a an existng connection to the application
server, it will not be marked down.
- Retryable requests that were in-flight are retried by the WAS
Plug-in, [as permitted](#retry).
- If the backend servers use memory to memory session replication (ND
only), the WLM component will tell the WAS Plug-in to use a specific
replacement affinity server.
- If the backend servers use any kind of session persistence, the
failover is transparent.
- Session persistence is available in all websphere editions.
- New requests, with or without affinity, are routed to remaining
servers..
- After the `RetryInterval` expires, the WAS plug-in will try to
establish new connections to the server. If it remains down, failure
will be relatively fast, and put the server back into the markd down
state.
### What flexibility around timeouts is available?
APAR
[PM94198](http://www-01.ibm.com/support/docview.wss?uid=swg1PM94198)
enhances the WAS Plug-in timeouts in a few different ways, mostly
limited to Apache-based webservers.
- A timeout while waiting for the "Extended Handshake" response,
"100-Continue" response, or during the SSL handshake now always
results in a markdown of the server.
- Timeouts at this stage imply a severe resource shortage on the
application server.
- The timeout used during "Extended Handshake", "100-Continue", and
SSL handshakes can use the (typically shorter) configured
ConnectTimeout by setting the per-request environment variable
`websphere-shorten-handshake` (Apache only).
- ServerIOTimeout can be overridden by setting the per-request
environment variable `websphere-serveriotimeout` (Apache only).
- Apache-based webservers can override ServerIOTimeoutRetry by setting
the per-request environment variable
`websphere-serveriotimeoutretry` (Apache only).
Note, conditional setting of per-request environment variables in Apache
can be accomplished with the `SetEnvIf` or `RewriteRule` directive.
Examples:
```apache
SetEnvIf Request_URI "^/app1/" websphere-serveriotimeout=10
SetEnvIf Request_URI "^/app1/SlowServlet" websphere-serveriotimeout=60
SetEnvIf Request_URI "^/app1/" websphere-serveriotimeoutretry=2
SetEnvIf Request_URI "^/app1/" websphere-shorten-handshake=1
SetEnvIf Request_URI "\.(jpe?g|gif|css|js)$" skipwas=1
````
### Why might a server appear to be used or marked up before `RetryInterval`
There are a handful of reasons a server may appear to be used or marked up before `RetryInterval has been exceeded:
1. The use of the server may be in a different process from where the markdown occurred
2. The server may have been selected prior to the markdown. If the request completes successfully, the server will additionally be marked `up`.
3. The config may have been reloaded, or the webserver restarted, subsequent to the markdown.
## Backend connection management
### Can I disable SSL for backend connections?
In 9.0.0.4 and later, PI77874 allows the Plugin to use HTTPS backend connections for inbound HTTPS requests. This is not available in V8R5.
See [http://www-01.ibm.com/support/docview.wss?uid=swg1PI77874](http://www-01.ibm.com/support/docview.wss?uid=swg1PI77874)
### Why does it sometimes take 1-2x the ServerIOTimeout setting to report 'ServerIOTimeout fired'?
If the plugin is waiting for I/O while the webserver is sent a signal,
such as during a webserver stop or some types of child process exit such
as MaxRequestsPerChild or MinSpareThreads, the poll() system call that
waits for I/O with the specified timeout will be restarted. The pluin
will have only waited for a portion of ServerIOTimeout, and continues to
wait for a full ServerIOTimeout.
### What's the story with 100-Continue?
- Forward Expect: 100-Continue if present from client
- Add it if there's a request body
- Add it when there's no body if WaitForContinue=true (non-default) is
set for the selected server.
See ExtendedHandshake for something that is done pre-flight on new
connections only. Note that Liberty does not appear to support the
extended handshake.
### How does MaxConnections work?
MaxConnections limits the number of connections the WAS Plug-in will
open to a single application server *from a single webserver child
process*. In practice, the per-process limitation severely limits the
ability to pick a useful number.
- Crossing MaxConnections does not result in a markdown.
- MaxConnections applies even to affinity requests.
- It is usually better to drastically reduce the TCP listen backlog in
the application server and reject workload that way
### How to debug a "connection refused" error in the http_plugin log imply?
1. Verify the hostname and port in the message are correct for your JVM.
1. Verify that the JVM was running at the time of the error, if it wasn't, the error is expected.
1. Verify that the JVM had idle web container threads at the time of the message.
1. Verify a command-line client can access the hostname and port from the webserver system.
1. If this fails, Verify there is no firewall between IHS and WAS and no local firewall rules that block access.
1. Verify that if multiple IP addresses are associated with the transport hostname,
that all lead to the WAS host and the application server listens for connections
on all of them.
## Misc FAQS
### Why are my OSGI context roots not present in plugin-cfg.xml?
OSGI applications need to be mapped/targetted to a webserver just as a
normal web module, however the WAS admin console interface is slightly
different.
### Why might the plugin not reload plugin-cfg.xml on the fly / require a restart?
- No request has arrived past the `RefreshInterval`.
- When "IM for WebServers" or the "ODRLIB" is enabled, dynamic
reloading is disabled.
- In Apache-based servers, reloads of plugin-cfg.xml happen at
request-processing time after the webserver has switched to an
unprivileged userid. Ensure the full path to plugin-cfg.xml is
readable by this ID.
### Does IHS support websockets? Does the WebSphere Plug-in support websockets?
### General
- The WAS Plug-in supports websockets as of 8.5.5.3 and 9.0.0.0 and
all later releases/versions(PI17642). This support applies only to
Apache-based webservers. WebSockets connections currently have a
unique requirement that the backend transports host and port need to
be specified as host aliases in the virtual host the websocket
application is mapped to.
- IBM HTTP Server does not include any other websockets related
modules (such as mod\_proxy\_wstunnell)
- mod\_proxy\_wstunnel in vanilla Apache 2.4 should interoperate with
WebSphere Application Server websockets applications, but no support
is provided by IBM for generic HTTP proxy servers.
### Operational concerns
Using websockets presents some special concerns for Apache-based servers running
the WAS WebServer Plug-in. Each established websocket connection ties up a webserver
processing thread for the duration of the websocket connection.
The primary ways that a websocket connection terminates are:
- Client or server initiated close
- Idle timeout enforced by the WAS Plug-in
- Restart (non-graceful) of the Apache-based server instance
- Crash or other abnormal termination of the Apache-based webserver process
Beyond scalability concerns, perpetually open websockets connections will block the
_graceful_ termination of Apache processes. These occur when `MaxSpareThreads` is
less than `MaxClients`, when `MaxRequestsPerChild` is non-zero, and when the administrator
performs a "graceful restart" of the server. These very same features are often used
to limit the memory usage of Apache-based servers, but when the termination is
delayed it will have a counter-productive effect.
This leads to the following general guidance:
- WebSocket clients should recover (reconnect) in the event of TCP closure or other error.
- WebSocket backend applications should accomodate clients that reconnect on error.
- If the client/server implement a keepalive mechanism, avoid webserver features that
trigger graceful process termination:
- set `MaxSpareThreads` to the same value as `MaxClients`
- ensure `MaxRequestsPerChild` is 0
- avoid `apachectl graceful` in these environments, as they will leave processes
from the prior generation that need to be forcibly killed.
- If using z/OS or 9.0 on Linux, set ServerLimit to a value larger than MaxClients/ThreadsPerChild
- This allows exiting processes to not take up space needed by replacement processes.
### What are known limitations in large POST requests?
- The WAS Plug-in can't forward request bodies that specify a
Content-Length of 2GB or greater, until 9.0.0.0.
- The WAS Plug-in can forward a \> 2GB POST body if it is sent with
Transfer-Encoding: chunked, until 9.0.0.0.
Note that only WebSphere Liberty with servlet-3.1 can parse requests
with a 2GB or greater Content-Length header.
### What are known ESI limitations?
The ESi processor has a number of known limitations:
- Responses are buffered before being sent to the client.
- Non-text content types are scanned for ESI
includes.
### What maintenance do I need to see milliseconds in the http\_plugin.log?
PM76364 (6.1.0.47, 7.0.0.28, 8.0.0.6, 8.5.0.2) adds milliseconds on z/OS
and unix
platforms.
## Session Management questions
### What controls whether the Plug-in switches back after a failover?
By default, affinity switches back when the original affinity server
comes back. This switch back occurs because new clones are appended to
the session cookie, and the Plugin tries them left to right. WAS session
management can prepend the cookie instead, causing requests to NOT
switch back. This is recommended when any persistence with time-based
writes is used. See property "NoAffinitySwitchBack" in topic
"rprs\_custom\_properties".
### How can I avoid having the JSESSIONID cookie blown away when going between different applications
By default, a cookie path of '/' is specified and sessions are unique to
each application. The JSESSIONID, as opposed to the session itself, can
be shared by setting the Session management custom property
"HttpSessionIdReuse".
When you access the 2nd application, it will add its clone ID (according
to ["NoAffinitySwitchBack"](#switch)) but will not change the rest of
the session
cookie.
### How can IHS ignore URL parameters inserted by WebSphere URL session rewriting?
IBM HTTP Server treats the URL rewriting as part of the filesystem path
of static resources being requested. mod\_rewrite can be used to remove
this information from URL's, but care has to be taken to only change
requests that really will not be sent to WebSphere.
The first method involves putting the rewrite rules in existing
\ containers, because these will never affect WebSpehre
Requests.
- Enable mod\_rewrite by uncommenting the `LoadModule
rewrite_module...` line in `httpd.conf`.
- Find the `` container for your `DocumentRoot` and for any
Alias you've added that will be requested with a URL-rewritten
session ID.
- Inside *each* existing `` container for your DocumentRoot
and Alias'es, add this ruleset, substituting the `RewriteBase`
parameter as dictated by the URL-path of the `` context
being updated:
```
# Note: this must be in an existing container
RewriteEngine on
# The RewriteBase is "/" in the case that this block is for the DocumentRoot
# If this block is for an Alias, the RewriteBase below
# Should be the same as the first argument to the Alias directive.
RewriteBase /
RewriteRule (.*); /$1 [PT]
```
The second method puts the mod\_rewrite rules in \
context, which simplifies configuration in one way but complicates it in
that the user-supplied pattern must be used to restrict the rewrite to
static (local IHS) content. This effectively needs to mimic the
WebSphere Plugin processing to determine which URLs to remove the
URL-rewritten session info from.
- Enable mod\_rewrite by uncommenting the `LoadModule
rewrite_module...` line in `httpd.conf`.
- Determine what prefix or filename extension you want to limit the
rewriting to, and express it in a Perl Compatible Regular Expression
(PCRE).
- Inside *each* existing `` container, and once at the
bottom of httpd.conf, add this ruleset, substituting prefixes and
file extensions are your requirements dictate.
```
RewriteEngine on
RewriteRule (.*\.(?:gif|jpg|css); $1 [PT]
RewriteRule (/someprefix/.*\.(?:txt|pdf)); $1 [PT]
```
### What options are there for removing a server from routing?
The primary consideration is what happens to requests in-flight, new affinity requests,
and new non-affinity requests. Every approach listed below is safe for in-flight requests.
- Performing a normal stop of the application server is the simplest and
most effective way to stop it from taking traffic. During stop
processing, the application server immediately closes its listening
sockets and idle keepalive connections, while allowing in-flight
requests to complete for 3 minutes.
- With closed listening sockets and keepalive connections, the Plug-in
will "fail fast" when trying to route to this server and quickly mark it
down.
- If you set a servers (static/configured) weight to zero, and generate and propagate the
plugin-cfg.xml, the WAS Plug-in will only send affinity requests.
- This method of "draining" can be slow because affinity is maintained by the presence of
a session cookie which by default has no timed expiration. Long dormant browsers with expired
server-side sessions may return to the webserver w/ the draining cloneID.
- This change can optionally be done in plugin-cfg.xml manually.
- This method is used by the single-cell rollout procedure discussed
[here](_static/0412_vansickel-pdf.pdf)
- If you make a server a "backup" server in the cluster, it will not be
used while any primary (non-backup) servers are available, even if
affinity is established with the backup server. This will cause HTTP session failovers.
- Backup servers are not load-balanced, so this is not recommended if
taking multiple servers out of load balancing at the same time.
- In the UI, the "server role" is under a path like `WebSphere application server clusters > mycluster > Cluster members > member1 > Web server plug-in properties`
- In wsadmin terms:
```
plgProps = AdminConfig.list('WebserverPluginSettings', getServerByNodeAndName(nodename, servername)
AdminConfig.modify(plgProps, [['Role', backup_or_primary]])
```
- This change can optionally be done in plugin-cfg.xml manually.
- Removing (or commenting) a `` entry from plugin-cfg.xml will stop all subsequent requests,
causing affinity requests to failover.
- If you change an application servers cloneID, and generate/propagate plugin-cfg.xml,
affinity requests using the old cloneID will be re-balanced. See the `HttpSessionCloneId`
HTTP Session custom property.
- This change *cannot* be effectively done in plugin-cfg.xml manually.
- If Intelligent Management for webservers is enabled, putting the server
into one of the two maintenance modes either sends it only affinity
requests, or never sends it any subsequent requests. Pending requests
are permitted to complete.
- Using a "maintenance page" in the webserver is a way to quickly bleed traffic from all application servers.
### How to safely roll out updates to enterprise applications
There are a spectrum of methods used for rolling out an update. These methods generally try to address one or more of the
following requirements to varying degrees:
- Bleed sessions from servers running version N, for example by setting weights to zero.
- Validation of the new application version in production before all users are exposed to it
Examples include:
- Accessing the application server ports directly for validation
- Using an additional virtual host name in the updated version, possibly in an additional cluster or cell
- Fast rollback, for example by changing plugin-cfg.xml manually to pivot from servers running version N-1 to version N.
On z/OS, WAS supports "application rollout" as a first-class operation. Details of what happens during a rollout are available
[here](https://www.ibm.com/docs/en/was-nd/9.0.5?topic=environment-automatically-rolling-out-updates-high-availability-application)
Network Deployment, when used with an Intelligent Management (IM) enabled proxy servers, supports this via Application Editioning. An
explanation of what happens during rollout is available [here](https://www.ibm.com/docs/en/was-nd/9.0.5?topic=edition-algorithm-performing-rollout)
Other editions of WebSphere, or Network Deployment without an IM proxy server, can achieve many of the requirements with a series of
coordinated changes to WebSphere and manual changes to plugin-cfg.xml. More information is available [here](_static/0412_vansickel-pdf.pdf)
A basic but straighforward solution is to use a maintenance page on the proxy server in a low-traffic window.
## SSL Questions
### How is the plugin-key.kdb personal certificate used?
When a webserver definition is created in traditional WAS, a plugin-key.kdb is created that trusts
the deployment manager root CA. A personal certificate, signed by the same deployment manager CA,
is also created and assigned the label "default". This key store can be managed in the WAS admin console
and propagated to the webserver host.
In default configurations, this initial personal certificate is not used for any purpose.
In non-default configurations described below, the personal certificate may be used implicitly
or further configuration might be required.
In any case where TLS mutual auth is required, including XD_AGENT ports or collective dynamic routing:
1. The certificate label used by the Plugin can be customized by using the `CertLabel` property:
[KC Link](https://www.ibm.com/support/knowledgecenter/SSEQTP_9.0.5/com.ibm.websphere.base.doc/ae/rwsv_plugin_customprop.html#rwsv_plugin_customprop__CertLabel)
2. The certificate used must be issued by a CA that is trusted by the application server.
- If the deployment manager is removed or replaced, certificates will need to be manually created.
3. The WAS Plug-in personal certificate can be converted or replaced via the WAS admin console. Some steps related to
replacing older certificate types is documented [here](https://www.ibm.com/support/pages/node/6235840)
#### TLS Mutual Authentication required in WAS
For traditional WAS, connections from the WAS Plugin will fail if TLS mutual auth is enabled without taking one
of the following additional actions:
- Using Ikeyman, gskcapicmd, or gskcmd mark the certificate as the default certificate for the keystore.
- Note, this is different then having a label of "default".
- Customize the `CertLabel` as described in the preceding section.
For Liberty, plugin-key.kdb is manually created and administered by the end user. Obtain a certificate and ensure Liberty trusts its
issuer via Libertys configured key or trust stroe.
#### Intelligent Management for webservers (odrlib) enabled
For traditional WAS, the automatically generated certificate with label "default" will be used by default.
For Liberty collectives, a replacement certificate with separate properties is created during dynamic
routing setup.
### What's the deal with SSL breakage when moving past 8.5.5.6/8.0.0.11?
After APAR PI39126 (8.5.5.7, 8.0.0.12, 9.0.0.0), the WAS WebServer
Plugin uses modern defaults for SSL/TLS processing. This includes
disabling legacy protocols, ciphers, and certificate validation. This
may cause problems if WAS has been explicitly configured to use only
weak/export ciphers, or has been configured with a certificate chain
that does not meet contemporary standards.
In practice, problems observed after PI39126 have fit into the following
certificate processing related categories (See RFC5280 for complete
details):
1. BasicConstraints extension: All certificates used to validate
digital signatures (AKA issuers, signers, or CA's) must contain a
BasicConstraints extension with the "criticality" field set to TRUE.
2. CertificatePolicies extension: The CertificatePolicies extension
must be RFC5280 conformant across the certificate chain. The
algorithm is quite complex, but in a simplifed form an intermediate
signer cannot assert policies not also asserted by its own signer.
- A special case of this is certificate chains that list both
specific policies and the special `AnyPolicy` policy. These are
incorrectly rejected by GSKit prior to 8.0.55.10.
The certificate validation changes in PI39126 can be disabled, after
PI49893 (8.0.0.12, 8.5.5.8, 9.0.0.0) by setting the WAS Plugin custom
property certificate\_validation\_strict\_rfc5280="false".
The default of the property above is flipped in PH27968 (8.5.5.18, 9.0.5.6) so
that the checking is more tolerante by default.
### What SSL/TLS ciphers/protocols does the Plug-in support?
If the application and the WAS Plugin do not share any TLS ciphers or protocols, http_plugin.log will typically log a message with
GSK\_ERROR\_SOCKET\_CLOSED(420) and `PARTNER CERTIFICATE: DN=[No Information Available]`.
#### Protocols
- TLS 1.2 has been supported and enabled by default on all platforms since 8.5.5.10/9.0.0.0.
- TLS 1.3 is supported, but not enabled by default, on all platforms in 9.0.5.2 and later. For more information
see
#### Ciphers
- AIX, Linux, Windows, HP-UX, Solaris:
- In 9.0.5.2 and later, WAS plugin uses all available FIPS-approved ciphers by default.
- With `AutoSecurity="False"`, the WAS plugin uses a small subset of weak and moderate historical ciphers.
- `StrictSecurity` has no affect.
- In 8.5.5.7 (and later) and 9.0.0.0-9.0.5.1, the WAS plugin uses all available FIPS-approved ciphers by default.
- If `StrictSecurity="True"` is set in plugin-cfg.xml, only TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(C027) and TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(C02F) are used.
- With `AutoSecurity="False"`, the WAS plugin uses a small subset of weak and moderate historical ciphers.
- z/OS:
- In 9.0.5.2 and later, the WAS Plugin uses a broad set of ciphers by default including GCM and ECDSA based ciphers by default.
- It is possible to override the list of ciphers by setting the following native environment variables:
```
export WS_PLG_NO_CIPHERS_EXPANDED=1
export WS_PLG_GSK_V3_CIPHERS_CHAR4=1
# Multiple can be listed, see https://www.ibm.com/docs/en/zos/2.4.0?topic=programming-cipher-suite-definitions#csdcwh__telcsd for identifiers
export GSK_V3_CIPHER_SPECS_EXPANDED=C030
```
- If `StrictSecurity` has been explicitly set to `False`, the WAS plugin uses a small subset of weak and moderate historical ciphers.
- In 9.0.0.0-9.0.5.1, only TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(C027) and TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(C02F) are used by default.
- If `StrictSecurity` has been explicitly set to `False`, the WAS plugin uses a small subset of weak and moderate historical ciphers.
- In 8.5, the WAS plugin uses a small subset of weak and moderate historical ciphers by default
- If `StrictSecurity` has been explicitly set to `True`, TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256(C027) and TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256(C02F) are used.
### What does GSK\_ERROR\_BAD\_CERT (gsk rc = 414) mean?
`GSK_ERROR_BAD_CERT` has multiple causes/solutions, listed below in
order of frequency:
#### CA not trusted by plugin-key.kdb
The http_plugin.log will record a GSK_ERROR_BAD_CERT error when plugin-key.kdb doesn't trust
the root issuer of the application servers certificates.
In typical configurations where WAS issues certificates for the servers in the cell,
it's only necessary to do a one-time copy of plugin-key.kdb from the WAS
admin console. Use the button in the console under Webservers \> webserver1 \> Plug-in properties:
![](copykdb.jpg)
If self-signed certs are replaced with certificates issued by some internal or public CA,
the deployment managers copy of plugin-key.kdb will need to be updated with the new root CA
(intermediates not necessary)
- If the webserver is used with WebSphere Liberty, or used with traditional WAS but not managed by the WAS admin console
the plugin-key.kdb will have to be modified manually on the webserver system:
1. Obtain the new CA certiicate that issued your application server certificates and store it on the webserver (e.g. /tmp/ca.cer)
1. Find the path to plugin-key.kdb by checking plugin-cfg.xml
1. Add the trusted CA to plugin-key.kdb: *bin/gskcapicmd -cert -add -db <*/path/to/*>plugin-key.kdb -stashed -file */tmp/ca.cer* -label *new-ca*
1. Restart IHS.
- If the webserver is managed by WAS, the issuer certificate is maintained in a keystore in the deployment
manager, which must be synchronized with managed or unmanaged webservers via the admin console, wsadmin, or manually.
1. Obtain the new CA certificate and upload it to a temporary path on the dmgr (e.g. /tmp/ca.cer)
1. Open the WAS admin console and navigate to Webservers \> webserver1 \> Plug-in properties
1. Click 'Manage keys and certificates'
1. Click 'Signer Certificates', then 'add' to add your updated certificate authority. Click 'OK' and save any changes.
1. Propagate the plugin-key.kdb to your webserver using the instructions/image in the preceding section
1. Restart IHS.
##### Alternatives to the "copy to webserver keystore directory" button:
- Using wsadmin:
```python
def propagateKeyring(configRoot, cellName, nodeName, webServerName):
objNameString = AdminControl.completeObjectName('WebSphere:name=PluginCfgGenerator,*')
verb = 'propagateKeyring'
args = '[' + configRoot + ' ' + cellName + ' ' + nodeName + ' ' + webServerName + ']'
types = '[java.lang.String java.lang.String java.lang.String java.lang.String]'
print AdminControl.invoke(objNameString,verb,args,types)
```
- Manually: Find plugin-key.\* under your webservers dmgr config repository on the deployment manager and copy it to the path identified in plugin-cfg.xml
#### Other causes of GSK_ERROR_BAD_CERT
- On very old levels, or configurations with "AutoSecurity" set to False:
If you enable strict SP800-131a support in your backend application
server, you must also configure the `StrictSecurity` WAS Plug-in
custom property. It is strongly encouraged to instead upgrade (PI39126 8.5.5.7, 9.0.0.0)
and set "AutoSecurity" to True.
- Other generic invalid certificate processing errors that need
further investigation. Some may be resolved by PH27968 (8.5.5.18, 9.0.5.6).
#### Other more obscure GSKit errors
- GSK_ERROR_ASN at startup is usually caused by a corrupt plugin-key.rdb. If no CSR is outstanding, just delete the file.
## Intelligent Management for webservers (ODRLIB)
### Where can I learn more about IM for webservers, plug-in based ODR?
The best references for now is this topic and its children combined with
existing IM information and this EA module
* [IM for webservers](http://www-01.ibm.com/support/knowledgecenter/SSAW57_8.5.5/com.ibm.websphere.nd.doc/ae/cwve_implugin.html?lang=en)
(primary infocenter topic)
* [IM
overview](http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-dist&topic=cwve_xdovrvw)
* [8.5.5 education assistant module for Plug-in based
ODR](http://publib.boulder.ibm.com/infocenter/ieduasst/v1r1m0/topic/com.ibm.iea.was_v8/was/8.5.5.0/content/WAS855_IM_in_plugin.pdf?dmuid=20130807160921825483)
* [8.5.5 concepts and
planning](http://www.redbooks.ibm.com/abstracts/sg248022.html?Open)
(core Intelligent Management)
* [8.5.5 administration and
configuration](http://www.redbooks.ibm.com/abstracts/sg248056.html?Open)
(core Intelligent
Management)
### How do traditional configuration elements work once IM is enabled?
Most config elements specified globally in plugin-cfg.xml still apply.
Any configuration at the Server or ServerCluster in plugin-cfg.xml scope are not directly used once IM is enabled, but the same source of those values (servers configuration in the cell or values set in \ in a collective) provides the information to the IM-enabled webserver dynamically at runtime.
Where timeout related settings can be overridden in httpd.conf, they take precedence over the dynamic IM-enabled values.
* Example server-level options are handled dynamically: connectTimeout, serverIOTimeout, waitForContinue, wsServerIdleTimeout, wsServerIOTimeout
* With Liberty, the server level settings are read from \ in server.xml.
* Example cluster-level options are handled dynamically: ServerIOTimeoutRetry
* In traditional WebSphere, server weights are handled dynamically. With
Liberty, the load balancing algorithm does not use weights.
### When I enable global security, IM is no longer able to handshake with WAS for the control/REST connection
There is a limitation that prevents the IM-enabled Plugin from using the
"default keystore certificate" to communicate with the XD\_AGENT port
when that port requires TLS client authentication.
Here is what works:
- Use a certificate label of 'default' in your KDB for the cert
expected to be used on the XD agent port.
- Specify a global property (just like KeyFile and Stashfile) in
plugin-cfg.xml naming any other label. This can be set via custom
property of the same name to be included in a generated
plugin-cfg.xml
### What are the firewall requirements/issues around "Intelligent Management for WebServers?"
In original Intelligent Management topologies where IHS uses a static
plugin-cfg.xml generated by the On Demand Router (ODR), IHS only needs
to be able to access the ODR HTTP/HTTPS ports.
The ODR needs access to all of the individual application servers on
their web container ports AND the XD\_AGENT\_PORT on the node agents and
dmgrs.
In topologies with an IM-enabled IHS (i.e. using ODRLIB), the IM-enabled
IHS needs to be able to talk to all of the individual application
servers on their web container ports AND the XD\_AGENT\_PORT on the node
agents and dmgrs. There are several ways to manage this requirement:
- Permit the traffic between the IM-enabled IHS and the cell.
- Use a dedicated proxy server (such as WebSphere Datapower) in the
DMZ and move the IM-enabled IHS into a network segment where the
traffic will be permitted.
- Use a statically configured plugin-cfg.xml in the DMZ that routes to
a IM-enabled IHS deeper in the network where the traffic will be
permitted (simulating the IHS + Java ODR topology described above)
## Logging FAQs
### Can I rotate the http\_plugin.log?
PI16910 and later allows http\_plugin.log to be rotated if running in an
Apache-based server.
## Liberty-specific questions
### How can the liberty-generated plugin-cfg.xml be customized?
Some elements of plugin-cfg.xml can be specified in server.xml in the
\ element. See [the KC for
details](https://www.ibm.com/support/knowledgecenter/SSD28V_9.0.0/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/rwlp_config_pluginConfiguration.html)
Keys can be changed or added to the top-level \ element of
plugin-cfg.xml:
```
```
### What options are available for plugin-cfg.xml generation in Liberty?
Historically, plugin-cfg.xml generation was problematic under liberty,
but it improved greatly in late 2016.
- In 16.0.0.3 and later, individual Liberty servers automatically
generate a plugin-cfg.xml at startup / when new applications are
loaded. Just copy it into place.
- The collective controller provides a ClusterManager MBean that can
be used to generate a plugin-cfg.xml for a configured Liberty
cluster. The script `bin/pluginUtility` can call this bean from
the controller or an application server.
- Servers with the collectiveMember or clusterMember feature can use
IM-enabled WAS WebServer Plug-in, in which case only collective
controller metadata lives in plugin-cfg.xml.
More information is available here: [Setting up Dynamic Routing for
Liberty collectives](http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil&product=was-nd-dist&topic=twlp_wve_enabledynrout).
- Historical interest only: Individual/standalone liberty application
servers expose an MBean over JMX that writes a plugin-cfg.xml out to
the local filesystem.
- jconsole is included with every java runtime and can be used to
invoke the
mbean
More information about the WAS Plugin on Liberty is available here:
[https://www.ibm.com/support/knowledgecenter/SSAW57_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_admin_conf_webserver_plugin.html](https:////www.ibm.com/support/knowledgecenter/SSAW57_liberty/com.ibm.websphere.wlp.nd.multiplatform.doc/ae/twlp_admin_conf_webserver_plugin.html)
### What options are available for plugin-cfg.xml merging in Liberty?
What options are available for plugin-cfg.xml merging in Liberty?
- In 16.0.0.3, a merge utility was added to Liberty. The executable is
bin/pluginUtility. It provides merging of the plugin-cfg.xml
generated on standalone servers. 16.0.0.4 is required for the
function to work in non Liberty-ND distributions.
- The Liberty collective controller can generate pre-merged
plugin-cfg.xml files for the servers it manages.
- Individual/standalone liberty application servers plugin-cfg.xml
files can be merged by the scripts provided within traditional WAS
- Individual/standalone liberty application servers plugin-cfg.xml
files can be merged by the standalone command-line merge tool
available here: .
More information about the WAS Plugin on Liberty is available here:
http://www14.software.ibm.com/webapp/wsbroker/redirect?version=phil\&product=was-nd-dist\&topic=twlp\_admin\_webserver\_plugin
## Alternatives to the WAS WebServer Plug-in
While the typical configuration uses a WAS-aware reverse proxy tier for
load balancing, failover, and offload (like IHS + the WAS WebServer
Plug-in), generic HTTP reverse proxies can be used with only a handful
of functional differences.
These generic proxies range from appliance-based HTTP load balancers to
open source reverse proxy servers. Some might even have explicit WAS
exploitation, but the depth of that exploitation would need to be
discussed with the vendor. The info below assumes the server is
backend-agnostic.
Directly fronting WebSphere with a layer 3 or layer 4 device ("IP
sprayer" or NAT forwarding load balancer) that doesn't even terminate
HTTP is also an option, albeit not a very flexible one.
### Information about specific alternatives
- [Comparison of WAS-related HTTP
proxies](https://www.ibm.com/docs/en/was-nd/9.0.5?topic=wspi-selecting-front-end-your-websphere-application-server-topology)
- [https://www.nginx.com/blog/using-nginx-to-load-balance-websphere/](https://www.nginx.com/blog/using-nginx-to-load-balance-websphere/)
### What's missing in a generic HTTP reverse proxy?
- Filtering of private HTTP headers used by WAS. These are relatively
well known to some administrators who have had to look at WAS
Plug-in traces, but are strictly speaking not managed as a public
API. Headers beginning with $WS should be filtered from untrusted
clients (the WAS WebServer Plug-in and On-Demand-Router can both
whitelist IP addresses that are permitted to supply $WS headers).
- *Setting* of private headers used by WAS. An HTTP Server can
communicate various bits of data via the private headers listed
above, many of which become accessible in HTTPServletRequest APIs
(client address, client certificate). See Web Container custom property
`httpsIndicatorHeader` for one example where a generic HTTP reverse proxy
might need additional configuration if it does HTTPS offload.
- When memory-to-memory session persistence is enabled, the semantics
of the session ID cookie change with an added level of indirection.
A table of partition to clone mappings can be requested from the
application server which may change over time. Third-party proxies
are unlikely to request/parse this table and may do their own
affinity, ignoring DWLM in the application server.
- Most Intelligent Management functionality requires either an
On-Demand Router or IM-enabled WAS Plug-in to be involved in the
topology.
- Cell and collective managed generation of the reverse proxy
configuration.
- Dynamic Clustering (in either traditional or liberty profiles)
communicates the cell topology to On-Demand Routers, Datapower, or
IM-enabled WAS Plug-in in a way that third-party proxies would be
unlikely to be able to interpret. This feature would be difficult to
use without one of the above enumerated proxy servers in the request
pipeline.
#### What's missing with no HTTP proxy at all?
- Static file off-load
- HTTP Caching
- Header Manipulation
- Cookie based session affinity
- SSL OffLoad
### What is likely to work?
- Basic HTTP reverse proxy (offload, load balancing / clustering,
security enforcement, affinity)
- Websockets with Liberty
### What's better?
Some contemporary dedicated proxies are highly vertically scalable and
use very low memory. There is also a benefit to standardization if there
already exists a preferred enterprise proxy solution.
If an HTTP proxy is built into a Platform as a Service, or part of an
otherwise orchestrated container technology, it may already know how to
deal with the dynamic registration and deregistration of HTTP servers.
### What's worse?
- No IBM support of the proxy server tier. Not only would support be
arranged separately, any problems now become multi-vendor
investigations.
- Unlimited use and commercial support of IHS and the WAS WebServer
Plug-in come with every application server edition, whereas other
commerical alternatives may be separately licensed.
- Beyond true dynamic clustering, frequent changes in applications or
application servers is much simpler with generated plugin-cfg.xml
rather than hand-tended WAS-agnostic proxy server configurations.
## Debugging questions
### Why does my application get the wrong client or remote IP address or not pass authentication?
HTTPServletRequest.getRemoteAddr() and HTTPServletRequest.getRemoteHost() rely on the
WAS Plug-in passing private HTTP headers to the server. Passing of TLS client
authentication / mutual authentication between IHS and WAS also depends on this
trust being established.
- The 9.0.5.1 WAS Plug-in fails to propagate these values correclty due to PH17449. Upgrade past 9.0.5.1.
- If you recently upgraded past 8.5.5.16, 9.0.0.11, or 19.0.0.3 for the first time, configuration is required in WAS to trust
the WAS Plug-in to pass this information along:
[technote](https://www.ibm.com/support/pages/potential-websphere-application-server-problems-when-deployed-behind-websphere-aware-proxy-server)
Keywords: trustedSensitiveHeaderOrigin, trustedHeaderOrigin
### How do I troubleshoot a connection refused or ConnectTimeout error?
There are several high levels reasons this symptom might be reported, they should be
debunked or confirmed in the order presented below, which is a combination of ease in checking
and likelyhood.
A "connection refused" error implies one of the following:
1. There is no process listening on the target IP:port. Confirm
the JVM was up at the time the error was reported.
2. A firewall on either host, or an intermediate device, has blocked the connection.
3. The server is hung or overloaded to the point where it has no threads available
to accept new connections, and the OS buffer for new connections is full.
A "ConnectTimeout" error implies one of the following:
1. The application server is hung or overloaded to the point where it peroiodically has no
threads available to accept new connections in a timely fashion.
2. A firewall on either host, or an intermediate device, has blocked the connections but
is doing so in a way that silently discards packets.
3. The connect() attempts are being retransmitted due to network failures.
#### related messages
The primary errors for a refused connection looks like:
```
ws_common: websphereGetStream: Failed to connect to app server on host ...
Connection to %s:%d failed (%d) : %s (local port %d)
```
On some platforms connection timeout or connection refused errors may be reported
in non-straightforward ways, based on how the specific operating system returns
errors. On AIX for example, the following message almost universally is the same as
"connection refused". A packet capture should confirm that an `RST` was sent by the
application servers host operating system in response to the initial `SYN`.
```
ws_common: websphereExecute: AIX Non Blocking connect and first read failed; port is downn
```
The operative part of all the permutations of these messages is that a connect() operation
has failed.
#### Consequences
A connection related error always results in a markdown. If there are happening rarely, but the
loss of capacity is problematic, you can drastically reduce the `RetryInterval` to reduce the impact
of the markdown.
### How do I troubleshoot a ServerIOTimeout?
There are several high levels reasons a ServerIOTimeout might be reported, they should be
debunked or confirmed in the order presented below, which is a combination of ease in checking
and likelyhood.
1. Confirm the server was not hung or out of threads at the time
- The most common cause of a ServerIOTimeout is simply a hung application
or hung application server. This can cause a delay at any point in the processing
of a response, from establishing a connection, performing a handshake, reading the
first line of the response, or reading any subsequent byte.
- You should be able to track the free threads in the application server over time and
correlate with the timeout.
- It is also helpful to get a sequence of javacores around the time of the timeout to look
for suspect blocking operations in the application.
2. Confirm the timeouts are reasonable for the URL that failed
- The ServerIOTimeout must exceed the amount of time between I/O being available from
the application server. There is no end-to-end timeout. Check every server in plugin-cfg.xml,
sometimes they may be inconsistent.
3. Confirm the timeouts are actually occurring
- There may be a defect where the timeout being used is not the one configured, or it otherwise
pops prematurely. This can be monitored by WAS Plug-in trace and/or adding response time to the
webserver access log.
4. Confirm no intermediate network device is dropping connections
- If there is any load balancer or firewall between IHS and WAS, it could
disturb connections sitting in the WAS Plug-in's outbound connection
pool that can give the appearance of a timeout.
- If there is such a device, any timeouts it has configured with should exceed the configured
ServerIOTimeout by at least several seconds. Further, it should be configured to actively
close connections during any kind of timeout or resource shortage rather than throwing away
packets from each side of the connection.
### TRACE/DEBUG message explanations
#### Messages related to reading and writing the _response_ body
The messages below are in the chronological order they normally appear in. The
leading number/letter indicate when a choice of multiple messages may follow the previous.
- #1: DEBUG: lib_htresponse: htresponseGetContentBlock: Getting the next content block
- This message is issued immediately before the plugin will try to read any piece
of the response body from the application server.
- #2a DEBUG: ws_esi: readResponseBody, len_p=65536
- If ESI is enabled and not being "bypassed" on this response due to the Surrogate-Control
response headers, this message is issued when a read in #1 has completed. #1 and
#2a will often loop tightly due to ESI buffering the response.
- #2b: TRACE: lib_rio: Blocking for read(2), waiting 900
- This message is written if the attempted read for #1 cannot be satisfied immediately. It
is actually printed any time data is needed from the AppServer yet not available, for example
when awaiting the first line of the response. In the latter case it is much more common to see
this traced as there will often be some delay for the initial response and rarely for the N-th byte
of the response.
- #2c DEBUG: ws_common: websphereReplyToBrowser: Read body of length |65536|
- This message means the read in #1 has completed and is ready to be written to the client.
- #3: DEBUG: mod_was_ap22_http: cb_write_body: In the write body callback writing 65536
- This message is written immediately before the Plugin will pass the block read from the application
server to Apache/IHS to pass through any Apache/IHS filters and ultimately write to the client.
## Docker related questions
### Can the WAS WebServer Plug-in be run inside a docker container?
The WAS Plug-in can itself be run in a docker container, in a supported webserver
such as IBM HTTP Server. Dockerfiles are provided by IBM that include
IHS and the WAS Plug-in:
If the application server also runs in a container environment, see the entry
below.
### Can the WAS WebServer Plug-in be used to frontend WebSphere running in a container such as docker?
The WAS WebServer Plug-in can be used to frontend WebSphere Application Server
running anywhere, but the integration and responsibility of generating, merging,
and distributing the configuration and SSL trust stores varies depending on the
tools and infrastructure used.
If the WAS containers are relatively static, it is a supported configuration. Management of the
plugin-cfg.xml and plugin-key.kdb is manual.
In kubernetes environments, it is not practical nor supported to use IHS and the WAS Plugin as a
gateway. More information is available here: .
-----