IBM WebSphere Application Server Performance Cookbook

POSIX

Shells

Current Shell

There is a convention that if the logged in user is root, then the shell prompt character is #, while non-root users show $.

To print which shell you're currently using, try one of these options:

$ echo $0
bash
$ echo $SHELL
/bin/bash
$ ps -p $$
 PID TTY          TIME CMD
6549 pts/4    00:00:00 bash

Cross-Shell Topics

POSIX-compatible shell scripting

POSIX defines a standard for shell scripts.

Variable Expansion

Variable expansion supports the following operations:

  • Return the value of a variable or a default value: ${VARIABLE:-DEFAULT}
  • Return the value of a variable if non-null or a DEFAULT value if null: ${VARIABLE:-DEFAULT}
  • Return the length of a variable value: ${#VARIABLE}
  • Return the value of a variable with the specified suffix removed: ${VARIABLE%SUFFIX}
  • Return the value of a variable with the specified prefix removed: ${VARIABLE#SUFFIX}
Create a file in one command

Without variable expansion (single or double quotes around the heredoc word):

cat > diag.sh << 'EOF'
#!/bin/sh
OUTPUTFILE="diag_$(hostname)_$(date +%Y%m%d_%H%M%S).txt"
echo "[$(date)] Started command" >> "${OUTPUTFILE}" 2>&1
uname >> "${OUTPUTFILE}" 2>&1
echo "[$(date)] Finished command" >> "${OUTPUTFILE}" 2>&1
EOF

In most shells, a dash at the end of << will strip leading tabs (but not spaces); for example:

cat > diag.sh <<- 'EOF'
    #!/bin/sh
    OUTPUTFILE="diag_$(hostname)_$(date +%Y%m%d_%H%M%S).txt"
    echo "[$(date)] Started command" >> "${OUTPUTFILE}" 2>&1
    uname >> "${OUTPUTFILE}" 2>&1
    echo "[$(date)] Finished command" >> "${OUTPUTFILE}" 2>&1
EOF

To use variable expansion, remove the single or double quotes around the heredoc word.

If a wrapper command is needed such as sudo:

sudo sh -c "cat > diag.sh" << 'EOF'
#!/bin/sh
OUTPUTFILE="diag_$(hostname)_$(date +%Y%m%d_%H%M%S).txt"
echo "[$(date)] Started command" >> "${OUTPUTFILE}" 2>&1
uname >> "${OUTPUTFILE}" 2>&1
echo "[$(date)] Finished command" >> "${OUTPUTFILE}" 2>&1
EOF
test

The test command is often used in boolean expressions and the short-hand is []:

  • Check if a string is empty (-z could be used but this is more readable): 
    if [ "${1}" = "" ]; then
  echo "First argument missing"
  exit 1
fi
  • Check if a string is non-empty (-n could be used but this is more readable): 
    if [ "${1}" != "" ]; then
  echo "First argument missing"
  exit 1
fi
  • Check string equality (note that == is not standard but supported by many modern shells): 
    if [ "${1}" = "somestring" ]; then
  echo "First argument is somestring"
  exit 1
fi
  • Check if file exists: 
    if [ -f "${1}" ]; then
  echo "File exists and is a regular file"
  exit 1
fi
  • Check if file doesn't exist: 
    if ! [ -f "${1}" ]; then
  echo "File does not exist"
  exit 1
fi
  • Check if directory exists: 
    if [ -d "${1}" ]; then
  echo "Directory exists"
  exit 1
fi
Globs

When using a glob (*) in a terminal to expand to a list of files and/or directories, by default, files beginning with a dot (.), also known as dotfiles or hidden files, are not included in the list. To include these, first disable this behavior by running:

  • bash: 
    shopt -s dotglob
  • zsh: 
    setopt GLOB_DOTS
set

set is a command to configure options for the current context. One or more flags may be specified at the same time.

  • set -e exits the script if a command receives an error.
  • set -x prints each command to stderr before it is run.

It is common to put set -e at the top of a script to exit if an error is observed:

#!/bin/sh
set -e
case

The case structured is describe in Case Conditional Construct. Each case supports wildcards. Example:

case "${somevariable}" in
  ta0*)
    echo "First branch";
    ;;

  itc*)
    echo "Second branch";
    ;;

  *)
    echo "Default branch";
    ;;
esac
POSIX-compliant one-liners
  • Generate random character sequence of a-z, A-Z, or 0-9 (replace -w 22 with the count): 
    cat /dev/random | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 22 | head -n 1
  • Loop through a list of numbers (replace MAX=10 with the maximum; exclusive): 
    i=1; MAX=11; while [ "$i" -ne $MAX ]; do echo $i; i=$(( $i + 1 )); done
Script template
#!/bin/sh
# Comment about this script

usage() {
  printf "Usage: %s [-v] [-d DELAY] COMMAND [ARGUMENTS...]\n" "$(basename "${0}")"
  cat <<"EOF"
             -d: DELAY between executions in seconds.
             -v: verbose output to stderr
EOF
  exit 22
}

DELAY="30"
VERBOSE=0

OPTIND=1
while getopts "d:hv?" opt; do
  case "$opt" in
    d)
      DELAY="${OPTARG}"
      ;;
    h|\?)
      usage
      ;;
    v)
      VERBOSE=1
      ;;
  esac
done

shift $((OPTIND-1))

if [ "${1:-}" = "--" ]; then
  shift
fi

if [ "${#}" -eq 0 ]; then
  echo "ERROR: Missing COMMAND"
  usage
fi

printInfo() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S.%N %Z')] $(basename "${0}"): ${@}"
}

printVerbose() {
  echo "[$(date '+%Y-%m-%d %H:%M:%S.%N %Z')] $(basename "${0}"): ${@}" >> /dev/stderr
}

printInfo "started with delay ${DELAY}"

[ "${VERBOSE}" -eq "1" ] && printVerbose "Commands: ${@}"

"$@"

printInfo "finished"

bash

Change the command prompt to include more information:

$ export PS1="[\u@\t \w]\$ "

Search command history using Ctrl+R and typing and recall with ESC or execute with ENTER:

$ less /etc/hosts
$ othercommand1
$ othercommand2
$ # Now type Ctrl+R and type less
(reverse-i-search)`less': less /etc/hosts

Recall and execute the last command with !!:

$ /opt/IHS/bin/apachectl restart
$ sudo !!

Recall and execute the first previous command containing a search string with !?:

$ /opt/IHS/bin/apachectl restart
$ !?apachectl

Recall and execute the last command but replace all instances of one expression with another:

$ !!:gs/stop/start/

Use the last argument of the previous command with !!:$, or all arguments of the previous command with !!:*

Search history and re-run a particular command based on its position in history:

$ history | grep startServer.sh
527 /opt/IBM/WebSphere/profiles/node01/bin/startServer.sh server1
543 /opt/IBM/WebSphere/profiles/node01/bin/startServer.sh server2
$ !543

Print the name of the current user:

whoami

Extract all .tar.gz files into subdirectories using the name of the file without the extension:

for i in *.tar.gz; do mkdir `basename $i .tar.gz` && mv $i `basename $i .tar.gz` && pushd `basename $i .tar.gz` && tar xzvf $i && rm $i && popd; done

Extract all .tar files into subdirectories using the name of the file without the extension:

for i in *.tar; do mkdir `basename $i .tar` && mv $i `basename $i .tar` && pushd `basename $i .tar` && tar xvf $i && rm $i && popd; done

Extract all .zip files into subdirectories using the name of the file without the extension:

for i in *.zip; do mkdir `basename $i .zip` && mv $i `basename $i .zip` && pushd `basename $i .zip` && unzip $i && rm $i && popd; done

Gunzip all .gz files in a directory:

find . -type f -name "*gz" -print | while read line; do pushd `dirname $line`; gunzip `basename $line`; popd; done

Change to a directory by replacing a part of the current directory:

cd ${PWD/Dmgr01/AppSrv01}

Recall the last word of the previous command using Alt+.

Command line navigation

^ refers to the Ctrl key. The ⎇ refers to the Alt key.

Move cursor:

  • Beginning: ^a or ^xx
  • Beginning of word: ⎇b
  • End: ^e
  • End of word: ⎇f
  • First occurrence of character X to the right: ^]X
  • First occurrence of character X to the left: ^⎇X

Delete content:

  • Delete to the beginning: ^u
  • Delete to the end: ^k
  • Delete a word to the left: ^w
  • Delete a word to the right: ⎇d
  • Paste what was deleted: ^y
Global aliases

For truly global aliases, update the scripts for both interactive shells (/etc/profile.d/*) and non-interactive shells (/etc/bashrc or /etc/bash.bashrc, depending on the distribution).

First, create a shell script with the commands you want to run and place it in a common location such as /etc/globalprofile.sh:

#!/bin/sh
alias x="exit"
alias l="ls -ltrh"
export PS1="[\u@\t \w]\$ "

Then add the execute permission:

# chmod +x /etc/globalprofile.sh

Finally, append the following line to both interactive and non-interactive shell script locations (e.g. /etc/bashrc):

source /etc/globalprofile.sh

zsh

Completion

Initialize completion by putting the following at the top of ~/.zshrc:

autoload -U compinit; compinit

When you start a new terminal tab, if you receive the error "zsh compinit: insecure directories, run compaudit for list.", then run compaudit and, for each directory, run:

sudo chown -R $(whoami):staff $DIR
sudo chmod -R 755 $DIR

Script to watch netstat to create file triggers

  1. watchoutbound.sh:

    #!/bin/sh
SEARCH="10.20.30.100:50000"
SOCKETTHRESHOLD="150"
SLEEPINTERVAL="30"

while true; do
  currentcount="$(netstat -an | awk '{print $5}' | grep -c "${SEARCH}")"
  if [ ${currentcount} -ge ${SOCKETTHRESHOLD} ]; then
    echo "$(date) Threshold exceeded with ${currentcount} outbound connections to ${SEARCH}" >> /tmp/trigger.txt
  fi
  sleep ${SLEEPINTERVAL}
done

  2. chmod +x watchoutbound.sh

  3. nohup ./watchoutbound.sh &

awk

This section has been moved to awk.

Truncating Logs

While some operating systems have commands specifically for truncation (e.g. "truncate" on Linux), it is simpler and more cross-platform to simply write /dev/null on top of a file to truncate it:

cat /dev/null > file.log

This does not work with sudo because the redirection operator occurs outside of the sudo. In that case, you can use tee:

cat /dev/null | sudo tee file.log

Defunct Processes

Use ps -elf | grep defunct to monitor defunct processes.

"Processes marked <defunct> are dead processes (so-called "zombies") that remain because their parent has not destroyed them properly." (http://man7.org/linux/man-pages/man1/ps.1.html)

"A defunct process, also known as a zombie, is simply a process that is no longer running, but remains in the process table to allow the parent to collect its exit status information before removing it from the process table. Because a zombie is no longer running, it does not use any system resources such as CPU or disk, and it only uses a small amount of memory for storing the exit status and other process related information in the slot where it resides in the process table." (http://www-01.ibm.com/support/docview.wss?uid=isg3T1010692)

Defunct processes are processes that have exited and are waiting for the parent process to read its exit code. Most of the resources of the exited process are released; however, the PID, exit code, and process table entries are still resident and a persistent and large number of defunct processes can limit scalability. Every process will be defunct, but normally it is only for a short period of time. Normally, persistent defunct processes mean that the parent process is hung. In the case of WAS, this is usually the nodeagent process. To remove defunct processes, kill the parent process. Before doing this, gather diagnostics on the parent process such as performing activity on it to see if it is still alive, requesting a thread dump, and finally requesting a core dump. Killing the parent process will cause the parent process of the defunct process to become the init (1) process which will then read the exit code and allow the defunct process to finish.

Example C Program that Crashes

test.c:

#include <stdio.h>

int main(int argc, char** argv) {
  char *p = 0;
  printf("Hello World\n");
  printf("p: %d\n", *p);
  return 0;
}

Run:

$ gcc -g test.c
$ ./a.out

SSH

To bypass any configured private keys:

$ ssh -o PubkeyAuthentication=no user@host

SSH Port Forwarding

Often you can SSH into a box but other ports that you want to access are blocked by firewalls. You can create an SSH tunnel that takes traffic on your machine at some port, forwards it through an SSH connection and sends it to a different port on the target server. For example, let's say hostX has something that is listening on port 8879 which you can't access from outside that box. You can create a tunnel like this:

$ ssh -L 9999:hostX:8879 sshuser@hostX

Now you should have a listening port on localhost port 9999. You can access this port through your client program as you would access port 8879 on hostX.

This can also be done with programs such as putty by using the tunnel option:

kill

kill is used to send signals to processes. The general format of the command is:

$ kill -${SIGNAL} ${PID}

${SIGNAL} is either a number or the name of the signal.

For example, to send the equivalent of Ctrl+C to a process 123:

$ kill -INT 123

less

less is a common command to browse files and input:

$ less input

Tips:

  • Jump to the beginning of input: g
  • Jump to the end of input: G
  • Jump to a line number N: Ng
  • Go to the next input: :n
  • Search for something: /SEARCH
  • Find next: n
  • If you jump to the end of input and it says "Calculating line numbers," press Ctrl+C if you don't need to do this to stop the calculation.
  • Start tailing a log: Shift+F
  • Enable any command line option after the file has been loaded: -OPTION } Enter. Common ones:
    • Show line numbers: -N
    • Show percent progress in file: -m
      • If it shows byte X then less doesn't know how many bytes are available in total so it can't calculate a perecentage. Go to the end and then the beginning: G } g
    • Show file name: -M
    • Don't wrap lines: -S

tail

tail may be used to skip the first N lines of input using -n (N+1). For example, to skip the first line of input:

$ tail -n +2 input

sort

Sort by a particular column using -k:

$ sort -k 3 input

Sort numerically:

$ sort -k 3 -n input

bc

paste and bc may be used to sum a set of numbers from input:

$ cat input | paste -sd+ | bc

sed

sed and bc may be used to do simple math on input:

$ cat input | sed 's/$/*1024/' | bc
  • Print lines 4-10: sed -n '4,10p'
  • Delete first 5 lines: sed '1,5d'
  • Delete blank lines: sed '/^$/d'

Perl

perl is a commonly used scripting language. A perl script normally has the .pl extension and starts with this shebang line:

#!/usr/bin/env perl

Useful command line options:

  • perldoc perlrun: Man page of executing the perl interpreter.
  • -e: Specify perl code on the command line rather than a perl file.
  • -p: Run specified perl command on each line of standard input.
  • -n: Same as -p except that each line is not also printed.

For example, to convert a POSIX date epoch into a human-readable time:

$ date +%s | perl -ne 's/(\d+)/localtime($1)/e;'

The $_ variable will contain each line of the file.

Code may be run at the start and end using BEGIN {} and END {} blocks, respectively:

$ date +%s | perl -ne 'BEGIN { print "Starting\n"; } s/(\d+)/localtime($1)/e; END { print "Finished\n"; }'

Useful things to remember in perl:

  • $x =\~ /$REGEX/: Return true if $REGEX matches $x
  • $x =\~ s/$REGEX//g: Replace all occurrences of $REGEX in $x with nothing.

Commonly used regular expression tokens:

  • Match zero or more: *
  • Match one or more: +
  • Any character: .
  • White space character (space, tab, or newline): \s
  • Opposite of white space character: \S
  • Word character (a-z, A-Z, 0-9, or underscore): \w
  • Non-word character: \W
  • Digit character (0-9): \d
  • Non-digit character: \D

wget

wget may be used to execute an HTTP request:

$ wget http://ibm.com/
Saving to: "index.html"

When multiple URLs are passed to wget, if possible, wget will attempt to re-use the same TCP socket. Use Perl to automate generating the same URL many times on the command line. In the following example, 64 requests will be attempted over the same socket:

$ wget -O/dev/null `perl -e 'print "http://ibm.com/ " x 64;'`

To review response headers and (short) bodies, a useful one-liner is:

$ wget -qS http://ibm.com/ -O-

netcat (nc) / openssl s_client

When you want more control over what goes into the HTTP/HTTPS request, you can use printf and netcat or openssl s_client:

$ printf "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n" | nc 0 80
$ printf "GET / HTTP/1.1\r\nHost: example.com\r\n\r\n" | openssl s_client -connect 0:443 -ign_eof

openssl

Show a remote TLS certificate (replace both instances of localhost):

echo | openssl s_client -showcerts -servername localhost -connect localhost:8880 2>/dev/null | openssl x509 -inform pem -noout -text

find

The /usr/bin/find command searches for files recursively based on their name or metadata. Check the bottom of the Linux manual for examples.

$ find /opt/IBM/WebSphere -name server.xml
$ find /opt/IBM/WebSphere -size +100M (note: the M suffix is not portable)
$ find . -name server.xml|grep -vi Templates|xargs grep startupTraceSpecification

gpg

File Encryption

Encrypt a file for storage or transit:

$ gpg --s2k-mode 3 --s2k-count 65536 --force-mdc --cipher-algo AES256 --s2k-digest-algo sha512 -o ${OUTPUTFILE}.pgp --symmetric ${INPUTFILE}

File Decryption

Decrypt a PGP-encrypted file:

$ gpg --output ${OUTPUTFILE} --decrypt ${INPUTFILE}.pgp

touch

changes the timestamp of a file (=access/modification time) to the current date and time:

$ touch input

Tip: "touch [non-existing filename]" will create a new empty file (no directories). You can also create several new files using "touch [newfile1 newfile2 newfile3]"

Filenames and special characters

Special characters in the terminal include $ <> & |;"'\

If you'd like to use them, you'll need to precede "escape" them with a \ [back slash]. There is no way you can create a filename with a / [forward slash] or null character.

Auto completion

Use "cd + [first few letters of your filename] + TAB {+ TAB}" to change directory to files starting with the letters you specified using auto completion:

$ cd + inp + TAB {or TAB + TAB to display all options}

"ls + TAB + TAB" will open up a list of suggestions which you can use with a given command.

Keyboard shortcuts:

Tips:

  • Move to the beginning of a line: CTRL+A
  • Move to the end of a line: CTRL+E
  • Move one word backwards at a time: ALT+B
  • Delete character at cursor location: CTRL+D
  • Cut text from the cursor location to the end of the line (beginning of the line): CTRL+K (U) and use CTRL+Y to paste it back; "kill" text and "yank" it.
  • Convert the current word [at the beginning of a word] to lowercase (uppercase): ALT+L (U)

tac

Show contents of filename1 and filename2 in a reverse order:

$ tac [filename1][filename2]

nl

Add line numbers to input lines.

Types of commands

To list all available built-in shell commands for a particular shell use:

$ compgen -b

To display your command type you can use "type [command name] OR file [absolute path to command]":

$ type cd 
OR type ifconfig and file /sbin/ifconfig

Alternatively, you can use the "which" command to display an executable location/the absolute command path (not working for shell built-ins):

$ which ifconfig

To check if a command name is already in use or may be used as an alias you may use "type [potential alias name]":

$ type myalias

Combining commands

  • To combine 1stcommand and 2ndcommand etc., use "1stcommand; 2ndcommand;...." Correct commands will be executed unless you use exit: 
    $ date; cal
  • Alternatively, you can use "&&" to combine commands. However, this will only execute until it encounters an error ("short circuit evaluation"): 
    $ date && cal

Wildcards

  • Represents or matches any characters: *
  • Represents or matches a single character: ?
  • Match a range of characters: [range of characters]
  • Not match a range of characters: [!range of characters]
  • Any number from numbers of the digit range:[digit-digit]*
  • Uppercase: [[:upper:]]
  • Lowercase: [[:lower:]]
  • Digit: [[:digit:]]
  • Alphabetical: [[:alpha:]]
  • Alphanumeric: [[:alnum:]]
  • NB: To negate this use [![....:]]

htpasswd

  • Create 10 users with random passwords: 
    i=1; MAX=11; OUTPUT=users.htpasswd; while [ "$i" -ne $MAX ]; do PASSWORD="$(cat /dev/random | LC_ALL=C tr -dc 'a-zA-Z0-9' | fold -w 22 | head -n 1)"; htpasswd $(if ! [ -f "${OUTPUT}" ]; then printf "%s" "-c"; fi) -B -b "${OUTPUT}" "user${i}" "${PASSWORD}"; echo "user${i} = ${PASSWORD}"; i=$(( $i + 1 )); done

Makefiles

make uses a Makefile to automate tasks.

Makefile Basics

A Makefile is a set of rules of the form:

$TARGET: $PREREQUISITES
$TAB$COMMAND
$TAB$COMMAND
[...]

Each $COMMAND is run in its own shell.

For example:

a.out: test.c
    gcc test.c

Make will only run a rule if the last modification time of $TARGET is older than the last modification times of its $PREREQUISITES. A target without $PREREQUISITES always evaluates to needing to be executed; for example:

clean:
    rm a.out

Comments begin with #.

The first $TARGET is the default target when running make. Otherwise, specify the targets with make $TARGET...; for example, make a.out.

To ensure POSIX-compliance, the first non-comment line should be the following:

.POSIX:

Makefiles may use macros of the form $(MACRO) and these may be nested. Default values may be specified with assignment; for example:

CFLAGS=-g

Macros may be overridden with environment variables in the calling shell or on the command line with MACRO=value; for example, make CFLAGS="-g -O". The $< macro expands to the prerequisite.

Makefile Example

.POSIX:

CC=gcc
CFLAGS=-W -O -g -fno-omit-frame-pointer
PREFIX=/usr/local

all: a.out
install: a.out
    mkdir -p $(DESTDIR)$(PREFIX)/bin
    cp -f a.out $(DESTDIR)$(PREFIX)/bin
a.out: test.c
    $(CC) $(LDFLAGS) $<
clean:
    rm -f a.out

PHONY

The special .PHONY target specifies a list of targets which will be executed, if requested, even if a file name matching the target exists. For example:

When one directory contains multiple programs, it is most convenient to describe all of the programs in one makefile ./Makefile. Since the target remade by default will be the first one in the makefile, it is common to make this a phony target named ‘all’ and give it, as prerequisites, all the individual programs. For example:

all : prog1 prog2 prog3
.PHONY : all

prog1 : prog1.o utils.o
        cc -o prog1 prog1.o utils.o

prog2 : prog2.o
        cc -o prog2 prog2.o

prog3 : prog3.o sort.o utils.o
        cc -o prog3 prog3.o sort.o utils.o