IBM WebSphere Application Server Performance Cookbook

Linux network issues

Choose either the basic analysis with low overhead to check for a few common issues or advanced analysis with higher overhead for deeper diagnostics.

Basic analysis

The following data gathering has trivial performance overhead and is completely safe to do in production.

  1. If a browser client is involved, start a HAR file collection in the browser if possible.
  2. Reproduce the problem
  3. Gather the following on all sides of the suspect conversation(s) during the suspected problem (set TARGET to the relevant destination host or IP): 
    TARGET=...
netstat -s > diag_netstat_s_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
netstat -i > diag_netstat_i_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
netstat -antop > diag_netstat_antop_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
cat /proc/net/dev /proc/net/snmp* /proc/net/netstat /proc/net/tcp* /proc/net/udp* /proc/net/if_inet6 /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/cpu/*/*/cpu.stat > diag_proc_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
vmstat -tn 2 2 > diag_vmstat_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ping -c 10 -n $TARGET > diag_ping_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
mtr --report-wide --show-ips --aslookup --report-cycles 10 $TARGET > diag_mtr_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
  4. After the above commands complete, wait 30 seconds
  5. Gather another set of everything in step 3 above
  6. Upload all the diag* files, HAR files if gathered, and describe the approximate time of the issue and all related IP addresses, ports, and whether communication is TCP or UDP.

Advanced analysis

The following data gathering may have some performance overhead:

  1. Start network trace, ideally on all sides of the conversation(s) at the same time:
    1. If traffic is encrypted, then only gather packet headers with -s 100. Set INTERFACE to the relevant network interface name (or use any although this can cause issues). -C is maximum size per file in MB and -W is maximum files, so this example is up to a 10GB rolling collection: 
      INTERFACE=...
sudo sh -c "date >> diag_nohup_$(hostname).txt && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 100 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap >> diag_nohup_$(hostname).txt 2>&1 &) && sleep 3 && cat diag_nohup_$(hostname).txt"
    2. If traffic is not encrypted and overhead is acceptable, capture entire packets with -s 0. Replace $INTERFACE with the relevant network interface name (or use any although this can cause issues). -C is maximum size per file in MB and -W is maximum files, so this example is up to a 10GB rolling collection: 
      INTERFACE=...
sudo sh -c "date >> diag_nohup_$(hostname).txt && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 0 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap >> diag_nohup_$(hostname).txt 2>&1 &) && sleep 3 && cat diag_nohup_$(hostname).txt"
    3. For an OpenShift pod, perform the above using nsenter.
  2. If a browser client is involved, start a HAR file collection in the browser if possible.
  3. Reproduce the problem
  4. Stop network trace: 
    sudo pkill -INT -f diag_capture
  5. Upload all the diag* files, HAR files if gathered, and describe the approximate time of the issue and all related IP addresses, ports, and whether communication is TCP or UDP.

Analysis

Common things to check for:

  1. TCP:
    1. Increases in "segments retransmitted" relative to "segments sent out". The retransmission percentage is (segments retransmitted / (segments sent out + segments retransmitted)) * 100. In general, a retransmission rate greater than ~1% for LAN traffic is concerning. 
      Tcp:
    5863279049 segments sent out
    447776 segments retransmitted
  2. UDP:
    1. Increases in "receive buffer errors" relative to "packets received": 
      Udp:
    191344993 packets received
    61085 packet receive errors
    61085 receive buffer errors
    2. Increases in "send buffer errors": 
      Udp:
    368 send buffer errors