IBM WebSphere Application Server Performance Cookbook

Linux network issues

Choose either the basic diagnostics with low overhead to check for a few common issues or advanced diagnostics with higher overhead.

Basic diagnostics

The following data gathering has trivial performance overhead and is completely safe to do in production.

  1. If a browser client is involved, start a HAR file collection in the browser if possible.
  2. Reproduce the problem
  3. Gather the following on all sides of the suspect conversation(s) during the suspected problem: 
    netstat -s > diag_netstat_s_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
netstat -i > diag_netstat_i_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
netstat -antop > diag_netstat_antop_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
cat /proc/net/dev /proc/net/snmp* /proc/net/netstat /proc/net/tcp* /proc/net/udp* /proc/net/if_inet6 /proc/loadavg /sys/fs/cgroup/cpu.stat /sys/fs/cgroup/cpu/*/*/cpu.stat > diag_proc_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
vmstat -twn 1 4 > diag_vmstat_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ip -color=never address > diag_ipaddress_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ip -color=never route show > diag_iprouteshow_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ip -color=never -s link > diag_iplink_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ulimit -a > diag_ulimit_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ss -amponetOi > diag_ss_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
  4. If there is a particular path of interest set TARGET to the IP address on each side and run: 
    TARGET=...
ping -c 10 -n $TARGET > diag_ping_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
mtr --report-wide --show-ips --aslookup --report-cycles 10 $TARGET > diag_mtr_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
ip route get $TARGET > diag_iprouteget_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1
  5. After the above commands complete, wait 30 seconds
  6. Gather another set of everything in the steps above
  7. Gather Linux kernel logs: 
    journalctl -S yesterday > diag_kernel_$(hostname).txt 2>&1
cat /proc/cmdline /proc/cpuinfo /proc/meminfo /var/log/messages /var/log/syslog >> diag_kernel_$(hostname).txt 2>&1
sysctl -a >> diag_kernel_$(hostname).txt 2>&1
  8. Create a tar of all the output files: 
    tar czhvf diag_$(hostname).tar.gz diag*
  9. Upload diag*tar.gz, HAR files if gathered, and describe the approximate time of the issue and all related IP addresses, ports, and whether communication is TCP or UDP.

Advanced diagnostics

The following data gathering may have some performance overhead:

  1. Start network trace, ideally on all sides of the conversation(s) at the same time:
    1. List the network interfaces: 
      ip a
    2. If traffic is encrypted, then only gather packet headers with -s 100. Set INTERFACE to the relevant network interface name (or use any although this can cause issues). -C is maximum size per file in MB and -W is maximum files, so this example is up to a 10GB rolling collection: 
      INTERFACE=...
sudo sh -c "date >> diag_nohup_$(hostname).txt && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 100 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap >> diag_nohup_$(hostname).txt 2>&1 &) && sleep 3 && cat diag_nohup_$(hostname).txt"
    3. If traffic is not encrypted and overhead is acceptable, capture entire packets with -s 0. Replace $INTERFACE with the relevant network interface name (or use any although this can cause issues). -C is maximum size per file in MB and -W is maximum files, so this example is up to a 10GB rolling collection: 
      INTERFACE=...
sudo sh -c "date >> diag_nohup_$(hostname).txt && (nohup tcpdump -nn -v -i $INTERFACE -B 4096 -s 0 -C 1024 -W 10 -Z root -w diag_capture_$(hostname)_$(date +%Y%m%d_%H%M%S).pcap >> diag_nohup_$(hostname).txt 2>&1 &) && sleep 3 && cat diag_nohup_$(hostname).txt"
    4. For an OpenShift pod, perform the above using nsenter.
  2. If a browser client is involved, start a HAR file collection in the browser if possible.
  3. Reproduce the problem
  4. Gather all of the above Basic Diagnostics
  5. Stop network trace: 
    sudo pkill -TERM -f diag_capture
  6. Upload all the diag* files, HAR files if gathered, and describe the approximate time of the issue and all related IP addresses, ports, and whether communication is TCP or UDP.

Analysis

Common things to check for in the basic diagnostics:

  1. TCP:
    1. A high rate of "segments retransmitted" relative to "segments sent out". The retransmission percentage is (segments retransmitted / (segments sent out + segments retransmitted)) * 100. In general, a retransmission rate greater than ~1% for LAN traffic is concerning. 
      Tcp:
    5863279049 segments sent out
    447776 segments retransmitted
  2. UDP:
    1. Increases in "receive buffer errors" relative to "packets received": 
      Udp:
    191344993 packets received
    61085 packet receive errors
    61085 receive buffer errors
    2. Increases in "send buffer errors": 
      Udp:
    368 send buffer errors

Common things to check for in the advanced diagnostics:

  1. TCP:
    1. The rate of non-spurious retransmissions (tcp.analysis.retransmission && !tcp.analysis.spurious_retransmission). In general, a non-spurious retransmission rate greater than ~1% for LAN traffic is concerning.
    2. TCP zero window packets
    3. Given a known expected latency between hops, multiply the number of packets in any direction times the average latency to understand basic throughput limits. TCP congestion control receive & send windows essentially control concurrent outstanding packets, so driving the increase of the windows (e.g. application send/receive buffer tuning, window size tuning, initial window size tuning, congestion control algorithms, OS tuning, etc.) may help compensate for the known latency by sending more concurrent traffic, if there's available bandwidth. Note that this may help even without seeing TCP window full.