IBM WebSphere Application Server Performance Cookbook

Linux eBPF memleak Recipe

On Linux >= 4.1 (e.g. RHEL >= 7.6), bcc-tools includes a memleak eBPF tool that can be used to dynamically attach to a process and monitor unfreed native memory allocations:

  1. Ensure you're logged in as root.
  2. Install iovisor BCC: 
    dnf install kernel-devel bcc-tools bpftool
  3. Once the RSS is above a concerning amount, attach BCC memleak, replacing $PID with the process ID of the JVM: 
    nohup /usr/share/bcc/tools/memleak -p $PID 300 &> diag_memleak_$(hostname)_$(date +"%Y%m%d_%H%M%S").txt &
    • Monitor the disk usage of diag_memleak*txt. The example uses a 300 second interval, so if that is producing too much output but the growth is slow, then detach and re-attach with a larger interval.
  4. After the RSS has grown at least a few GB, detach memleak: 
    pkill -TERM -f "memleak -p"
  5. Upload diag_memleak*
Background
  1. On Linux < 4.6, the command is memleak.py rather than memleak.
  2. In some cases, eBPF may not be able to perform stack walks. This is usually due to libraries being compiled with frame pointer omission (FPO). IBM Java and IBM Semeru Runtimes are compiled with FPO. Consider using LinuxNativeMemoryTracker instead (available with IBM Java 8 or contact IBM Support for IBM Semeru Runtimes), although eBPF may still be useful as a first pass to find issues in non-FPO librarires or to get an idea of where the issue is since LinuxNativeMemoryTracker requires a restart whereas eBPF can attach/detach dynamically.
  3. Each iteration prints the 10 largest unfreed allocations (-T 10), so if you know the leak is currently happening, you can just set the number of iterations to 1 so that it exits after the duration; for example: 
    nohup /usr/share/bcc/tools/memleak -p $PID 300 1 &> diag_memleak_$(hostname)_$(date +"%Y%m%d_%H%M%S").txt &
  4. Linux eBPF is a very low level kernel interface so there are rare risks of process crashes or kernel panics.
  5. Consider testing the performance overhead in a test environment. If it's too high, test the --combined-only and/or -s SAMPLE_RATE options.
  6. Compiling iovisor BCC for non-RHEL systems
  7. RHEL9 discussion of iovisor BCC
  8. bcc-tools is available in RHEL since 7.6
  9. Creating custom eBPF scripts
Script to convert memleak output to CSV
  1. Create linuxebpfmemleak.awk: 
    #!/usr/bin/awk -f
# Example output:
# [18:39:56] Top 10 stacks with outstanding allocations:
#    134217728 bytes in 1 allocations from stack
#        0x00007fbb3b697777  new_heap+0xa7 [libc.so.6]

want_frame {
  want_frame = 0;
  if (justlibrary) {
    driver = $NF;
  } else {
    driver = $(NF-1) " " $NF;
  }
  gsub(/[\[\]]/, "", driver);
  if (!(driver in seen)) {
    seen[driver] = 1;
    cols[++ncols] = driver;
  }
  val[nrows, driver] += bytes;
}

/stacks with outstanding allocations:/ {
  nrows++;
  row_time[nrows] = substr($0, 2, index($0, "]") - 2);
}

/allocations from stack/ {
  bytes = 0 + $1;
  want_frame = 1;
}

END {
  printf "Time";
  for (c = 1; c <= ncols; c++) {
    printf ",%s", cols[c];
  }
  printf "\n";

  for (r = 1; r <= nrows; r++) {
    printf "%s", row_time[r];
    for (c = 1; c <= ncols; c++) {
      printf ",%d", ((r, cols[c]) in val) ? val[r, cols[c]] : 0;
    }
    printf "\n";
  }
}
  2. Make it executable: 
    chmod +x linuxebpfmemleak.awk
  3. Run it on the memleak output: 
    ~/linuxebpfmemleak.awk diag_memleak*