IBM WebSphere Application Server Performance Cookbook

Kubernetes detailed container memory usage

  1. After the pod starts, open a terminal into each container. For example: 
    kubectl exec -n mynamespace -it podname -- bash
  2. Change directory to /tmp: 
    cd /tmp
  3. Write the following shell script: 
    cat > memorystats.sh << 'EOF'
#!/bin/sh
SLEEPTIMESECONDS=300
OUTPUTFILE="diag_memorystat_${HOSTNAME}_$(date +"%Y%m%d_%H%M%S").txt"
while getopts "s:" opt; do
  case "$opt" in
    s)
      SLEEPTIMESECONDS="${OPTARG}"
      ;;
  esac
done
echo "Writing output to ${OUTPUTFILE}"
while true; do
 echo "[$(date)] Performing iteration" >> "${OUTPUTFILE}" 2>>"${OUTPUTFILE}"
 for FILE in $(ls /proc/meminfo /sys/fs/cgroup/memory* /sys/fs/cgroup/*/*/memory* /proc/[0-9]*/smaps*); do
  echo "[$(date)] Gathering ${FILE}" >> "${OUTPUTFILE}" 2>>"${OUTPUTFILE}"
  cat "${FILE}" >> "${OUTPUTFILE}" 2>>"${OUTPUTFILE}"
 done
 echo "[$(date)] Sleeping for ${SLEEPTIMESECONDS} seconds..." >> "${OUTPUTFILE}" 2>>"${OUTPUTFILE}"
 sleep ${SLEEPTIMESECONDS}
done
EOF
  4. Make the script executable: 
    chmod +x memorystats.sh
  5. Start the script in the background, changing the interval in seconds as needed: 
    nohup ./memorystats.sh -s 300 &
  6. Write the script's PID into a file: 
    echo $! > pid_memorystats.txt
  7. Reproduce the issue
  8. After the issue has been reproduced, stop the script: 
    kill -TERM $(cat pid_memorystats.txt)
  9. List the resulting output file; for example: 
    $ ls diag*
diag_memorystat_liberty1-b6cddccb4-m4c62_20260811_172419.txt
  10. Exit the pod terminal
  11. Download the file locally: 
    kubectl cp -n mynamespace -it podname:/tmp/diag_memorystat_liberty1-b6cddccb4-m4c62_20260811_172419.txt diag_memorystat_liberty1-b6cddccb4-m4c62_20260811_172419.txt

Notes:

  1. Watch the disk usage in the script output directory. If a pod hits an ephemeral disk limit, it may be killed. Log volume will be a function of the interval.
  2. Instead of manually starting the script after the pod starts, this can be done automatically by writing and executing the script in a postStart configuration; for example (be careful with escaping quotes properly): 
    spec:
  containers:
  - name: [...]
    lifecycle:
      postStart:
        exec:
          command: ["/bin/sh", "-c", "cd /tmp/; echo -e '#!/bin/sh \n SLEEPTIMESECONDS=600; OUTPUTFILE=\"stdout_memorystat_${HOSTNAME}_$(date +\"%Y%m%d_%H%M%S\").txt\"; sleep 5; echo \"[$(date)] Started $(basename \"${0}\") in background (PID $$)\" >> ${OUTPUTFILE}; while true; do echo \"[$(date)] Performing iteration\" >> ${OUTPUTFILE}; for FILE in $(ls /proc/meminfo /sys/fs/cgroup/memory* /sys/fs/cgroup/*/*/memory* /proc/[0-9]*/smaps*); do echo \"[$(date)] Gathering ${FILE}\" >> ${OUTPUTFILE}; cat ${FILE} >> ${OUTPUTFILE}; done; echo \"[$(date)] Sleeping for ${SLEEPTIMESECONDS} seconds...\" >> ${OUTPUTFILE}; sleep ${SLEEPTIMESECONDS}; done' > memorystat.sh; chmod +x memorystat.sh; ./memorystat.sh &> stderr_memorystat.txt &"]