Linux syscall times

Linux perf trace -s tracks system call times for all processes and then, on SIGINT / Ctrl+C, outputs statistics for each syscall for each process.

  1. Install perf if it's not installed.
  2. Start perf trace -s in the background:
    nohup perf trace -s > diag_perftracesummary_$(hostname)_$(date +%Y%m%d_%H%M%S).txt 2>&1 &
  3. Reproduce the problem
  4. After the problem is reproduced, stop perf trace -s:
    pkill -INT -f "perf trace"
  5. Upload diag*

Example output:

bash (1), 738 events, 0.0%

   syscall            calls  errors  total       min       avg       max       stddev
                                     (msec)    (msec)    (msec)    (msec)        (%)
   --------------- --------  ------ -------- --------- --------- ---------     ------
   pselect6              62      0  7268.012     0.000   117.226  1695.909     29.78%
   rt_sigaction         112      0     0.594     0.004     0.005     0.027      4.33%
   write                 46      0     0.432     0.005     0.009     0.022      4.93%
   [...]