IBM WebSphere Application Server Performance Cookbook

Troubleshooting macOS

Signals

Available signals may be listed with kill -l:

$ kill -l
 1) SIGHUP      2) SIGINT    3) SIGQUIT    4) SIGILL    5) SIGTRAP
 6) SIGABRT     7) SIGEMT    8) SIGFPE     9) SIGKILL  10) SIGBUS
11) SIGSEGV    12) SIGSYS   13) SIGPIPE   14) SIGALRM  15) SIGTERM
16) SIGURG     17) SIGSTOP  18) SIGTSTP   19) SIGCONT  20) SIGCHLD
21) SIGTTIN    22) SIGTTOU  23) SIGIO     24) SIGXCPU  25) SIGXFSZ
26) SIGVTALRM  27) SIGPROF  28) SIGWINCH  29) SIGINFO  30) SIGUSR1
31) SIGUSR2

xattr

Files downloaded from the internet have a quarantine flag automatically added that may cause security issues. Open a terminal and run:

xattr -d com.apple.quarantine *
xattr -d com.apple.metadata:kMDItemWhereFroms *

By default, crashes are summarized in *.crash files in ~/Library/Logs/DiagnosticReports (or /Library/Logs/DiagnosticReports) and they may be read with a text editor or opened with the Console application.

To create full core dumps, ensure the /cores/ directory exists and is writable by the user creating the dump. By default, /cores/ is only writable by the root or wheel users:

drwxr-xr-x   2 root  wheel    64 Aug 24  2019 cores

To make it writable:

sudo chmod a+rwx /cores/

The location of core dumps is controlled with kern.corefile:

% sudo sysctl -a | grep kern.corefile
kern.corefile: /cores/core.%P

In addition, ensure that the core ulimit is unlimited. By default, the soft limit is 0 which means cores are not created (unless it's a program that increases the limit from hard to soft at runtime):

% ulimit -a
-c: core file size (blocks)         0
% ulimit -aH 
-c: core file size (blocks)         unlimited

To set the core dump soft and hard ulimits to unlimited by default, apply the following and restart:

sudo /usr/libexec/PlistBuddy /Library/LaunchDaemons/corelimit.plist -c "add Label string com.launchd.corelimit" -c "add ProgramArguments array" -c "add ProgramArguments: string launchctl" -c "add ProgramArguments: string limit" -c "add ProgramArguments: string core" -c "add ProgramArguments: string unlimited" -c "add RunAtLoad bool true" -c "add ServiceIPC bool false"

You may also need to add the user creating the dump to the wheel group. First check whether your user is a member of wheel:

groups

If not, add to wheel:

sudo dseditgroup -o edit -a $USER -t user wheel

lldb

Process Thread dump

The equivalent of pstack is to use lldb to print all thread stacks. For example:

% echo "thread backtrace all" | lldb -p $PID
* thread #1, queue = 'com.apple.main-thread', stop reason = signal SIGSTOP
  * frame #0: 0x00007fff20569e7e libsystem_kernel.dylib`mach_msg_trap + 10
[...]

Core dumps

lldb may be used to analyze macOS core dumps. For example:

% lldb -c /cores/core.11243
(lldb) bt

Common commands:

Cause of core dump and current thread/instruction: process status
List threads: thread list
List loaded libraries: image list
Variables for the current stack frame: var
Print a specific stack frame variable: frame variable $NAME
Read a block of memory in some format (e.g. with bytes and ASCII): memory read -fY -c$BYTES 0x$ADDRESS --force
Print a specific stack frame variable in some format (e.g. with bytes and ASCII): frame variable $NAME -f Y
Print current registers: register read
Current assembly: dis
Current code: list
Find type structure: type lookup $NAME
Write a block of memory to a file: memory read -o mem.bin -c$BYTES 0x$ADDRESS

Tips

Long click on the maximize button to do split-screen tiling
Finder
- Show all files: defaults write com.apple.finder AppleShowAllFiles TRUE

Open Application from Terminal

Open a well known application (in /Applications/):

open -a Wireshark.app

Open an application in the current directory:

open Eclipse.app

Open multiple instances of an application with the -n flag.

Common alias:

alias code="open -a '/Applications/Visual Studio Code.app'"

Network

mtr

mtr is available through Brew:

brew install mtr

However, it must be run as root and it's on the sbin path. For example:

sudo /opt/homebrew/sbin/mtr --report-wide --show-ips --aslookup --report-cycles 30 example.com

Security

Some security error codes above 10000 may be 10000 + an errno-style code.

Print the entitlements of an application:

% codesign -d --ent :- $EXECUTABLE

Termination Reason: CODESIGNING

For crashes due to Termination Reason: CODESIGNING, a workaround may be to force re-sign:

find Eclipse.app -name "*" -execdir sudo codesign --force --deep --sign - {} \;

System Integrity Protection

System Integrity Protection (SIP) is a security feature that includes restricting runtime attachment to system processes.

SIP Status

Show the SIP status with:

% csrutil status
System Integrity Protection status: enabled.

Disable System Integrity Protection

To disable SIP, boot into the Recovery Partition and run csrutil disable.

Enable Kernel Symbolication

May require disabling SIP first.
```
sudo nvram boot-args="keepsyms=1"
```
Reboot
Without Activity Monitor running, get a spindump:
```
sudo spindump -reveal -noProcessingWhileSampling
```
Review /tmp/spindump.txt

Recovery Partition

Booting to the recovery partition:

Turn off Mac, then:
1. ARM:
  1. Press and hold the power key
2. Intel:
  1. Hold the ⌘R keys
  2. Turn on Mac
  3. When you see the "macOS Recovery" window, release the ⌘R keys
Log in with a user
At the top of the screen, click Utilities } Terminal

Run desired commands; for example:

% csrutil disable
System Integrity Protection is off.
Restart the machine for the changes to take effect.

At the top of the screen, click the Apple icon } Restart

After restarting, ensure SIP has been disabled:

% csrutil status
System Integrity Protection status: disabled.

Enable Non-Maskable Interrupt

From the recovery partition, enabling Non-Maskable Interrupt (NMI) allows for creating kernel dumps for a hung system.

Add debug=0x4 to the current settings:

% sudo nvram boot-args="$(nvram boot-args) debug=0x4"

Add User to a Group

For example, to add $USER to the group wheel:

sudo dseditgroup -o edit -a $USER -t user wheel

Comparing and Merging Files

Eclipse has a good compare and merge functionality
1. Create a project (any kind)
2. Right click on the project } Import... } General } File System
3. Select the two files under the project
4. Right click } Compare With } Each Other
FileMerge is a visual diff tool available with open /Applications/Xcode.app/Contents/Applications/FileMerge.app

Kernel core dumps

It appears that kernel core dumps may only be configured to be sent over a network to another machine (although this may be a virtual machine) rather than produced locally, and the types of network interfaces that may be used are limited (e.g. not over WiFi).

Generate kernel core dump: sudo dtrace -w -n "BEGIN{ panic();}"

See https://developer.apple.com/library/archive/technotes/tn2004/tn2118.html

Previous Section (Troubleshooting Windows) | Next Section (Troubleshooting Solaris) | Back to Table of Contents