Why do I see the <defunct> zombie processes when running ps with a collector process running in the background?

You may see <defunt> process like the following:

# ps axo stat,ppid,pid,comm | grep -w defunct 
Z 28572 2724 java <defunct> 
Z 28572 2862 java <defunct> 
Z 28572 4263 java <defunct> 
Z 28572 5029 java <defunct> 
Z 28572 7150 java <defunct> 
Z 28572 8215 java <defunct> 

Typically this could occur if the JVM becomes very slow, which can happen under heavy resource use (memory, CPU utilization). If the JVM does not have the CPU cycles to run, it will delay a response back to our Collector wrapper process. At this point, the wrapper may think the JVM is hung and will attempt to stop and then restart the JVM. If the JVM process does not properly shut down during this process, due to the current resource limitations, then you may see this become a defunct process. Performing a manual restart of the Collector should clear these processes.

If this issue persists after a restart of the Collector. Please perform the following steps:

1.) Update or un-comment the following entry found within in /<sumo_home>/config/wrapper.conf file, which will allow for additional debugging information to be printed within the collector.out.log file. 

  # wrapper.debug=TRUE

2.) Restart the Collector. 

3.) Let the Collector run until multiple zombie instances are observed.

4.) Once zombie processes are seen please zip and send the contents of the Collector logs directory to Sumo Logic support for further analysis.