Collector Warning message: Unable to forward message to <ip address>-<Port>

Problem:

At times I see below error messages in the collector.log. 

2018-02-27 11:54:33,818 +1100 [Thread-112] WARN com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender - Unable to forward message to 10.65.85.21-514
java.util.concurrent.RejectedExecutionException: Task com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender$$anon$2@4892d719 rejected from java.util.concurrent.ThreadPoolExecutor@8110635[Running, pool size = 1, active threads = 1, queued tasks = 512, completed tasks = 144926]
at java.util.concurrent.ThreadPoolExecutor$AbortPolicy.rejectedExecution(ThreadPoolExecutor.java:2063)
at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:830)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1379)
at com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender.process(SyslogForwardingPipeline.scala:152)

Resolution:

This error means the current volume of messages matching the Syslog forwarding rule is too high for the Collector to keep up with.

There is a single thread that polls from a queue (default size of 512 items). Upon dequeue, it will use a single-threaded TCP/UDP connection to send the event. The issue is that the 512-capacity queue is full, meaning that the Collector is not sending fast enough.

If the data flow varies over time (high throughput to low throughput), you may benefit by increasing the queue size to give more room for backpressure.

1. Stop the collector service

2. Add the following line to collector.properties (Found within the Collector Installation directory under the /config folder) (default is 512):

forwarding.syslog.maxCapacity = 1024

3. Start the collector service