At times I see below error messages in collector.log. What does it mean?
2018-02-27 11:54:33,818 +1100 [Thread-112] WARN com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender - Unable to forward message to 10.65.85.21-514
java.util.concurrent.RejectedExecutionException: Task com.sumologic.scala.collector.blade.forwarding.syslog.SyslogSender$$anon$2@4892d719 rejected from java.util.concurrent.ThreadPoolExecutor@8110635[Running, pool size = 1, active threads = 1, queued tasks = 512, completed tasks = 144926]
This error means the current volume of messages matching the syslog forwarding rule is too high for the Collector to keep up with.
There is a single thread that polls from a queue (default size of 512 items). Upon dequeue it will use a single-threaded TCP/UDP connection to send the event.
The issue is that the 512-capacity queue is full, meaning that collector is not sending fast enough.
If the data flow varies over time (high throughput to low throughput), you may benefit by increasing the queue size to give more room for backpressuring.
1. Stop the collector service
2. Add the following line to collector.properties (Found within the Collector Installation directory under the /config folder) (default is 512):
forwarding.syslog.maxCapacity = 1024
3. Start the collector service