Docker logs re-shipped after container restart
We have a recurring issue where we have a number of containers that may require a restart for one reason or another. The issue is that when we restart these containers, the collector picks up all of the old logs (that have already been shipped to Sumo) and ships them again which blows out our daily quota and creates numerous duplicate log entries. We have our collector configured with the "cutoffRelativeTime" set to "-1h" in hopes of preventing or at least reducing the volume hit, but Sumo support has told us that setting is only utilized upon source creation. Their suggestion was to use "cutoffTime", but that wouldn't work unless we were constantly reconfiguring the collector and pushing the timestamp forward. In the most extreme cases, we will re-ship weeks of logs. Are others experiencing this? How are you working around it/preventing it?
TIA
-
Official comment
Sorry for the delay on this one. If you haven't already, definitely open a support ticket (or keep working the existing one if already opened) so we can help track this down. I would include details such as collector version, docker version, etc. A lot of improvements have been made recently around containers.
Anecdotally, I've tried to reproduce this, using Docker CE 18.06.0-ce-mac70 and Collector version 19.227-4. My collector is on the Docker host, and I made a source to capture all containers as per the Collect-Events-and-Statistics-for-the-Docker-App topic. I was using Tomcat, logging in the container but also tried -v to map to local drive, etc.
Comment actions
Please sign in to leave a comment.
Comments
1 comment