Any recommendations on gathering the logs from AWS ECS containers?

Follow

edgar nielsen

• July 26, 2018 15:57

Currently my site has been using a standard write to a log file on the EC2 container host and a sumo collector is sending the data to Sumo.

What are the alternatives?

• I see there is a way to hook docker logs and sumo together - https://github.com/SumoLogic/sumologic-docker-logging-driver - anyone using that with ECS?

 

• And I guess one could send the logs to cloudwatch and then pull into Sumo from cloudwatch. (https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html and https://help.sumologic.com/Send-Data/Collect-from-Other-Data-Sources/Amazon-CloudWatch-Logs)

0

• 

  Graham Watts

  • July 27, 2018 15:56
  • Edited

  Hey Edgar,
  
  There are a few ways you can collect logs from ECS, each has its strengths and weaknesses:
  
  

  1. Install a collector directly on the ECS host

  • Allows you to use our host metrics source to measure things like CPU, Memory, Network, Disk, and TCP of the underlying host
  • Use Docker sources to collects logs from workloads running in containers, and metrics (performance stats) about those containers
  • Use a local file source to get OS logs, ECS logs, and any other logs directly from the host's file system

  2. Use the Sumo Logic Docker collector container

  • Good option when you are using a different orchestration tool (Rancher, for example) that doesn't allow you access the docker host
  • Mount directories for Operating System logs or ECS output
  • The main disadvantage is that you won't get host level metrics, although this may not matter if you don't need to manage the host

  3. Agentless - Use the Sumo Logic Docker logging driver

  • Good option if you don't want to use an agent on the host or as a container
  • Stream logs to our HTTPS endpoint (HTTP Source)

  4. Send to CloudWatch, then use our Lambda function to stream data from CloudWatch log groups to Sumo

  • Least common option due to additional storage costs in CloudWatch and added delay to get logs into Sumo
  • Easy approach if you already log to CloudWatch and don't want to change your logging meethod

  0

  Comment actions Permalink

Please sign in to leave a comment.