Using Logstash Logback Encoder for a Java Application
When creating JSON structured logs using the Logstash Logback encoder(https://mvnrepository.com/artifact/net.logstash.logback/logstash-logback-encoder/4.11) has anyone had any trouble with SumoLogic parsing the log files?
I will want to be able to query the logs using JSON fields as parameters similar to in the example below.
(_source=container-logs ) AND _sourcename = RApp*
| json field=_raw "response-time" as responseTime
| json field=_raw "url" as url
| avg(responseTime) group by url
Are there any special formatting you have to do or configuration so that the logs are properly ingested and parsed?
-
Official comment
Update: We now have an official Logback appender:
https://github.com/SumoLogic/sumologic-logback-appender
____________________________________________
Hi Chris,
In order to use JSON parsing in Sumo Logic, you just need to ensure the message is a valid JSON object. I see in that link that the Logback encoder will output events as Logstash-compatible JSON, so assuming this is normal JSON we shouldn't have any issues parsing it.
Also, some customers use this open source Logback appender to log straight to Sumo Logic: https://github.com/relateiq/sumo-logback-appender
Let me know if this answers your question, I'm happy to assist further. If you use Slack, you can join our public Slack channel here slack.sumologic.com - I'm @grahamwatts-sumologic.Comment actions -
Graham,
Thanks for the response.
We will be logging valid JSON objects to our log files. However, the whole log file itself will not be a valid JSON.The log file will just be valid JSON log objects separated by a newline. Will this work, or does the entire log file itself have to be a valid JSON document. i.e. A JSON array of our JSON log objects.
Example log file:
{
"@time":"2017-08-14 17:43:44.630",
"level":"INFO",
"logSource":"com.core.logging.ControllerLogging",
"msg":"Request TrackerId=tracking:1234: End request for resource requestMethod=POST path=/api/. Status statusCode=500. Total time: duration=183 ms",
"jsonMsg":null,
"thread":"eventloop-thread-0",
"OG-TrackerId":"tracking:1234",
"requestMethod":"POST",
"path":"/api/",
"statusCode":500,
"duration":183,
"context":"default"
}
{
"@time":"2017-08-14 17:43:44.630",
"level":"INFO",
"logSource":"com.core.logging.ControllerLogging",
"msg":"Request OG-TrackerId=tracking:12345: End request for resource requestMethod=POST path=/api/. Status statusCode=500. Total time: duration=183 ms",
"jsonMsg":null,
"thread":"eventloop-thread-0",
"OG-TrackerId":"tracking:12345",
"requestMethod":"POST",
"path":"/api/",
"statusCode":500,
"duration":183,
"context":"default"
} -
Our team considered: https://github.com/relateiq/sumo-logback-appender but decided against it.
Factors in our decision include:
- Last commit was over 2 years ago
- Only 1 contributor and 30 commits
- Lack of a README or any documentation
It did not seem like a mature, well supported or well documented tool.
-
Thanks for the info Chris. Here is another more recently updated Logback to Sumo Logic Appender: https://github.com/vital-software/sumo-logback
The correct way to send logs into sumo is one valid JSON object per message. If you are sending multiple messages, you might be able to use boundary regex to have Sumo Logic split them, as long as they are not wrapped in [brackets].
Please sign in to leave a comment.
Comments
6 comments