Send entire json file as single message?
So i set up a LocalFile source to send a specific file every time it gets updated. This file consists of a json message which starts with [ and ends with ]. However in the sumologic UI im seeing every single line of the json file as a new message. All the brackets basically are a new message.
Is it possible to send this whole file as a single json message? I was going to try matching the regex but the [ character occurs very often in json so I dont think that will work. Am i missing something here?
-
Hi Yuriy!
Thank you for reaching out! This is Peter Woch from Sumo's Customer Success team.
You can definitively import entire JSON as a single log message, however, there are two caveats:
1) You need to disable multi-line processing in your source configuration
2) There is a limit of how big a single log-message can be (64k)
You can read more on multi-line logs processing here:
Please let me know if this was helpful or if you could use any additional assistance.
Thank you!
-
Hi Piotr thanks for the reply!
So I have these in the config but it still sends everything as single lines.
{
"api.version":"v1",
"sources":[{
"name":"custodian-test",
"automaticDateParsing":true,
"multilineProcessingEnabled":false,
"useAutolineMatching":true,
"encoding":"UTF-8",
"blacklist":[],
"sourceType":"LocalFile"
"pathExpression":"/opt/cloud-custodian-policies/accounts/*/us-east-1/*/resources.json",
}]
}Im guessing its because the message is too long then? The logs show this:
2019-01-23 18:50:23,988 +0000 [HTTP Sender - 1] INFO com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '796', bytes: '20188', encoded: '153079', compressed: '8471', by transmitter: '0', sender: 'LogSender'
2019-01-23 18:50:28,360 +0000 [QuotaEnforcer] INFO com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Senders) - Requested: 19 KB (1 KB/s). Remaining capacity: 1004 KB (until Wed Jan 23 17:48:08 UTC 2019)
2019-01-23 18:50:28,452 +0000 [QuotaEnforcer] INFO com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Inputs) - Requested: 9 KB (1002 bytes/s). Remaining capacity: 355 MB (until Wed Jan 23 17:48:08 UTC 2019)
2019-01-23 18:50:52,986 +0000 [HTTP Sender - 1] INFO com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '3', bytes: '3', encoded: '544', compressed: '247', by transmitter: '0', sender: 'LogSender'
2019-01-23 18:52:07,886 +0000 [Collector Scheduled Executor] INFO com.sumologic.scala.collector.EventFlowController - Total events in: '799', batches in: '2', events out: '799', batches out: '2', sender: 'LogSender' -
My apologies for the confusion. Enabling the multiline processing is actually REQUIRED for the collector to properly ingest logs messages spanning more than 1 line.
So contrary to my previous message, you actually need to ENABLE multiline processing.
Then the question becomes, whether Sumo's built-in, automatic line-boundary finding logic is able to automatically handle your JSON data. My recommendation is to try the automated way first and if this doesn't work, try specifying a regular expression that will unequivocally match the line separators in your data (eg. "}" in a separate line).
Please let me know if you get stuck or need any further advice.
Thank you!
-
Thank! It did help a little but unfortunately its still splitting up it up after every 3rd or 4th line. Im not really sure how regex can help since the json file starts with [ and will contain multiple instances of [ and ] before closing with ]. Ill try to experiment with jq command to create a single line file but dont know if that will work yet.
-
Is the ending square bracket ("]") occurring as the sole character in a separate line?
If so, try this expression:
^\]$
If not, is it at least always the last character in the line (and it does not end any line in the middle of the JSON)?
If so, try this expression:
\]$
I recommend using this online tool for real-time regex testing: https://regex101.com/
You can also copy-paste a representative example of the original JSON message in this post and I can help come up with the regex.
-
HI, This is how the my log file might look and might be more bigger. All my jsonlogs are coming as single separate line.
^\]$ . matched the end ']' and ^\[ matched the starting '['.
can you help me to get a regex or some easy format to send the log file as single log,
[
{
"AvailabilityZone": "us-east-1c",
"Attachments": [
{
"AttachTime": "2019-02-06T17:47:34+00:00",
"InstanceId": "i-0cae8378cad",
"VolumeId": "vol-0fd7ec51d7",
"State": "attached",
"DeleteOnTermination": true,
"Device": "/dev/sda1"
}
],
"Tags": [
{
"Value": "test",
"Key": "Test"
},
{
"Value": "Resource does not meet policy: delete@2019/02/26",
"Key": "Cloud Governance"
},
{
"Value": "",
"Key": "Creator"
}
],
"Encrypted": false,
"VolumeType": "gp2",
"VolumeId": "vol-0fd7ec51d7db8dabc",
"c7n:MatchedFilters": [
"tag:Asset",
"tag:Application",
"tag:Environment",
"tag:Contact",
"tag:Name"
],
"State": "in-use",
"Iops": 100,
"SnapshotId": "snap-0e78738cb3b8e2467",
"CreateTime": "2019-02-06T17:47:34.119000+00:00",
"Size": 10
}
]
Please sign in to leave a comment.
Comments
6 comments