Send entire json file as single message?



  • Avatar
    Piotr Woch

    Hi Yuriy!


    Thank you for reaching out! This is Peter Woch from Sumo's Customer Success team.

    You can definitively import entire JSON as a single log message, however, there are two caveats:

    1) You need to disable multi-line processing in your source configuration

    2) There is a limit of how big a single log-message can be (64k)

    You can read more on multi-line logs processing here:

    Please let me know if this was helpful or if you could use any additional assistance.

    Thank you!

  • Avatar
    Yuriy Denysov

    Hi Piotr thanks for the reply!


    So I have these in the config but it still sends everything as single lines. 















    Im guessing its because the message is too long then? The logs show this: 

    2019-01-23 18:50:23,988 +0000 [HTTP Sender - 1] INFO  com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '796', bytes: '20188', encoded: '153079', compressed: '8471', by transmitter: '0', sender: 'LogSender'

    2019-01-23 18:50:28,360 +0000 [QuotaEnforcer] INFO  com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Senders) - Requested: 19 KB (1 KB/s). Remaining capacity: 1004 KB (until Wed Jan 23 17:48:08 UTC 2019)

    2019-01-23 18:50:28,452 +0000 [QuotaEnforcer] INFO  com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Inputs) - Requested: 9 KB (1002 bytes/s). Remaining capacity: 355 MB (until Wed Jan 23 17:48:08 UTC 2019)

    2019-01-23 18:50:52,986 +0000 [HTTP Sender - 1] INFO  com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '3', bytes: '3', encoded: '544', compressed: '247', by transmitter: '0', sender: 'LogSender'

    2019-01-23 18:52:07,886 +0000 [Collector Scheduled Executor] INFO  com.sumologic.scala.collector.EventFlowController - Total events in: '799', batches in: '2', events out: '799', batches out: '2', sender: 'LogSender'
  • Avatar
    Piotr Woch

    My apologies for the confusion. Enabling the multiline processing is actually REQUIRED for the collector to properly ingest logs messages spanning more than 1 line.

    So contrary to my previous message, you actually need to ENABLE multiline processing.

    Then the question becomes, whether Sumo's built-in, automatic line-boundary finding logic is able to automatically handle your JSON data. My recommendation is to try the automated way first and if this doesn't work, try specifying a regular expression that will unequivocally match the line separators in your data (eg. "}" in a separate line).

    Please let me know if you get stuck or need any further advice.

    Thank you! 

  • Avatar
    Yuriy Denysov

    Thank! It did help a little but unfortunately its still splitting up it up after every 3rd or 4th line. Im not really sure how regex can help since the json file starts with [ and will contain multiple instances of [ and ] before closing with ]. Ill try to experiment with jq command to create a single line file but dont know if that will work yet. 

  • Avatar
    Piotr Woch

    Is the ending square bracket ("]") occurring as the sole character in a separate line?

    If so, try this expression:


    If not, is it at least always the last character in the line (and it does not end any line in the middle of the JSON)?

    If so, try this expression:


    I recommend using this online tool for real-time regex testing:

    You can also copy-paste a representative example of the original JSON message in this post and I can help come up with the regex.


  • Avatar
    Murthy Chitturi

    HI, This is how the my log file might look and might be more bigger. All my jsonlogs are coming as single separate line.

    ^\]$ .  matched the end ']' and ^\[ matched the starting '['.


     can you help me to get a regex or some easy format to send the log file as single log,


    "AvailabilityZone": "us-east-1c",
    "Attachments": [
    "AttachTime": "2019-02-06T17:47:34+00:00",
    "InstanceId": "i-0cae8378cad",
    "VolumeId": "vol-0fd7ec51d7",
    "State": "attached",
    "DeleteOnTermination": true,
    "Device": "/dev/sda1"
    "Tags": [
    "Value": "test",
    "Key": "Test"
    "Value": "Resource does not meet policy: delete@2019/02/26",
    "Key": "Cloud Governance"
    "Value": "",
    "Key": "Creator"
    "Encrypted": false,
    "VolumeType": "gp2",
    "VolumeId": "vol-0fd7ec51d7db8dabc",
    "c7n:MatchedFilters": [
    "State": "in-use",
    "Iops": 100,
    "SnapshotId": "snap-0e78738cb3b8e2467",
    "CreateTime": "2019-02-06T17:47:34.119000+00:00",
    "Size": 10

Please sign in to leave a comment.