Send entire json file as single message?

Comments

6 comments

  • Avatar
    Piotr Woch

    Hi Yuriy!

     

    Thank you for reaching out! This is Peter Woch from Sumo's Customer Success team.

    You can definitively import entire JSON as a single log message, however, there are two caveats:

    1) You need to disable multi-line processing in your source configuration

    2) There is a limit of how big a single log-message can be (64k)

    You can read more on multi-line logs processing here: 

    https://help.sumologic.com/03Send-Data/Sources/04Reference-Information-for-Sources/Collecting-Multiline-Logs

    Please let me know if this was helpful or if you could use any additional assistance.

    Thank you!

    0
    Comment actions Permalink
  • Avatar
    Yuriy Denysov

    Hi Piotr thanks for the reply!

     

    So I have these in the config but it still sends everything as single lines. 

    {

    "api.version":"v1",

    "sources":[{

      "name":"custodian-test",

      "automaticDateParsing":true,

      "multilineProcessingEnabled":false,

      "useAutolineMatching":true,

      "encoding":"UTF-8",

      "blacklist":[],

      "sourceType":"LocalFile"

      "pathExpression":"/opt/cloud-custodian-policies/accounts/*/us-east-1/*/resources.json",

      }]

    }

     

    Im guessing its because the message is too long then? The logs show this: 

    2019-01-23 18:50:23,988 +0000 [HTTP Sender - 1] INFO  com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '796', bytes: '20188', encoded: '153079', compressed: '8471', by transmitter: '0', sender: 'LogSender'

    2019-01-23 18:50:28,360 +0000 [QuotaEnforcer] INFO  com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Senders) - Requested: 19 KB (1 KB/s). Remaining capacity: 1004 KB (until Wed Jan 23 17:48:08 UTC 2019)

    2019-01-23 18:50:28,452 +0000 [QuotaEnforcer] INFO  com.sumologic.scala.collector.quota.BlockingQuotaEnforcer - BlockingQuotaEnforcer(Inputs) - Requested: 9 KB (1002 bytes/s). Remaining capacity: 355 MB (until Wed Jan 23 17:48:08 UTC 2019)

    2019-01-23 18:50:52,986 +0000 [HTTP Sender - 1] INFO  com.sumologic.scala.collector.CommonsHTTPSender - Publishing message piles: '1', messages: '3', bytes: '3', encoded: '544', compressed: '247', by transmitter: '0', sender: 'LogSender'

    2019-01-23 18:52:07,886 +0000 [Collector Scheduled Executor] INFO  com.sumologic.scala.collector.EventFlowController - Total events in: '799', batches in: '2', events out: '799', batches out: '2', sender: 'LogSender'
    0
    Comment actions Permalink
  • Avatar
    Piotr Woch

    My apologies for the confusion. Enabling the multiline processing is actually REQUIRED for the collector to properly ingest logs messages spanning more than 1 line.

    So contrary to my previous message, you actually need to ENABLE multiline processing.

    Then the question becomes, whether Sumo's built-in, automatic line-boundary finding logic is able to automatically handle your JSON data. My recommendation is to try the automated way first and if this doesn't work, try specifying a regular expression that will unequivocally match the line separators in your data (eg. "}" in a separate line).

    Please let me know if you get stuck or need any further advice.

    Thank you! 

    0
    Comment actions Permalink
  • Avatar
    Yuriy Denysov

    Thank! It did help a little but unfortunately its still splitting up it up after every 3rd or 4th line. Im not really sure how regex can help since the json file starts with [ and will contain multiple instances of [ and ] before closing with ]. Ill try to experiment with jq command to create a single line file but dont know if that will work yet. 

    0
    Comment actions Permalink
  • Avatar
    Piotr Woch

    Is the ending square bracket ("]") occurring as the sole character in a separate line?

    If so, try this expression:

    ^\]$

    If not, is it at least always the last character in the line (and it does not end any line in the middle of the JSON)?

    If so, try this expression:

    \]$

    I recommend using this online tool for real-time regex testing: https://regex101.com/

    You can also copy-paste a representative example of the original JSON message in this post and I can help come up with the regex.

     

    0
    Comment actions Permalink
  • Avatar
    Murthy Chitturi

    HI, This is how the my log file might look and might be more bigger. All my jsonlogs are coming as single separate line.

    ^\]$ .  matched the end ']' and ^\[ matched the starting '['.

     

     can you help me to get a regex or some easy format to send the log file as single log,

     

    [
    {
    "AvailabilityZone": "us-east-1c",
    "Attachments": [
    {
    "AttachTime": "2019-02-06T17:47:34+00:00",
    "InstanceId": "i-0cae8378cad",
    "VolumeId": "vol-0fd7ec51d7",
    "State": "attached",
    "DeleteOnTermination": true,
    "Device": "/dev/sda1"
    }
    ],
    "Tags": [
    {
    "Value": "test",
    "Key": "Test"
    },
    {
    "Value": "Resource does not meet policy: delete@2019/02/26",
    "Key": "Cloud Governance"
    },
    {
    "Value": "",
    "Key": "Creator"
    }
    ],
    "Encrypted": false,
    "VolumeType": "gp2",
    "VolumeId": "vol-0fd7ec51d7db8dabc",
    "c7n:MatchedFilters": [
    "tag:Asset",
    "tag:Application",
    "tag:Environment",
    "tag:Contact",
    "tag:Name"
    ],
    "State": "in-use",
    "Iops": 100,
    "SnapshotId": "snap-0e78738cb3b8e2467",
    "CreateTime": "2019-02-06T17:47:34.119000+00:00",
    "Size": 10
    }
    ]

    0
    Comment actions Permalink

Please sign in to leave a comment.