json field extraction in SumoLogic AWS Guardduty template not working

Follow

Kenneth Ord

• April 02, 2019 09:29
• Edited

AWS GuardDuty produces log messages in JSON that have multiple interesting fields nested under "service.action", like 

• networkConnectionAction.remoteIpDetails.ipAddressV4
• networkConnectionAction.localPortDetails.port
• networkConnectionAction.remoteIpDetails.geoLocation.lon
• networkConnectionAction.remoteIpDetails.geoLocation.lat

SumoLogic provides templates for parsing the logs. In the template the fields are each parsed like this:

| json field=_raw "accountId", "region", "partition", "id", "arn", "type","service.serviceName","service.detectorId","service.action","severity","title","description" nodrop

| json field=%service.action "networkConnectionAction.remoteIpDetails.ipAddressV4","networkConnectionAction.localPortDetails.port" as ip, localPort nodrop

But it doesn't work! To work I have to directly reference each field with this command instead:

json field=_raw "service.action.networkConnectionAction.remoteIpDetails.ipAddressV4","service.action.networkConnectionAction.localPortDetails.port" as ip, localPort nodrop

I'm confused about the intended meaning of "%service.action" in the template. There's no mention of using a "%" anywhere in the Support documentation. It looks like the first json command is trying to extract the whole "section.action" contents and store it in "%service.action" and is then using that to parse again, but whatever it's pulling into "%service.action" isn't true JSON. 

Anyone help me understand, please? I'm having to edit every template by hand to get them to work and it's a real pain!

 

0

• Official comment

  

  Matt Sullivan

  • April 09, 2019 20:32

  I tried your code vs. some sample data. It does seem to work, just that elements in the service.action field will vary based on the action so the nodrop keeps a lot of messages with empty ip and localPort fields. If you append these two lines to the end of your query, hopefully you do see ip and localport for at least some of the messages
  
  | count ip, localPort
  | order by _count
  
  If you want to exclude messages that don't have service.action, you could remove the nodrop statements and the json parse will act as a filter. additionally you could include the target action name in the scoping section of the query as keywords to the search.

  The % fyi is just a way to allow field names with normally disallowed characters in the syntax, in this case the period.  The article describing probably could be easier to find but it's here: https://help.sumologic.com/05Search/Get-Started-with-Search/Search-Basics/Reference_a_Field_with_Special_Characters .
  
  Sorry to hear that there was pain involved. 

   

  Permalink

Please sign in to leave a comment.