Use parse regex on the whole ingest log

Follow

Oleksii Motorykin

• July 29, 2022 16:37

Hello! Very new to SumoLogic, but have good programing skills.

When using multiple Parse Regex operators in a row, I notice that consecutive operators try to find data within the first operator's _message. For example, 

| parse regex "Recent\\(?<protocol>.*?)<\/td>"
| parse regex "Cycler:<\/B><\/td><td>(?<cyclerID>.*?)<\/td>"

| parse regex "TimeBegin:<\/B><\/td><td>(?<timeBegin>.*?)<\/td>"
| parse regex "TimeEnd:<\/B><\/td><td>(?<timeEnd>.*?)<\/td>"

All operators work on their own, but not together. Can I somehow use the other parses, but NOT within the result of the first?

Related to this, do I need to specify Processing Rules for Logs so that all this information is within one log message?

Thank you!

0

• Official comment

  

  Anna Truta

  • November 02, 2022 14:51

  Hello Oleksii,

  Processing rules might not be the best option here, as it filters data by:

  • Excluding messages that match
  • Including messages that match
  • Hashing messages that match
  • Forwarding messages that match

  It’s not focused on extracting fields from your logs, but on taking an action if a message matches a pattern.

   

  I think the option that would be more beneficial for you is FER – Field Extraction Rules. Field extractions allow you to parse fields from your log messages at the time the messages are ingested, which eliminates the need to parse fields at the query level. However, keep in mind that when you define a scope of FER (e.g., partition, source category), all the messages will be parsed in that way – so this option is good for regularly used fields in an organization, not to worsen search performance.

  In FER or in your log search, to perform regex that will not include the result of the first outcome but will perform within the same message, you can include multiple regex patterns in one query.

  For example, for the following two log lines:

  Oct 11 18:20:49 host123.example.com 16234563: Oct 11 18:20:49: %SEC-6-IPACCESSLOGP: list 101 denied tcp 10.1.2.3(1234) \> 10.1.2.4(5678), 1 packet
  Oct 11 18:20:49 host123.example.com 16234564: Oct 11 18:20:49: %SEC-6-IPACCESSLOGP: list 101 accepted tcp 10.1.2.5(4321) \> 10.1.2.6(8765), 1 packet
  
  

  you can write the following query to extract the "protocol":

  | parse regex "list 101 (accepted|denied) (\<protocol\>.*?) "

  So, you'd write:

  | parse regex "list 101 (?:accepted|denied) (\<protocol\>.*?) "

  But if you mean to also capture whether it is an "accepted" or a "denied" into an alias, then you'd include:

  | parse regex "list 101 (\<statu\>accepted|denied) (\<protocol\>.*?) "

  In the same way, you can extract your fields in one regex parsing. The above example is taken from here - you can find Regex Parsing covered in more detail.
  
  Hope that helps!

  Comment actions Permalink

Please sign in to leave a comment.