Use parse regex on the whole ingest log


1 comment

  • Official comment
    Anna Truta

    Hello Oleksii,

    Processing rules might not be the best option here, as it filters data by:

    • Excluding messages that match
    • Including messages that match
    • Hashing messages that match
    • Forwarding messages that match

    It’s not focused on extracting fields from your logs, but on taking an action if a message matches a pattern.


    I think the option that would be more beneficial for you is FER – Field Extraction Rules. Field extractions allow you to parse fields from your log messages at the time the messages are ingested, which eliminates the need to parse fields at the query level. However, keep in mind that when you define a scope of FER (e.g., partition, source category), all the messages will be parsed in that way – so this option is good for regularly used fields in an organization, not to worsen search performance.

    In FER or in your log search, to perform regex that will not include the result of the first outcome but will perform within the same message, you can include multiple regex patterns in one query.

    For example, for the following two log lines:

    Oct 11 18:20:49 16234563: Oct 11 18:20:49: %SEC-6-IPACCESSLOGP: list 101 denied tcp \>, 1 packet
    Oct 11 18:20:49 16234564: Oct 11 18:20:49: %SEC-6-IPACCESSLOGP: list 101 accepted tcp \>, 1 packet

    you can write the following query to extract the "protocol":

    | parse regex "list 101 (accepted|denied) (\<protocol\>.*?) "

    So, you'd write:

    | parse regex "list 101 (?:accepted|denied) (\<protocol\>.*?) "

    But if you mean to also capture whether it is an "accepted" or a "denied" into an alias, then you'd include:

    | parse regex "list 101 (\<statu\>accepted|denied) (\<protocol\>.*?) "

    In the same way, you can extract your fields in one regex parsing. The above example is taken from here - you can find Regex Parsing covered in more detail.

    Hope that helps!

    Comment actions Permalink

Please sign in to leave a comment.