Consider the following log format:
TIME | LEVEL | IDENTITY | STUFF | MESSAGE
The items here are broken up using pipes. Suppose an automatic rule extracts the first couple items as fields when the data is ingested:
"parse | * | * | as level, identity"
This parse rule will skip over time (which we already have) and then parse out level and identity. We don't want to parse "STUFF" at all times so we don't have a field extraction rule for that. Now, suppose I'm going to build a query and I now want STUFF pulled out as a field. Here's what I've got:
"parse | * | * | * | as ignore1, ignore2, stuff | -fields -ignore1 | fields -ignore2"
This allows me to get a new variable for "STUFF" via anchor parsing and skip over the existing fields that were already parsed out. I only want to parse out STUFF, but I'm unable to anchor parse without pulling those other items again. I saw a community request concerning this topic and the answer was what I used above (to parse the items again and then drop them immediately). This seems wasteful and ugly, however I understand this is a limitation of the system (anchor parsing doesn't support wildcards unless those wildcards are pulled out as variables).
That introduces the question - is it faster to do this ugly anchor parse and then drop things just so I can get to the value I want, or is it faster to just use a Regex statement? I understand anchor parsing is faster in general, but for this specific case I'm not sure if it is.
Please sign in to leave a comment.