query for logs that do not have a matching entry in a later log
Say you have two log lines that can be parsed like this:
parse "Foo *," as fooId
parse "Bar *," as barId
("fooId" and "barId" are the same id, just named differently in the query fragments here.)
In the happy path case, one of the "Foo" entries should be followed a short time later by a matching "Bar" entry (where fooId == barId).
I want to query for problem cases where we have a "Foo" entry that does not have a matching "Bar" entry. Is there a way to do that with subqueries or maybe some other way?
I've tried something like this and it seems to match all the "Foo" entries -
_collector="xxx"
| parse "Foo *," as fooIds
| where ![subquery:_collector="xxx"
| parse "Bar *," as barId
| where fooId == barId
| count by barId | fields barId
| compose barId
]
-
in order to do a transaction type use case you need to define the id field with the same name in both then use one of the sumo transaction operators (such as transactionize ... merge ...) or just do an aggregation like below example.
so for example:
_collector="xxx" (foo or bar)
| parse "Foo *," as fooId nodrop| parse "Bar *," as barid nodrop
| if(isempty(fooid),0,1) as f
| if(isempty(barid)),0,1) as b
// we want to always have a unifying id field
| if(isempty(fooid),barid,fooid) as id
| min(_messgetime) as earliest, max(_messagetime) as latest,max(f) as f,max(b) as b by id
| where f >0 and b = 0You could enclose the above in a subquery if you wanted to get the log lines with this method of using the id as keyword in the parent:
_collector=xxx (foo or bar)
[subquery:
_collector="xxx" (foo or bar)
| parse "Foo *," as fooId nodrop| parse "Bar *," as barid nodrop
| if(isempty(fooid),0,1) as f
| if(isempty(barid)),0,1) as b
// we want to always have a unifying id field
| if(isempty(fooid),barid,fooid) as id
| min(_messgetime) as earliest, max(_messagetime) as latest,max(f) as f,max(b) as b by id
| where f >0 and b = 0 | count by id | compose id keywords] -
Thanks, that makes sense.
I had not looked at the transaction operators before, looks like something useful to dig into.
one question about subqueries, I don't understand why all the examples have a "count by" clause preceding "compose". Is that required for some reason? An optimization?
Please sign in to leave a comment.
Comments
3 comments