query for logs that do not have a matching entry in a later log

Comments

3 comments

  • Avatar
    Rick Jury

    in order to do a transaction type use case you need to define the id field with the same name in both then use one of the sumo transaction operators (such as transactionize ... merge ...) or just do an aggregation like below example. 

    so for example:

    _collector="xxx" (foo or bar)
    | parse "Foo *," as fooId nodrop

    | parse "Bar *," as barid nodrop

    | if(isempty(fooid),0,1) as f
    | if(isempty(barid)),0,1) as b
    // we want to always have a unifying id field
    | if(isempty(fooid),barid,fooid) as id
    | min(_messgetime) as earliest, max(_messagetime) as latest,max(f) as f,max(b) as b by id
    | where f >0 and b = 0

     

    You could enclose the above in a subquery if you wanted to get the log lines with this method of using the id as keyword in the parent:

    _collector=xxx (foo or bar)

    [subquery: 

    _collector="xxx" (foo or bar)
    | parse "Foo *," as fooId nodrop

    | parse "Bar *," as barid nodrop

    | if(isempty(fooid),0,1) as f
    | if(isempty(barid)),0,1) as b
    // we want to always have a unifying id field
    | if(isempty(fooid),barid,fooid) as id
    | min(_messgetime) as earliest, max(_messagetime) as latest,max(f) as f,max(b) as b by id
    | where f >0 and b = 0 | count by id | compose id keywords]

    0
    Comment actions Permalink
  • Avatar
    Ryan O'Rourke

    Thanks, that makes sense.

    I had not looked at the transaction operators before, looks like something useful to dig into.

    one question about subqueries, I don't understand why all the examples have a "count by" clause preceding "compose". Is that required for some reason? An optimization?

    0
    Comment actions Permalink
  • Avatar
    Rick Jury

    if you run the subquery without the subquery but just compose you will see what is going on. for example
    error | count by _sourcecategory | compose _sourcecategory 

    vs

    error | count by _sourcecategory | compose _sourcecategory keywords

    0
    Comment actions Permalink

Please sign in to leave a comment.