LogCompare: Detecting Patterns and Changes Across Environments and Time

LogCompare can be used to automatically expose and alert on anomalous patterns in your logs. First, determine the target dataset you want to inspect. This is usually the 'broken' environment - the tool, service, or application exhibiting non-normal behavior.

In this example, lets assume we have an active-active architecture, with a travel app load-balanced across resources running in a datacenter and in AWS. We notice our checkout service is experiencing an outage, specifically in our datacenter. Our initial search (or Target Query) is:

_sourceCategory=datacenter1/travel/checkout "error"

We can then use the LogReduce Custom form to build a LogCompare search. The below settings show how to:

• Find error signature in my datacenter1/travel/checkout environment in the last 60 minutes
• Compare them to some other environment, like the AWS environment, 24 hours ago

After running the above custom LogCompare, we will have the following search:

_sourceCategory=datacenter1/travel/checkout "error" 
| logcompare timeshift -24h baseline (_sourceCategory=aws/travel/checkout "error")

Immediately we can see that there are over 900 traceback errors and SSL certificate issues in the datacenter environment ("new") that were not in the AWS environment 24 hours ago.

Alerting on LogCompare Signature: New, Gone, +/- Percent Change

What if we want to automatically detect changes in Signatures like the one we found above? We can use a `where` filter to trigger alerts on the LogCompare value that tells us what percent a signature has changed, or if it's gone or new.

The parameters for alerting and filtering LogCompare results are:

• _deltapercentage
• _isnew
• _count

The below search looks for Signatures that have increased more than 100%, are not new, and have at least 10 logs per signature:

_sourceCategory=aws/travel/checkout "error" 
| logcompare timeshift -1h baseline (_sourceCategory=datacenter1/travel/checkout "error")
| where _deltapercentage>100 and !_isnew and _count>10

Then, you can set up a scheduled search or alert to automatically notify you when these conditions are met.