How to Compare and Alert on Historical Data

The Time Compare operator can be used to overlay and compare historical data with current data. For example, if you wanted to show a trend of errors, today vs. yesterday, you could use:

"error"
| timeslice by 15m
| count as today by _timeslice
| compare timeshift 1d as vs_yesterday

You can see that there are some big spikes in errors today (dark blue) vs. a relatively steady amount of errors yesterday (light blue).  

But what if you wanted to alert on this? You can create a new field that represents the difference (delta) in yesterday’s data vs. today’s data:

"error"
| timeslice by 15m
| count as today by _timeslice
| compare timeshift -1d as vs_yesterday
// create a delta field that represents the difference between historical and current data
| (today - today_vs_yesterday) as delta
// use an Outlier to statistically monitor spikes or dips in the delta
| outlier delta

Now you can see the spikes in the difference between yesterday’s errors and today’s errors, as noted by the Outlier markers (pink triangles):

To alert on the deviations from the baseline (historical) error behavior, add one more line to push alerts only when an outlier is identified:

"error"
| timeslice by 15m
| count as today by _timeslice
| compare timeshift -1d as vs_yesterday
// create a delta field that represents the difference between historical data and current data
| (today - today_vs_yesterday) as delta
// use an Outlier to statistically monitor spikes or dips in the delta
| outlier delta
// now filter out and return just the outliers
| where delta_violation >0

Then use Save As > Schedule This Search to push alerts via Slack, Hipchat, Pagerduty, email, or your favorite alerting tool that accepts webhooks! If you don't yet have a webhook connection set up, see this link for instructions.

Missing Data: fillmissing Operator

When you run a standard group-by query, Sumo Logic only returns non-empty groups in the results. For example, if your query is grouping by timeslice, then only the timeslices that have data are returned.

This can be a problem because:

• The lack of data is sometimes also an interesting event, but there is no easy way to capture this information. For example, the outlier operator cannot catch anomalies arising from missing data because it can only mark an existing timeslice as anomalous.
• Missing data can lead to misleading visualizations. For example, if you plot a line chart across timeslices with missing data, the chart will interpolate across the missing timeslices and represent them deceptively as nonempty.

The fillmissing operator addresses this shortcoming by allowing you to specify groups that should be represented in the output, even if those groups have no data.

Alternatively, you can choose longer timeslices to ensure each one has data.

Custom TimeCompare: Alert on Today vs. Average of Last 7 Days

You can use a custom TimeCompare to compare current values vs the average value across the last 7 days, for example.

Then, you can add the same delta and outlier query lines as above:

"error"
| timeslice by 1h
| count as today by _timeslice 
| compare with timeshift 1d 7 avg as vs_last_wk_avg
| (today - today_vs_last_wk_avg) as delta
| outlier delta

Finally, add the 'where' filter to the end of the query to set up the query for alerting:

"error"
| timeslice by 1h
| count as today by _timeslice 
| compare with timeshift 1d 7 avg as vs_last_wk_avg
| (today - today_vs_last_wk_avg) as delta
| outlier delta
| where delta_violation >0