Data Ingestion: Rolling Standard Deviation, Accumulation, Backshift, etc.

Follow

Dwayne Hoover

• October 23, 2014 14:51

Wanted to share an example query that I put together for a customer recently.  The request was for a search that will look at the amount of data ingested, per hour by a single collector.  He wanted to set up alerts if it seemed to be lagging or spiking.  This is what I came up with.

* _index=sumologic_volume|where _sourceCategory="collector_volume"

| parse regex "\"(?<collector>(?:[^\"]+)|(?:\"\"))\":{\"sizeInBytes\":(?<bytes>\d+),\"count\":(?<count>\d+)}" multi

| where collector matches "collector name here"

| timeslice 1h

| bytes/1024/1024 as mbytes| sum(mbytes) as mbytes by collector, _timeslice

| sort + _timeslice

| backshift mbytes, 1 as prevMB

| smooth prevMB,24 as movingAVG

| rollingstd mbytes, 24 as rollingStd

| movingAvg + (3 * rollingStd) as upper

| movingAvg - (3 * rollingStd) as lower

| accum mbytes as totalMB

Few things to note:  

• This is calculating in MB, not GB.  You'll need to add another "/1024" and do some renaming if you want to see this in GB.
• You need to enable the data volume index

Drop that last accum statement (comment the line out with // ) and it makes a nice visualization on a line or area chart.

Add some where clauses and this could easily be used for alerting purposes.

• for example: where rollingStd > 1

Please comment with any questions or suggestions

3

• 

  1 - Sumo Admin

  • December 18, 2014 09:37

  Question, if I wanted to look at the average bytes sent out of my network by the top ten talkers, and then watch to see if any individual talker exceeds the std dev of these top ten, how would I do that. I can't seem to figure that out.

  0

  Comment actions Permalink
• 

  Dwayne Hoover

  • December 19, 2014 20:17

  Hey Chuck - that's a tricky one.  Since your top ten talkers will be variable (or could be) this type of search might not work for you.  Never fear, there are definitely some ways around that.  I'll reach out to you via email to talk through it and we'll get you squared away.

  0

  Comment actions Permalink

Please sign in to leave a comment.