Histogram of the elements in a list in a JSON log
Edit: I think I am asking the same thing that is being asked here:
E.g., series of JSON logs:
{
'list-of-things': [ 'thing1', 'thing2', ...]
}
And I want to produce a histogram of all the things in the list-of-things across all the log entries in a given period of time.
It happens that these are Cloudflare logs and I want a histogram of triggered Cloudflare WAF rules so I can see which rules are being triggered most often. The triggered rules are in a list { 'firewallMatchesRuleIDs': [] }.
-
Guessing I need some regex magic to make this work. What I have below is counting each unique set of triggered rules, but not counting the individual rules within each set, which is what I am shooting for.
.
.
.
Uh, for some reason I cannot copy & paste into this dialog box. MacOS 10.14.5, Chrome 75. When I paste, all I get are a bunch of blank lines. Tried copying into a text document first, and then copying here, but same results. :(
-
Mark,
Fortunately there is an App (out-of-the-box content) that is already doing this. Check out this link for info on the App and how to install it:
In particular, look for the "Cloudflare - Security (WAF)" Dashboard which gives you info on WAF rules triggered. Keep in mind that you can always edit Panels in that dashboard (go to the query behind each Panel) to tweak it for your specific needs.
Cheers,
Mario
-
Hi Mario. I am using the Cloudflare app, but the WAF Rules Triggered query is just pulling the WAFRuleID field, which is not what I am looking for. The most common WAFRuleID is "981176", which is an aggregate rule triggered as a result of a number of other rules being triggered by the source.
The actual rules behind a 981176 event are listed in the FirewallMatchesRuleIDs field. That field is a JSON list object. What I would like to do is get a histogram of all the rule IDs in the list object across all the logs in a given time period.
{
.
.
.
"ClientRequestUserAgent":"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/75.0.3770.100 Safari/537.36",
.
.
.
"EdgeResponseStatus":403,
.
.
.
"FirewallMatchesRuleIDs":["07","01","900001","900002","900004","900006","900012","900018","900021","960024","981133","950901","981311","959073","981257","981248","981245","981247B","981243","981136","973300","973335","973334","973333","973332","2000001","2000003","2000004","2000006","981176"],
.
.
.
"WAFRuleID":"981176",
.
.
.
} -
Got it.
OK, here's an idea. Since we do not know the number of rules you will have in each message, we can use the parse multi option to have Sumo find a given pattern as many times as it exists in the message.
By the way, my Regex skills are not the best, so you might be able to optimize this, but here's a thought:
_sourceCategory=<your_cloudflare_logs>
// Let's make a list of all the rules (call it all_my_rules)
| parse "\"FirewallMatchesRuleIDs\":*"as all_my_rules
// Use multi to parse out all rules that are between 1 and 9 digits)
| parse regex field=all_my_rules "(?<rule>\d{1,9})" multi
//This previous command creates a new message/line for each rule
//A count by rule with the histogram can identify trends
| count by ruleLet me know how this works.
-
That put me on the right path. Here is what I ended up with (apologies for the link ... cut and paste is not working for me and I'm not going to type it out here): https://pastebin.com/DKkJQZAv
I was going to post a link to screen recording of me trying to cut and paste the log query code here, but of course cut and paste is not working for me, so I included it in the pastebin link.
Please sign in to leave a comment.
Comments
5 comments