how do we eliminate/filter duplicate lines based on a field

Follow

sridhar krishnamurthy

• May 15, 2014 16:15

consider the following sample log lines

.... 123 abc def

.... 123 xxx yyy

.... 345 qwert aaaa

.... 456 qqq vvv

.... 456 bbb zzz

(question a) i need the output messages to be

.... 123 abc

.... 345 qwert

.... 456 qqq

(question b) i need to "do something on this with aggregate functions so that a monitor

can be added to the dash board which plots the numbers on the y axis and the names on the x axis

0

• 

  Kevin Sumo

  • May 16, 2014 22:40

  You will first need to parse the "number" and the "name" from the message.

  For example:

  * | parse regex "(?<number>\d+)\s+(?<name>\D+)"

  A. ) You can then use the "first" or "last" operator to get the first or last occurrence of the name field, by number.

  * | parse regex "(?<number>\d+)\s+(?<name>\D+)"
  
  | first(name) as name by number

  This query will give you an output similar to the following which could be presented in the charts.

  123 | abc
  
  345 | qwert
  
  456 | qqq

  B.) You can get then chart on the fields by summing up the number by name, since there is only going to be one number per name (due to the first/last) the sum will equal the value.

  | parse regex "(?<number>\d+)\s+(?<name>\D+)"
  
  | first(name) as name by number
  
  | sum(number) as number by name

  0

  Comment actions Permalink

Please sign in to leave a comment.