Is it possible to exclude partial keywords from search
I would like to exclude log messages for which I only partially know what string that I want to exclude will contain. Is this possible within the keyword portion of the search?
For example: My log file contains the following 3 entries:
Jan 15 21:47:47 47-e100 workerthread: be60: b6794ab0 00000000 bb4ab480 2ac99000 8000000 b6975b00 bb4ab480 2ac99000
Jan 17 21:47:48 47-d312 workerthread:
Jan 17 21:47:49 47-04a1 workerthread: Internal Error
I would like to exclude them from my search results in the keyword search if possible. For example to exclude the last line I would use the following in my keyword search for example:
(_sourceCategory=log/logmessages AND "workerthread") AND !"Internal Error"
How can I exclude anything in the logs that contains "workerthread: b"? (Log line 1) because the following doesn't work:
(_sourceCategory=log/logmessages AND "workerthread") AND !"workerthread: b*"
How can I exclude anything that is blank after a workerthread:? (Log line 2)
Thank you!
-Tony P.
-
Hi Tony,
Due to the way data is indexed and how keyword searches are performed, this isn't possible in the way you are looking to do this. An overview of this may be found here: https://help.sumologic.com/Search/Search-FAQs/Searching-by-keyword-returns-no-results-even-though-the-keyword-exists-in-messages
Another caveat is that we don't allow the mix of the wildcard with special characters. When attempting to do so you will receive the following error:
Unquoted search term can't mix wildcard, punctuation, and alphanumeric.
However back to your question, a way you can do this is by using the inherent filtering capability of our parsing operators. For example, using your data, the following will exclude events where there is no value for the workerthread:
| parse regex "workerthread:\s+(?<workerthread_details>.+)"
-
I responded to your comment via email a few days ago. I am going to copy what I sent here...
How would I incorporate the parse regex into a query. I have tried the following (trying to exclude empty workerthreads form being returned in the results by using “- fields” and it is not working: I do know that he query has grammar errors as well.(_sourceCategory=log/logmessages AND "workerthread”)AND !"Internal Error"| parse regex "kernel:\s+(?<workerthread_details>.+)"| if(isEmpty(workerthread_details), 1, 0) as nada2| parse “47-* " as id| count by id| sort by _count| where _count > 25| fields - nada2Thank you!-Tony P. -
Hi Tony,
Stepping through the example query you've sent through (and understanding there may be some errors in what you've posted)
(_sourceCategory=log/logmessages AND "workerthread”) AND !"Internal Error"
| parse regex "kernel:\s+(?<workerthread_details>.+)"I can see in the parse statement you've changed workerthread to kernel. This may be an issue as on the very first line you have specifically asked for only events that contain the text workerthread.
Assuming your base search *is* returning results, at this point, only events that have a value for workerthread_details will be available for use. This is due to the filtering/dropping nature of the parse regex statement, specifically any event that does not contain the text kernel:, followed by one-or-more spaces, and then followed by one-or-more of *any* character (apart from spaces or newlines) will be dropped.
| if(isEmpty(workerthread_details), 1, 0) as nada2
As per the previous note, at this point you should only have events with a non-empty value for workerthread_details (i.e. nada2 will only evaluate to 0 here). Also, seeing as you don't use nada2 for aggregation or filtering later in this query, this statement doesn't actually need to be included.
| parse “47-* " as id
| count by idThis will further filter the events you are looking at to those containing the string 47- followed by a string (followed by a space). Assuming there is matching data, the output of this would be a table containing values for id and a corresponding value for _count.
| sort by _count
| where _count > 25Yep, no problem here. Standard sort with a conditional statement.
| fields - nada2
At this point, there would not be a field called nada2 to filter out due to the previous aggregation, so this is asking Sumo to not display a field it doesn't actually have to display in the first place.
I hope this is helpful :-)
RE: the email, unfortunately email responses back to community threads aren't forwarded through, so please feel free to either reply back within the community forum, or alternatively join our public Slack channel at http://slack.sumologic.com where you can get assistance from myself, other Sumo Logic engineers, and other customers as well in real-time.
Cheers!
-
I did not realize that the parse regex "kernel:\s+(?<workerthread_details>.+)" actually took care of omitting the results. I thought that I had to do something with the workerthread_details in order to omit them.
THANK YOU!!!
This was blocking some major work.
Regards!
-Tony Petronelli
-
No problems! Just to round this conversation out, in the event you did need to keep the events that didn't match the parse statement (e.g. for additional parsing or aggregation), then you would use the nodrop qualifier in the parse statement.
| parse regex "kernel:\s+(?<workerthread_details>.+)" nodrop
-
Yes, I was just going to send you an email regarding that...
However, the regex statement catches anything after "kernel:\s+ and stores it in workerthread_details, right? Because I am only looking to drop workerthread_details that are blank. There are many others that I have yet to parse and don't want dropped and wasn't able to drop by using a keyword search expression and thus must use a regex against.
-
Hi Ryan,
Any chance we can setup a call? I must be missing something with the way parse regex works, because I am unable to get multiple various parse regex statements to work in sequence for example:
| parse regex "workerthread:\s+(?<blank_messages>.+)"
| parse regex "workerthread:\s5[a-z0-9]{3}:(?<non_blank_5>.+)"
| parse regex "workerthread:\sb[a-z0-9]{3}:(?<non_blank_b>.+)"Thank you!
-Tony P.
-
Hi Tony - I've reached out to you via our Public Slack channel at http://slack.sumologic.com. Will post any solution we come to in here for the benefit of anyone else who may be have a similar issue in future.
Please sign in to leave a comment.
Comments
8 comments