Real time alerts not catching error
Hi!
I have a real time alert set up to catch critical alerts in some log files.
However, when I ran the search manually in the morning, I noticed an alert was fired off around 3 am that the alert did not catch.
Do I need to tweak my settings to make sure these types of errors are caught?
Any insight is appreciated!
Chris
-
Chris,
the issue here is with the timezone settings for this source. The data seems to be created in EST and the timestamp in the message does not tell us what timezone the logs are. With your current config we default to UTC, which is the collector and source default if not told otherwise.
To address this issue you need to change the source configuration to EST. For each source sending these events. change the Time Zone setting to EST
Currently this data is being sent to the past (3am), the data does not arrive until 9am, when this error occurred.
Let me know if you have any questions
Olaf
-
Hi Chris,
The real-time alerts trigger based on the parsed message time from the logs. If the log message your seeing had not yet come in, or came in prior to the interval the real-time search was querying when it ran, then it's possible the alert missed the message.
Looking at your log sample, the log you have does not appear to have a timezone designation. One common cause of this type of issue is a mis-parsed time due to a missing timezone. If your Source is configured with a backup timezone to use when a timezone is not detected within the logs and this timezone does not match the actual timezone of the logs it's possible the message was received just after creation but the parsed message time is incorrect. In this case because the parsed time does not match the current real-time interval the message may have been dropped.
The best way to check for this is to run the same query that you used to show your message but check the "Use Receipt Time" box just under the time range selection. This will run your query by the received time instead of parsed time and will allow you to compare the parsed "Time" to the "Receipt Time." Usually comparing these to the time shown within the raw log message will help show if there is a timezone issue. Another way to show the receipt time of the message is to add the following operation to your current query.| formatDate(_receipttime, "MM/dd/yyyy HH:mm:ss.SSS") as receipt
Within your sample the parsed time is 20:00:40 PST but the log shows 03:00:40. If that time in your logs is actually 03:00:40 PST (UTC-0700) but your Source is configured to interpret this time as UTC then your parsed time of 20:00:40 may be incorrect. If this is the case a quick update to your Source configuration that is pulling in these logs to set a matching timezone setting should address this issue moving forward. -
Okay. I see what you are saying.
The logs are generated here on the West Coast (Los Angeles). This is why I set the timezone for the scheduled search to Pacific Standard Time. However, our headquarters are in New York, which probably accounts for the difference. I double checked this by running Kevin Keech's test and saw there is a timezone issue.
Would forcing the time zone to PST help? That is the time zone in which the log files originate.
"forceTimeZone": "America/Los_Angeles",
-
@Chris, Setting the Source configuration to use PST when a timezone is not found in the logs should resolve the issue. If you are setting this via a local JSON file, with a JSON managed config, the parameters to set should be the following.
"forceTimeZone":false,
"timeZone":"America/Los_Angeles",
Please sign in to leave a comment.
Comments
5 comments