Question:
My Live Dashboard is displaying a message that says "No Data to Display", but the interactive query returns results in the Messages tab.
Answer:
This issue is usually due to a timezone misconfiguration within your Source configuration(s) or a timestamp problem in your logs. If the timezones in your messages are not parsed automatically, are missing a timezone value, or the Source is not configured with a default timezone, Sumo Logic applies UTC as the timezone. This may result in the "parsed time" showing as either hours in the future or hours behind the actual message time from the logs. This effects how Panels interpret these messages.
Live Panels are active queries that only query data as it is being received into the system prior to messages being indexed, and they only look for messages with a parsed timestamp 10 minutes forward in time or within the window of the current Panel time. Basically, as the messages are received, the Panel will ask, "Are these messages between x hours/minutes ago to 10 minutes from now?" If they are, then the message gets added to the Panel. If the message parsed time does not fall within the current Panel range, they are excluded.
So if your message times are in PST time, but Sumo Logic thinks they are UTC, the Panels will skip the messages as not being current. For example, if the current time is 17:00 UTC, and the log messages coming in have a timestamp of 10:00 (PT), and the service parsed them as 10:00 (UTC) due to a timezone misconfiguration, then the Panel will not show these messages, as the parsed time is 7 hours behind the current time and may be outside the current Panel window.
An interactive search is a bit different. Interactive searches query the log messages post processing and indexing, and will find messages that have a parsed timestamp that falls within the selected time range, regardless of when they were received by the service. With an interactive search, a message that was received 7 hours prior to the parsed message time, which now falls within the query timerange, will still be found by the current query.
The easiest way to check if a timestamp parsing problem or delayed ingest could be causing this problem is to compare the parsed time "Time" field to the time the service received the message. Sumo Logic provides an option just under the time range selector called "Use Receipt Time". With this option, you can search by the time Sumo Logic received the messages instead of the time parsed from the logs. This option displays both the parsed time as well as the receipt time, so you can compare the values. If you see hours of difference between these values, then you most likely have a time parsing problem and may need to update your Source configurations.
Query For Checking Offset of Receipt Time and Message Time
The following query can be run within your account and will display a count of Collectors, Sources, and SourceNames that have a receipt time and parsed message time which is greater than 1 hour. This query should be run over a very small time range with the "Use Receipt Time" option for the query selected.
* | _receipttime - _messagetime as difference
| difference/1000/60 as diff_minutes
| where diff_minutes < -60 or diff_minutes > 60
| count by _collector, _source, _sourceName
Comments
0 comments
Please sign in to leave a comment.