Is the search results "Export Results" limited?
If a query has more than 10,000 records, it seems to only export 10,000 records to the CSV File.
For example, I have query that has 263,741 records in the result set. When I click "Export Results" I get "Export successful" message. Then I click download and regardless of if I choose download to file or load directly into Excel, I get a CSV file with 10,001 rows (the first being the headings).
Nothing in the docs or online queries indicates a limit. Is there a way to get all the results? Excel will happily take 1 MM records.
-
I should note that this query had counts and I was trying to export from the messages tab. I noticed if I try to browse too far into the records set, I get an error telling me I need to view results in new tab. Maybe there's a limitation that applies only if query has aggregate counts?
-
Steven,
There are a couple of things to note. From the UI, we only support downloading 10K rows, however if you use the search API to access results, you can download more.
Also, if all you want is the raw message tab, you can/should remove the aggregate operators like "group by". When aggregate operators are used, we think that the user is more interested in the aggregate results, and hence only show a subset of raw logs.
Hope that helps.
kumar
-
Thanks for the response. The problem with running w/o aggregates is that the query pauses every few minutes unless that's been changed.
I would just do the sort of pivot table work I'd do in Sumo if it was faster. But for now its generally much quicker to look at variations in Excel.
While one might think 10k records is more than enough, we have cases where we see numbers like 250k of even a single error combination. 10k is a very small number of events for a large scale service.
The search API seems rather unattractive since I'd need to build my own tools to pull results. I'll look into it though.
I think once pre-indexed Sumo Views are available I may be able to perform the aggregations I need quickly the enough not to have to load the data is into Excel.
Thanks for clarifying though!
-
Steven,
I have a hunch you already know this - but you should consider creating a view (Manage => Scheduled Views) and create one with this query. Queries done on top of the views will be much faster. You can access those views in your query by using _view=<name of the view>.
Please lets us know if we can help with a use case, or help optimize the queries. If you can file a support ticket with the query you are trying to use, we can certainly review those and help you get to the answers efficiently.
best,
kumar
-
Dmitry,
It is documented here : http://help.sumologic.com/Help/Default.htm#Exporting_Search_Results.htm?Highlight=csv
-
- Sorry, the page at https://help.sumologic.com/Help/Default.htm could not be found.
Can you please route me to the API / Export data documentation ?
-
Nischal,
The link above to export through the UI has been updated, the correct link is: https://help.sumologic.com/Search/Get_Started_with_Search/Search_Basics/Export_Search_Results
As for API documentation, you can find that here: https://help.sumologic.com/APIs/About-the-Search-Job-API
Cheers,
Mario
Please sign in to leave a comment.
Comments
10 comments