------ Start of Edit -------------
EDIT 1: Use Case
- The production server is sends analytics events to Splunk as tagged, log entries. I have a Python script which runs every 5 mins and searches for specific analytics tags within the logs and ingests that into our data warehouse. This incremental search is now htting the 50k limit
EDIT 2: Python Code
def search(self, text, options):
search_string = 'search ' + text
kwargs_oneshot = {"earliest_time": options['start_time'],
"latest_time": options['end_time']}
oneshotsearch_results = self.service.jobs.oneshot(search_string, **kwargs_oneshot)
lazy_results = results.ResultsReader(oneshotsearch_results)
return [l for l in lazy_results]
-------- End of Edit -------------
Hi,
In my use case, I can't lose data during search. I'm currently hitting the max limit of 50000 result rows. I am currently reducing the time interval for my searches but our log data is scaling up. The documentation says the following about maxresultrows.
maxresultrows =
* This limit should not exceed 50000. Setting this limit higher than 50000
causes instability.
Other posts in this forum (can't post links, low karma) talk about increasing the limit to 100000 for e.g. But according to the above documentation, won't we lose results?
If we do lose results if we go over 50k, how can I get above that? Is there some kind of paging mechanism which allows me to go above this limit without losing data?
... View more