|
I'm having trouble with backfilling a scheduled search that populates the summary index. Here is the basic search:
When I run this search on a small dataset I get the expected results – there are 10-minute wide timestamps, and one entry for each timestamp. A small sample follows:
This search is scheduled as follows:
The idea here is that I want to roll up some statistics into 10-minute buckets, and I want the job to run every hour, and look back in a window slid backwards in time by 10 minutes, in order to allow data to make its way through the LWF into the Indexer. The problem is that when I run the backfill, I am only seeing a very limited number of entries in the summary index, and the few entries that exist are all of the form hh:50:00 which means they are all for the 10-minute window spanning 50-59 minutes after the hour. There is definitely a full range of entries covering the entire hour, so it's not a data problem. Can you figure out why this is happening and tell me how to correct the search and/or the schedule configuration data? I think I must have something configured incorrectly, though damned if I can figure out what it is. Thanks! |

How are you running the backfill?
To perform summary indexing, the best would be to use "sistats" (or "sitop", "sitimechart") commands as this will also store additional informations about the aggregated fields which are useful for further and later aggregations.
When you run the scheduled search manually, do you get the results you want/expect?
BTW, you would probably be better off running this every 10 minutes over a 10-minute window in order to smooth out the load on the system, instead of hourly over an hour. The same total amount of work pretty much needs to be done in either case.
Also, what you're doing generally seems right, so there is probably something deeper configured wrong.