Refine your search:

I'm having trouble with backfilling a scheduled search that populates the summary index. Here is the basic search:

index=cdnmanager | bucket _time span=10m | stats sum(Bytes) as NumBytes, sum(Elapsed_Time) as ElapsedTime by Base_Content,_time

When I run this search on a small dataset I get the expected results – there are 10-minute wide timestamps, and one entry for each timestamp. A small sample follows:

_time Base_Content NumBytes ElapsedTime 
1   1/24/11 8:00:00.000 PM  a_e       6636911       2116055
2   1/24/11 8:00:00.000 PM  ae        8554          368082
3   1/24/11 8:00:00.000 PM  amc       16695         10710
4   1/24/11 8:00:00.000 PM  cnn       8666          8111
5   1/24/11 8:00:00.000 PM  comedy    7530079       1285838

This search is scheduled as follows:

action.summary_index = 1
action.summary_index.report = bandwidth_by_service
cron_schedule = 0 * * * *
dispatch.earliest_time = -70m@m
dispatch.latest_time = -10m@m

The idea here is that I want to roll up some statistics into 10-minute buckets, and I want the job to run every hour, and look back in a window slid backwards in time by 10 minutes, in order to allow data to make its way through the LWF into the Indexer.

The problem is that when I run the backfill, I am only seeing a very limited number of entries in the summary index, and the few entries that exist are all of the form hh:50:00 which means they are all for the 10-minute window spanning 50-59 minutes after the hour. There is definitely a full range of entries covering the entire hour, so it's not a data problem. Can you figure out why this is happening and tell me how to correct the search and/or the schedule configuration data? I think I must have something configured incorrectly, though damned if I can figure out what it is. Thanks!

asked 31 Jan '11, 17:11

beaumaris's gravatar image

beaumaris
24619
accept rate: 50%

1

How are you running the backfill?

To perform summary indexing, the best would be to use "sistats" (or "sitop", "sitimechart") commands as this will also store additional informations about the aggregated fields which are useful for further and later aggregations.

(31 Jan '11, 22:03) Paolo Prigione

When you run the scheduled search manually, do you get the results you want/expect?

(01 Feb '11, 07:10) gkanapathy ♦

BTW, you would probably be better off running this every 10 minutes over a 10-minute window in order to smooth out the load on the system, instead of hourly over an hour. The same total amount of work pretty much needs to be done in either case.

(01 Feb '11, 07:13) gkanapathy ♦

Also, what you're doing generally seems right, so there is probably something deeper configured wrong.

(01 Feb '11, 07:14) gkanapathy ♦
Be the first one to answer this question!
toggle preview

Follow this question

Log In to enable email subscriptions

RSS:

Answers

Answers + Comments

Markdown Basics

  • *italic* or _italic_
  • **bold** or __bold__
  • link:[text](http://url.com/ "Title")
  • image?![alt text](/path/img.jpg "Title")
  • numbered list: 1. Foo 2. Bar
  • to add a line break simply add two spaces to where you would like the new line to be.
  • basic HTML tags are also supported

Tags:

×299
×46
×40

Asked: 31 Jan '11, 17:11

Seen: 482 times

Last updated: 31 Jan '11, 17:11

Copyright © 2005-2012 Splunk, Inc. All rights reserved.