Speed up a duration search?

[mark_chuman]

Here is my search. I'm trying to get a report on the duration between an ESXi host sync task in vCenter logs. The search works ok in searches that are looking over say 15 minutes or so, but seriously slows down when going to 4 days and beyond. Any idea how to speed things up? Thanks!

index=distributed sourcetype=vcenter_* "task-internal" | rex "task-internal-(?\w+)" | transaction TASKID maxspan=-1 | timechart avg(duration)

Here are example log entries -

11/21/14
3:59:43.000 PM

2014-11-21T15:59:43.760-05:00 [01928 info 'commonvpxLro' opID=HB-host-52@132635-1b9acc5a] [VpxLRO] -- BEGIN task-internal-574059 -- host-52 -- VpxdInvtHostSyncHostLRO.Synchronize --
eventtype = nix-all-logs
11/21/14
3:59:43.000 PM

2014-11-21T15:59:43.758-05:00 [11912 info 'commonvpxLro' opID=HB-host-901@183888-5f049582] [VpxLRO] -- FINISH task-internal-574059 -- host-901 -- VpxdInvtHostSyncHostLRO.Synchronize --
eventtype = nix-all-logs

[martin_mueller]

There are several steps towards speeding up this search:

1. Remove unnecessary events from the search: index=distributed sourcetype=vcenter_* "task-internal" (BEGIN OR FINISH)
2. Make the transaction more targeted: ... | transaction TASKID startswith="BEGIN" endswith="FINISH"
3. As suggested in the comments, avoid running one huge transaction and either switch to stats or chunk up the work with summary indexing. Beware though of transactions that start in one chunk and finish in the other, you will need to write your search to cover that overlap smartly.

I'd test #1 and #2 together first, see if that speeds up the search well enough. If not, consider #3.

Additionally, do post the timing info at the top of the job inspector for us to look at what's taking up the most time.

[mark_chuman]

Aside from speeding things up is there a better way to chart the duration between say these two examples (between a BEGIN and FINISH log entry)?

[somesoni2]

Is your task id unique everytime? If yes then your can use stats (first and last of _time ) to calculate duration and that would be way faster than transaction.

[aljohnson_splun]

Is this answer of any help? They use min & max.

[aljohnson_splun]

Have you checked out summary indexing? Your search would also be a lot faster if you can find a way to avoid using the transaction command, which is notoriously slow (e.g. using stats and a by clause)

[mark_chuman]

yes, the transaction command is the culprit on the slow down. I'll take a look at summary indexing

[somesoni2]

Agree, also, saving your field extraction to props.conf would also speed up your search.