I have a bunch of events coming in the format of the below example. They are random in the time it takes from start to end.
I want to query a term in the 'processing' event and have it return the entire session. I have been doing so by doing a subsearch for the term and getting the session id, searching for the session ID and transactioning the result. This is very CPU intensive, slow, and causes my subsearches to time out so I can only search a very limited time frame.
I have 2 thoughts on how to solve this issue:
- Correlate at index time - I am not entirely sure how to do this since the events are streamed in with no predefined start and end, and it makes me very wary of data loss.
- Correlate in a summary index - I could run the transaction command every hour to populate a summary index. However, if a session isn't complete, I assume I would lose that data in my query. If I overlap than that would lead to duplication. I am not sure if there is some way to use the overlap command to help with this?
Event 1: session_1 Start
Event 2: session_1 Processing
Event 3: session_1 Finish
Event 4: session_2 Start
Event 5: session_2 Processing
Event 6: session_2 Processing
Event 7: session_2 Finish
Any help trying to figure this out would be much appreciated.
... View more