Not sure if I am 100% clear in the question, but here is what I am looking to do. I have a stream of incoming messages which have an identifier which can be duplicated (updated statuses). I have a flag which indicates whether the message is a new message and a flag which indicates whether the message is a close of the event. I want to run a dedup to have the current status of the message, but I want to count all of the new messages and compare them to all of the open and closed messages. I cannot seem to find a mechanism for counting a set of events and then counting a subset of those events created through the dedup command. I cannot use transactions as the number of events can range into the millions and the number of messages per event can range in the 20-30s.
Here are the two searches that I want to combine:
index=dccmtdit
| `GetDatesAndTimes`
| stats sum(IsNewDispatch) as DispatchesOpened by QuarterCreated
And:
search index=dccmtdit
| `GetDatesAndTimes`
| dedup DpsNum sortby -IsClosed -CurrentStatusDateEpoch
| stats sum(IsOpen) as OpenDispatches, sum(IsClosed) as ClosedDispatches, by QuarterCreated
I want to take those results and work further like this:
| search OpenDispatches!=0 OR DispatchesOpened!=0
| eval PercentOpen=exact(OpenDispatches/DispatchesOpened*100)
| eval PercentClosed=exact(ClosedDispatches/DispatchesOpened*100)
GIve this a try
index=dccmtdit
| `GetDatesAndTimes`
| eventstats sum(IsNewDispatch) as DispatchesOpened by QuarterCreated
| dedup DpsNum sortby -IsClosed -CurrentStatusDateEpoch
| stats sum(IsOpen) as OpenDispatches, sum(IsClosed) as ClosedDispatches first(DispatchesOpened) as DispatchesOpened by QuarterCreated
| search OpenDispatches!=0 OR DispatchesOpened!=0
| eval PercentOpen=exact(OpenDispatches/DispatchesOpened*100)
| eval PercentClosed=exact(ClosedDispatches/DispatchesOpened*100)
GIve this a try
index=dccmtdit
| `GetDatesAndTimes`
| eventstats sum(IsNewDispatch) as DispatchesOpened by QuarterCreated
| dedup DpsNum sortby -IsClosed -CurrentStatusDateEpoch
| stats sum(IsOpen) as OpenDispatches, sum(IsClosed) as ClosedDispatches first(DispatchesOpened) as DispatchesOpened by QuarterCreated
| search OpenDispatches!=0 OR DispatchesOpened!=0
| eval PercentOpen=exact(OpenDispatches/DispatchesOpened*100)
| eval PercentClosed=exact(ClosedDispatches/DispatchesOpened*100)