Help with Splunk Transaction dealing with multiple startwiths and single endswith events?

[jphillips24744]

I have run across an edge case dealing with some f5 data.  Some times a nodes down can be reported one or more times before the nodes up occurs.  Currently setting up a transaction on pool and member name which should be unique, I end up with orphans records which aren't really orphans.  Is there some way to only have one transaction open per unique fields and skip the next match closing the transaction when it finds the endswith?  I know I could set keeporphans=false, but that would negate the whole purpose of this report which is to determine if a node is down.

Here is what I am trying to do.

| makeresults
| eval _raw = "Nov 19 2022 00:24:37 mcpd[9745]: 01070638:5: Pool /den-dmz/ShapePool member /den-dmz/Shape-Prod-East:443 monitor status down. [ /den-dmz/ShapeMonitor: down; last error: /den-dmz/ShapeMonitor:  Response Code: 307 (Moved Temporarily) @2022/11/19 00:24:37.  ]  [ was up for 0hr:0min:36sec ]"
   | eval _time=strptime("1668835477","%s" )
| append
    [| makeresults
    | eval _raw="Nov 19 2022 00:25:22 mcpd[9745]: 01070727:5: Pool /den-dmz/ShapePool member /den-dmz/Shape-Prod-East:443 monitor status up. [ /den-dmz/ShapeMonitor: up ]  [ was down for 0hr:0min:5sec ]" 
    |eval _time=strptime("1668835522","%s" ) ]
| append
    [| makeresults
    | eval _raw="Nov 19 2022 00:25:17 mcpd[9745]: 01070638:5: Pool /den-dmz/ShapePool member /den-dmz/Shape-Prod-East:443 monitor status down. [ /den-dmz/ShapeMonitor: down; last error: /den-dmz/ShapeMonitor:  Response Code: 307 (Moved Temporarily) @2022/11/19 00:25:17.  ]  [ was up for 0hr:0min:31sec ]"
   |eval _time=strptime("1668835517" , "%s" )  ]

| append
    [| makeresults
    | eval _raw="Nov 19 2022 00:25:17 mcpd[9745]: 01070638:5: Pool /den-dmz/ShapePool member /den-dmz/Shape-Prod-OTHER:443 monitor status down. [ /den-dmz/ShapeMonitor: down; last error: /den-dmz/ShapeMonitor:  Response Code: 307 (Moved Temporarily) @2022/11/19 00:25:17.  ]  [ was up for 0hr:0min:31sec ]"
   |eval _time=strptime("1668835600" , "%s" )  ]

`comment("Find Pool Name")`
| rex field=_raw "(: Pool | ltm pool )(?<pool>.*?)( member| {)"
  
`comment("Determine which member of Pool")`
| rex field=_raw "(member |members delete { )(?<member>.*?)( monitor status| })"

  
`comment("Determine Actually Status")`
| rex field=_raw "monitor status (?<status>.*?)\."

`comment("deal with up down time")`
| eval timedown=if(status=="down", _time, null())
| eval timeup=if(status=="up", _time, null())
| fieldformat timedown=strftime(timedown,"%F %T")
| fieldformat timeup=strftime(timeup,"%F %T")
| sort 0 _time desc

| transaction pool, member startswith=eval(status=="down") endswith=eval(status=="up")  keeporphans=true

| eval down_duration=if(isnull(timeup),now() - timedown, timeup - timedown)
| fieldformat down_duration=tostring(down_duration,"duration")

| table _time, pool, member, timedown, timeup, down_duration

[bowesmana]

Take a look at streamstats/eventstats, e.g.

| streamstats range(_time) as r by pool member
| eventstats max(r) as max_r by pool member
| transaction pool, member startswith=eval(status=="down") endswith=eval(status=="up")  keeporphans=true

| eval down_duration=if(isnull(timeup),now() - timedown, max_r)
| fieldformat down_duration=tostring(down_duration,"duration")

| table _time, pool, member, timedown, timeup, down_duration max_r

given your example search, insert the first two lines before your transaction and then use max_r for the duration calculation.

This will give timedown as 45 seconds in your example rather than 5, which is what I assume you're after.

Generally you will probably be able to solve the problem using just eventstats/streamstats/stats without having to resort to transaction as well, which is always a tricky command to find a valid use case for given its memory/performance constraints.

With streamstats you can use reset_after/before to reset range values when you get an up state, but you may not need those.