I have 2 logs. The first statement gets logged when a pod dies. The second gets logged when my app gets notified. Sometimes, the pod dies and my app doesn't get notified. I want to write an alert when the pod dies but my application doesn't get notified.
Log1 (when a pod dies):
index=log1 "Forced deletion of orphaned Pod" | rex "podnamespace/(?<machineName>(.*?))\s"
Log2 (when my app gets notified):
index=conversation "*Clearing DMC pod" sourcetype="cui-orchestration-log" podname=<podNameWhichDied>
I tried several options, but I am unable to refer to the field 'machineName' created by rex in the Log1 inside Log2 even though machineName has the right pod name:
index=log1 "Forced deletion of orphaned Pod" | rex "podnamespace/(?<machineName>(.*?))\s" | stats count as podsCrashedCount by machineName| appendcols [search index=log2 "App is deleting pod" podname=$machineName| stats dc(podname) as deletedInApp] | where podsCrashedCount!=deletedInApp
... View more