@PickleRick - unfortunately, not able to make it work.  The eventtype that is present for every log is still showing up in events tab and timechart tab comes out empty.  ... View more

@PickleRick - read through the documentation and tried search first as suggested and then timechart count by eventtype, it did not work. Not sure what i'm doing wrong. any help is appreciated. the eventtype that is present in every events is still present in the chart.  eventtype=heartbeat namespace::my-namespace | search eventtype=heartbeat | timechart count by eventtype span=1m ```only want to see eventtype heartbeat``` ... View more

@tread_splunk  thank you so much for looking into this. I will try out the query you just provided. After thinking through, I realized that I should just provide real data with clean-up and masking.  Here is the real data.  https://drive.google.com/file/d/1If2G2JNFm7NljWR7WuIU_VavIBxH9jqE/view?usp=drivesdk  ... View more

@tread_splunk @somesoni2  I realized having a sample data set would be helpful. For that reason, I am attaching a sample data set and an explanation below: We have 4 Pods. Each of these pods receive 3-5 messages in Every Minute. Now these messages are NOT evenly distributed. Meaning, it's not like time: 0s, 15s,30s,45s etc.  We have noticed, one or two of these pods goes in Zombie state. Meaning, for say 3 minutes, these Pods are not writing this event.  Overall objective is to find the query > create a dashboard panel > generate simple alerts when we detect these Zombiness.  Now the explanation of the dataset. Below is the top level query: index namespace message="incoming events" pod=* To detect the bad actors with human eyes, I am adding timespan so I can detect the anomaly. The sample data that I am providing comes from the below query (and not the starter query): index namespace message="incoming events" pod=* |  timechart count by pod  span=1m Since data from raw data set is in JSON objects, I am not attaching here. But I can of course provide the true raw data set if it helps in our investigation.  From this dataset, this is the bad Pod: Bad Pod: pod-a 2021-10-14T21:01:30.000+0000 event count 3 2021-10-14T21:02:30.000+0000 event count 0 ............. ........... 02021-10-14T21:05:00.000+0000 event count 0 02021-10-14T21:05:30.000+0000 event count 4 Duration of being in bad state: 3 mins I am trying to get the query that I can utilize in creating a dashboard (and alert), where me or any of my team mate can simply run the query and detect: 1. Bad Pod 2. Duration in bad state ( time > 1m).   I appreciate your help and Thank you.  And as I said, I can always provide the true raw data (from the starter query) if it helps in this investigation.       CSV File  ... View more

@somesoni2 - yes ... View more

@tread_splunk  - yes ... View more

@tread_splunk  the expected behaviour is in every minute, every pod receives 2 to 4 heartbeats messages.  the Zombie behaviour is a POD(s) not either receiving/ writing logs for several minutes.  ultimate goal :  figure out which pod(s) and how many minutes did not register heart beats. then setup alert based on that. ... View more

yes. I have a list of all pods. ... View more

this is what I am trying to detect. The time range or minute(s) when we are missing the heart beat for any given pod.  ... View more

@somesoni2 - sorry, it did not work. ... View more

This worked 100%. Thank you so much!!!! ... View more

@oscar84x :: Yes. Within the same time frame (e.g., Last 24 hours, Last 7 days), I'm seeing search results that are not consistent. However, The alerts I receive based on "status" is accurate. To extract the http status, like 200, 500 etc., I used the regular expression to create the "status" field extraction.  Since, "| stats count by status" does not bring the 500 statuses, my dashboard is off not much use. ... View more

@thambisetty , I am sorry, my post probably was not very clear. Let me rephrase... Original query:   index=xyz | stats count by eventtype where All_logs encompasses every log in the search (100% coverage). Current Result: eventtype count All_logs 14 Error 2 Login 4 Auth 8   Expected Result: eventtype count Error 2 Auth 8   Appreciate all your help. ... View more

@thambisetty , still seeing All_logs and Login events in the stats count 😞 ... View more