I have two events: items received, and items acted on. I want to set an alert when the count by transactionID is not equal for the two searches. I have the search set up like so:   index=myIndex source=mySource | search criteriaForItemsReceived | stats count as itemsReceived by transactionID | append [ search index-myIndex source=mySource | search criteriaForItemsProcessed | stats count as itemsProcessed by transactionID ] | stats values(*) as * by transactionID   ok, it's a little more complicated, but this is the important part. So then need to compare itemsReceived to itemsProcessed to determine if there should be an alert. I have tried   ... stats values(eval(itemsReceived-itemsProcessed)) as Difference ... | search Difference != 0   as well as doing the eval before the stats, but everything I try ends up with null values for the eval, even though the table is properly populated with the values for itemsReceived and itemsProcessed (I have also tried convert num(itemsReceived) and tonumber(itemsReceived,10) in the event Splunk was not recognizing these fields as numbers, but each time, the fields are null). What am I doing wrong here? ... View more

I have a column chart that works great, but I want to add a single value to each column. The columns represent the sum of run times for a series of daily sub-jobs. Jobs are variable, but lets say for example there are 5 jobs that run, and maybe 5 sub-jobs. If I run my stats and chart using:   stats sum(subJobRunTime) as totalRunTime by Job Date | xyseries Date, Job, totalRunTime   I get a nice column chart. The table looks like this:  Date Job 1 Job 2 Job 3 Job 4 Job 5 6/1/2020 78.7 11.11666667 3.366666667 48.01666667 2.283333333 6/2/2020 93.55 13.71666667 39.88333333 53 3.233333333 6/3/2020 99.01666667 11.86666667 5.7 35.91666667 2.666666667 6/4/2020 77.36666667 34.98333333 86.31666667 50.78333333 2.183333333 6/5/2020 117.2166667 12.25 58.65   2.133333333   I would like to simply add a row at the bottom that is the average plus one standard deviation for each column, which I would then like to add as an overlay on the chart as a "limit line" that the user can use as a visual of "above this, job is taking too long."  I have tried append, appendpipe, and even appendcols, as well as trying eventstats to add the average as an additional field to each event, but these either add no value or worse, make the chart unnecessarily complicated (converting my columns into runTimeJob1, avgRunTimeJob1, runTimeJob2, avgRunTimeJob2, etc., so I don't have a single series "average" that I can overlay on the column chart. Would like something like:  Date Job 1 Job 2 Job 3 Job 4 Job 5 6/1/2020 78.7 11.11666667 3.366666667 48.01666667 2.283333333 6/2/2020 93.55 13.71666667 39.88333333 53 3.233333333 6/3/2020 99.01666667 11.86666667 5.7 35.91666667 2.666666667 6/4/2020 77.36666667 34.98333333 86.31666667 50.78333333 2.183333333 6/5/2020 117.2166667 12.25 58.65   2.133333333 Average+SD 100 20 60 55 3.5   (and yes, the values in the last row are totally made up, but I hope you're seeing my point). Then I'd have the ability to chart job runtime by date, with an overlay of "Average+SD" Any suggestions? ... View more

I haven't seen much on creating a bell curve in Splunk. I've created a query that returns 30,000 events for 40+ associates over a month. Each event contains the number of minutes they've worked a specific activity. I then use stats to sum the time each associate works: stats sum(hoursWorked) by Associate but I want to use bins to create a bell curve to show the "normal" distribution of each associate's work. I have tried several ways with no success. I'm basically trying to show the number of associates that fall into each bin of number of hours worked. I want it to be something like: bin span=5 hoursWorked | stats count(sum(hoursWorked) by Associate) by hoursWorked but I realize I'm trying to count a table there. Help? ... View more

I have events with a lot of possible fields to search. I made a dashboard for business (non-technical) users to be able to see those events in a meaningful way. Depending on their business unit, they will most likely only want to see a subset of the data, so I've included checkboxes for the potential fields they'd like to include in the table. But there are a lot of checkboxes, so it makes the "inputs" portion of the dashboard very long: What I'd like to be able to do is reformat it to distribute the fields in columns (and even group the other three inputs in a single column to the left), such as: Is there a way to do that? I have checked the xml documentation at docs.splunk.com/Documentation/SplunkCloud/latest/Viz/PanelreferenceforSimplifiedXML but not found any solution. From what I see, the fieldset node doesn't permit rows or panels, and the fieldset node cannot be a child of a row or panel. Any help is greatly appreciated. ... View more

So I have a single log event that captures the request and the response JSONs. As a user I'd like to be able to write a query that will capture the fields from the JSONs, but the field names are the same in the request and the response, so when I search: index="myIndex" sourcetype="mySourceType" "Keywords to search for only request and response events" | rex field=_raw "This event received the following request (?<requestJson>.*) and sent the following response (?<responseJson>.*)" | spath input=requestJson | spath input=responseJson When I get the results of this search, I get one field with two values (request and response values): "clientId":[123, 123] <-----searched by "name":[null, "Joe Schmoe"] "ssn":[null, "123-45-6789"] .....etc. What I'd really like to be able to do is get a response more like: "request.clientId":123 "request.name":null "request.ssn":null "response.clientId":123 "response.name":"Joe Schmoe" "response.ssn":"123-45-6789" I tried renaming the fields in "requestJson" after using spath: spath input=requestJson | rename * as request.* but that doesn't seem to work unless I use at least one letter before the wildcard (*), such as: spath input=requestJson | rename a* as request.* How can I rename these fields generated dynamically by spath-ing my JSONs? Or, alternative I may be missing: how can I differentiate between the request and response values even though they have the same field name? ... View more

Cog in a larger machine, I have asked my Splunk team to improve the parsing on some of our logs, but it hasn't happened yet, so I need a way to parse complex events in my search. I have run my raw event(s) through rex and gotten several fields of interest (eg. field1, field2, field3, field4). Field 3 is a json, which I can then run through spath to get the key:value pairs, but field4 is a different series of key:value pairs (formatted as field-name=[fieldValues]). My question is, I run my rex, then I pipe, then I run spath, then... how do I run extract? Example event: 2019-03-20 19:25:16.010 Container: 32 Fred {json}, {Date=[some date and time and zone], Another-field=[values=may have; subvalues=with spaces; but no internal closing bracket], and-so-on=[until the last field]} Note: I am trying to keep it all simple other than field4, which is the basis of my question; I am not having a problem with rex, so don't focus on my rex expressions (I won't even put them), but I can easily extract the other fields with rex, as well as the json fields with the spath: rex field=_raw "Container: (?<field1>AppropriateRex) (?<field2>AppropriateRex) (?<field3>AppropriateRex),\{(?<field4>AppropriateRex)\}" | spath input=field3 so I would get: field1=32 field2=Fred field3={json} field4 Date=[some date and time and zone], Another-field=[values=may have; subvalues=with spaces; but no internal closing bracket], and-so-on=[until the last field] and let's say all 50 of the fields from my json are properly extracted via spath. How do I add an "extract" command to get the key:value pairs from field4? I'd like to: | extract input=field4 ... but extract doesn't have an "input" parameter. Thank's in advance for your help. ... View more