About tpflicke

tpflicke · ‎09-21-2015

The pivot command currently (v6.2.1) support the median but not other percentiles. However, if you simply want to have the reports using datamodel acceleration rather than the drag and drop features of the pivot UI the you could use tstats. The tstats command will use the tsidx files of your accellerated datamodel and you can use perc , exactperc and upperperc . | tstats p25(Duration) AS p25 p50(Duration) AS p50 p75(Duration) AS p75 from datamodel=SomeDataModel BTW, for something more complex you might create a pivot via the UI, run it and inspect the job which gives you the tstats command which you can then modify / develop further.

tpflicke · ‎03-24-2015

I had the same message and resolved it by single-quoting the fields inside the eval statement so the following might do the trick: ... | eval n = if('Probe.Class'==1, "true", "false") | ...

tpflicke · ‎02-26-2015

While a far better solution would be fixing the log format, I wonder if a heavy forwarder could be employed to modify the date string before sending it to the indexing tier. If necessary, the forwarder could be configured to ignore the timestamp, it should simply apply the transformation, e.g. via SEDCMD, then send the data on.

tpflicke · ‎02-24-2015

For larger amounts of HTML I found it useful to format multi-line with \, e.g: login_content = This is a <b>production server</b>. \ If you are running some expensive searches, try one of the following servers:\ <a href="http://server1:8000">server1</a> or <a href="http://server2:8080">server2</a>

tpflicke · ‎02-24-2015

I found the following in $SPLUNK_HOME/etc/users//user-prefs/local/user-prefs.conf of Splunk 6.2.1: eai_app_only = False eai_results_per_page = 100 What do they do? There doesn't seem to be any documentation.

tpflicke · ‎02-22-2015

From yannK's answer I understand the problem is that you've got clock hour [1;24] rather than hour [0;23] so you need to shift everything back by 1 hour from clock hour to hour as Splunk has no support for clock hour. The query below shows how this can be done using rex mode=sed and a timezone offset with the 24 clock hour being handled separately. Essentially what happens is: "28.02.2015 24:05:03:100" => "28.02.2015 23:05:03:100-0000" => "28.02.2015 23:05:03.100" "28.02.2015 20:32:45:123" => "28.02.2015 20:32:45:123+0100" => "28.02.2015 19:32:45.123" | gentimes start=-1 | eval testdata="something 28.02.2015 24:05:03:100 else#something 28.02.2015 20:32:45:123 else" | table testdata | makemv testdata delim="#" | mvexpand testdata | eval _raw = testdata | rex mode=sed field=_raw "s/(\d\d\.\d\d\.\d{4}) 24(:\d\d:\d\d:\d+) /\1 23\2-0000 /g" | rex mode=sed field=_raw "s/(\d\d\.\d\d\.\d{4}\s+\d+:\d\d:\d\d:\d+) /\1+0100 /g" | rex field=_raw "(?<date_str>\d\d\.\d\d\.\d{4}\s+\d+:\d\d:\d\d:\d+[+-]\d+)" | eval n = strftime(strptime(date_str, "%d.%m.%Y %k:%M:%S:%3N%z"), "%d.%m.%Y %H:%M:%S.%Q") | table testdata date_str n You should also be able to specify the manipulations in props.conf using SEDCMD which would be applied at indexing time: SEDCMD-clockhour24 = s/(\d\d\.\d\d\.\d{4}) 24(:\d\d:\d\d:\d+) /\1 23\2-0000 /g SEDCMD-clockhourXX = s/(\d\d\.\d\d\.\d{4}\s+\d+:\d\d:\d\d:\d+) /\1+0100 /g

tpflicke · ‎02-14-2015

With regards to the regular expression - the first group should be non-greedy to prevent it from gobbling up too much. Here's my test query producing the desired result for the given examples: | gentimes start=-1 | eval temp="DATABASE-DTA-PRD#APACHE-SCM-PRD-TST#SERVERS-PRD" | table temp | makemv temp delim="#" | mvexpand temp | rename temp as _raw | rex field=_raw "(?<result>.*?)(?:-DTA-PRD|-PRD-TST|-TST|-DTA|-PRD)" | table _raw result If you are not anchoring the regular expression to the end with $ you shouldn't need the -PRD-TST and -DTA-PRD matches. Under the condition that the -SCM tag is always the first tag and that there is always at least one tag you might simplify the query a bit, e.g. | rex field=_raw "(?<result>.*?)(?:-DTA|-PRD|-TST)" Alternative 1 Using rex mode=sed would allow you to discard specific tags which also works if the tag order differs. | gentimes start=-1 | eval temp="DATABASE-DTA-PRD#APACHE-SCM-PRD-TST#SERVERS-PRD#APACHE-PRD-TST-SCM" | table temp | makemv temp delim="#" | mvexpand temp | rex field=temp mode=sed "s/-(PRD|TST|DTA)//g" | table temp Alternative 2: Now, if things turn really complicated the you could resort to using a lookup table full_fieldname,short_fieldname DATABASE-DTA-PRD,DATABASE APACHE-SCM-PRD-TST,APACHE-SCM APACHE-PRD-SCM-TST,APACHE-SCM SERVERS-PRD,SERVERS ... and use like: | eval full_fieldname=... | lookup name_translation full_fieldname OUTPUT short_fieldname | eval fieldname=if(isNull(short_fieldname), full_fieldname, short_fieldname) | stats count by fieldname All of these would work if you do the manipulation in the query, if you want to do this in props.conf this might not be the case.

tpflicke · ‎02-14-2015

The following should do the trick though producing the regular stats output format: ... | rex field=_raw "\{state,\"(?<state>\w*)" | stats count by state | sort -count or with timechart: ... | rex field=_raw "\{state,\"(?<state>\w*)" | timechart count by state

tpflicke · ‎02-14-2015

Here's a suggestion trying to get to the same result though via a different approach and one that would not require to effectively make an all time search every hour. To remember already discovered combinations of field1, field2, field2 long term and even exceeding your normal data retention period you could use either a summary index or a lookup table. In general I find working with lookup tables easier so here's an suggestion facilitating one: In CSV format the table, I named discovered_combinations, could look like: discovered_timestamp,field1,field2,field3 1423922204,AAA,"",123 1423923303,"",bbb,435 1423924444,"RRR,bbb,"" The timestamp would provide a return value but might also be used in queries and generally help keep track of when field combinations were discovered. A search scheduled every hour would find events that don't match a field combination in the lookup table, dedup and send the alert so you get the first event for any new combination. index=mydata earliest=-1h@h latest=@h | dedup field1, field2, field3 | lookup discovered_combinations field1 field2 field2 OUTPUT discovered_timestamp | where isNull(discovered_timestamp) A second search scheduled afterwards would then update the lookup table using ... | outputlookup discovered_combinations append=true . There might be a way of doing this all in a single query which would be neater.

tpflicke · ‎02-13-2015

One approach would be to set the alert action for your search to a script. The script can then send you an email and also interact with Splunk via the REST API. http://dev.splunk.com/view/SP-CAAADQ5 provides an example for disabling a search via the API: curl -k -u admin:changeme -d "disabled=1" -d 'search="fail+OR+error"' https://localhost:8089/servicesNS/admin/search/saved/searches/web_errors You also create, delete and otherwise manipulate saved searches without having to interact with the UI. It's also possible just disable the alert action.

tpflicke · ‎02-13-2015

To deal with all the various examples in this thread and all other possible cases such as new domains like .london, I think it will need something more than a reasonably short regex line. I would probably go down the route of calling a Python script to deal with the cases to my satisfaction and being able to lay out the logic in a maintainable way. Maybe there is a splunk app or add-on that provides such functionality, if not, it could make a nice exercise to create one. A few test cases: conductor.io.com => io.com support.expedia.co.uk => expedia.co.uk 0.52.channel.facebook.com => facebook.com 0.52.channel.facebook.london => facebook.london

tpflicke · ‎02-13-2015

I am not sure if you want to return events with only public or only private IP addresses. How about index=<someindex> source=<somesource> " connected to 10\." to only get the private ones or index=<something> source=<somesource> "connected to " NOT "connected to 10\." to get the public ones. This will only look consider 10.x.x.x as private (not other ranges like 192.168.x.x etc) which might be sufficient for what you want. This should also be quick as it's filtering on the indexer.

tpflicke · ‎02-10-2015

To elaborate on the answer provided by thomrs: I've recently set up an alert to keep an eye on lag (including 'negative lag' caused by incorrect timezone setting or server time). Note that the query is looking only at the latest event for each source by using dedup and does not attempt to determine lag by indexer. index=<index_name> | dedup host source | eval indexed_time=strftime(_indextime, "%+") | eval file_timestamp=strftime(_time, "%+") | eval lag = round(_indextime - _time) | eval abs_lag=abs(lag) | table host file_timestamp indexed_time lag abs_lag source | sort -num(abs_lag) | search abs_lag > 30 | rename lag as "lag in seconds" | rename abs_lag as "lag in seconds (unsigned)" | rename indexed_time as "log event indexed at" | rename file_timestamp as "timestamp from log event (adjusted)"

tpflicke · ‎02-10-2015

The following eval clause might do the trick though it will result in field Logon being present with a blank value if neither phrase is found: eval Logon=case(match(_raw,"user logged on"), "yes", match(_raw,"user failed logon"),"no",1==1,"") As jrodman wrote this could be used as part of the search and should also work for creating a calculated field. Another approach would be using rex mode=sed to change the text: rex mode=sed field=_raw "s/user logged on/Logon=yes/" | rex mode=sed field=_raw "s/user failed logon/Logon=no/" You might need a kv or | rex field=_raw "Logon=(?<Logon>yes|no)" to pick up the name value pair.

tpflicke · ‎05-27-2014

Thanks, exactly what I need.

tpflicke · ‎05-24-2014

I've got a large number of logs which look similar to: INFO com.this.that.SomeLogger 2014-05-08 08:29:49,997 [CSP-12.3.4.3245432] [pool-10-thread-1] {"metaData":{"sourceURI":"/search/" ....}} and INFO com.this.that.SomeLogger 2014-05-08 08:29:49,997 [CSP-12.3.4.3245432] [pool-10-thread-1] payload=<event type="x12">...</event> in addition to log entries with a freeform text or name-value pair load. It the whole event was JSON then Splunk would format this nicely in the event viewer panel but this doesn't work for the above mixed entries. Changing the logs to all JSON is at present not an option. One way seems to be creating custom events viewers that would deal with displaying above mixed events so users awkwardly end up copy-pasting payload into a JSON or XML formatter. I hope it's possible to improve presentation of events with mixed content with limited effort, e.g. by facilitating an already existing Splunk app or deriving custom events viewers from existing ones. What are my options?

tpflicke · ‎05-24-2014

I frequently use the length of the raw data - more often than readily extracted fields punctuation and linecount I do so be extracting the field in the query like ... | eval raw_length=len(_raw) | ... What are the options to do this as a predefined field extraction at either search time or even index time?

tpflicke · ‎05-19-2014

Excellent, specifying the prefix to avoid collisions was precisely what I was missing. Thanks a lot.

tpflicke · ‎05-19-2014

I wonder if it is possible to perform more than one inline drilldown. Here is what I would like to do: run a query A listing the top x error codes during a selected time frame, e.g. 10 minutes display the results in a table A when a table A row is clicked run a query B against an error code - bug ticket lookup table (there can be multiple tickets for a given error code) display the results in a second table B when clicking on a row in table B perform some action using the data of that row, e.g. show details, run another search ... I have been using the Table - inline drilldown from the Sideview utils (3.2.6) as starting point. I understand how to pass data from the clicked row of table A to narrow search B. However, I struggle to see how step 5. could be made to work given the $row.fields.xxxx$ mechanism of passing data.

tpflicke · ‎05-19-2014

The answer to this question is actually the same as for the field menu: this can be achieved using workflow actions. (Credits to bwooden)

tpflicke · ‎05-09-2014

Thanks, that's exactly what I was looking for.

tpflicke · ‎05-08-2014

When a field is selected to be shown in the results, the field appears with a collapsed dropdown menu containing the items 'Tag = ' and 'Report on Field'. I would like to extend this menu with additional items for some fields. Example I extract and show the user_id in the results. In the menu I want to display an item that, when clicked, opens the URL http://somedomain/display?user_id=6787765685 in a new browser tab/window BTW, ideally I want to build the URL from more than just one field value, e.g. http://somedomain/display?user_id=6787765685&date=20140131&firstname=Fred The HTML of the menu (before modifications) looks like: <div class="outerMenuWrapper splShadow splMenu splMenu-primary" style="display: none; top: 556px; left: 975px;"> <ul> <div class="innerMenuWrapper"> <li class=""> <a href="javascript:void(0);" tabindex="-1" class="menuItemLink" s:fieldname="user_id" s:fieldvalue="6787765685"> Tag user_id=6787765685</a></li> <li class=""><a href="javascript:void(0);" tabindex="-1" class="menuItemLink" s:fieldname="user_id" s:fieldvalue="6787765685">Report on field</a></li> </div> </ul> </div>

tpflicke · ‎05-07-2014

I want to extend the Event Options Menu which is located beside the result records. The idea is to add a link containing some data from the result record beside which the menu is located. When clicking on the extra menu item the link should open in a new browser tab. Is this achievable with relatively limited effort? Example for Illustration I a result record contains the field userid then the Event Options Menu should contain an item Display in XYZ which would invoke a URL like https://some_domain/display_user?date=20140203&userid=12434233433 The date would be derived from _time and userid from a field in the results, e.g. customer_id. While the Event Options Menu is appealing for this enhancement, I am open to other suggestions too The HTML of that menu (without any modifications): <div class="outerMenuWrapper splShadow splMenu splMenu-primary" style="display: none; top: 411px; left: 243px;"> <ul> <div class="innerMenuWrapper"> <li class=""><a href="/en-US/etb?sid=1399554203.68&offset=0&namespace=search" tabindex="-1" class="menuItemLink" target="_blank">Build Eventtype</a></li> <li class=""><a href="/en-US/ifx?sid=1399554203.68&offset=0&namespace=search" tabindex="-1" class="menuItemLink" target="_blank">Extract Fields</a></li> <li class=""><a href="/en-US/app/search/show_source?sid=1399554203.68&offset=0&latest_time=" tabindex="-1" class="menuItemLink" target="_blank">Show Source</a></li> </div> </ul> </div>

tpflicke · ‎11-30-2013

I've created the following query for a range of -12h...+12h with data 1,2,3,4 weeks ago and the data of the last 12h creating a timechart with 10m span and snapping to 10min. The query works ok with minor caveats such as ignoring summer/winter time changes. index=x somesearchfields earliest=-684h latest=+12h | eval interval_sec=600 | eval week1_end = relative_time(now(), "-156h") | eval week1_start = relative_time(now(), "-180h") | eval week2_end = relative_time(now(), "-324h") | eval week2_start = relative_time(now(), "-348h") | eval week3_end = relative_time(now(), "-492h") | eval week3_start = relative_time(now(), "-516h") | eval week4_end = relative_time(now(), "-660h") | eval today_start = relative_time(now(), "-12h") | eval dataset = case( _time < week4_end,"4 weeks ago", _time > week3_start AND _time < week3_end, "3 weeks ago", _time > week2_start AND _time < week2_end, "2 weeks ago", _time > week1_start AND _time < week1_end, "1 week ago", _time > today_start AND _time<=now(),"last 12h", 1==1, null) | eval stop=interval_sec*floor(now()+(12*3600)/interval_sec) | eval start=interval_sec*ceil((now()-(12*3600))/interval_sec) | eval _time = if(dataset=="1 week ago", _time+(3600*24*7), _time) | eval _time = if(dataset=="2 weeks ago", _time+(3600*24*14), _time) | eval _time = if(dataset=="3 weeks ago", _time+(3600*24*21), _time) | eval _time = if(dataset=="4 weeks ago", _time+(3600*24*28), _time) | where _time>start AND _time<stop | timechart span=600s fixedrange=f count by dataset The Question: How do I modify the query to get a timechart displaying the following: the last 12h of data (as currently showing as 'last 12h') the data for the interval a week earlier (as currently showing as '1 week ago') the average of the data 1,2,3 and 4 weeks earlier (i.e. avg('1 week ago' + '2 weeks ago' + '3 weeks ago' + '4 weeks ago'))

tpflicke · ‎11-29-2013

Ah, didn't think of trying metadata or inputlookup as first element of a sub-query. This certainly opens up possibilities!

Posts	35
Solutions	6
Karma Given	11
Karma Received	20
Member Since	‎11-01-2013

Online Status	Offline
Date Last Visited	3 weeks ago

What do eai_app_only and eai_results_per_page in u...

Custom event viewer for records that have XML, JSO...

How to extract event raw size just like linecount

Multiple inline drilldown

How to extend the dropdown menu beside a field in ...

How to extend the Event Options Menu

Timechart with data 1 week earlier and average of ...

How to avoid subsearch auto-finalize in query perf...

Re: Is it possible to get percentiles in pivots?

Re: Search on data model object, weird behavior

Re: Date not parsed if the hour is 24

Re: Splunk Login Page Banner

What do eai_app_only and eai_results_per_page in u...

Re: Date not parsed if the hour is 24

Re: Regex question: cut of several tags and any co...

Re: timechart count by -not a legit field-

Re: How to create an hourly alert when never seen ...

Re: Need a query to run until it gets results and ...

Re: Extracting domain name out of a url

Re: Differentiate between public IP and private IP...

Re: How to create an alert to identify if there is...

Re: How to create a new key-value pair name and va...

Re: How to extract event raw size just like lineco...

Custom event viewer for records that have XML, JSO...

How to extract event raw size just like linecount

Re: Multiple inline drilldown

Multiple inline drilldown

Re: How to extend the Event Options Menu

Re: How to extend the dropdown menu beside a field...

How to extend the dropdown menu beside a field in ...

How to extend the Event Options Menu

Timechart with data 1 week earlier and average of ...

Re: How to avoid subsearch auto-finalize in query ...