Doing an extraction in Splunk Stream and get an error when trying to use (?i) in my regex: (?i)x-forwarded-for([:\s]+[\d\.:\s,]+) Gives this error: Invalid regex Expression This works fine in a rex command, not sure what the Splunk Stream syntax is for case insensitive. Thanks. ... View more

We have Enterprise Security installed for a specific Search Head and would like the _audit logs in a different location than the main Search Heads. The ES SH is used for doing security investigations and we do not want the searches executed readable by the masses. However, we don't want to lock down everything in _audit. I'd think the simplest thing to do is have the _audit logs for that one SH sent to a different index?? Is that even possible?? Thanks. ... View more

After our SSO migration, users have reported instances where a single tab will re-authenticate which causes a cascading re-authentication across all tabs. This wouldn't be so bad if it happened once, but for some reason the re-authentication cycles between the tabs and happens continually. Here is a sample of the events that show up in the audit log: 5/15/19 9:52:05.198 AM 127.0.0.1 - 111111 [15/May/2019:15:52:05.198 +0000] "GET /en-US/splunkd/__raw/services/authentication/users/111111?output_mode=json&_=111222333 HTTP/1.0" 200 1904 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.111.111.111 Safari/537.36" - 111aaa222bbb333ccc 0ms host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd_ui_access.log sourcetype = splunkd_ui_access 5/15/19 9:52:05.160 AM 127.0.0.1 - 111111 [15/May/2019:15:52:05.160 +0000] "GET /en-US/splunkd/__raw/services/authorization/roles?output_mode=json&count=0&_=111222333 HTTP/1.0" 200 6593 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/111.111.111.111 Safari/537.36" - 111aaa222bbb333ccc 0ms host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd_ui_access.log sourcetype = splunkd_ui_access 5/15/19 9:52:05.039 AM Audit:[timestamp=05-15-2019 15:52:05.039, user=111111, action=change_authentication, info=denied ][n/a] host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = audittrail sourcetype = audittrail 5/15/19 9:51:57.179 AM 05-15-2019 15:51:57.179 +0000 ERROR UiAuth - Request from 127.0.0.1 to "/en-US/splunkd/__raw/servicesNS/111111/aaabbbcccddd/search/jobs" failed CSRF validation -- expected "111aaa222bbb333ccc444ddd", but instead cookie had "111aaa222bbb333ccc444ddd" and header had "555eee666fff777ggg888hhh" host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 5/15/19 9:51:57.123 AM 05-15-2019 15:51:57.123 +0000 ERROR UiAuth - Request from 127.0.0.1 to "/en-US/splunkd/__raw/servicesNS/111111/aaabbbcccddd/search/jobs" failed CSRF validation -- expected "111aaa222bbb333ccc444ddd", but instead cookie had "111aaa222bbb333ccc444ddd" and header had "555eee666fff777ggg888hhh" host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 5/15/19 9:51:57.071 AM 05-15-2019 15:51:57.071 +0000 ERROR UiAuth - Request from 127.0.0.1 to "/en-US/splunkd/__raw/servicesNS/111111/aaabbbcccddd/search/jobs" failed CSRF validation -- expected "111aaa222bbb333ccc444ddd", but instead cookie had "111aaa222bbb333ccc444ddd" and header had "555eee666fff777ggg888hhh" host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 5/15/19 9:49:49.400 AM Audit:[timestamp=05-15-2019 15:49:49.400, user=111111, action=change_authentication, info=denied ][n/a] host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = audittrail sourcetype = audittrail 5/15/19 9:49:42.259 AM 05-15-2019 15:49:42.259 +0000 ERROR UiAuth - Request from 127.0.0.1 to "/en-US/splunkd/__raw/services/search/jobs/111111__111111__aaabbbcccddd__search3_111111111.145588_0000-000-000-000/control" failed CSRF validation -- expected "999iii000jjj111kkk222lll", but instead cookie had "999iii000jjj111kkk222lll" and header had "333mmm444nnn555ooo666ppp" host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd 5/15/19 9:49:42.252 AM 05-15-2019 15:49:42.252 +0000 ERROR UiAuth - Request from 127.0.0.1 to "/en-US/splunkd/__raw/services/search/jobs/111111__111111__aaabbbcccddd__search9_222222222.145594_0000-000-000-000/control" failed CSRF validation -- expected "999iii000jjj111kkk222lll", but instead cookie had "999iii000jjj111kkk222lll" and header had "333mmm444nnn555ooo666ppp" host = sh-i-111aaa222bbb.aaabbbccc.splunkcloud.com source = /opt/splunk/var/log/splunk/splunkd.log sourcetype = splunkd This cycle of Change Auth -> Fail -> CSRF Validation -> Change Auth will just repeat continually until the user closes all tabs and starts over. Somehow one of the tabs gets a new token and stores it in the cookie without telling his friends. Then, instead of reusing the updated token, the other tabs all try to get their own token and store it in the cookie. Certainly sounds like a defect, but looking for guidance. Thanks. -Tyler ... View more

Currently, we have about 100 applications writing about 50 million events to a logging index/sourcetype per day. It works fine when you are looking for the specific application at specific times, which is the most common scenario. It works okay when you want to get a more a general view of the application over the course of about week. Much beyond that and things start to get slow and annoying. I tried setting up a summary index that would only pull unique combinations of logging events — something like this: index=logging sourcetype=mylogs | sistats count by application, eventcode, eventtext, host The summary ran daily and brought the events from ~50 million to a little over 1 million. Success!! Now the users can run queries like this: index=summary source=loggingSummary application=myApplication | bin _time span=1d | stats sum(count) by _time This worked just fine for the first couple weeks of data but then things got slow again. Also, the summary job that runs takes over an hour and we get warnings in the audit log like this: 11-02-2018 00:33:32.701 +0000 WARN AggregatorMiningProcessor - Too many events (1200K) with the same timestamp: incrementing timestamps 12 second(s) into the future to insure retrievability - data_source="loggingSummary", data_host="mySearchHead", data_sourcetype="stash_new" Also, because the job takes so long to complete, the last time we did a search head restart the job got killed and the data lost. Ultimately the users want to know the following: When was the first time this event happened? When was the most recent time this event happened? What is the total count of these events (daily count is a nice to have)? Keep at least one year of data Be able to compare current data with historical data to find new events that have never happened before Thought about maintaining a running total of the events and the dates in a lookup, but seems like we have way too many rows for a CSV. Also considered a KV store, but haven't tried that yet. Might also need to just give up on putting all the application data in one spot and find a way to divide it up. Could be that 1 million+ rows of summary data per day is just too much. Thanks. ... View more

I'm hoping to get a single summary index query that I can then use to pull data in different ways. I would prefer to roll the data up daily but there are about 150 million events in a day. Normally that wouldn't be an issue but I'm also wanting to group the data by lots of different fields like this: index=cif | fields ApplicationName, DataCenter, Environment, ServerType, host, ErrorCode, MessageText, _time | eval dateOnly = strftime(_time, "%x") | fields dateOnly, ApplicationName, DataCenter, Environment, ServerType, host, ErrorCode, MessageText | fillnull value="" | stats count as messageCount by dateOnly, ApplicationName, DataCenter, Environment, ServerType, host, ErrorCode, MessageText The goal is to count the number of times a particular message occurs. On the backside, when this summary is done the user would select the data back like this: index=summary source=mySource ApplicationName=foo DataCenter=foo Environment=bar ServerType=bar host=* | stats count as by dateOnly On retrieval the user will know the various filter fields which is a much smaller set of data. So if I group by the filter fields when building the summary index then I can use them to filter later. I like that this gets me a single summary index job but the query takes like 2.5 hours to complete. Am I better off running more summary jobs and filtering up front?? Will mean more Summary Index sources and more jobs, which is annoying but maybe necessary?? Thanks. ... View more