Hi Splunkers,  we have a Windows log source with a UF installed on it. We have no access to this log source: we only know that we collect Windows logs via UF and it works properly. Collected logs are the usual one: Security, Applications, and so on. Starting from today, we need to add a monitor input: some files are stored in a folder and we need to collect them. So, on our DS, we created another app, inside deployment-app folder, with a proper inputs.conf and props.conf and then we deployed it. Why we created another app and does not simply added a monitor stanza in inputs.conf for Windows addon? Simply because Windows addon is deployed on many host; on the other side, we need to monitor the path only on 1 specific host, so we preferred to deploy another dedicated app, with its server class and so on. DS give no error; app is shown as deployed with no issues. At the same time, we got no error looking on splunkd.log and/or _internal index. By the way, logs are not collected. For sure, we are going to reach Host owner and perform basic checks, like: Is provided path the right one? User in charge of execute UF has read permission on that folder? In UF app folder, is the one deployed by us viewable?  But before this, there is a doubt I have: above point 2, in case of permission denied, I should see in _internal logs some error message, right? Because currently I don't see any error message related to this issue. The behavior is like the inputs.conf we set in deployment app is totally ignored: searching on _internl and/or splunkd.log, I cannot see anything related to path we have to monitor. ... View more

Hi Splunkers, I have a doubt about underscores and path in props.conf. Suppose, in my props.conf, I have: [source::/aaa/bbb/ccc_ddd] As you can see, in my path I have an underscore in path name. This could be a problem? I mean: can I put the underscore without problem or I have to use backslash to escape it? ... View more

Hi Splunkers, I have a problem with a Per-Event Index Routing use case. In involved environment, there are some data currently collected in a index named ot. Here we have some logs that must be splitted and redirect to other indexes, with naming convention ot_<tecnology>. Inputs.conf involved file is placed under a dedicated app, named simply customer_inputs. The procedure to use is very clear for us: we created, inside above app, props.conf and transforms.conf and worked with key and regex. The strange behavior is this: if we work to redirect one kind of logs, it works perfectly. When we add another log subset, nothing works properly. Let me share you an example.  Scenario 1 In this case, we want: Windows logs must go on ot_windows index. All remaining logs still must go to ot index. We can identify involved logs based on ports; they are coming as network input on port 514 udp, with CEF format. First, our props.conf   [source::udp:514] TRANSFORMS-ot_windows = windows_logs   Second, our transofrms.conf   [windows_logs] SOURCE_KEY = _raw REGEX = <our_regex> DEST_KEY = _MetaData:Index FORMAT = ot_windows   This configuration works fine: Windows logs goes in ot_windows index, all remaining ones still go on ot index. Then, we try another configuration, explained on second scenario. Scenario 2 In this case, we want: Nozomi logs must go on ot_nozomi index. All remaining logs still must go to ot index. Again, we can identify involved logs based on ports; they are coming as network input on port 514 udp, with CEF format. First, our props.conf   [source::udp:514] TRANSFORMS-ot_nozomi = nozomi_logs   Second, out transforms.conf   [nozomi_logs] SOURCE_KEY = _raw REGEX = <our_second_regex> DEST_KEY = _MetaData:Index FORMAT = ot_nozomi   Again, this conf works fine: all Nozomi logs go on dedicated index, ot_nozomi, while all remaining one still go on ot index.  ISSUE So, if we set one of above conf, we got expected behavior. By the way, when we try to merge above confs, nothing works: logs, both Windows and Nozomi, continue to go on ot index. Due they work fine when they are "single", we suspect error is not on regex used, but on how we perform merge. Currently, our merged conf files looks like this: props.conf   [source::udp:514] TRANSFORMS-ot_windows = windows_logs TRANSFORMS-ot_nozomi = nozomi_logs   transforms.conf   [windows_logs] SOURCE_KEY = _raw REGEX = <our_regex> DEST_KEY = _MetaData:Index FORMAT = ot_windows [nozomi_logs] SOURCE_KEY = _raw REGEX = <our_second_regex> DEST_KEY = _MetaData:Index FORMAT = ot_nozomi   Is our assumption right? If yes, what is the correct merge structure? ... View more

Hi Splunkers, I have a strange behavior with a Splunk Enteprise Security SH. In target Environment, we have a Indexer's Cluster queried by 2 SH: a Core one and a Enteprise Security one. For a particular index, If we perform a search on ES SH, we cannot see data. I mean, even if we perform the simplest query possible, which is: index=<index_name>   we go no result. Perhaps, if I try the same search on Core SH, data are shown. The behavior in my mind is very strange because it happened only with this specific index; all other remaining indexes return the same identical data, both  performing query on ES SH and Core SH. So in a nuthshell we can say: Index that return result on SH Core: N Index tha return result on ES Core: N - 1   ... View more

Hi Splunkers, today I have a problem about understanding how and where Log Sources sends logs to Splunk. In this particular Splunk On Prem environments, no documentation has been done, except the HLD. So, we have to understand, for each log source, what Splunk component it reaches and how. For example, if I have a Domain Controller, we must establish: Where it sends logs? Directly to Indexers? To a HF? A UF is installed on it? If not, how it send logs? WMI? WEF? Other And so on. Now, List of servers sending logs to Heavy forwarder is a community discussion where I started from @scelikok suggested search, changed it in:     index=_internal component=TcpOutputProc | stats count values(host) as host by idx | fields - count     and it helped me a lot: I'm able, for each Splunk Component of env (IDS, HF and so on) to understand what Log sources send them data. So, what's the problem? The above search return data forwarded by another Splunk component. I mean, in the output, field idx has always format ip/hostname:9997, so it means that data are coming from a server with UF or from another Splunk host (we have some intermediate forwarder, so sometimes I can see data ingested by an HF coming from another HF). What about data sent not with a Splunk agent/host? For example, suppose I have this flow: Log source with Syslog -> Splunk HF receive on port 514 With above search, I cannot see those sources (and I know for sure they exist on our env). How can I recover it? The syslog is only an example, the key point here is: I must complete my search with all log sources that do not use UF and/or any other Splunk element, but other forwarding tool/protocol (syslog, API, WEF, and so on). ... View more

Hi Splunkers, I have to calculate daily ingested volume in a Splunk Enteprise environment. Here on community I found a lot of post, and related answer, to a similar question: daily license consumption, but I don't know if it is what I need. I mean: we know that, once data are ingested by Splunk, compression factor is applied and, in a non clustered environment, it is more or less 50%. So, for example, if I have 100 GB data ingested by day, final size on disk will be 50 GB . Well, I have to calculate total GB BEFORE compression is applied. So, in my above example, search/method I need should NOT return 50 GB as final result, but 100 GB. Moreover, in my current env, I have an Indexers cluster.  So, what is not clear is: daily consumed License, is what I need? I mean: when I see daily consumed license by my environment, GB returned are the ingested one BEFORE compression, or the Compressed one?   ... View more

Hi Splunkers, today I have a very strange case to manage. I'm going to try right now to be more clear possible. The scenario is a full on prem Splunk Enterprise environment, with many components. For this customer, we are not the starting provider; another company was on charge before us and developed a full custom app. About this application: No doc has been shared by previous provider It states now some error messages that are not completely clear. So, in a nutshell, we have to try to understand why we got those errors and try to fix them. Now of course I'm not here to ask you "Ehy magic guys, give me the magic solution!"; the purpose of this topic is ask your help to understand data we have (we have only a GUI little dashboard with a short app description and how it works) and try to understand how we can fix those errors. The app analyze Indexers and their indexes. Its purpose is to understand if indexes are retaining the correct amount of historical data; do achieve this, it investigate the index retention status. So, how this investigation is done? The app analyze the currentTimePeriodDay value against the frozenTimePeriodDay. To state if an error is found, the app consider 2 possible cases: currentTimePeriodDay > frozenTimePeriodDay + 45: this case is considered unhealthy because indexes are retaining more historical data than expected currentTimePeriodDay < frozenTimePeriodDay:  this case is considered unhealthy because indexes are retaining insufficient historical data. For both cases, the suggested workaround is a generic retention and disk space settings tuning. Of course there are more specific error message for each index on every Indexers (we have a menu to select specific Indexers) but this, by my point of view, is a further analysis step; what is not clear, for my team and me, is the foundation logic of app. I mean: how comparison between currentTimePeriodDay and frozenTimePeriodDay should help us to check a good index retention? How are they related? Why if one of them is greater than the other one, this could be an unhealthy symptom?  ... View more

Hi Splunkers, I have a problem with timestamp on our platform. Here some assumption and acquired knowledge. Knowledge _time =  is the event time (the time which is present in the event. In other words: the time when the event was generated. _indextime = is the index time or, if you prefer, the time when the events have been indexed. Issue with timezone shown can be related to user settings, that can be changed under username -> Preferences -> Timezone. Environment: a Splunk Cloud SaaS platform with logs ingested in different ways: Forwarder (both UF and HF) API Syslog File monitoring Issue: If I expand the event and I examinate the _time field:  Why, in my case, time event and time shown are different? Important additional Info Our user settings timezone are set on GMT+1 (due we are in Italy) for all users. You see a Windows events as sample, but the problem is present on all logs: it doesn't matter what log source I consider and how it is sending events to Splunk. Every log show time difference. The difference between _time and time shown is always on 1 hour, for every events on every log sources. I searched here on community and I found other topics about this issue, some of them has been very useful to gain a basic knowledge like Difference Between Event Time and _time  but, due we are on cloud (with limited chance to set some file and parameter that are involved) and the issue is for all events, I'm still locked on this problem.    ... View more

Hi Splunkers, I have a doubt about setting for Splunk Enterprise Security. As usual when I put a question here, let me share a minimal of context and assumption. Environment: A completely on prem Splunk Enterprise (no Slunk Cloud SaaS). Currently, only one SH Clustered indexers Task:  Install and configure a SH with Splunk Enterprise Security. Assumption: I know the full installation procedure (doc + Splunk Enterprise Admin course) I know how to manage a cluster environment (doc + Architect course). For example, I know that if I have to set a Splunk instance as SH I can use, from CLI: > splunk edit cluster-config -mode searchhead -manager_uri https://<manager node address> -secret <cluster secret>   Questions: This syntax is still valid to add a SH with ES installed on it? The doubt is if the ES presence should lead me to use a different approach to tell "Hey, SH wth ES: indexers to query are those". SH with ES component should be  add as single SH (so, decoupled from already existing SH) or should I create a SH Cluster with normal SH + ES ES? ... View more