1. You can look for the source using metadata command | metadata type=sources or even | metadata type=sources index=your_index Alternatively you can use tstats | tstats count where index IN (some, subset, of, your, indexes) source="your_source" by index 2. The data may not be findable due to a host of possible issues: a) The data is indexed outside of your search timerange due to either data itself or wrong timestamp recognition b) The configuration can be filtering/redirecting events to another index c) The data may be being sent to a non-existent index and you don't have last-resort index defined d) The source might be overwritten on ingestion. ... View more

It seems a bit like an overkill to use Splunk for this if all you send are errors. 😉 But anyway, you should just search for events with continuous scheduling and you're set (just take into account possible delay in indexing). ... View more

As @deepakc already pointed out - you can't find something that isn't there so unless some external source reports those events to Splunk, Splunk doesn't know about it. While you might try to set up some forms of auditing in Windows alone you'll typically end up with either too little information or too much (you can of course even set up procmon to run all the time and try to ingest its output but that's... not very convenient). And that's why you end up paying big bucks for DLP systems (which can have the nice feature of enforcing policy, not just detecting when someone violates it). ... View more

Aren't you overcomplicating it a bit? Just render the date to a field | eval day=strftime(_time,"%F") and you're ready to go | stats min(_time) as earliest max(_time) as latest by day ... View more

It's the other way around - you might need table if you didn't have stats. If you do stats it produces a results table from your summarized events. And yes, join will probably be the way to go. As you seem to have different sets of fields, you simply extract them _before_ doing stats values(interesting_field1) as interesting_field1 values(interesting_field2) as interesting_field2 [...] by common_field ... View more

The question is what actually is in this event. After all Splunk wouldn't escape a newline character. Depending on props for the data it would either break input stream into separate events or break the line into multiline event. It would not get rendered as \n. I think similar thing goes for \x80 and \x9c - they are not (at least in ascii-derivative encodings) control characters but extended characters). So unless I'm missing something, this means that this is not Splunk escaping data (as it does sometimes with control characters) but these are actually raw character sequences received on input. I also don't understand why it should break json parsing. After all these are properly escaped characters so they should be parsed out of the json data... My bet would be that someone didn't rely on automatic json extraction but instead fiddled with regexes to parse fields out of this sourcetype. Edit: No. I forgot one thing. Json specification only allows specifying extended characters by unicode as \uXXXX. The \xXX notation is not allowed. That's why json parsing fails. ... View more

Ok. If you have two searches with a significantly different performance (one is taking much longer to complete than the other or returns way more events) and you deal with subsearches, you want to put the "smaller" one into subsearch. (Subsearches have their limitations and with a smaller search it's kess probable that the subsearch will get silently finalized returning wrong/incomplete results). As to replacing join, the typical way is to use stats. In a similar case to yours (we don't know your events and the search you presented is incorrect) it would be something like <first search conditions> | fields a b c d | rename d as common_field | append [     <second search conditions>     | fields e f g h     | rename g as common_field ] | stats values(*) as * by common_field This is more or less the only way if you need to transform your data in both searches separately (for example have different stats aggregations in them). If you have only streaming commands, you can try to either use multisearch to overcome the limitations of the append command or alternatively search with both sets of conditions and conditionally set/chose/transform fields you need depending on which of result sets they come from. (This is probably the most effective way in some cases but not always applicable and can be quite messy to write) ... View more

It doesn't have to be. This is a dataset for the tutorial examples. Noone said there must be a single Buttercup Games dataset. As far as I remember, different trainings use different datasets. Buttercup Games is just a fictional company so that your data has some "real-world" context. ... View more

No. You can define a output group and load-balance your event between both indexers. They don't have to be cluster members. ... View more

Ok. We have no context. You're writing as if we were supposed to know what you are talking about. You're posting in a Splunk Enterprise section of this forum, which is meant for questions specific to on-premise software functionality and issues. But you selected a specific add-on as a product you're referring to. In such case you should have posted in the 'All Apps and Add-ons' section. We do not have glass orbs and don't know what you mean 😉 ... View more

+1 to what @richgalloway wrote - the official requirements are a bit... imprecise here and noone really knows how to interpret them. From my personal experience, it means that: 1) All cluster members should be running on the same operating systems - 100% Linux cluster or 100% windows cluster 2) All member should run on the same architecture (I don't remember if there are 32-bit versions available anymore but back when they were it might have mattered so you mustn't mix 32-bit and 64-bit; and of course don't try to add to the mix any ARMs if/when they become available) 3) As long as the cluster members are properly set up on each respective OS they should work but it is a good practice to keep things homogenous - it saves you on maintenance and troubleshooting. Also Splunk Support can reject cases if you have mixed environment especially if an issue is present on one OS and not showing on another. ... View more

Honestly? I have no idea what you're talking about. Could you be more specific? ... View more

It's a bit vague what you're trying to do. You can't get two separate result sets from one search. ... View more

First you have to think what exactly components in your dashboard would search for. Then you have to check if you have data to search from. After that comes time to write searches which do search for those things. Last step is making the results from those searches into some visualizations or make them dynamic based on dashboard inputs contents. So where are you in this process? ... View more

That is puzzling. If I understand correctly, you're talking about the host_regex setting of the monitor input, right? The docs don't say that there is any kind of escaping required. If it is however, it would be great if you posted a docs feedback (there is a form at the bottom of https://docs.splunk.com/Documentation/Splunk/latest/Admin/Inputsconf ) describing your situation and how it differs from the described state. ... View more

Your regex seems pretty OK. You could try to simplify it a bit (the character class is not needed if you want just one character, slashes don't need escaping and {1,} can be replaced by +)  so you could do something like this: (?:/[^/]*)+/(\w*) But you can simplify it even further (?:.*)/(\w*) You could take one thing into account though - a valid hostname can contain a dash which is not included in \w. Also, depending on your environment, if it's a FQDN, it can contain dots.   ... View more

1) splunk list monitor and splunk list inputstatus 3) Remember that Splunk searches by _time, which typically is the timestamp extracted from the event. In order to verify how much data you ingest during given period of time you need to aggregate it over _indextime. That's why I was asking about time parsing. You can also check metrics on your HFs but if you have many sources, those UFs might not show in metrics.log at all if they don't fall into the "most active" subset. 4) It's not about the time formats being consistent or not. It's about how they are parsed in Splunk and if they fit the proper time. Otherwise the events might get indexed at a completely different time than you'd expect. And one more thing - even on your syslog receiver, the events can be delayed and if those files are rotated with logrotate (as they probably are) you might be doing a summary of a day's worth of events but those events could be shifted in time versus the "midnight to midnight" period. Did you verify that?   ... View more

Well... there are several things to consider here. 1. Are all files being read properly (check status of inputs on the UF, check for errors, verify that you're not hitting some limits on opened files and so on)? 2. Are other files from the same UF (the typical candidates for cross-checking would be UF's own logs) getting ingested properly? 3. How did you verify the discrepancy between those numbers? 4. Are your time parsing rules properly set up? That can heavily influence _where_ (or rather "when") the events are indexed. So you might be getting the events ingested "properly" but you might just not be seeing them while searching. 5. Do you have any rules (props/transforms, ingest actions) on your HFs/indexers that filter the events (or move them to other indexes). There are many things that could affect your ingestion process. ... View more

A bit late to the party 😉 But seriously - getting the size of the index is one thing but before that we need to define what we mean by that. Index can be measured by many different parameters. 1. Cumulative size of all indexed events (that's what license usage counts as well) 2. Size of raw event files (compressed or not) 3. Cumulative size of everything related to just events (raw data, metadata) 4. Cumulative size of data regarding events as well as summaries created for given index. 5. Any of the above points but expressed not in terms of file sizes but in terms of usage of underlying storage (as in block-aligned or similar). ... View more

This requires more thorough debugging. You need to look into your splunkd.log on indexers and possibly on the CM and look for errors. ... View more

And each of your indexers is in different site? And all your indexers are online and OK? (BTW, this seems a bit overly complicated; unless you're planning on some further expansion of your environment, you could perfectly well get away with a single-site setup). ... View more

It depends on your architecture and your config. How many indexers you have and in what sites are they? What is your siteRF/siteSF configuration? ... View more

As @gcusello already pointed out, the Universal Forwarder by default has a limit on data throughput so if you have too many events coming in, the UF might not keep up with sending them out sufficiently quickly (the same can happen if your network bandwidth is too low). First question though is where the latency appears - look into Forwarded Events log on your WEC machine and verify if those events you see are current or delayed - that's first hint where to troubleshoot it. There are also two different modes of how WEF operates - in push mode the source machines send the events to the WEC machine but in the pull mode the WEC machine actively pulls the events from the source machines with given schedule (I'm not sure if push mode is continuous only or does it work with scheduled periods as well). That's something you should discuss with your Windows admins. (I suppose there can be also different factors possibly causing WEF delays). Another thing that shows when you exceed given performance level is that WinEventLog sources seem to get capped at some point and you can't go over some performance level using single input (even though the machine itself is perfectly capable of handling additional load). In such case the solution is to create additional EventLog channels beside the "normal" Forwarded Events and split the events from subscriptions into multiple channels (and of course ingest them with UF from those channels). But that's a relatively advanced topic (on the Windows side). ... View more