So here is my understanding and the way that I've got our on-prem instance configured. hot buckets are stored on a local flash array.  When the bucket closes, it keeps the closed bucket on the flash drive and writes a copy to the S3 storage.  The S3 storage copy is considered to be the 'master copy'.   I try not to use the term 'warm bucket', but instead use 'cached bucket'.  All searches are performed on either hot or cached buckets on the local flash array.  Cached buckets are eligible for eviction from local storage by the cache manager.  So if your search needs a bucket that is not on the local storage, it will evict eligible cached buckets, retrieve the buckets from S3 storage and then perform the search. The frozenTimePeriod defines our overall retention time.  We use hotlist_recency_secs to define when a cached bucket is eligible for eviction.  That is. buckets less than the hotlist_recency_secs age are not eligible for eviction.  Our statistics show that probably 90% of the queries have a time span of 7 days or less (research gosplunk.com for query).  Thus, by setting the hotlist_recency_sec to 14 days, we are ensured that the search buckets are on local, searchable storage w/o having to reach out to the S3 storage (which is slower). One last thing.  We need a 1 yr searchable retention.  However, we also need to keep 30 months total retention.  To accomplish this, I use ingest actions to the S3 storage.  Ingest actions will write the events in compressed json format by year, months, day, and sourcetype.   Hope this helps. ... View more

These visualizations looks great.  However, I'm on version 7.1.1 and I don't see the visualizations.  Is there any special configurations/conditions required to get them to display? ... View more

That script, I believe, moves frozen buckets to an S3 storage location.  However, I what I want is to use the ingest actions  (Settings / Data / Ingest actions) to write all raw events in compressed JSON format.  Ingest actions is per source type and with nearly 100 or more source types, manually creating one for each is onerous.   Ingest actions use RULESET in the props to define how to store on the Remote File System (RFS). ... View more

To clarify.  Splunk +9 provides 'ingest actions' to filter, mask and route data.  This can be done on a heavy forwarder or indexer.  Heavy forwarders would receive their configuration from a deployment server, but clustered indexers receive their configuration from the cluster manager (I guess stand-alone indexers could get their configurations from a deployment server too).  It was that in the class documentation, they only mentioned that 'ingest actions' were deployed through the deployment server and not that they would be deployed by the cluster manager if you had clustered indexers. ... View more

I may get slapped for this answer since Splunk technically says it's deprecated, but you could consider using the LightWeightForwarder.  The LWF disables the parsing & index queues on the HF, but gives one the HEC inputs and Python (for running apps such as eStreamer, dbConnect, etc) as an intermediate relay.   A LWF disables the web interface, but that can be turned back on in a higher precedence web.conf. I use a LWF in the DMZ which gives me all the above, but also provides a deployment server for assets in the DMZ. ... View more

I realize this is a very old post, but as I was browsing I didn't see an answer to your questions.  The best answer is probably to use a LightweightForwarders.  Yes, I know it's supposedly deprecated, but it does pretty much what you want.  That is, it sends cooked, but unparsed/unindexed data to the indexer.  It also gives you all of the functionality such as Python, HEC inputs, etc.  By default, the LightweightForwarder will disable the web interface, but that can be turned though the web.conf setting.  I use this configuration in my DMZ.  ... View more

Correlation rules from sources such as ESCU will have a filter macro.  For example, the rule "ESCU - Access LSASS Memory for Dump Creation - Rule" has a macro named "access_lsass_memory_for_dump_creation_filter".  You should modify the macro rather than the rule itself.  Sometimes rules are updated in the ESCU.  If one modifies the rule directly, the search will be saved in the 'local' folder and have precedence over an updated rule in the default folder. ... View more

Seems like Splunk should incorporate that feature; however, as far as I can tell they haven't. My work around has been to: 1. Run Splunk https on port 8443. 2. Install lighttp light web server that redirects http port 80 to https port 8443. 3. Run firewalld and redirect port 443 to port 8443. ... View more

I'm seeing something similar with my v9 intermediate forwarders.  I have not upgraded my indexer cluster to v9 yet (8.2.5) and was wondering if the forwarders being at a higher version than the indexer was causing the issues.  I'm seeing events from the problem relay servers such as:    Invalid ACK received from indexer=aaa.bbb.ccc.ddd:9997. Got unexpected ACK with eventid=53825 Are you seeing anything such as that?   index=_internal sourcetype=splunkd ack ... View more

I'm running into the same issue, but with Ricoh printers.  But my direction will be what you outlined with sending to SYSLOG, writing to file and forwarder to read that file.   ... View more

Seems like Splunk should incorporate that feature; however, as far as I can tell they haven't.  My work around has been to: 1.  Run Splunk https on port 8443. 2.  Install lighttp light web server that redirects http port 80 to https port 8443. 3.  Run firewalld and redirect port 443 to port 8443. ... View more

The query is something like this: The query uses CoreLight data and excludes local and well known sources. | timechart count as s  span=5ds | fillnull value = 0 | eval time_interval = 0.5 | eval sequence_number = 1 | streamstats current=f sum(sequence_number) as seq | streamstats sum(time_interval) as time | eval time = time - time_interval | head=4096 | table time, s | fit FFT s from time Still working with it ....  Do you have any suggestions, improvements?   ... View more

So here's a sample dashboard. ... View more

So here's a scatter chart plotting the resultant magnitude.  I find a scatter chart a little easier to see the dominant frequencies (those that show stacked columns).  Clearly there is a strong beacon at 1 Hz and even stronger one at 1/2 Hz (every 2 sec).   There are probably others to inspect.  The data was generated looking at DNS traffic from Corelight data.  The data could have come from Splunk Stream just as easily, but we already have a Corelight infrastructure.  The query excludes internal DNS traffic and includes only A, AAAA, TXT DNS records.  Of course there's a lot of other factors such as  DNS caching and rotating ads to consider.  Now on to some addition hunting to find and exclude benign sources and hopefully find nothing!   As an aside, if anyone wants to see an fun use of the Fourier series, lookup "Fourier" and "Homer Simpson" on YouTube and see how Fourier series can draw Homer.   ... View more