Hi all,   Im trying to understand how rotation certificates used for SSO works in a search head cluster. We have a searchhead cluster where we have SSO working already. As for initial setup, I understand we can download SPmetadata.xml file from splunk SAML settings page. However, during rotation, how do we create this as we are using a cert thats already existing and we want to rotate the server side certificate? If we just download SPmetadata.xml for creating request for IDP, this will have same cert as we are using. If we rotate the cert first at our side so we can download SPmetadata.xml  to create request for IDP, then this will end up in error as IDP wont detect server side certificate during this, obviously.   ... View more

Hi all,   Im analysing event counts for a specific search criteria and I want to know how the count of values changed over time.  Below search is not good enough to see whats going on as many usernames have huge number of events and some with small numbers are barely noticeable (Im interested in rate of change and not count itself) ``` index=test_index "search string" | timechart span=10m count(field1) by username ``` So I want to see a rate of change of the count rather than simple count, by username field. How can we achieve this? ... View more

Hi all, We are seeing a scenario where there are a lot of unoptimised searches, dashboards etc which when run are exhausting our CPU on indexers. If some users run resource intensive adhoc searches/dashboards etc simultaneously, this is becoming a problem as so many searches running together resulting in 'server busy' error at indexer.   1. Is there any way we can throttle CPU/memory usage per user/role/searches? 2. Are there any documents on optimising searches for better performance and less resource usage? ... View more

Hi all, We have been facing some errors with Splunk indexers, where it says something like below. ``` Failed processing http input, token name=<HECtoken>, channel=n/a, source_IP=, reply=9, events_processed=62, http_input_body_size=47326, parsing_err="Server is busy" ``` And I found in some discussions that increasing queue sizes may help sometimes. We are indexing ~400GB per day and it makes sense to increase the queue sizes as default values might not be good enough in this case. However, the splunk docs doesnt have a detailed explanation of which queues can be set in server.conf and what are the proportions that we need consider. Can someone help with understanding this? ... View more

Hi All, Im looking for a way to share a non expiring search with other users. If we use the ''share job" option or just use the URL from address bar - it will get expired once the job expires. But I want to share a link that will not expire. Of course in such a case the search needs to run again from the users end, but the time stamps and search query are the things I want to share with a link. Is there a way to do this? ... View more

We have an index, say 'index1' that has log retention upto 7 days. As the log volume is huge, we dont want to retain all logs there for more than 7 days. However, there is also requirement to retain some logs for later use, say some errors logs that we want to inspect later. So the solution we though of is to use 'collect' and have it in a separate index say 'index2' which has a greater retention, say 6 months. So I planned on using the below way for this   index=index1 level=ERROR | collect index=index2 output_format=hec   *using output_format=hec because we want to use exact same source and source type to have the field extractors working exactly like the original index However there are some questions I have with this. 1. Does this method use license? The doc says below statements which is kind of confusing. a)Allows the source, sourcetype, and host from the original data to be used directly in the summary index. b)No license is counted for the internal stash source type. License is counted when the original source type is used instead of stash in output_mode=hec https://docs.splunk.com/Documentation/Splunk/9.0.2/SearchReference/Collect 2. The document also mentions "This command is considered risky because, if used incorrectly, it can pose a security risk or potentially lose data when it runs" But its not clear how is it risky and what are the things to make sure to avoid problems? 3. It looks like 'collect' command can be used by any user. I tried removing 'run_collect' capability and it also doesnt prevent a role from using collect. How to only allow certain roles to use 'collect' command? 4. The collect command is basically writing to an index. Is there a way to restrict a role from writing data to an index using 'collect' or any other command? ... View more

Hi all, We have a an index (say log_index) where the log retention is only 7 days. We can not have this increased to larger values due to disk space restrictions. Now, we have a  requirement where we would like to retain small parts of the logs in log_index for future reference, like search result for "index=log_index level=ERROR" for a 10 minute window or something. Is it possible to copy a search result to another index which has a longer log retention? I know we could export events, but it would be better to have these in a separate index so everyone will be able to make use of the same splunk log analytics tools on these. Also I dont want to reindex logs since that will again be using up license available. ... View more

We recently upgraded our cluster from splunk 8.1.0.1 to splunk 9.0.2 and the KVstore on SH cluster were manually upgraded to WiredTiger. We could see that cluster manager and peer nodes were automatically upgraded to WiredTiger mostly, however some indexer peers failed in this. Please find the related error messages from the mongodb.log below.    Its not clear why exactly this happened. Is there a manual way to recover and migrate? --------------- 2023-01-12T08:44:16.890Z I STORAGE [initandlisten] exception in initAndListen: Location28662: Cannot start server. Detected data files in /usr/local/akamai/splunk/var/lib/splunk/kvstore/mongo created by the 'mmapv1' storage engine, but the specified storage engine was 'wiredTiger'., terminating 2023-01-12T08:44:16.890Z I REPL [initandlisten] Stepping down the ReplicationCoordinator for shutdown, waitTime: 10000ms 2023-01-12T08:44:16.890Z I NETWORK [initandlisten] shutdown: going to close listening sockets... 2023-01-12T08:44:16.890Z I NETWORK [initandlisten] Shutting down the global connection pool 2023-01-12T08:44:16.890Z I - [initandlisten] Killing all operations for shutdown 2023-01-12T08:44:16.890Z I NETWORK [initandlisten] Shutting down the ReplicaSetMonitor 2023-01-12T08:44:16.890Z I CONTROL [initandlisten] Shutting down free monitoring 2023-01-12T08:44:16.890Z I FTDC [initandlisten] Shutting down full-time data capture 2023-01-12T08:44:16.890Z I STORAGE [initandlisten] Shutting down the HealthLog 2023-01-12T08:44:16.890Z I - [initandlisten] Dropping the scope cache for shutdown 2023-01-12T08:44:16.890Z I CONTROL [initandlisten] now exiting 2023-01-12T08:44:16.890Z I CONTROL [initandlisten] shutting down with code:100 --------------- Cluster details – splunk multisite ------------------------- 4 SH (site1) 4 SH (site2) 11 IDX (site1) 11 IDX (site2) Master1(Site1) Master2(Standby at site2) ... View more

Can anyone help with understanding the latest openssl vulnerability CVE-2022-0778 and how/if it affects Splunk installations? https://www.cve.org/CVERecord?id=CVE-2022-0778 ... View more

Hi all, I'm testing multisite indexer clustering with below configuration and found an undesired behaviour in the case of a site failure. available_sites = site1,site2 site_replication_factor = origin:2,site1:2,site2:2,total:4 site_search_factor = origin:1,site1:1,site2:1,total:2 As you can see I have configured the replication factor "origin:2,site1:2,site2:2,total:4" so that I will have 2 replicas in both sites. But, in the case of a site failure, I am observing that splunk will try to replicate locally in the site that is up and complete the 'total:4' condition. I think this can be a problem when the available disk space on the machines is less. Let's say site2 indexer machines are at 80% disk space usage and site1 fails - now when splunk tries to create 4 replicas in the same site (site2) due to site failure, it can easily exhaust the disks. As per update from splunk support, this is default behaviour, but I feel there needs to be additional control over this. Any advise or suggestions around this issue will be really helpful. Thank you. ... View more

Hi,   We are trying to move from single site to multisite splunk cluster. Although , its not clear how the SH clustering is supposed to work.   1. As per documentation, the recommended way is to have two separate SH clusters - But it doesn't look like we will have knowledge bundle (configs/user knowledge objects etc) replication between the two SH clusters formed. If this is the case then I don't get the point of suggesting multisite as a DR solution. When site1 fails, users connecting to site 2 wont have their knowledge objects and settings on the new SH cluster!? https://docs.splunk.com/Documentation/Splunk/8.2.0/Indexer/Multisitearchitecture   2. The other thing thats suggested  to have knowledge bundle and search artifact replication is to have a SH cluster spanning both the sites - BUT this also cant be suggested as a DR solution since in this case whenever the site with majority(or with same number) of SHs fail completely, the SH machines at the other site wont be able to form a cluster since they wont have majority. A work around is suggested here to deploy a static caption instead. https://docs.splunk.com/Documentation/Splunk/8.2.0/DistSearch/DeploymultisiteSHC ... View more

We are trying to setup a new cluster and move from Splunk single site to multisite. Could someone help with all the ports that needs to be allowed between the two sites for this to function properly? ... View more

Searches return errors like below. The indexers IPs returned seems to be changing on different attempts of the same search. The searches were run over long time periods like 15-20 days on an index with relatively large number of events.   ---- 3 errors occurred while the search was executing. Therefore, search results might be incomplete. Hide errors. Unknown error for indexer: <INDEXER_IP1>. Search Results might be incomplete! If this occurs frequently, check on the peer. Unknown error for indexer: <INDEXER_IP2>. Search Results might be incomplete! If this occurs frequently, check on the peer. Server error ----   Inspecting the job says   ---- warn : Socket error during transaction. Socket error: Success error : Unknown error for indexer: <INDEXER_IP1>. Search Results might be incomplete! If this occurs frequently, check on the peer. error : Unknown error for indexer: <INDEXER_IP2>. Search Results might be incomplete! If this occurs frequently, check on the peer. ----   Related entries found from search.log for one of the indexer IPs   ---- 07-29-2020 05:46:53.900 INFO TcpOutbound - Received unexpected socket close condition with unprocessed data in RX buffer. Processing remaining bytes=5792 of data in RX buffer. socket_status="Connection closed by peer" paused=1 07-29-2020 05:47:00.543 ERROR HttpClientRequest - HTTP client error=Success while accessing server=http://<INDEXER_IP1>:8089 for request=http://<INDEXER_IP1>:8089/services/streams/search?sh_sid=1596001558.314297_64CB0758-30F3-4D5E-9CC0-DA1DD06754ED. 07-29-2020 05:47:09.734 WARN SearchResultParserExecutor - Socket error during transaction. Socket error: Success for collector=<INDEXER_IP1> ----   From some other discussion, I saw it maybe related to ulimit value. However Im not seeing any ulimit/thread/socket errors in splunkd.log.   ulimit -n value is 1024 (which I believe is a softlimit)on the indexers, but the splunk uses 100K as per the startup log and    ---- splunkd.log.3:07-08-2020 10:19:48.050 +0000 INFO ulimit - Limit: open files: 100000 files ---- ... View more