Have you tried using the Embedded Reports features of Splunk? Embed scheduled reports - Splunk Documentation ... View more

Based on your description it sounds like you're wanting something pretty custom for your environment.  There's not quite this type of data-splitting, alerting, and re-display framework in Splunk.   First suggestion: check out Splunkbase for any add-on's having to do with alerting.  Maybe one of these has a good-enough implementation for what you need.  Here's the results of the keyword "alert" for all the apps out there: https://splunkbase.splunk.com/apps?keyword=alert   From what I understand of your description, a big part of what you want is a meaningful display of information to the person handling the alert.  One way to solve is to create a Splunk dashboard that expects inputs via the URL - just like the ?keyword=alert in the above URL.  When you include values like that in the URL for a dashboard you can access those as tokens (within your SimpleXML, for example).   A lot of times there is the SPL that goes into triggering an alert, and those results have a lot of "plumbing" data in them that helped trigger the alert.  So that result set isn't very actionable by the responder.  This is why you would create a custom dashboard that expects token inputs (like a timeframe and hostname), and then it renders visualizations for that host in that timeframe that helps them troubleshoot in response to the email. If you haven't already, install the Splunk Dashboard Examples app - it has a lot of good tips and tricks for creating dashboards. ... View more

Yes, you can do this using the Splunk HTTP Event Collector.   Here's a blog post showing an example of Splunk receiving data from a Zoom webhook: Splunking Webhooks with the HTTP Event Collector | Splunk ... View more

Sounds like you need this app from Splunkbase: Treeview Viz | Splunkbase ... View more

Does this page help at all?  There's some gotchas with this configuration depending on how the account was setup for the Slack integration: Slack Integration Guide | VictorOps Knowledge Base ... View more

If you're looking to reset the Splunk UF's user password, you can follow the instructions under the "Reset a lost administrator password" section on this page: Create secure administrator credentials - Splunk Documentation   ... View more

That error implies that your OpenAPI Org Id was not configured properly during the setup of the ChatGPT 4 Splunk app.  This app expects to find this info within Splunk's built-in password storage (where it was stored during the setup of this TA) Here's the python code behind the search command: ChatGPT-4-Splunk/TA-openai-api/bin/openai.py at main · bentleymi/ChatGPT-4-Splunk · GitHub   Also, @jkat54 is the author of that particular app.  Just tagging them here if they have any additional suggestions.  See also their session at .conf23. ... View more

Let's assume your sourcetype is called WindowsEventSourcetype, then you will want to add some lines to that sourcetype's definition in props.conf and transforms.conf:   props.conf [WindowsEventSourcetype] TRANSFORMS-t1=eliminate-4624-4634-3 transforms.conf [eliminate-4624-4634-3] REGEX=(?m)EventCode\s*=\s*(4624|4634).*?Type\s*=\s*3\s DEST_KEY=queue FORMAT=nullQueue  A couple things to note: These configurations need to be deployed to where your data is "cooked" by Splunk, not searched or the UF.  So this means these should be going to Heavy Forwarder(s) and Indexer(s) in your environment that would be ingesting this windows event log data. I might be slightly off on the regular expression - I can't recall the exact format of the logs.  If you could post a couple samples I could tighten this up.  Right now the regex is doing something like, "Use multiline mode, look for EventCodes 4624 or 4634, then some more stuff, then Type 3" - I don't recall how the Login_type is labeled within these particular events from Windows. ... View more

  This is a good resource page:  Splunk Cloud Platform Service Details - Splunk Documentation - Is there a message informing the license is about to expire? You can view information about your license entitlements.  Here's the docs that explain that for Splunk Cloud. - After the expiration date, is there any grace period provided? Assume no, but open a support ticket and work with your account manager on this situation. - In case I decide to not renew the license, are we able somehow to download the company data before its total removal? Or after my license is expired I lose all indexed data? Per the page under Data Handling, retrieving your data is suggested to be a Splunk Professional Services engagement, which often can mean "this is kinda hard if you're new to Splunk."   If you require your ingested data to be moved into your control before the termination of your subscription, this is accomplished through a Splunk Professional Services engagement. Some data can be moved into your control by enabling Dynamic Data Self-Storage to export your aged data to your Amazon S3 or Google Cloud Storage account in the same region There are data egress notes on that page, too, and this page details getting your data out of your Splunk Cloud envirnment using Dynamic Data Self Storage (DDSS).  You could technically "age out" your data to your own S3 buckets, for example. This also could be a good discussion for the #splunk_cloud channel on the splunk-usergroups.slack.com workspace. ... View more

That particular error is referring to a hiccup during the execution of the search with the search peers - aka the Splunk Indexers - that were involved in your query.   Here is a diagram of a simple Splunk enterprise deployment - you were initiating your query on a Search Head, and that query is shared out to the Indexer(s) that make up your deployment.  It sounds like one of the Indexers had an issue: To find more info on what your issue is, you can open your search results, and click on the Inspect Job option under the Job menu on the result page:     And then open the search.log from the Job Inspector page that pops up:     Within that log file that opens you can find more details on what happened during your search to trigger the warning message "Search results might be incomplete" that you saw. If you have more info on any of the error/warning logs in that file we could help you figure out what could be causing your issue.         ... View more

You should have complete results based on your description and a couple of assumptions:   I am assuming that you are connecting to a remote Splunk instance (probably a Search Head (SH) - aka the Splunk website), and you are not running Splunk locally on your computer.  In that case, you would have complete results.  Your computer does not have to stay connected to a Splunk SH for a background search to complete properly. ... View more

Assuming that you configured your HEC port to be 8088, that URL looks correct to me.  Also, that endpoint expects JSON data - is that what Blue Prism is sending? There are several REST endpoints you can post to for the Splunk HEC documented here. Also, here is a page with some curl commands you can use to test out the HEC endpoint locally to rule out where the issue is - that is, if it works with curl, then is there a network issue in between (e.g. firewall). HTTP Event Collector examples - Splunk Documentation       https://mysplunkserver.example.com:8088/services/collector   ... View more

Did you see any change in the data being ingested when you made the TRUNCATE value change?  Also, if you change it to something specific to test it, like 10237.  Does that limit it to 10237 bytes?  This is mainly just to see if this particular TRUNCATE setting is what is limiting your data, and maybe we can help rule it out as the culprit so we know to dig further. ... View more

Based on your description it sounds like you are configuring the Blue Prism side of things to point at the HTTP Event Collector (HEC).  To get the URL and token, you'll need to make sure the HEC configuration has been completed on the Splunk side so Blue Prism has a place to send its data. Here's the documentation that pertains to configuration of the HEC component in Splunk: https://docs.splunk.com/Documentation/Splunk/latest/Data/UsetheHTTPEventCollector  ... View more

IIRC Splunk Stream doesn't have truncation settings and this ends up being caught by the truncation settings for your sourcetype within props.conf.  Can you share what your stanza is for your sourcetype?  Is TRUNCATE=1000000?  You might need to change to TRUNCATE=0 to force Splunk to include all of the event. ... View more

What your subsearch is going to do is create a table of values for POID, IP, IPRISKSCORE, RISKTYPE, EVIDENCEDETAILS and add those to the criteria of your outer search.  It's not adding data to your results stream...rather, just adding criteria (and in your case a bunch of additional AND'ed criteria on your outer search clause).  At first glance the way you structured your subsearch doesn't look like it is doing what you expect it to do since a subsearch in this situation is used to generate dynamic criteria for your search (see more here). If this overall search is returning the data you want, and just not the fields you already extracted with rex, then you're on the right track...you just need to get those fields generated again.  In that case, do that rex command again before your timechart command. Also consider if these extractions should be put in a props.conf configuration if you always want them occuring and don't want to use the rex command everywhere.  Don't worry too much - regex extractions like this fail pretty fast and are rather performat if they come across a non-matching event since you're anchoring your match with such a distinct string of "Detected Fraud IP for the User with POID:" versus starting your regex with a greedy term like ".*" ... View more

I don't know off the top of my head if you can get to the PDF generation functionality from within Python - I've never tried in python and I haven't been able to find any documentation within Splunk's documentation if they officially expose those calls in their Python SDK. The PDF stuff in Splunk is based off of ReportLab, and lives in %SPLUNK_HOME%\Python-3.7\Lib\site-packages\reportlab, so you can poke aroundthere a bit to see if that helps. I wasn't finding anything obvious of Splunk-written python calling the ReportLab stuff to generate a PDF, your luck might be better. You can get a PDF generated from a REST endpoint - but do keep in mind that the overall PDF generation in Splunk is dashboard-centric, not search-centric.  So you will need to create a Dashboard that renders the results for your SID, and then have that be generated as a PDF.  Here are a few posts of people generating the PDF via a REST call which you can do from the CLI, within Python, etc.  Once you have those bytes you can send those on to where you need: Solved: Re: Generate PDF from View in REST API - Splunk Community Trigger a PDF via the command line - Splunk Community Historically the built-in PDF generation of Splunk has left something to be desired.  It's just been OK...There are some apps on Splunkbase that help with various PDF things you could look into as well: PDF Apps: https://splunkbase.splunk.com/apps?keyword=pdf      ... View more

Does this post help? Solved: How to format X axis label in the timechart? - Splunk Community ... View more

What do you mean by support? For example, do you want to write Scripted Inputs in Coretto?   ... View more

Assuming things are configured "correct" for what you expect a Splunk Cluster to be, I would assume you're fine for right now. For example, the Cluster Master probably has eyes on that missing indexer and is waiting for it to come up, and has been communicating to all of your Splunk deployment the necessary info like "Hey UF, don't send your data to this Indexer" and "Hey SH, this indexer is no longer a search peer." Can you describe how you have your cluster configured - replication/search factors?  How many indexers do you have (e.g. is this one of two)?  That could give us a hint on what you should expect when things do come back up.  For example, if that indexer comes up and says hello to your Cluster Master, then that CM is going to start doing any replication/balancing of buckets.  That's going to suck a chunk of your network pipe that you might not want when this data center comes up, so if that indexer is lower priority you leave Splunk shut down until your higher priority stuff in that data center is recovered. ... View more

When you say, "upload the pdf of  search results based on the alert to the ticket in an automated way" are you wanting to take the PDF that the code creates (the file /etc/apps/Splunk_Ivanti/local/ticket.pdf) and post it to some other endpoint?  If the answer is yes, then you'd need to write the python to do that send.  If you want an example how this is done, take a look at the %SPLUNK_HOME%/etc/apps/alert_webhook/bin/webhook.py.  That's the code behind the Webhook Alert Action and it does a fairly simple send of data to a URL.  ... View more

Can you describe your deployment a bit - # search heads, # indexers, are they on VMs or bare metal, maybe share a screenshot where you are seeing this high IOWait vs CPU (e.g. are you seeing this in a Splunk dashboard or from OS tools). ... View more

What type of Splunk instance are you getting this notification on - your Indexer, or are you using a single Splunk instance? I'm also assuming this notification is coming from outside of Splunk - like an OS monitor or something like that. For example, an Indexer is generally going to put lots of stuff into $SPLUNK_HOME/var/lib/splunk because that's where the indexes are stored.   The first thing I could think of is your indexer drive is almost full, and you index more data causing you to dip below a Splunk-configured max index size.  This causes Splunk to then move/delete old buckets of data because it dipped below the configured threshold.   Here's docs on this this concept:  Configure maximum index size - Splunk Documentation Also, these configs would be in one of your indexes.conf files. ... View more

When doing a high-availability with multiple cluster managers (CM), you should have a load balancer (LB) in front of them.  The manager_ui configuration then points to the LB - not an individual CM.  The CM's will keep their bundles in sync, and the LB will help ensure that your indexers always reach a "good" one that is up.   Here's the docs that explain how to put an LB in front of the CMs in this scenario. ... View more

I don't have your answer...but it might be helpful to cross-post your question here: Alert Manager Enterprise - Splunk Community   That is the "place" where questions about the Alert Manager Enterprise app on Splunkbase would go now, but I don't think there is any way to link this post with app right now. Also, the folks at Datapunctum AG might have their eyes on that area for there app, and not here, for answering any questions.   I'm going to tag one person I know at Datapunctum that I think worked on this app:  @my2ndhead  ... View more