Here are a few top of mind questions answered during the live Tech Talk Q. What is the best approach to select 'key' fields from a data source? This is where performing a Baseline hunt may help. The outcome of that hunt should be a data dictionary of data fields available in a given data source along with their values. By looking at the values, you can determine which have the most relevance to the target/technique/payload you are hunting - these become the "key" fields.    Q. Do you convert successful hunts to correlation searches at least until something is mitigated? What does that process look like?  If it's an immediate concern or poses an imminent risk, we will escalate and have it mitigated as soon as possible. If it can be turned into a correlation search with high fidelity/low noise, we will have a detection created for. Otherwise, it may be used as a candidate for an L1 hunt because it is too noisy for a detection.   Q. During the Act phase of the framework, what type of factors do you consider when creating notables/alerts vs. recurring searches/dashboard panels for future hunts? One of the factors we consider when creating notables/alerts vs. recurring hunts is how easy it is to distinguish anomalous non-malicious from anomalous malicious events. If it requires a human to determine if something is malicious, say an unexpected scheduled backup, or unexpected deletion of software, we want a human to determine if the activity should be happening. Another factor is the significance of the asset. If the asset in question has significant importance to Splunk, we may want an analyst to review events related to that asset to confirm they are expected. Our goal when creating a notable/alert is to minimize false positives but if there is a risk that a true negative that could have a large impact to our security could slip through we want to have a human look at the events to make sure vs. depending on an alert to fire.   Q. When you talk about sharing your results, who is the intended audience of these reports?  It will depend. Typically, we share with other security operational teams and leadership. We may also share with relevant stakeholders like tool owners.   Q. How much time should be dedicated to threat hunting per week as a SOC analyst?   Organizations should prioritize allocating sufficient time for threat hunting, balancing it with other team priorities. We've observed that analysts require a minimum uninterrupted window of 4 hours, ideally a full day, to focus solely on hunting. Context switching between reactive ticket handling and proactive hunting proves challenging and may impede success. Establishing dedicated hunting days, where analysts are free from interruptions, is optimal. Additionally, ensuring adequate shift overlaps can help mitigate the backlog of tickets, allowing analysts to focus on hunting without concern for pending tasks.   Q. Does PEAK help to prioritize which topic I should hunt?   The PEAK framework doesn't address how to prioritize hunt topics. Instead, we prioritize them by considering factors such as the environment's importance to the business, our current detection coverage, threat intelligence team recommendations regarding recency or known adversaries, and any recent incidents related to the technique.   Q. How do you avoid going down a rabbit hole?  Crafting a well-defined hypothesis that outlines the target, technique, and expected payload, while also completing the ABLE table categories of Actor, Behavior, Location, and Evidence, is crucial for maintaining focus during a hunt. Straying too far from the hunt's objective is a common pitfall, and having a clear hypothesis helps prevent such diversions. In threat hunting, you'll encounter rabbit holes, and experience helps decide when to stop pursuing unproductive leads.   Q. For any new environment how do you prioritize what to hunt/hypothesis?  For a new environment, it may be helpful to start with a Baseline hunt to understand the data and would be valuable to hunt. Here is a blog post with additional information. https://www.splunk.com/en_us/blog/security/peak-baseline-hunting.html   Q. What learning sources are there to follow for effective hunts? There is a book with a wealth of Threat Hunting goodness called Bluenomicon. You can download it for free here: https://www.splunk.com/en_us/form/the-network-defenders-compendium.html   Q. The acquisition of Splunk by Cisco has the potential to enhance Splunk, making it a better tool?  We're enthusiastic about the possibilities stemming from our acquisition by Cisco. Currently, we're evaluating how to effectively utilize our combined expertise, tools, technologies, and data.   Q. How much time did it take M-ATH to process the data and output results?   For this one SPL example, 3-5 minutes. This was done many times for this hunt. It's always going to depend on your SPL scope.   Q. What skills are good to focus on for self-development that would help with getting into Threat Hunting (ie. SPL, general OS knowledge, etc.)? What are "must-have" or most critical skills/knowledge to have as a Threat Hunter? Curiosity and a desire to understand why things are the way they are. If someone says "that's just how it is", and your response is "yeah, but why?", that's a great quality for a threat hunter. Some of the best findings come from questioning something that seems benign, but is just slightly different. Also, the innate ability to see patterns in things. Embracing learning about things you don't know about and enjoying learning new things. Understanding the foundation of how things work, e.g., the OSI model, networking, DNS, common protocols, cloud infrastructure, having a strong foundational knowledge related to technology and common security attacks helps when you need to pick a topic and dig deep to learn more about it. Being able to collaborate with other teams who may be experts in a topic you are trying to hunt is also important.   Q. Do you use custom data models to enable threat hunting in Splunk? Is there a core data set you find "necessary" for threat hunting? We do have some custom data models and customized out of the box data models, but we are not using them as primary sources in our hunting. In terms of a core data set, it depends on the topic of the hunt. For example, cloud logs are necessary for hunting in the cloud, endpoint logs are necessary for digging into endpoing activity, which may also result in pivoting into network logs depending on the goal.   Q. What is the measure of success for the overall hunt program? The goal is to make the organization more secure. While we document metrics in terms of hunts completed, findings, knowledge shared, etc. we are continually trying to determine impact. There is a great PEAK blog post on this where you can read more about the topic here: https://www.splunk.com/en_us/blog/security/peak-threat-hunting-metrics.html   Q. What is the overlap with Red Teams/Pentest teams? How do you collaborate? The Red Teams colloborate with us when we have hunt topics that would benefit from attack-related events that we do not have in our environment. They will emulate an attack so that we can see what it looks like in the logs to better determine how to hunt. We also work closely with them on Purple Team engagements where we have multiple red and blue teams working together to simulate a threat actor.   Q. How are you grouping hunts with similar scope/TTP? What is the criteria/process? We follow a hunt intake process to assess anything new, grading them based on criteria such as relevance to our internal environment and whether they're related to incidents. Additionally, we sometimes organize group hunts involving multiple participants that share a similar scope.   Q. It seams like there is a lot of Security/Splunk engineering going on that you do not need to deal with prior to and after hunting. Do you have any advice for getting logs parsed and CIM compliant if you are the hunter and engineer? Log directly into tools that would send data to Splunk and do your hunting in the tool first to help prioritize and inform what should be onboarded to Splunk. In terms of parsing and CIM compliance, our professional services team can help with that. From a hunt perspective, hunting without CIM compliance is not a showstopper and though it is not optimal, you can also fix poorly parsed fields in SPL with Rex and Regex, or through our built-in data parser.   Q. Was the use of the PEAK threat hunting framework for managed service providers considered, or is it very much applied to internal use for an organization? The PEAK threat hunting framework can definitely be used by managed service providers, in fact, we encourage it. The framework is applicable to any organization seeking to create, perform, or manage threat hunting.   Q. Was the use of the PEAK threat hunting framework for managed service providers considered, or is it very much applied to internal use for an organization? The PEAK threat hunting framework can definitely be used by managed service providers, in fact, we encourage it. The framework is applicable to any organization seeking to create, perform, or manage threat hunting.   Q. Do you have any example threat reports?  We don't currently publish public reports as we are internally facing. We may in the future. ... View more

Here are a few questions asked during the live Tech Talk   Q. What else should I be doing to prepare for the Python upgrade? A.The Splunk Docs page https://docs.splunk.com/Documentation/Splunk/latest/Python3Migration/AboutMigration is the latest and greatest resource for learning more about how to upgrade. We also recently released Splunk Python SDK 2.0.0 with added support for 3.9. That resource is here: https://dev.splunk.com/enterprise/docs/devtools/python/sdk-python/   Q. What would you recommend as a cadence of upgrade? We are tight on resources and need to plan in advance. A. I think it is worth noting that every customer will have different constraints and challenges, so there isn't a single right answer to this question. However, the general guidance here would be to think about a few dimensions: (1) It's important to stay on a supported Splunk version, and as the presentation highlighted, there is a 2 year support window after each release comes out. It is therefore prudent to be running on a version that gives sufficient support window prior to your next upgrade. (2) Moving to a new version likely involves some amount of qualification in your environment. How long that qualification may take, along with your organizations operations and critical periods (eg likely you wont want to do upgrades during a peak season (eg Tax Season, or Black Friday / Winter Holidays). Often then, we see customers planning to upgrade to a new major/minor version right after their busy season(s). Our goal has been to make this upgrade as easy as possible by providing the ability to upgrade to a new version without having to sequentially upgrade through all intermediate versions. Do make sure to check the release notes / upgrade guide (for example https://docs.splunk.com/Documentation/Splunk/9.2.1/Installation/HowtoupgradeSplunk) for more details on this as this may vary for specific relases. The major/minor version will include features as well as bug fixes and security fixes. The length of time your organization needs to qualify a release will vary based on your own specific system complexity and organization processes. It is however recomended to upgrade to a new major/minor version at least once per year. (3) Staying current with regard to security vulnerabilities is likely an important aspect for you to consider. Many enterprises have specific policy around security vulnerabilities and the time in which they need to be mitigated/remediated. With this in mind, our maintenance releases, which come out roughly every 8 weeks provide updates for issues published in our security advisories (see https://advisory.splunk.com/). The maintenance releases do not contain features and are intended to be much easier to upgrade to, given the surface area of changes is much smaller and thus should require less qualification prior to deployment. It is therefore recommended to stay current on the latest maintenance version for your given version. In summary then, your specific circumstances will likely differ, but it is recomended to upgrade to the latest major.minor version at least once per year, and then to stay current with the maintenance versions for that major/minor release on an ongoing basis.   Q. We are currently on 9.0. Can we upgrade to 9.2 directly, or do we need to upgrade to 9.1 and then 9.2? A.Yes, that is a supported version. You can always find information on what the supported ugrade paths are in Splunk Documentation. For Splunk Enterprise 9.2, what is located here https://docs.splunk.com/Documentation/Splunk/9.2.0/Installation/HowtoupgradeSplunk   Q. Are you planning to discontinue "classic Dashboard"? A. As mentioned in the Webinar, the future vision is for Dashboard Studio to become the defacto standard and tool for creating Splunk dashboards. We know many customers are currently on Classic, and are working to get as much feature parity as possible so that customers can migrate.   Q. Will we get to see any more detail about how the certificate store integration works? A. You can find information about Trust Store integration in our Splunk Documentation here: https://docs.splunk.com/Documentation/Splunk/9.2.0/Security/ConfigTLSCertsS2S   Q. Are there any real changes expected to apps when we go up to 3.9 from 3.7? A. In our internal testing across the platform and Splunk-supported apps, we have not seen significant breakage or compatibility issues. We expect customers to be able to migrate to Python 3.9 relatively easily, but we do recommend customers testing it before deploying into Production. The version we are targeting after Python 3.9 is expected to be Python 3.12 - which we know will be a major effort. We continue to publish more information on Splunk Docs as it becomes available.   Q. Will Splunk be able to use the Windows Certificate Store and how do we select which certificate will Splunk use? A. You can find information about Trust Store integration in our Splunk Documentation here: https://docs.splunk.com/Documentation/Splunk/9.2.0/Security/ConfigTLSCertsS2S   Q. Are the Splunk Containers going to be released on the same cadence as the main product? We have seen many more CVEs in the containers than in Splunk. A. Yes, the docker-splunk team maintains the same cadence as Splunk. The Splunk Operator for Kubernetes, which utilizes Docker-Splunk for deployment in Kubernetes environments, also strives to adhere to this cadence. However, occasionally, it may require more time to align with the desired schedule.   Q. What versions of Splunk can we upgrade directly from to 9.2.1? A. You can always find information on what the supported ugrade paths are in Splunk Documentation. For Splunk Enterprise 9.2, what is located here https://docs.splunk.com/Documentation/Splunk/9.2.0/Installation/HowtoupgradeSplunk   Q. Aren't the DSs using DFS shares for the apps? A. For our internal testing, we have tested with NFS and can confirm it works. It should extend and work using DFS as well.   Q. Since it can be put behind a load balancer, does replication occur between Deployment servers?   A. All of the DSs are mounted with the same network driver folder for phonehome, client events and apps files. Since the folders are pointing to the same network driver files for all DSs, they can be share each other’s info.   ... View more

Key Resources Register for Community Office Hours: Threat Detection and Response on March 27  Check out the Threat Research Team's blogs  Register for Community Office Hours: Generative AI on March 13  Link to monthly ESCU update posts  Read the PEAK Threat Hunting Framework  ... View more

Your Questions Answered Q. Is there a place to see a demo and more resources on the use of Attack Range?  A.  Absolutely, we're glad to see your interest in Splunk's Attack Range! For a hands-on demo and comprehensive resources, the best places to start are our latest release blog post on Attack Range v3.0 here, the GitHub project here for code and detailed guidance, and our documentation here for in-depth usage instructions. If you're looking for a quick setup, using Docker is highly recommended, with a guide available here.   Q. I've heard as part of best practices, we should clone correlation searches so that ECSU updates don't automatically update or change our customization (to account for other sources, unique exclusions, search timing, etc.). Is that a true best practice and how does this impact your ability to assess what coverage you have on correlation searches and analytic stories? A.  Certainly! Cloning correlation searches in Splunk is a recommended best practice to protect your custom modifications when Enterprise Security Content Update (ESCU) releases new updates. This approach safeguards your tailored settings, like unique data source adjustments and specific exclusions, from being overwritten. However, it does present a challenge in tracking and assessing your security posture, as it's harder to keep up with which searches you've customized and how they align with ESCU's analytic stories. To manage this, maintain thorough documentation of your changes, use version control for easy tracking, regularly review your custom searches against ESCU updates, and adopt clear naming conventions. This strategy ensures you benefit from ESCU's enhancements while keeping your specific security needs addressed.   Q. Do you still recommend M365 and Azure data collection via Add-on for MSCS and Office 365, etc. Or using Event Hub and "Microsoft Defender Advanced Hunting Add-on for Splunk" Add-on? A.  Choosing between specific Splunk Add-ons for Microsoft 365 (M365) and Azure data collection or integrating via Azure Event Hubs with the "Microsoft Defender Advanced Hunting Add-on for Splunk" largely depends on your specific needs and the volume of data you're dealing with. Microsoft recommends Azure Event Hubs for its scalability and efficiency in handling vast amounts of log data, making it ideal for organizations with extensive Microsoft service usage. This method, especially with the Defender Advanced Hunting Add-on, enhances security analytics by pulling in detailed threat intelligence. However, direct Add-ons for services like MSCS and Office 365 might still be preferable for targeted data collection needs or ease of setup in certain environments. Ultimately, the best approach could involve leveraging both options to align with your data collection and analysis goals, ensuring a comprehensive and efficient integration with Splunk.   Q. I could see there is a email logs delay in my environment. Is there any pre-built addons or apps to threat detections queries? A.  For enhancing threat detection in your environment, you can utilize the Splunk Security Content (ESCU) available on Splunkbase and further explored at Splunk Research. ESCU offers a comprehensive collection of pre-built detection queries and analytics stories designed to help you efficiently identify and mitigate security threats. Integrating ESCU with your Splunk setup enables you to leverage advanced security analytics and improve your overall security posture.   Q. Can we get any source office365 threat detections? A.  Yes, we have content for Office 365 threat detections. You can explore detailed analytics stories and detection techniques for Office 365 at the following links on Splunk Research: Office 365 Account Takeover Office 365 Collection Techniques Office 365 Persistence Mechanisms ... View more

Your Questions Answered Q. How significant of an accuracy improvement is achieved with fine-tuned LLMs as compared to traditional approaches?  In order to measure improvements in accuracy, it’s important that you establish baselines before employing LLMs. Then, you can start implementing and fine-tuning your LLMs and measure the difference in accuracy. While the overall improvement may be small, we found that typically the model performed well for harder instances.   Q. What are the cost considerations while deploying LLMs?  One important cost consideration to keep in mind is the GPUs required to run LLMs. In general LLMs are resource hungry but with time, we are seeing more focused and efficient implementations.   Q. What are the privacy considerations when using Generative AI models for security use cases?  Preserving privacy while learning from sensitive data is always a challenge. However, these risks can be mitigated by adding carefully calibrated noise while training. Techniques such as Differential Privacy allow us to provide strong privacy guarantees.   Q. Probably not ML related, but is there a way/site to differentiate photos AI created on sites like this-person-does-not-exist from real ones?  This is very challenging. Once the adversary becomes aware of the defenses employed to filter out AI-generated images being used for malicious intent, they can adjust how they train the LLM to evade these defenses. So while today, my machine learning classifier may be able to correctly identify real versus fake images, the adversaries could adapt tomorrow.   Q. Beyond DGA and mis-information can you think of any user uses of generative AI by adversaries ? can you give some practical examples of generative models applied over data in the security field? or maybe a reference/link to such examples and use cases?  TWhen it comes to adversaries’ use of generative AI, there are two things to keep in mind: 1. Not every adversary is equipped to use LLMs and generative AI, given how sophisticated they are. 2. For those adversaries that are employing these technologies, it’s a bit of a cat and mouse game. For example, just like we can train LLMs to help detect certain threats or threat indicators, adversaries can then train their own LLMs to evade detection. Security focused Generative AI use cases: 1. Detecting DGAs 2. Detecting Typosquatting 3. Detecting spam 4. Detecting phished webpages 5. Privacy-friendly data generation to train models 6. Data generation to supplement sparse datasets The above mentioned use cases can also be used by the adversary to strengthen their attacks.   ... View more

Key Resources Community Office Hours: Generative AI on March 13 Enterprise Security Content Update (ESCU) | New Releases Splunk Blogs ... View more

@wslow  We apologize for our delay in response to you and also wanted to check to see if you were still experiencing issues with finding the release?  The GA date was actually in November to be timed along with our Tech Talk on this topic.  Hopefully you are up and running with it now, but please let me know if I can assist! Thank you, Whitney ... View more

Here are a few top of mind questions from the live Tech Talk   Q. How long does it take to upgrade from 7.1.1 to 7.2.0 ? A. It depends on your architecture and how many other components also require upgrades. The core Enterprise Security search head upgrade app runs in about 10 minutes, however backups and other app updates add to the time.   Q. Do you have any detection search to find anomaly login? A. We have a list of detections in this analytic story: Suspicious AWS Login Activities, that cover various detections related to anomalous login activity, you can repurpose this SPL logic to a data source of your choice. Q. Can you list the recommended steps for upgrading ES to version 7.2? A. Please see our documentation here: https://docs.splunk.com/Documentation/ES/7.2.0/Install/Beforeupgrading Q. My incident review looks nothing like this. How can I get this view? A. In ES 7.2.0, follow these steps to turn on the enhanced analyst workflows on the Incident Review page: From the Splunk ES menu bar, select Configure > General > General Settings. Locate the card for Enhanced Incident Review Workflows. Select Turn on to turn on the ability to use shared views, table filters, and table columns on the Incident Review page. ... View more

Please find materials to continue your journey below: Splunk Security Content for Threat Detection & Response: Q2 Roundup Amadey Threat Analysis and Detections Defending the Gates: Understanding and Detecting Ave Maria (Warzone) RAT Sharing is Not Caring: Hunting for Network Share Discovery Mockbin and the Art of Deception: Tracing Adversaries, Going Headless and Mocking APIs See More, Act Faster, and Simplify Investigations with Customizable Workflows from Splunk Enterprise Security 7.2 ... View more

Continue your journey AI Use Cases for Observability Guide 5 Big Myths of AI and Machine Learning Debunked AI and Machine Learning in Your Organization 6 Myths of AIOps Debunked ... View more

You have questions, we have answers! Here are the questions and answers from our Tech Talk:   Q: What products do I need for this? A: Splunk Core (version 8.2.x and above) and Chrome Browser Cloud Management.    Q: Do I need to have Splunk Enterprise Security? A: Splunk Enterprise Security is not required. All events are CIM tagged so any Splunk Enterprise Security content built on the data models that an event is tagged to will populate with the tagged event.   Q: Where can I find this app?  Will it be Splunk supported? A: In Splunkbase. The chrome add-on and app are built by "Splunk Works", Splunk Field Solutions (internal Splunk teams). They don't carry the official Splunk SLAs but there is an internal team that actively works on it, bug fixes and enhancements.    Q: Per splunkbase, this app is not supported by Splunk.  Safe to assume it is? A: ^Same answer as above.   Q: Will there be automatic updates of the add-on for users, so that new security issues are continually being protected against? A: Yes. We will monitor the list of chrome events that are being released by Google Chrome team and build new use cases or enhance existing ones within Splunk.   ... View more

Continue your journey Getting Started with the Google Chrome App for Splunk Guide Configuring Alert Actions Guide Using Splunk to Enhance Enterprise Security Capabilities of Google Chrome Check the solution out here           ... View more

How to Get Started with Google Chrome Add-On for Splunk Navigate over to Splunkbase to install the Google Chrome Add-on for Splunk and Google Chrome App for Splunk. If you need helping getting started, take a look at our resources below: Getting Started Guide for the App and FAQ Chrome Browser Cloud Management set up, enrollment, and admin console Eradicate the risks that come from risky browser behavior and make your enterprise more resilient by installing the Google Chrome Add-on for Splunk and the Google Chrome App for Splunk today!   ... View more