Threat Hunting Unlocked: How to Uplevel Your Threat Hunting With the PEAK Framework and Splunk

[WhitneySink]

WATCH NOW

As AI starts tackling low level alerts, it's more critical than ever to uplevel your threat hunting to find sneaky and elusive threats. This tech talk shares how the Splunk Threat Hunting team seamlessly integrated the PEAK Threat Hunting Framework into their workflow while leveraging Splunk. See how they also joined forces with the SOC to turn non-hunters into cyber sleuths. Don't have a dedicated hunt team (or even if you do) – explore Splunk’s end-to-end processes with tips and tricks to unleash a pipeline of hunters and turn the PEAK Threat Hunting framework from a concept into a powerful tool in your organization.

Watch Splunk threat hunters Sydney Marrone and Robin Burkett to learn about:  

• The PEAK threat hunting framework 
• How you can customize PEAK for your environment 
• How to enable your SOC analysts to be successful threat hunters
• Real-world examples of PEAK hunt types for actionable insights

WATCH THE REPLAY

[WhitneySink]

Here are a few top of mind questions answered during the live Tech Talk

Q. What is the best approach to select 'key' fields from a data source?

This is where performing a Baseline hunt may help. The outcome of that hunt should be a data dictionary of data fields available in a given data source along with their values. By looking at the values, you can determine which have the most relevance to the target/technique/payload you are hunting - these become the "key" fields. 

Q. Do you convert successful hunts to correlation searches at least until something is mitigated? What does that process look like? 

If it's an immediate concern or poses an imminent risk, we will escalate and have it mitigated as soon as possible. If it can be turned into a correlation search with high fidelity/low noise, we will have a detection created for. Otherwise, it may be used as a candidate for an L1 hunt because it is too noisy for a detection.

Q. During the Act phase of the framework, what type of factors do you consider when creating notables/alerts vs. recurring searches/dashboard panels for future hunts?

One of the factors we consider when creating notables/alerts vs. recurring hunts is how easy it is to distinguish anomalous non-malicious from anomalous malicious events. If it requires a human to determine if something is malicious, say an unexpected scheduled backup, or unexpected deletion of software, we want a human to determine if the activity should be happening. Another factor is the significance of the asset. If the asset in question has significant importance to Splunk, we may want an analyst to review events related to that asset to confirm they are expected. Our goal when creating a notable/alert is to minimize false positives but if there is a risk that a true negative that could have a large impact to our security could slip through we want to have a human look at the events to make sure vs. depending on an alert to fire.

Q. When you talk about sharing your results, who is the intended audience of these reports? 

It will depend. Typically, we share with other security operational teams and leadership. We may also share with relevant stakeholders like tool owners.

Q. How much time should be dedicated to threat hunting per week as a SOC analyst?  

Organizations should prioritize allocating sufficient time for threat hunting, balancing it with other team priorities. We've observed that analysts require a minimum uninterrupted window of 4 hours, ideally a full day, to focus solely on hunting. Context switching between reactive ticket handling and proactive hunting proves challenging and may impede success. Establishing dedicated hunting days, where analysts are free from interruptions, is optimal. Additionally, ensuring adequate shift overlaps can help mitigate the backlog of tickets, allowing analysts to focus on hunting without concern for pending tasks.

Q. Does PEAK help to prioritize which topic I should hunt?  

The PEAK framework doesn't address how to prioritize hunt topics. Instead, we prioritize them by considering factors such as the environment's importance to the business, our current detection coverage, threat intelligence team recommendations regarding recency or known adversaries, and any recent incidents related to the technique.

Q. How do you avoid going down a rabbit hole? 

Crafting a well-defined hypothesis that outlines the target, technique, and expected payload, while also completing the ABLE table categories of Actor, Behavior, Location, and Evidence, is crucial for maintaining focus during a hunt. Straying too far from the hunt's objective is a common pitfall, and having a clear hypothesis helps prevent such diversions. In threat hunting, you'll encounter rabbit holes, and experience helps decide when to stop pursuing unproductive leads.

Q. For any new environment how do you prioritize what to hunt/hypothesis? 

For a new environment, it may be helpful to start with a Baseline hunt to understand the data and would be valuable to hunt. Here is a blog post with additional information. https://www.splunk.com/en_us/blog/security/peak-baseline-hunting.html

Q. What learning sources are there to follow for effective hunts?

There is a book with a wealth of Threat Hunting goodness called Bluenomicon. You can download it for free here: https://www.splunk.com/en_us/form/the-network-defenders-compendium.html

Q. The acquisition of Splunk by Cisco has the potential to enhance Splunk, making it a better tool? 

We're enthusiastic about the possibilities stemming from our acquisition by Cisco. Currently, we're evaluating how to effectively utilize our combined expertise, tools, technologies, and data.

Q. How much time did it take M-ATH to process the data and output results?  

For this one SPL example, 3-5 minutes. This was done many times for this hunt. It's always going to depend on your SPL scope.

Q. What skills are good to focus on for self-development that would help with getting into Threat Hunting (ie. SPL, general OS knowledge, etc.)? What are "must-have" or most critical skills/knowledge to have as a Threat Hunter?

Curiosity and a desire to understand why things are the way they are. If someone says "that's just how it is", and your response is "yeah, but why?", that's a great quality for a threat hunter. Some of the best findings come from questioning something that seems benign, but is just slightly different. Also, the innate ability to see patterns in things. Embracing learning about things you don't know about and enjoying learning new things. Understanding the foundation of how things work, e.g., the OSI model, networking, DNS, common protocols, cloud infrastructure, having a strong foundational knowledge related to technology and common security attacks helps when you need to pick a topic and dig deep to learn more about it. Being able to collaborate with other teams who may be experts in a topic you are trying to hunt is also important.

Q. Do you use custom data models to enable threat hunting in Splunk? Is there a core data set you find "necessary" for threat hunting?

We do have some custom data models and customized out of the box data models, but we are not using them as primary sources in our hunting. In terms of a core data set, it depends on the topic of the hunt. For example, cloud logs are necessary for hunting in the cloud, endpoint logs are necessary for digging into endpoing activity, which may also result in pivoting into network logs depending on the goal.

Q. What is the measure of success for the overall hunt program?

The goal is to make the organization more secure. While we document metrics in terms of hunts completed, findings, knowledge shared, etc. we are continually trying to determine impact. There is a great PEAK blog post on this where you can read more about the topic here: https://www.splunk.com/en_us/blog/security/peak-threat-hunting-metrics.html

Q. What is the overlap with Red Teams/Pentest teams? How do you collaborate?

The Red Teams colloborate with us when we have hunt topics that would benefit from attack-related events that we do not have in our environment. They will emulate an attack so that we can see what it looks like in the logs to better determine how to hunt. We also work closely with them on Purple Team engagements where we have multiple red and blue teams working together to simulate a threat actor.

Q. How are you grouping hunts with similar scope/TTP? What is the criteria/process?

We follow a hunt intake process to assess anything new, grading them based on criteria such as relevance to our internal environment and whether they're related to incidents. Additionally, we sometimes organize group hunts involving multiple participants that share a similar scope.

Q. It seams like there is a lot of Security/Splunk engineering going on that you do not need to deal with prior to and after hunting. Do you have any advice for getting logs parsed and CIM compliant if you are the hunter and engineer?

Log directly into tools that would send data to Splunk and do your hunting in the tool first to help prioritize and inform what should be onboarded to Splunk. In terms of parsing and CIM compliance, our professional services team can help with that. From a hunt perspective, hunting without CIM compliance is not a showstopper and though it is not optimal, you can also fix poorly parsed fields in SPL with Rex and Regex, or through our built-in data parser.

Q. Was the use of the PEAK threat hunting framework for managed service providers considered, or is it very much applied to internal use for an organization?

The PEAK threat hunting framework can definitely be used by managed service providers, in fact, we encourage it. The framework is applicable to any organization seeking to create, perform, or manage threat hunting.

Q. Was the use of the PEAK threat hunting framework for managed service providers considered, or is it very much applied to internal use for an organization?

The PEAK threat hunting framework can definitely be used by managed service providers, in fact, we encourage it. The framework is applicable to any organization seeking to create, perform, or manage threat hunting.

Q. Do you have any example threat reports? 

We don't currently publish public reports as we are internally facing. We may in the future.