SOC Analyst's work with Splunk ES

[splunky_diamond]

Hello, Splunkers!

I hope there are some SOC analysts around who are using Splunk Enterprise and Splunk ES in their work. I've been learning Splunk for the past month and I have worked with Splunk ES a bit and tried configuring some correlation searches with automated notable generation along with email notification alerts.

I now have to present some cases in my test lab, where I have an attacker who performs some malicious activity that triggers some of the correlation searches that I have configured, and then I need to demonstrate the full investigation process from SOC analyst's POV. 

The problem is, I have almost 0 knowledge of how SOC operates and if they were to use Splunk Enterprise and Enterprise Security app, what would they do exactly? Would they just go over all the new notables and look at the drill-down searches trying to understand what notables are related to other notables? Would they try to correlate the events by time? Would they only work around Splunk ES, or would they also go to the dashboards and search for some data there? 

I would appreciate it if someone could explain how SOC works with Splunk ES in case of some simple, uncomplicated attacks, that trigger 2-3 correlation searches max.  

Also small question, since I have the email notifications configured, who is usually the one receiving the email notifications about triggered correlation searches, is it a SOC director, or analyst, or someone else?

Please let me know if more information is required, I would love to provide as many details as needed, as long as I get the best answer that would help me.

Thanks in advance for taking the time to read and reply to my post!

[gcusello · Accepted solution]

Hi @splunky_diamond ,

the normal activity flow for a SOC Analyst is the following:

• there a defined monitoring perimeter and some Correlation Searches that monitor the above perimeter to find some possible threat,
• if one or more CS thigger an alert, it creates a Notable,
• I think that an eMail notification can be useful only for night monitoring because during the day, the SOC Analysts should be aways connected to ES,
• when a Notable is triggered (a Notable is one or more events that match a condition to check, not a securty indicent!), a SOC Analyst takes in care the Notable ana ivestigate using the investigation panels and eventually its own searches,
• He/she could also use the other ES dashboards, even if I never saw this!
• based on the investigation the SOC Analyst defines if:
  • it's a real security incident,
  • it's a false positive,
  • the Notable requires an escalation for adeeper check,
• if it's a false positive THE SOC Analist closes the case ,eventualy adding a suppression rule,
• if the Notable requires an escalation check, the SOC Analyst passes to case following the indication of the related playbook,
• if it's a real security indicent, the SOC Analyst apply the predefined playbook actions or passes the activity to the colleagues enabled to intervene.

This is a general fow, and it depends on the internal processes of the SOC.

Only one additional information: if (as usual) in your SOC there are few SOC Analysts, it could be a good idea, doesn't associate to a Correlation Search a Notable but a Risk Score addition; in this way the SOC Analyst is informed in delay of a threat, but the SOC has to manage less Notables, in other words, if there are three SOC Analysts and the SOC receive 10,000 Notables/day they cannot check a of them.

Ciao.

Giuseppe

[PickleRick]

It's also worth noting that typically SOC (similarily to NOC, support and similar groups) is organized hierarchically.

Regardless of the actual tool used (there might be ES or any other SIEM, there could be a SOAR tool in place to simplify the process or automate some of its steps), the 1st line operator's task is to check the actual alert (it might be a Notable in ES, it might be an asset exceeding risk score threshold, it might be anything that is defined procedurally for a given SOC) and then verify it (typically according to a predefined playbook), react to it if it's something "standard" and reaction procedure is known and possibly pass the further processing to the 2nd line if the situation cannot be handled within the playbook defined parameters. 1st line operator will typically use some set of predefined dashboards if using ES or might be just having some playbooks defined in SOAR solution and not have to touch SIEM solution at all.

2nd line analyst has usually more knowledge about the company environment and access to more tools. Since it's this analyst's task to get more insight into a situation when there is an alert which cannot be handled in a predefined way, this analyst will typically use ES and Splunk in general along with other tools to find more information about context of the possible threat.

Most cases end at 2nd line analysts. If everything else fails 3rd line experts are called in (many smaller companies due to cost reason's don't even employ 3rd line analysts in-house but rather have some amount of man-hours purchased as a subscription service from an external service provider) and they will utilize everything at their disposal, including of course digging through the data in Splunk as well as reaching out to the solutions that generated those events and wil generally try to do whatever humanly possible to either stop the threat or - if the attack has already succeeded - limit its aftermath and restore the environment to normal state.

In other words - the more advanced work you do in the incident handling process, the more you'll probably be dealing with ES and Splunk in general.

[splunky_diamond]

Thank you so much for such a detailed addition @PickleRick !