I must be high on Easter chocolate, as I just can't get this to work right.
Problem: I have nmap verbose output (such as from running the nse scripts) for a scan across a subnet. Thus, I want each host to be an event. The issue should be, in theory, as simple as saying that with the right line_breaker, the event will be properly seen.
I thought that putting this in my props.conf would do the trick:
[nmap-verbose] LINE_BREAKER = Nmap scan report for (\d+\.\d+\.\d+\.\d+) TRANSFORMS-nmap=nmap-host
And my transforms.conf:
[nmap-host] REGEX = Nmap scan report for (\d+\.\d+\.\d+\.\d+) FORMAT = dst_ip::$1
I've tried this without the transform, and all sorts of combinations on the regex, but don't get anywhere.
Raw data looks like this:
Nmap scan report for 220.127.116.11 Host is up (0.015s latency). PORT STATE SERVICE 21/tcp open ftp Nmap scan report for 18.104.22.168 Host is up (0.014s latency). PORT STATE SERVICE 21/tcp open ftp | ftp-anon: Anonymous FTP login allowed (FTP code 230) |_drwxr-xr-x 2 0 0 4096 Mar 08 05:54 pub
Ideally, I'd like to get the IP address in the "scan report" line seen as the destination IP of the scan as a field I can use, but the way I'm doing it now either results in thousands of events, pretty CRLF separated, or, a handful events with many hundreds of lines per event. Also tried with and without the linemerge true/false set...so I'm obviously missing something..
asked 23 Apr '11, 18:44
I found a better way to get what I want, and am including my search here so that others may hopefully benefit. I've run my nmap scans with -oG to generate the "greppable" format. With that being read into Splunk, the following search generates some pretty useful ways of looking at/for data:
The raw input looks like:
22.214.171.124 () Ports: 21/closed/tcp//ftp///, 22/closed/tcp//ssh///, 23/closed/tcp//telnet///, 80/open/tcp//http//Microsoft IIS httpd/, 139/open/tcp//netbios-ssn///
And my search:
index=nmap open | rex "\\tPorts:(?P<ports>[^\\t]+)" | makemv delim="," ports | rex "^Host: (?<target>\d+\.\d+\.\d+\.\d+)" | rex field=_raw "\s(?<port>\d+)/(?<status>[^/]+)/(?<proto>[^/]+)//(?<daemon>[^/]+)//(?<desc>[^/]+)/" | search port="80" | stats count by desc
Translating this into plain English, I have all of my nmap output in a specific index, and am only looking for the lines which have something open. A system which reports everything as closed could be interesting as well, but, that's not what I was after.
From there, I extract the Ports bit of nmap's output and run that through makemv to break down the individual port and status //// combinations.
After that, I break down those into their respective components, which then leads to the ability to search for specific things like port=80, or 22 or whatever, and finally create a nice table of values of descriptions from the -sV flag in nmap.
There are no doubt far more graceful ways of doing this, but, if someone else comes to the splunkbase looking for nmap hints, maybe this will help.
answered 04 May '11, 21:43
You should use either:
The former is much more efficient, but the latter may be easier to understand.
Event breaking occurs in two steps:
If the second step does not run, then the events are simply the same as the individual lines. The first step is relatively efficient, while the second is relatively slow. If you are clever with the
This (howyagoin's) post helped me greatly in working out how to get it in operation - those interested in parsing greppable Nmap with Splunk may want to check it out (below and more information in my full post).
Additional answer for this issue - to extract all fields (and shorten your search) Field Extraction Help Gnmap and Troubleshooting
You can then search on port, state, daemon and banner in a more succinct search. You may have issues with a very small number of daemons (e.g. nfs and rpc) as the nmap output is slightly incorrect for those services at this point in time - inconsistent use of the field separators "/" (A work-around sed command is in my full post).
answered 08 Oct '12, 16:14