Summary of Indexes and SourceTypes

[MattKr]

Hi,
the size of my Splunk database is at around >1TB+.

I would like to know about all available Indexes and especially all of the associated SourceTypes
and the amount of it.

The search in WebUI works no problem for the last 24hrs but searching for all of the data
takes forever and times out.

I'm aware that saved searches would be an option but i'm curious to know if  a script
would work which recursive scans the database and process all SourceTypes.data file

like

< /opt/splunk/var/lib/splunk/sampledb/db/db_1680195600_1672423200_0/SourceTypes.data
< /opt/splunk/var/lib/splunk/sampledb/db/db_1698782400_1680199200_1/SourceTypes.data
...
...

Would this be a feasable option?

Many thanks

[danspav · Accepted solution]

Hi @MattKr,

Here's an option that will run from the UI.

| rest /services/data/indexes splunk_server=local
| stats count by title | rename title as index
| map  [| metadata type=sourcetypes index=$index$ | eval index="$index$"] maxsearches=100

• In the first line, make sure splunk_server=<NAME OF INDEXER>,  for Splunk Cloud local is fine.
• Make the maxsearches=XXX match the total number of indexes you have. 

This uses the metadata command to get the sourcetypes, and earliest/latest times, and the number of matching events.  The one drawback is that the index isn't included in the results, so I've set it up via the map command so it will run the metadata search for each index.

Couple of things to note:

• This will run as many searches as you have indexes - so be careful.
• The metadata search is lightening fast as it only runs on the index metadata (hence the name) so there's no real data being brought back - just data about the index.
• You need to run it as an all-time search to get all of your data... Pick a time to do this to reduce any impact.

I ran the search on a small cloud environment with 52 indexes over all time and it completed in 4.9s. 

Give that a go.

[MattKr]

Hi danspav,

thank you so much, the query took around 300 sec. on around 10 indexes, 4TB db size and returns what i'm looking for, perfect!

[bharathkumarnec]

@MattKr What is your retention period of logs? Also you can look at _internal logs by sourcetype to get the required data but these internal logs are stored only for 30days by default.