how can I check 7 years of old data from splunk?

[prasireddy]

Hi Team,
how can I check 7 years old data that means the first ingestion was on 26 dec of 2016 I need total data size from starting date to Jun 30 2023.
I have tried with following Query ,when I run that its showing some 
1."DAG Execution Exception Error ":search has cancelled 
2.search Auto-cancelled 

the Query which I have used 

index=wineventlog source=security command_type!="METER_ALERT"
|eval size=len(_raw)
| eval raw_len_KB= round(size/1024,3)
| eval raw_len_MB = round(size/1024/1024,3)
| eval raw_len_GB = round(size/1024/1024/1024,3)
| table size,raw_len_KB,raw_len_MB ,raw_len_GB,index
| stats count sum(size) as Bytes sum(raw_len_KB) as KB sum(raw_len_MB) as MB sum(raw_len_GB) as GB by index


please help on this ?

Thanks In Advance 
Bala

[PickleRick]

Even assuming that you indeed have the data (you can check it in various ways, from looking at your actual bucket directories to tstats and dbinspect) your search is very inefficient and will probably get auto-canceled due to resource exhaustion.

Also, instead of manually calculating those stats, you can get most of the same info from the dbinspect command.

[prasireddy]

Hi @PickleRick ;
I have used dbinspect command  but here I need sourcetype and even the total count and size of command_type="METER_ALERT" and NON "METER_ALERT" separately .

Query :   index=service_audit sourcetype=SMWAN command_type !="METER_ALERT"
|eval size=len(_raw)
| eval raw_len_KB= size/1024
| eval raw_len_MB = size/1024/1024
| eval raw_len_GB = size/1024/1024/1024
| table size,raw_len_KB,raw_len_MB ,raw_len_GB,index
| stats count sum(size) as Bytes sum(raw_len_KB) as KB sum(raw_len_MB) as MB sum(raw_len_GB) as GB by index

[prasireddy]

Yes, For shorter periods it is working fine.
How can I check retention time grater than seven years 
and it is not a summary index .

Please could you help on this

[gcusello]

Hi @prasireddy,

you should check in the indexes.conf file where the index is defined, what's the retentin period of the logs contained in that index.

Retention is define using the option "frozenTimePeriodInSecs".

If it isn't defined, you have the default six years retention period.

Ciao.

Giuseppe

[prasireddy]

Hi ,
I have checked indexex.conf but here I did not find Retention option like  "frozenTimePeriodInSecs". then it means default is 6years .
Even when I'm giving 6 yr periods I did not see the data why ?

[gcusello · Accepted solution]

Hi @prasireddy,

Did you checked in the indexes.conf where the index is defined?

To be more sure, you should check using the btool:

$SPLUNK_HOME/bin/splunk camd btools indexes list --debug

maybe the option in another indexes.conf.

In addition you could use the Monitoring Console to see the retention in your index and how old is the earliest event in your index.

Ciao.

Giuseppe

[gcusello]

Hi @prasireddy,

good for you, see next time!

Ciao and happy splunking

Giuseppe

P.S.: Karma Points are appreciated 😉

[prasireddy]

I will check come back 

thank you so much 

[gcusello]

Hi @prasireddy,

i suppose that you have a retention time grater than seven years for your data modifying the default value (6 years), otherwise it isn't directly possible,  the only solution is storing aggegated results in a summary index (with a retention grather than 7 years) and then run searches on this summary index.

Anyway, have you results to your search using e shorter period?

Ciao.

Giuseppe