How to get an answer from search command in Splunk Enterprise?

[gjhaaland]

Hi,

Splunk has been working for a long period without any trouble. When I changed settings yesterday (can't remember what I did) the search command dos not work as before (no answer). 

If I go to settings - indexing   _audit, _internal , _introspection,  _telemtry, _history + main area all of them is disabled.

I also google, and it says that it perhaps has something to do identical id under db directory. We have same id on some files with .sentinel  

example:

db_123_345_12

db_123_345_12.rbsentinel 

If I run following command:

run netsat -an | grep 9997 we have many tcp session establised . 

Have of course rebooted, restarted splunk server several times.  It does not help much. 

Thanks in advance. Hope someone can give me a hint. 

Rgds

Geir

[gjhaaland]

Forgot to mention

When I open Data Summary it says "Waiting for results" but it never get/receive any data. Only Waiting for Results without ending.

Rgds

Geir

[gcusello]

Hi @gjhaaland ,

open a case To Splunk Support, it's the only way to have a quick answer.

ciao.

Giuseppe

[gjhaaland]

Giuseppe,

Thanks again, 

Yes, If I run search command and/or old reports we get  no answer at all. The splunk gui is running, but we don't get any answer if we run search - index=*. Normally we will see a long listing with output. 

I have not deleted any files.  All I have done is  some settings regarding field extraction. After a while I discovered that we did not receive any data at all.  So I must be some connection between fields (enable/disable) and fields extraction.

Rgds

Geir 

[gcusello]

Hi @gjhaaland,

if you run a search on _internal, did you have results?

have you any messages from Splunk?

Ciao.

Giuseppe

[gjhaaland]

Hi gcusello

Thanks for the answer.  No answer at all, even if I run “Usage Reporting Dashboard” the answer is empty. Since it work perfect yesterday I thinks/assume that some files are blocking stopping normal behavior .  

If I restart splunkd  I got following messages

1: Invalid key in stanza  [admin_external:configure]in /home/splunk/etc/apps/TA-eStreamer/default/restmap.conf, line 7: python.version

2: your indexes and inputs configurations are not internally consistent. For more info run splunk btool -check –debug

3: Validating installed files against hashes from /home/splunk/splunk/7.1……..-x86_64manifest’

Problems were found, please review your files and more customization to local

Starting splunk aerver deamon (splunkd)

Done

[OK}

Rgds

Geir

If I run splunk btool -check –debug

I got following error (cut/paste errors)

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/local/css_views.co

No spec file for: /home/splunk/etc/apps/TA-eStreamer/local/encore.conf

No spec file for: /home/splunk/etc/apps/eStreamer/local/estreamer.conf

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/default/css_views.conf

No spec file for: /home/splunk/etc/apps/Splunk_CiscoSecuritySuite/default/eventgen.conf

No spec file for: /home/splunk/etc/apps/TA-eStreamer/default/encore.conf

Invalid key in stanza [admin_external:configure] in /home/splunk/etc/apps/TA-eStreamer/default/restmap.conf, line 7: python.version  (value:  python3).

No spec file for: /home/splunk/etc/apps/eStreamer/default/estreamer.conf

No spec file for: /home/splunk/etc/apps/firepower_dashboard/default/appsetup.conf

No spec file for: /home/splunk/etc/apps/firepower_dashboard/default/umbrella.conf

No spec file for: /home/splunk/etc/system/default/conf.conf

No spec file for: /home/splunk/etc/system/local/migration.conf

[gcusello]

Hi @gjhaaland,

the error messages aren't relevant.

Let me better understan: the search doesn't run or you have always no results?

When you say that yesterday worked perfectly, are you meaning: that yesterday the searches  run or that running  today a search on yesterday data the are ok?

Probably the only solution is to opena a case to Splunk Support that can access your system (with you) and debug the situation.

Ciao.

Giuseppe