Refine your search:

2
1

I wanted to see how Splunk would index my data, so I configured it to index a few files into a 'test' index. Now that I have it configured properly, I want to re-index that same data into the 'main' index. I cleaned the test index - ./splunk clean eventdata index test - and removed the index = test from inputs.conf, but Splunk doesn't automatically re-index the files - why not?

asked 09 Apr '10, 00:51

Mick's gravatar image

Mick ♦
4.0k1327
accept rate: 52%

edited 04 Oct '11, 02:24

jlaw's gravatar image

jlaw ♦
20113


One Answer:

removing index=test probably tripped you up.

the logic for this feature is:
if:
- indexing a file F
- splunkd's last record of F is from time T1
- the creation time of the destination index is T2
- T1 < T2
then:
- begin reading F from position 0 again.

i'm assuming you didn't clean index main here, so its creation date is well before our last fishbucket record for that file - thus T1<T2 is false, and we don't re-read the file.

link

answered 09 Apr '10, 00:58

amrit's gravatar image

amrit ♦
1.2k38
accept rate: 62%

even if I leave 'index = test' the files don't get re-read, but based on what you're saying, I would have to create a brand new index for the data to get re-read? That doesn't make sense to me, I want to add it to my existing index once I'm happy that it will be indexed correctly. What if I have to tweak settings several times before I get it right, do I need to create a new 'test' index each time?

(09 Apr '10, 01:04) Mick ♦
2

maybe you should RTFM and get it right the first time. jk!! <3 ...did you clean before you had index=test in the conf? if you cleaned test, restarted, added index=test, and restarted again, you'll encounter the above fail. anyway, good point about finalizing in a different index. right now this isn't possible (although, you could add a bogus crcSalt and leave it there forever...), but we can add something. the idea would be something like "splunk reset filepos /path/to/file", which would use btprobe to zero out our record of the file. this feature does not exist currently.

(09 Apr '10, 01:22) amrit ♦

that would be a very very very good feature to have.

(09 Apr '10, 03:20) gkanapathy ♦

i forget one very. very.

(09 Apr '10, 03:21) gkanapathy ♦

See --reset @ http://docs.splunk.com/Documentation/Splunk/latest/Troubleshooting/CommandlinetoolsforusewithSupport#btprobe

(11 Apr, 11:19) amrit ♦
Post your answer
toggle preview

Copyright © 2005-2012 Splunk, Inc. All rights reserved.