|
Can Splunk read data to index off an NFS mount? |
|
There are two questions here:
The answer to 1 is "yes", you can read it like any other file, though some NFS implementations occasionally can corrupt their data if there are multiple writers to a file. The answer to 2 is "sort of". You can definitely store the cold database onto NFS. You will not be able to store the hot/warm database partition onto most NFS implementations without significant chances of data corruption, as well as probable major performance problems, so we recommend against the practice in general. That is not to say that it is impossible in very limited and special circumstances. The requirements for storing a Splunk hot/warm database onto NFS will be similar to storing high-volume RDBMS data files onto NFS, i.e., low latency, high thruput, TCP-based NFS, a rock-solid NFS client implementation, correct caching settings, etc. But these are complex and require NFS infrastructure and background that is very rare, so for most people, the practical answer is "you can store cold, but not hot". |
|
it depends on what you're actually asking--you can store indexes on an NFS mount if you don't need to search them very rapidly or frequently. if you intend to search more frequently and want fast response, NFS is not recommended. there's a lot more information about index sizing and capacity planning in this documentation topic: http://www.splunk.com/base/Documentation/latest/Installation/CapacityplanningforalargerSplunkdeployment with respect to indexing from an NFS mount: NFS is usually a poor choice for Splunk indexing activity, for the same basic reasons that it's not really recommended to store your index across an NFS mount--it's slow and tends to be high latency, and not terribly resilient. In environments with very high bandwidth, very low latency links that are kept highly reliable, it can be an appropriate choice. Typically you achieve this by using a SAN which is accessed via the NFS protocol. hah we all answered this at the same time! good thing we didn't contradict each other.
(26 Jan '10, 19:18)
benstraw
|
|
It is not the best solution, but yes, you can store your index on an nfs mount. From the splunk documentation:
|
|
I know this is late to the game, but I thought I should lend some favor to using NFS for hotdb/warmdb indexing. This is the only way we do indexing within our environment, so let me share what we have in our infrastructure. - 120GB/day indexing - IBM blades - Linux - (2) quad core procs - 16Gb mem - 2nd 1Gb interface connected to Copper passthru in chassis hooked directly to a 1Gb NAS dedicated switch port. When originally mentioned to Splunk that we would use this setup they hesitated and asked us to run some bonnie++ testing. So I did. Results came back VERY favorable and actually surprised the resident Professional Services Expert who was a bit skeptical. He is now a believer in our environment. We have a VERY tuned NAS and network infrastructure that allows us to make this environment happen. If you think you can pull it off, run some bonnie++ testing and get the results to their engineers for analysis against what they spec. They might advise you that you can pull it off.....we did. Good luck. |
Can you please clarify which of the two questions you were really asking?