|
I set up a splunk instance on a server with a local csv file that updates 1/min. Using the add data wizard, it auto detected all the appropriate timestamp, metadata and value fields. I then set splunk to forward to another instance (to test forwarding), and the data forwards fine, but its all in raw format. I looked for a props.conf file on the original server to see if the wizard created something I could copy over, but no luck. Also, if you can help with setting this up on the reciever instance, can you also mention if there is a way to go through all the data that has already been indexed and extract the fields into the indexes? Sorry, totally new to splunk, just trying to build a business case and do some DD before strapping to it as a platform! |
|
Good news - field extraction is done at search time. This means that you can create fields for data that has already been indexed. If you selected Option 1 - Set sourcetype to csvHere is one way to do this. This technique has you set the sourcetype of the input to inputs.conf
Important - this will only affect new data. It will not change the sourcetype of data that has already been indexed. Option 2 - Set field extraction for a sourcetypeAnd here is another way. This technique assumes that the data has already been indexed, and has been assigned a sourcetype that is not You may need to create the props.conf and transforms.conf files. Put them under props.conf
transforms.conf
Now, as you add more data to splunk, you can continue to use sourcetype X and sourcetype Y, or create new sourcetypes as needed. Thanks. I was most interested in having this work from scratch, so I uninstalled splunk on both servers. Set up the first server again, used the wizard with Preview to add the file. Despite making sure both on the first screen and on more options that the sourcetype was csv, when the data source was saved, it assigned a sourcetype of csv-2. Set the primary up to forward to the newly reinstalled secondary, and the data is sent to the secondary server, but it does not break out the fields like on the primary server. Same data source, datatype, I have 21 fields on pri and 17 on sec. THX!
(27 Feb '12, 17:30)
bmgilmore
1
Right. The csv sourcetype is configured to use CHECK_FOR_HEADER, and that type of configuration generates AutoHeader config that ends up in '$SPLUNK_HOME/etc/apps/learned', and ends up trapped on the forwarder. So while the data itself gets forwarded, and mod the weird "foo-2" thing that CHECK_FOR_HEADER does to it's sourcetypes, arguably the sourcetypes come across, the field extractions do not come across to the indexer.
(13 Jun '12, 12:55)
sideview ♦
|