Splunk Cloud user here. I have defined a new sourcetype with the following properties:
Name: postgresql_csv
Destination app: Distributed Management Console
Category: Database
Description: PostgreSQL CSV log
Indexed Extractions: csv
Timestamp:
Extraction: Auto
Delimited settings:
Field delimiter: ,
Quote character: "
File preamble: (empty)
Field names: Custom
Comma separated field names: log_time,user_name,database_name,process_id,connection_from,session_id,session_line_num,command_tag,session_start_time,virtual_transaction_id,transaction_id,error_severity,sql_state_code,message,detail,hint,internal_query,internal_query_pos,con,query,query_pos,location,application_name
Then I conduct a search for sourcetype=postgresql_csv
, but it doesn't look like any of the fields I defined above appear in the fields bar on the left. After a bit of searching, 40+ new fields pop up, all with approximately 0% coverage, fields such as "database" or "average" or "AND_classid", which are AFAICT just guessed extractions from the actual log text, but aren't particularly useful.
Sample CSV below:
2015-08-30 00:50:31.539 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,2,"authentication",2015-08-30 00:50:31 UTC,2/1713,0,LOG,00000,"connection authorized: user=classroom_api database=classroom_dev",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,3,"PARSE",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.106 ms parse <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,4,"BIND",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms bind <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2015-08-30 00:50:31.541 UTC,"classroom_api","classroom_dev",17910,"127.0.0.1:48594",55e25357.45f6,5,"SET",2015-08-30 00:50:31 UTC,2/1714,0,LOG,00000,"duration: 0.011 ms execute <unnamed>: SET extra_float_digits = 3",,,,,,,,,""
2
What am I doing wrong here? I read in a couple of posts that it's not possible to disable CSV auto-learning without digging into props.conf
, which I don't have access to in Splunk Cloud. Please advise.
INDEXED_EXTRACTIONS only works when ingesting local files.
So you want to setup your props.conf with the sourcetype definition on the forwarder.
INDEXED_EXTRACTIONS only works when ingesting local files.
So you want to setup your props.conf with the sourcetype definition on the forwarder.
Looks like your suggestion works, I added a props.conf with a custom source type to system/local.
On that note, if I want to add a few custom field extractions to this type that only exists in props.conf at this point, do I create a sourcetype with matching name on my Splunk Cloud instance and add those extractions there?
Does your CSV have a header line? CSV parsing works much better if there is a header.
I don't think postgres ever leaves the header in the file itself, so that wouldn't be a possibility. I thought the system was able to use column ordering for identifying which column it's working with.
Could you add a line or two of the log file you are applying this to? Thanks.
Sure thing, done!