Getting Data In

Timestamp from file name

hartfoml
Motivator

I have file names like this "Patch-Data_2-1-2012.csv"

How do I use the date in the file name for the datestamp for all the lines/events in the file

This is a one shot import

Tags (2)
1 Solution

hexx
Splunk Employee
Splunk Employee

Here's what you need to do :

  • Write your own regular expression which will extract the month, the day and the year from your event source each in its own capture group.
  • Include your regex in a custom datetime.xml file. You should use $SPLUNK_HOME/etc/datetime.xml as a reference for the format to use for your own custom datetime.xml
  • Set up DATETIME_CONFIG in props.conf so that it would use your custom datetime.xml file.

Just to get you started, here's what you'll probably want to have in your own instance of datetime.xml to extract dates from your file source :



<![CDATA[(?:^|source::).*?_(0?[1-9]|1[012])-(0?[1-9]|[12]\d|3[01])-(20\d\d|19\d\d|[901]\d(?!\d))\.csv]]>



(...)


(...)


The easy way to go about this would be to copy $SPLUNK_HOME/etc/datetime.xml to your custom app and then add the regular expression definition at the end of the existing ones, and reference it under as shown above.

View solution in original post

duncanzhang
New Member

props.conf:::::
[mysourcetype]
TRANSFORMS-change_time= change_time

transforms.conf:::::
[change_time]
INGEST_EVAL = _time = strptime ( replace(source,".+(\d+-\d+-\d+).+", "\1"), "%H-%M-%S")

0 Karma

hexx
Splunk Employee
Splunk Employee

Here's what you need to do :

  • Write your own regular expression which will extract the month, the day and the year from your event source each in its own capture group.
  • Include your regex in a custom datetime.xml file. You should use $SPLUNK_HOME/etc/datetime.xml as a reference for the format to use for your own custom datetime.xml
  • Set up DATETIME_CONFIG in props.conf so that it would use your custom datetime.xml file.

Just to get you started, here's what you'll probably want to have in your own instance of datetime.xml to extract dates from your file source :



<![CDATA[(?:^|source::).*?_(0?[1-9]|1[012])-(0?[1-9]|[12]\d|3[01])-(20\d\d|19\d\d|[901]\d(?!\d))\.csv]]>



(...)


(...)


The easy way to go about this would be to copy $SPLUNK_HOME/etc/datetime.xml to your custom app and then add the regular expression definition at the end of the existing ones, and reference it under as shown above.

kiddo258
New Member

It seems a time is necessary at least.

"If no events in a source have a date, Splunk Enterprise tries to find one in the source name or file name. (This requires that the events have a time, even though they don't have a date.)"

0 Karma

gelica
Communicator

@asimagu
Hi, I've had the same problem, couldn't get it to work at all, Splunk used the modification date instead.

I ended up with overriding _time by using EVAL in props.conf. Not the most elegant solution, but it works for now.

However, it would be nicer to do it at index time as it should, so I really hope you fint a solution that works for me as well 🙂

0 Karma

asimagu
Builder

hi guys, I have tried replicating this same config and filename on Splunk 5.0.2 but it is not working for me

any known issues on Splunk 5 about this??

0 Karma
Get Updates on the Splunk Community!

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...

What's new in Splunk Cloud Platform 9.1.2312?

Hi Splunky people! We are excited to share the newest updates in Splunk Cloud Platform 9.1.2312! Analysts can ...

What’s New in Splunk Security Essentials 3.8.0?

Splunk Security Essentials (SSE) is an app that can amplify the power of your existing Splunk Cloud Platform, ...