|
Anyone have a good working python DB table dump scripts that keeps track of last row marker? I guess it would be in-efficient for every Splunk users to come up with their own scripts. Which I already have spent quete some to the get it to work since I am not a hardcore develper, but I am not quote 100% satisfied with the result. |
|
I have written a script which periodically checks the DB (up to once a second) for new transactional entries and writes the log. Like Lowell's one it is not a product: no installer or docs, only runs with SQL server, and queries are hardcoded into a module. Maybe if there's enough interest. It writes pipe-separated key-value pairs (K1=V1|K2=V2), with the timestamp key first. The files are named like /{basepath}/{queryname}/{YYYYMMDD}.log and rolled at midnight. A "timestamp" file maintains the date of the last row retrieved for a query, whose contents is used as the start date for the next query execution. A --start YYYYMMDD argument forces it to overwrite old log files for a delete-and-reindex operation. We have got 9 queries running to pull data from timestamped database tables. They join up to 8 other tables to pull in all possible related fields. Splunk then acts like a data warehouse: denormalised read-only transactional data for charting and statistics. |
|
I have written a python library to take care of what you're talking about. And your right, not everyone should have to reinvent the wheel on this. Unfortunately, I haven't had a chance to publish a demo app yet. I'm much closer than I was, but more cleanup is needed to remove any company specific info. What platform are you running on? And which database are you trying to connect to? If I could contact you offline, I would be willing to sent you a rough draft version of the app. It would be helpful to me to have some feedback from a less python-savvy individual since I would like to may a generic app that is fairly accessible to both python pros, and novices alike. (At the moment, what I have may be over engineered, so I'd be curious in hearing your requirements.) Unfortunately, I don't think this site has a user-to-use message feature.... (which I generally don't like anyways.) 1
Clyde772, send me an email at (my user name) + "87" at gmail dot com. (I'll be deleting this comment after you do)
(18 May '10, 14:36)
Lowell ♦
|
Personally, if there were to be a shared canonical DB scripted input, I think it should be written in Java, because of the far better DB interface support.