Splunk MySQL Connector -> Can do this?

I have a table of ~11 million rows and 53 columns. The table looks like:

entity_id | week_1 | week_2 | ..... | week_52 abcd | 35874 | 587489 |.......| 5478

While we discover new entities with splunk the table can grow up to ~30 million rows.

I was wondering the following: -. What is the maximum number of lookup output fields that MySQL connector can support? -. Can I have up to 52 or 365 look up output fields? -. what is the recommended maximum number of rows a lookup table can have? -. If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL? -. In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude? -. Is there any other approach you might recommend?

Thanks, Lp

asked 09 Feb '12, 04:55

lpolo
678●4●15
accept rate: 13%

One Answer:

oldest newest most voted

What is the maximum number of lookup output fields that MySQLconnector can support?
There is no inherent maximum number of output fields - if MySQL can support it so can the MySQL connector
Can I have up to 52 or 365 look up output fields?
Yes, you should be able to
what is the recommended maximum number of rows a lookup table can have?
This depends in a lot of things such as the MySQL server spec, MySQL tuning, schema design etc. With a commodity server and some tuning you should be able to get decent performance with hundreds of millions of rows
If new entities and weekly data are discovered can Splunk MySQL connector insert the new entities and weekly data in the lookup table that is stored in MySQL?
Yes, you should be able to - look in mysqloutput command
In case we need to update, delete or insert some weekly data can Splunk MySQL connector perform updates, delete, and insert of this magnitude?
Yes, you can use scheduled searches in Splunk to execute any table maintenance, look into mysqlquery command
Is there any other approach you might recommend?
As far as maintenance of the tables you can also use cron jobs to run maintenance scripts

link

answered 09 Feb '12, 09:26

Ledion Bitincka ♦
2.0k●4●7
accept rate: 33%

Post your answer

toggle preview

Follow this question

Email:

RSS:

Answers

Answers + Comments

Markdown Basics

*italic* or _italic_
**bold** or __bold__
link:[text](http://url.com/ "Title")
image?![alt text](/path/img.jpg "Title")
numbered list: 1. Foo 2. Bar
to add a line break simply add two spaces to where you would like the new line to be.
basic HTML tags are also supported

learn more about Markdown

Tags:

Asked: 09 Feb '12, 04:55

Seen: 891 times

Last updated: 09 Feb '12, 09:26

Splunk MySQL Connector -> Can do this?

Follow this question

Splunk Resources

Related questions