Refine your search:

4 ratings

Thanks For Downloading!

Review the documentation below and follow any custom installation steps. If no install steps are listed, most Splunk Apps and Add-ons can be installed as follows:

Windows: Decompress the downloaded file using a tool like 7-Zip and place the resulting folder into %PROGRAMFILES%\Splunk\etc\apps. Then restart Splunk using the splunk restart command or the GUI.

Unix/Linux: Decompress the downloaded file using a tool like tar -xvf and place the resulting folder into $SPLUNK_HOME/etc/apps. Then restart Splunk using the splunk restart command or the GUI.

Description

Big Data archiving and bulk data movement for Splunk. Supports HDFS, NFS, Amazon S3, and Amazon Glacier.

Shuttl provides data archive management for Splunk. It supports backend storage solutions such as: ApacheHDFS, Amazon S3, or NFS attached storage. Shuttl works on the bucket level, and leverages the standard Splunk mechanism for archiving data based on total data size or time expiration. Use of Shuttl eliminates the need for Splunk users to implement their own homegrown solution for bulk-moving data to storage backends.

In addition to Archiving, Shuttl is useful for both compliance needs of data retention, as well as improving performance of Splunk. Shuttl also supports archiving the data in CSV format, and therefore, when data is moved to HDFS, it opens up the data to other tools such as Apache Hive and Hadoop Map Reduce to do further data processing and analysis.

For more information see the following blog articles:

  • <http://blogs.splunk.com/2012/07/02/shuttl-for-big-data-archiving/>
  • <http://blogs.splunk.com/2012/09/04/unlocking-splunk-data-with-shuttl/>

Source code is available here: <https://github.com/splunk/splunk-shuttl>

Quickstart Guide is available here: <https://github.com/splunk/splunk-shuttl/wiki/Quickstart-Guide>

Setup video is available here: <http://www.youtube.com/watch?v=OP7IYNVR5ms>

For feedback, please email shuttl-dev at splunk.com.

Versions and Release Notes

Version 0.8.4 (current version - updated May 08, 2013)
release notes:

Now comes with both coldToFrozen AND warmToCold transfer retries!
This means that all failed transfers/shuttl's will be retried periodically (default every 60 secs), instead of "everytime any bucket is shuttl'ed".

User happiness is expected to go up by 137%!

show older versions »
Version 0.8.3.1 (updated May 02, 2013)
release notes:

Automatically retries to transfer coldToFrozen failures every 60 secs (configurable)

Version 0.8.2 (updated Apr 17, 2013)
release notes:

- Fixes Amazon S3 bug, where the aws.secret couldn't contain some characters. Now it can!
- Even more reliable bucket transfers!

Version 0.6.1.1 (updated Apr 17, 2013)
Version 0.8.1 (updated Mar 11, 2013)
release notes:

* Fixed bug where if an indexer (which is a search peer) does not have Shuttl installed, the resulting error would result in missing values in the UI.
* Further tests on Hadoop 1.1.1

Version 0.8.0 (updated Feb 28, 2013)
release notes:

Splunk Shuttl 0.8.0 release is another major milestone release.

The main new feature is distributed Shuttl operations from the Splunk Search Head. This means that all Shuttl operations will operate on the entire cluster, and not on a per-indexer basis.

Actions include:
* Listing of archived buckets
* Thawing of archived buckets
* Flushing of archived buckets

There is no special configuration that needs to be done on the Search Head. Shuttl when installed on a Search Head will query the Search Head for all Search Peers (the Search Peers should all have Shuttl installed of course), and all Shuttl operations will be distributed operations.

Extensive testing has been done for distributed failure scenarios. However, if problems are encountered, please report them ASAP for them to be addressed.

Happy Shuttling!

Version 0.7.2 (updated Jan 01, 2013)
release notes:

Shuttl version 0.7.2 represents a significant leap in functionality. Thanks to our robust user base for the feature request ideas. Now, Shuttl is easier to setup, supports a new backend (Glacier), reduced archive latency, and full support for the latest Splunk 5.0.

In summary, new features include:

- Amazon Glacier storage! Amazon Glacier is an extremely low-cost storage service that provides secure and durable storage for data archiving and backup.
- Splunk 5.0 Clustering Support. Since Splunk replicates buckets, when archiving, any archiving script will by default get duplicate buckets. Shuttl will dedup buckets and not waste storage space.
- Shuttl now will move data when buckets are moved to cold. This means significantly reduced latency times, and data that is shuttled is also still available for search by Splunk!
- New simpler and easier configuration! and a script to test out backends, to verify the configuration is correct, without going through the whole exercise of indexing data and forcing a bucket roll.

Version 0.6.3 (updated Sep 11, 2012)
release notes:

* Shuttl tested on CDH3. There is a wirelevel incompatibility with CDH and Apache, so there's documents on how to configure Shuttl to work with CDH3 on the github wiki. https://github.com/splunk/splunk-shuttl/wiki/System-Requirements

* Fix spurious error message when a thaw request is issued while thaw is in progress. No duplicates happen, and all requests are serviced.

* Simplified the failed bucket transfer dashboard.

* When doing thaw operation, the UI can "return" and list a table, before the thaw action completes - Fix is to do thaw operation asynchronously, so the UI returns immediately, while job happens in background.

* Amazon S3 tested, and works with no code change - slight display issue for bucket sizes in the UI, bug filed.

Version 0.6.2 (updated Sep 07, 2012)
release notes:

Updates:
* Support for Flushing of Thawed Splunk Buckets via the UI

Known Issues being worked on:
* Quirks in the Thaw request UI
* S3 is not yet tested
* Failed buckets dashboard may report false failures

posted 06 Sep '12, 18:58

Petter%20Eriksson's gravatar image

Petter Eriksson
24612
accept rate: 42%

new version 08 May, 21:46


2 Reviews
2 reviews, 4 ratings, average 5.00

Did you find this app useful?

Reviews related to version 0.8.3.1 (current is 0.8.4)

The current download (0.8.2) is a download to the sample test app.

The previous version (by date) is labeled as 0.6.1.1 and has no description.

The version before that is 0.8.1.

I have no clue which one is the latest version.

comments (2)

reviewed 02 May, 11:28

phemmer's gravatar image

phemmer
313
accept rate: 100%

No idea why 0.8.2 is the "test app".

Just uploaded the version 0.8.3.1. It should be available here on Splunkbase soon. The latest version is the one with the highest version number. 0.8.1 > 0.6.1.1.

Sorry for the inconvenience

(02 May, 16:22) Petter Eriksson

You can try downloading 0.8.3.1 before it's updated on this site, with this link: http://splunkbase.splunk.com/apps:startdownload/Shuttl/0.8.3.1/shuttl.spl

(02 May, 16:24) Petter Eriksson
Reviews related to version 0.7.2 (current is 0.8.4)

Great and useful app!

comments (0)

reviewed 07 Nov '12, 03:14

bizza's gravatar image

bizza
54118
accept rate: 0%

Copyright © 2005-2012 Splunk Inc. All rights reserved.