Best practices for scalable application development

Splunk provides great tools it its web interface for quickly creating an application, saving searches, adding dashboards, etc. What tools or best practices support development and maintenance of larger applications?

This summer I was tasked to create a "metrics application," which will expose summary data extracted from our log files. The list of metrics is 15 pages long.

Each individual view was easy to create, but it became difficult to keep them consistent, particularly when changes were requested.

• "Add a link on every page to ... ."
• "Display service times as hours instead of minutes."
• "Make all the charts a bit taller."

Most of the views have a similar structure, but differ in the details: the form fields, the searches, the chart labels, etc. There seems to be no way to abstract the overall page layout away from the detailed contents of each page. The advanced XML views, in particular, expose every tiny detail of the page layout; any change must be carefully repeated in every advanced XML file.

I would welcome hints from others who have developed large Splunk applications.