We have a system called Metaviewer that produces XML log files. I've read several answers about indexing and trapping events from an XML file. The problem in this case is that all of the events are combined into a single line. The only time a line break even enters the picture is when one of the <SQL> elements in the XML event contains a line break. Is there a way to index multiple events from a single line?
A stripped down sample log event from Metaviewer might look like this...
<?xml version="1.0" encoding="UTF-8"?><!--Log file generated byMetafile Event Logging Server--><EventLog xmlns="http://www.metafile.com/ns/eventlog/10" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"><CreateDate>6/17/2010</CreateDate><CreateTime>13:20:00</CreateTime><Logger>servername</Logger><Events><Event><EventID>256</EventID><Description>Modify User</Description><Category>Audit</Category><Source>MetaViewer Enterprise</Source><SubSource>MVENT</SubSource><UserName>DOMAIN\user</UserName><UserID>562</UserID><Computer>servername</Computer><Date>06/17/2010</Date><Time>13:24:33</Time><ObjectType>User</ObjectType><Details><SQL>UPDATE UserInfo SET CMAnnTlbx = 0,CMArchMaint = 0,CMArchSearch = 0,CMDocClass = 0,CMDocForms = 0,CMMetaViews = 0,CMMoveStrg = 0,CMPgStore = 0,CMPrefetch = 0,CMPurgeDoc = 0,CMQueries = 0,CMShortcuts = 0,CMSysInfo = 0,CMTasks = 0,CMUsers = 0,CMWorkGrps = 0,GroupID = 36,Notes = NULL,SIDType = 0,SIDValue = NULL,SigCode = 0,Attribs = '<Variables></Variables>',InactivityTO = 0,NameFirst = NULL,NameLast = NULL,NameUser = 'UserName' WHERE UserID =575</SQL><UserID>575</UserID><UserName>UserName</UserName></Details></Event></Events></EventLog>
In a full log, multiple <Event> elements would be slapped together on the same line. The <SQL> element may contain line breaks, but that's the only time you'll ever see them. So the challenge is to index each of the <Event>s that exist on the same line, while still trapping multiple lines when necessary.
I can't even seem to get Splunk to index multiple events on a single line. Here's what I'm trying right now...
[metaviewer_audit_xml]
TIME_PREFIX = <Date>
MAX_TIMESTAMP_LOOKAHEAD = 200
BREAK_ONLY_BEFORE = <Event>
BREAK_ONLY_BEFORE_DATE = False
MUST_BREAK_AFTER = </Event>
SHOULD_LINEMERGE = True
I feel like I know why this doesn't work, but I haven't found another way of doing it. Any suggestions?
... View more