How to group the messages by exception message present in stack_trace field of a json event?

[ghostrider]

I have json events/messages in my search result. There is a field or property called "stack_trace" in the json like below. I want to group the events and count them as shown below based on the Exception Reason or message. The problem is traces are multi lined and hence below query that I am using is, it seems not able to extract the exact exception message. Is there a way to achieve the expected output? 

Event

{

MESSAGE : Failed to send 
stack_trace : com.abc.xyz.package.ExceptionName: Missing A.
at random.package.w(DummyFile1:45)
at random.package.x(DummyFile2:64)
at random.package.y(DummyFile3:79)



}

Query I am using

MY_SEARCH | rex field=stack_trace "(?<exceptionclass>\w+): (?<exceptiontext>\w+)."
| stats count as Count by "exceptiontext"

Expected Output

Exception     Count

Missing A     3
Missing B     4
Missing C     1

[gcusello]

Hi @ghostrider,

you have to review your regex, please try this:

MY_SEARCH 
| rex field=stack_trace "ExceptionName: (?<exceptiontext>[^\.]+)"
| stats count as Count by "exceptiontext"

that you can test at https://regex101.com/r/OAJ4Iw/1

Ciao.

Giuseppe

[ghostrider]

Thank you!!. Issue is "ExceptionName" this is not same for all the exceptions. Is there a way to completely ignore this field and just get the exception message?

[gcusello]

Hi @ghostrider,

sorry but I don't understand you need: maybe you should categorize the messages to find all the regexes to extract the message.

If you could share a sample of the other kind of logs I could try to find a regex.

Ciao.

Giuseppe

[ghostrider]

Ok. So I am trying to say is currently we can have different exceptionnames in the events like below. In this case your query will not work since you are matching the ExceptionName literally. So is there any way to ignore the entire text till ":" and just extract the "Missing A" etc part?

Event 1

{

MESSAGE : Failed to send 
stack_trace : com.abc.xyz.package.ExceptionName: Missing A.
at random.package.w(DummyFile1:45)
at random.package.x(DummyFile2:64)
at random.package.y(DummyFile3:79)
}

Event 2

{

MESSAGE : Failed to send 
stack_trace : com.abc.xyz.package.OtherExceptionName: Missing B.
at random.package.w(DummyFile1:45)
at random.package.x(DummyFile2:64)
at random.package.y(DummyFile3:79)
}

{

MESSAGE : Failed to send 
stack_trace : com.abc.xyz.package.SomeOtherExceptionName: Missing C.
at random.package.w(DummyFile1:45)
at random.package.x(DummyFile2:64)
at random.package.y(DummyFile3:79)
}

Event 3 

[gcusello]

Hi @ghostrider,

you have to identify the string to capture.

Viewing your sample with the same regex, you can take all the messages as you can see at https://regex101.com/r/OAJ4Iw/2

It's not relevant if before ExceptionName there something else, it's important that there's "ExceptionName: ".

If you haven't this word it's difficoult because you have many colons in your logs so it isn't sufficient to identify the string to capture.

Ciao.

Giuseppe

[ghostrider]

Yes thanks your query is perfect. Just was curious is there any way to include in the regex one condition to extract the string till the current line only and not go to next line? Currently you are having "." as the limiting char till which we can read the string

[yuanliu]

@ghostrider have you tried just using

| rex field=stack_trace "^(?<classname>.+)\.\w+: (?<exceptiontext>.+)"

Using your sample data, output is like

classname	exceptiontext	stack_trace
com.abc.xyz.package	Missing A.	com.abc.xyz.package.ExceptionName: Missing A. at random.package.w(DummyFile1:45) at random.package.x(DummyFile2:64) at random.package.y(DummyFile3:79)

By default, rex stops at the first line.

[PickleRick]

You have to anchor your regex somewhere 🙂

Otherwise the regex processor will not know where to start or stop. That's why the constant part of ExceptionName. Regex is a simple tool which matches strings to patterns, it doesn't understand "business logic" and cannot guess what you want 😉

So you have to either anchor it with a specific constant term(s) or restrict it to a special pattern. There's no way around it. You could try extracting, for example, a second line from each matching event, but then you'd have to be sure it's always on the second line.

[gcusello]

Hi @ghostrider,

sorry but I don't understand, could you share and highlit what you want to extract?

Ciao.

Giuseppe