How i can apply mvdedup to stats values?

[karthi2809]

Hi All,

I have a message filed having multiple success messages .I am using stats values(message) as message .So i want to show any one of the success messages in the output.For that i used below query to restrict the other message values using mvdedup. But its not filtering.

| eval Result=mvdedup(mvfilter(match(message, "File put Succesfully*") OR match(message, "Successfully created file data*") OR match(message, "Archive file processed successfully*") OR match(message, "Summary of all Batch*") OR match(message, "processed successfully for file name*") OR match(message, "ISG successful Call*") OR match(message, "Inbound file processed successfully") OR match(message, "ISG successful Call*") ) ) 

[yuanliu]

If I understand your question correctly, you want group matching messages to be displayed as a single string like “file put successfully”, not separately as "Inbound file processed successfully GL1025pcardBCAXX8595143691007",  "File put Succesfully GL1025pcardBCAXX8595143691007", and so on.  This is a common requirement.  But in addition to unnecessary asterisks in regex's as @ITWhisperer points out, you should group them before performing stats.  Here is the code

| eval message = if(match(message, "File put Succesfully|Successfully created file data|Archive file processed successfully|Summary of all Batch|processed successfully for file name|ISG successful Call|Inbound file processed successfully|ISG successful Call"),
  "file put successfully", message)
| stats values(message) as message

Suppose you have events with the following values of message:

message

Inbound file processed successfully GL1025pcardBCAXX8595143691007

Inbound file processed successfully GL1025pcardBCAXX8595144691006

Inbound file processed successfully GL1025pcardBCAXX8732024191001

Inbound file processed successfully GL1025transBCAXX8277966711002

File put Succesfully GL1025pcardBCAXX8595143691007

File put Succesfully GL1025pcardBCAXX8595144691006

File put Succesfully GL1025pcardBCAXX8732024191001

File put Succesfully GL1025transBCAXX8277966711002

some unmatching value

some other unmatching value

The result will be

message

file put successfully some other unmatching value some unmatching value

Is this what you are looking for?

Here is an emulation that you can play with and compare with real data

| makeresults
| eval message = mvappend("Inbound file processed successfully GL1025pcardBCAXX8595143691007",
"Inbound file processed successfully GL1025pcardBCAXX8595144691006",
"Inbound file processed successfully GL1025pcardBCAXX8732024191001",
"Inbound file processed successfully GL1025transBCAXX8277966711002",
"File put Succesfully GL1025pcardBCAXX8595143691007",
"File put Succesfully GL1025pcardBCAXX8595144691006",
"File put Succesfully GL1025pcardBCAXX8732024191001",
"File put Succesfully GL1025transBCAXX8277966711002",
"some unmatching value",
"some other unmatching value")
| mvexpand message
``` data emulation above ```

[ITWhisperer]

match uses regex so the * at the end of each string is probably superfluous (unless you were matching for "File put Succesfull" or "File put Succesfullyyyyy")

Other than that, it looks like your mvdedup mvfilter should work. Please can you share some example events for which this is not working?

[karthi2809]

My Output :

Inbound file processed successfully GL1025pcardBCAXX8595143691007
Inbound file processed successfully GL1025pcardBCAXX8595144691006
Inbound file processed successfully GL1025pcardBCAXX8732024191001
Inbound file processed successfully GL1025transBCAXX8277966711002
File put Succesfully GL1025pcardBCAXX8595143691007
File put Succesfully GL1025pcardBCAXX8595144691006
File put Succesfully GL1025pcardBCAXX8732024191001
File put Succesfully GL1025transBCAXX8277966711002

In OR condition i mentioned both the keywords.why because some of the messages fields dont have Fileput successfully .That y i gave both the strings in the mvdedup

[ITWhisperer]

Your output looks correct. Is it not what you expected? If not, what did you expect?

[karthi2809]

Hi @ITWhisperer 

The output is correct I want any one the result in my output.either  “file put successfully ” or “inbound file processed”.but it showing both right.so that I want to dedup.

[ITWhisperer]

OK so I'll ask again another way, what output would you like, for example from the 8 lines you shared earlier?