Data Masking Before Ingestion

[anandhalagaras1]

Hi Team,

Want to mask two of the fields "password" and "cpassword" from the events which are getting written with the plain text information. So needs to be changed as #####.

Sample event information:

[2024-01-31_07:58:28] INFO : REQUEST: User:abc CreateUser POST: name: AB_Test_Max;email: xyz@gmail.com;password: abc12345679;cpassword: abc12345679;role: User;

[2024-01-30_14:05:42] INFO : REQUEST: User:xyz CreateUser POST: name: Math_Lab;email: abc@yahoo.com;password: xyzab54;cpassword: xyzab54;role: Admin;

So kindly help with the props.conf so that i can apply with SEDCMD-mask.

[marnall · Accepted solution]

Could you try this SEDCMD in the props.conf file? (Make sure that the stanza is changed to match the sourcetype of the logs)

[your_sourcetype]
SEDCMD-maskpasswords = s/password: ([^;]+);cpassword: ([^;]+);/password: ####;cpassword: ####;/g

[gcusello]

Hi @anandhalagaras1,please try this:

[your_sourcetype]
SEDCMD = s/password: ([^;]+);cpassword: ([^;]+);/password: (####);cpassword: (####);/gm

that you can test at https://regex101.com/r/ppaFZc/1

Ciao.

Giuseppe

[anandhalagaras1]

@gcusello 

We had two requirements for the same sourcetype. One involved line breaks, and the other required password masking during ingestion. As our Search heads are managed by Splunk Support and hosted in the Cloud, we created a custom app and deployed the props.conf in the default directory. After uploading the apps for the cloud vetting process, they were successfully installed. However, I've noticed that the logs are now being separated into individual events, which is acceptable, but the passwords are still visible and haven't been masked according to our requirement. I'm unsure about where exactly I may have missed it.

This is the props.conf file for reference. 

[sourcetype]
SHOULD_LINEMERGE = false
SEDCMD = s/password: ([^;]+);cpassword: ([^;]+);/password: (####);cpassword: (####);/gm

Sample log for reference: 

[2024-03-01_06:32:08] INFO : REQUEST: User:abc CreateUser POST: name: xyz;email: abc@gmail.com;password: xyz@123;cpassword: xyz@123;role: Administrator;

So kindly help on this requirement.

[gcusello]

Hi @anandhalagaras1,

regex substitution is correct.

Are you sure about the sourcetype?

is there any sourcetype replacement in your data?

are there some other Heavy Forwarders  before the one you used for the masking?

Ciao.

Giuseppe

[anandhalagaras1]

@gcusello ,

This is the exact and correct sourcetype and I have created a custom app and uploaded the App in our Search head. Since our Search head is hosted in Splunk Cloud managed by Support.

So I have uploaded the app in the upload app section and post vetting process completed i have installed the custom app into the Search head.

This is the custom app i have created "abc_app"

Under abc_app I have placed two folders "default" and "metadata"

Under default I have created the app.conf and props.conf

And under metadata I have created the default.metadata 

Refer screenshots for reference.

So kindly let me know where i am missing since the lines are getting segregated as separate events whereas password masking is not getting applied to the events. Hence kindly help on the same.

[gcusello]

Hi @anandhalagaras1,

what's the sourcetype to apply the masking?

I suppose that sourcetype in the props.conf stanza header it's only for example and that in your installation you have the correct sourcetype to apply the transformation.

ciao.

Giuseppe

[anandhalagaras1]

@gcusello  Indeed, I have applied the correct sourcetype there to ensure that events are appropriately divided. Nonetheless, the masking of passwords is not taking place as intended.

[anandhalagaras1]

@gcusello ,

Any inputs from your end since still i can see the events are getting ingested with the password information present in it.