did something change with the audit file keys between 4.2 and 4.3

[imacdonald2]

I tried copying over the audit keys from a 4.2 box to a 4.3 box and I am getting

 Last few lines of stderr (may contain info on assertion failure, but also could be old):
    2012-05-31 15:20:17.658 -0400 splunkd started (build 123586)
    terminate called after throwing an instance of 'RSAException'
      what():  Encrypt/Decrypt Exception: error:0906D06C:PEM routines:PEM_read_bio:no start line
    2012-05-31 15:21:34.142 -0400 splunkd started (build 123586)
    2012-05-31 15:26:22.825 -0400 Interrupt signal received
    2012-05-31 15:26:59.047 -0400 splunkd started (build 123586)
    terminate called after throwing an instance of 'RSAException'
      what():  Encrypt/Decrypt Exception: error:0906D06C:PEM routines:PEM_read_bio:no start line
    2012-05-31 15:28:55.387 -0400 splunkd started (build 123586)
    terminate called after throwing an instance of 'RSAException'
      what():  Encrypt/Decrypt Exception: error:0906D06C:PEM routines:PEM_read_bio:no start line

[feorlen · Accepted solution]

The pem files are generated for each Splunk instance on first start, so they aren't portable.

Basically that message is saying that splunkd can't decrypt something already encrypted with the certs it has. It's almost certainly the sequence number used for audit signing. If you don't have the originals and need to get audit signing working again, you can remove the encrypted sequence number in persistent storage:

$SPLUNK_HOME/var/lib/splunk/persistentstorage/seqno_db

and a new one will be created on startup, using the new certs. (Shut down to do this.) Once the original certs are gone, you won't be able to verify any of the old audit events however.

For the moment, this decryption failure results in an assert and splunkd terminates rather ungracefully. I'm looking at some ways to make that more friendly. But it only happens if something has gone wrong with the certs themselves.

[feorlen]

I've not tried with multiple instances this way, I'd be interested to see how it goes. I ran into this recently when a customer manually re-generated their certs and had the problem you saw.

[imacdonald2]

Thanks that explains a lot. We have multiple search heads and I need to be able to run the index=_audit | audit command, which has results from all three. since all the _audit trails feed into the same indexer, I thought I could just put the same pem files on all the search heads and be able to audit the data from a single location. I just turned on auditing so I don't to much about existing data.