I tried to get my indexer and forwarders communicating using SSL by following this guide. However, I found that I couldn't get it working without just throwing all of the certificates into auth folder. If I do that it works, but if I try to put the certificates outside of that folder it doesn't. The problem is that updating splunk will overwrite that folder, so I'm trying to keep my certificates in a different folder, which should be possible. I've tried setting the variables I can find to point to the new certificate location, but it provides the following error in splunkd.log and doesn't forward data.
12-02-2015 12:01:32.070 -0500 ERROR SSLCommon - Can't read key file /opt/splunkforwarder/etc/certs/server.pem errno=101077092 error:06065064:digital envelope routines:EVP_DecryptFinal_ex:bad decrypt.
Because server.pem works fine inside of the auth folder, I'm guessing that the problem is with supporting files. So far I've tried using outputs.conf sslCertPath and sslRootCAPath, as well as server.conf caPath, sslKeysfile, and caCertFile. Below is the contents of my auth folder.
-rwxr-x--- 1 splunk splunk 3050 Dec 2 12:54 ca.pem
-rwxr-x--- 1 splunk splunk 17 Dec 2 12:54 ca.srl
-rwxr-x--- 1 splunk splunk 1216 Dec 2 12:54 cacert.pem
-rwxr-x--- 1 splunk splunk 1834 Dec 2 12:54 cakey.pem
-rwxr-x--- 1 splunk splunk 1013 Dec 2 12:54 careq.pem
-rw------- 1 splunk splunk 1041 Dec 2 12:54 privKeySecure.pem
-rw------- 1 splunk splunk 566 Dec 2 12:54 req.pem
-rwxr-x--- 1 splunk splunk 4386 Dec 2 12:54 server.pem
-r-------- 1 splunk splunk 255 Dec 2 12:54 splunk.secret
drwx------ 2 splunk splunk 512 Dec 2 12:54 splunkweb
I provided ca.pem, ca.srl, cacert.pem, cakey.pem, careq.pem, server.pem from the certificate generation process; I mirrored all of my certificate names with the default file names. Those files (and only those files) are in the certs folder, which is the folder I made that I want to read certs from. Is there some other Certificate location I'm failing to point to? Or is this a problem somewhere else?
Here's my working outputs.conf:
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true
[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/auth/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/auth/cacert.pem
sslVerifyServerCert = true
Here's my working server.conf:
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk
[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3
Here's my failing outputs.conf:
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true
[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/certs/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/certs/cacert.pem
sslVerifyServerCert = true
Here's my failing server.conf:
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk
[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3
sslKeysfile = server.pem
caCertFile = cacert.pem
caPath = /opt/splunkforwarder/etc/certs
I ended up moving my certificate files into the auth folder to replace the existing files, and then copying the entire auth folder into a second folder I named macbAuth. I did not copy the splunkweb folder from the auth folder; I lacked the necessary permissions. However it seems those files aren't needed. The following configuration worked.
Here's my file structure:
C:\Program Files\SplunkUniversalForwarder\etc\macbAuth>dir
Volume in drive C is New Volume
Volume Serial Number is 3A13-D7AF
Directory of C:\Program Files\SplunkUniversalForwarder\etc\macbAuth
12/07/2015 11:07 AM <DIR> .
12/07/2015 11:07 AM <DIR> ..
11/12/2015 10:50 AM 1,024 .rnd
10/30/2015 04:08 AM 3,908 appsCA.pem
11/12/2015 10:32 AM 3,050 ca.pem
10/30/2015 04:08 AM 1,860 ca.pem.default
11/12/2015 10:51 AM 17 ca.srl
11/12/2015 10:32 AM 1,216 cacert.pem
10/30/2015 04:08 AM 928 cacert.pem.default
11/12/2015 10:32 AM 1,834 cakey.pem
11/12/2015 10:32 AM 1,013 careq.pem
12/03/2015 02:43 PM 1,041 privKeySecure.pem
12/03/2015 02:43 PM 570 req.pem
11/12/2015 10:51 AM 4,386 server.pem
12/03/2015 02:43 PM 255 splunk.secret
12/03/2015 02:43 PM <DIR> splunkweb
13 File(s) 21,102 bytes
3 Dir(s) 138,427,678,720 bytes free
Here's my outputs.conf:
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true
[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/macbAuth/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/macbAuth/cacert.pem
sslVerifyServerCert = true
Here's my server.conf:
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk
[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3
sslKeysfile = server.pem
caCertFile = cacert.pem
caPath = /opt/splunkforwarder/etc/macbAuth
I ended up moving my certificate files into the auth folder to replace the existing files, and then copying the entire auth folder into a second folder I named macbAuth. I did not copy the splunkweb folder from the auth folder; I lacked the necessary permissions. However it seems those files aren't needed. The following configuration worked.
Here's my file structure:
C:\Program Files\SplunkUniversalForwarder\etc\macbAuth>dir
Volume in drive C is New Volume
Volume Serial Number is 3A13-D7AF
Directory of C:\Program Files\SplunkUniversalForwarder\etc\macbAuth
12/07/2015 11:07 AM <DIR> .
12/07/2015 11:07 AM <DIR> ..
11/12/2015 10:50 AM 1,024 .rnd
10/30/2015 04:08 AM 3,908 appsCA.pem
11/12/2015 10:32 AM 3,050 ca.pem
10/30/2015 04:08 AM 1,860 ca.pem.default
11/12/2015 10:51 AM 17 ca.srl
11/12/2015 10:32 AM 1,216 cacert.pem
10/30/2015 04:08 AM 928 cacert.pem.default
11/12/2015 10:32 AM 1,834 cakey.pem
11/12/2015 10:32 AM 1,013 careq.pem
12/03/2015 02:43 PM 1,041 privKeySecure.pem
12/03/2015 02:43 PM 570 req.pem
11/12/2015 10:51 AM 4,386 server.pem
12/03/2015 02:43 PM 255 splunk.secret
12/03/2015 02:43 PM <DIR> splunkweb
13 File(s) 21,102 bytes
3 Dir(s) 138,427,678,720 bytes free
Here's my outputs.conf:
[tcpout]
defaultGroup = splunkssl
[tcpout:splunkssl]
server = 129.52.27.30:9997
compressed = true
[tcpout-server://129.52.27.30:9997]
sslAltNameToCheck = winsplunk
sslCertPath = /opt/splunkforwarder/etc/macbAuth/server.pem
sslCommonNameToCheck = winsplunk
sslPassword = totallyFunctionalHash
sslRootCAPath = /opt/splunkforwarder/etc/macbAuth/cacert.pem
sslVerifyServerCert = true
Here's my server.conf:
[lmpool:auto_generated_pool_forwarder]
description = auto_generated_pool_forwarder
quota = MAX
slaves = *
stack_id = forwarder
[lmpool:auto_generated_pool_free]
description = auto_generated_pool_free
quota = MAX
slaves = *
stack_id = free
[general]
pass4SymmKey = AnotherFunctionalHash
serverName = afemssplunk
[sslConfig]
sslKeysfilePassword = SameHashAsOutputs.confSSLPassword,CauseThey'reTheSamePassword
cipherSuite = TLSv1+HIGH:@STRENGTH
sslVersions = tls,-ssl2,-ssl3
sslKeysfile = server.pem
caCertFile = cacert.pem
caPath = /opt/splunkforwarder/etc/macbAuth
That guide is from 5 years ago. Splunk has more than likely changed since then 🙂
Check the documentation here:
http://docs.splunk.com/Documentation/Splunk/6.3.1/Security/Aboutsecuringdatafromforwarders (or whichever versionyou are using - theres a version picker at the top right of the page)
Your configs look different to the current config stanzas
You might go through this as reference.
http://www.georgestarcher.com/wp-content/uploads/2015/09/conf2015_DWaddle_DefensePointSecurity_deplo...