How do I set up SSL forwarding with new, self-signed certificates and authentication?

Here is a detailed procedure to use non-default (in this case, self-signed) SSL certificates with common-name-based authentication (for the indexer(s) only) in a splunk2splunk (indexer to forwarder) connection.

1 - On a Splunk instance, create your own root certificate :

(As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate_a_new_root_certificate)

NOTE : For clarity's sake, it is better to generate new certificates in another directory than $SPLUNK_HOME/etc/auth in order not to overwrite those that exist there. In our example, we will create and use $SPLUNK_HOME/etc/certs. This will also ensure that you can keep using the certificates that ship with Splunk in $SPLUNK_HOME/etc/auth for other Splunk components if you wish to do so.

# mkdir $SPLUNK_HOME/etc/certs

Point openssl to Splunk's openssl.cnf :

# export OPENSSL_CONF=$SPLUNK_HOME/openssl/openssl.cnf

Generate a new root certificate :

# $SPLUNK_HOME/bin/genRootCA.sh -d $SPLUNK_HOME/etc/certs

=> This will create a new certificate authority public certificate in $SPLUNK_HOME/etc/certs/cacert.pem This public CA certificate is to be distributed to all Splunk instances (indexers and forwarders) who will be checking server certificates signed with the root certificate we just generated (ca.pem).

2 - Generate a new self-signed server certificate for your indexer, specifying your indexer's host name as the common name recorded in the certificate :

(As per http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL#Generate_a_new_signed_certificate_and_private_key_pair)

In our example, let's assume that your indexer's host name is "splunk-idx-01.example.com". Let's also assume that you want to use "changeme" as the password for your indexer's server certificate.

# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n splunk-idx-01 -c splunk-idx-01.example.com -p

• Enter "changeme" as the PEM pass phrase.
• Enter "splunk-idx-01.example.com" as the common name

• All other values are optional.

  => This will create a new server certificate in $SPLUNK_HOME/etc/certs/splunk-idx-01.pem. We will copy this server certificate to the indexer "splunk-idx-01.example.com, who will present it to the forwarders. The forwarders will use the public CA certificate from step 1 (cacert.pem) to check the identity of the indexer.

3 - Generate a new self-signed server certificate for your forwarders :

In our example, let's assume that you want to use "changeme2" as the password for your forwarders' server certificates.

# $SPLUNK_HOME/bin/genSignedServerCert.sh -d $SPLUNK_HOME/etc/certs -n forwarder -p

• Enter "changeme2" as the PEM pass phrase.

• All other values are optional.

  => This will create a new server certificate in $SPLUNK_HOME/etc/certs/forwarder.pem. We will propagate this server certificate to all of our forwarders, who will present it to the indexer(s) when a connection is established. The indexer(s) will use the public CA certificate we generated in step 1 (cacert.pem) to check the credentials of the forwarders.

4 - Set up the indexer to use the newly created server certificate and to check the forwarders' certificates :

First, copy the $SPLUNK_HOME/etc/certs/splunk-idx-01.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your indexer and put them in a newly created $SPLUNK_HOME/etc/certs directory.

We will assume here that you will be using port 9997 to receive data from your forwarders.

In $SPLUNK_HOME/etc/system/local/inputs.conf, set up the following stanzas :

[SSL]
rootCA = $SPLUNK_HOME/etc/certs/cacert.pem
serverCert = $SPLUNK_HOME/etc/certs/splunk-idx-01.pem
password = changeme
requireClientCert = false

[splunktcp-ssl:9997]
compressed = true

Important note regarding requireClientCert: As of Splunk 4.2.4, setting requireClientCert = true in the indexer's inputs.conf will cause forwarding to fail! A bug (SPL-37637) is currently open to address this issue. In the meantime, keep requireClientCert set to "false".

Setting "requireClientCert = true" would require the following conditions to be met :

• "rootCA" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1.
• The forwarder's server certificate defined by "sslCertPath" in outputs.conf (see step 5) is signed by that CA.

• The forwarder has the password to read his own certificate ("sslPassword" in outputs.conf, as defined in step 5). This password is "changeme2" in our example.

  The purpose of this setup is to ensure that only forwarders that you have distributed a signed certificate to can connect to this indexer.

  Restart Splunk after making these changes. Note that this will hash the certificate password in inputs.conf.

# $SPLUNK_HOME/bin/splunk restart

5 - Set up your forwarder(s) to use the newly created server certificates and to check the indexer's certificate with common name authentication :

Copy the $SPLUNK_HOME/etc/certs/forwarder.pem and $SPLUNK_HOME/etc/certs/cacert.pem files to your forwarder(s) and put them in a newly created $SPLUNK_HOME/etc/certs directory.

Then, define the following stanzas in $SPLUNK_HOME/etc/system/local/outputs.conf. Let's assume that your indexer's IP address is 192.168.1.100 :

[tcpout]
defaultGroup = splunkssl

[tcpout:splunkssl]
server = 192.168.1.100:9997
compressed = true

[tcpout-server://192.168.1.100:9997]
sslRootCAPath = $SPLUNK_HOME/etc/certs/cacert.pem
sslCertPath = $SPLUNK_HOME/etc/certs/forwarder.pem
sslPassword = changeme2
sslVerifyServerCert = true
sslCommonNameToCheck = splunk-idx-01.example.com
altCommonNameToCheck = splunk-idx-01

Note that we have set "sslVerifyServerCert = true". This requires the following conditions to be met :

• "sslRootCAPath" must point to a file containing the CA's public key, without a password. In our example, it's the cacert.pem file we generated in step 1.
• The indexer's "serverCert" (as defined in inputs.conf on step 4) is signed by that CA.
• The indexer has the password to read his own certificate ("password" in inputs.conf, as defined in step 4). This password is "changeme" in our example.

• The common name on the indexer's server certificate ("serverCert" in inputs.conf, as defined in step 4) matches one of the "sslCommonNameToCheck" settings.

  This setup ensures that the client/forwarder only forwards to someone with a certificate signed by the CA and that that certificate was issued to someone with the expected name, i.e., so that the client only sends to someone to whom that certificate is specific. The name doesn't have to match the DNS, hostname, servername, or IP or anything outside of these files. (Thus this is not subject to DNS SSL attacks). All that needs to match is the name in the Splunk configuration file and the name signed into the indexer's certificate.

  If you are in a distributed environment and therefore have one server certificate per indexer, set up one [tcpout-server:...] stanza per indexer with "sslServerCertPath" pointing to each individual server certificate.

  Finally, restart the forwarder :

# $SPLUNK_HOME/bin/splunk restart

6 - You should now see the indexer and the forwarder establishing an SSL connection. Here's what it looks like in $SPLUNK_HOME/var/log/splunk/splunkd.log on both sides.

• on the indexer (with category.TcpInputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) :

(During start up...)

09-21-2010 18:47:29.735 DEBUG TcpInputProc - Initializing
09-21-2010 18:47:29.735 INFO  TcpInputProc - using queueSize 1000
09-21-2010 18:47:29.735 DEBUG TcpInputProc - creating tcp pipelineData queue
09-21-2010 18:47:29.735 DEBUG TcpInputProc - Reconfiguring
09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - clearing maps
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop enables2sHeartbeat=true
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop s2skeepaliveTimeout=600
09-21-2010 18:47:29.735 DEBUG TcpInputProc - global prop inputShutdownTimeout=90
09-21-2010 18:47:29.735 DEBUG TcpInputProc - readConfig - scanning configs
09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL serverCert=/opt/splunk/etc/certs/splunk-idx-01.pem
09-21-2010 18:47:29.735 DEBUG TcpInputProc - SSL rootCA=/opt/splunk/etc/certs/cacert.pem
09-21-2010 18:47:29.737 INFO  TcpInputProc - SSL cipherSuite=ALL:!aNULL:!eNULL:!LOW:!EXP:RC4+RSA:+HIGH:+MEDIUM
09-21-2010 18:47:29.737 INFO  TcpInputProc - supporting SSL v2/v3
09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL dhfile=
09-21-2010 18:47:29.737 DEBUG TcpInputProc - SSL requireClientCert=1
09-21-2010 18:47:29.740 INFO  TcpInputProc - port 61025 is reserved for splunk 2 splunk (SSL)
09-21-2010 18:47:29.740 INFO  TcpInputProc - Port 61025 is compressed
09-21-2010 18:47:29.740 DEBUG TcpInputProc - readConfig - creating acceptor for port 61025
09-21-2010 18:47:29.740 DEBUG TcpInputProc - Initing Acceptor with  SSL
09-21-2010 18:47:29.740 INFO  TcpInputProc - Registering metrics callback for: tcpin_connections

(...then when the forwarder connects)

09-21-2010 18:51:27.812 INFO  TcpInputProc - Connection in cooked mode from splunk-fwd-01.example.com
09-21-2010 18:51:27.816 INFO  TcpInputProc - Valid signature found
09-21-2010 18:51:27.816 INFO  TcpInputProc - Connection accepted from splunk-fwd-01.example.com

• on the forwarder (with category.TcpOutputProc=DEBUG in $SPLUNK_HOME/etc/log.cfg) :

(During start up...)

09-22-2010 16:22:12.017 DEBUG TcpOutputProc - groupName=[splunkssl] server=[192.168.1.100:9997]
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - creating group mapping for group splunkssl
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Validation complete
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : initialBackoff=2
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxNumberOfRetriesAtHighestBackoff=-1
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : maxBackoff=20
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Setting : backoffAtStartup=5
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will retry at max backoff sleep forever
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop maxNumberOfRetriesAtHighestBackoff=-1
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Using SSL for server 10.1.12.1:61025, sslCertPath=/opt/splunk/etc/certs/forwarder.pem
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - Key file password requires decrypting
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslRootCAPath=/opt/splunk/etc/certs/cacert.pem
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslVerifyServerCert=1
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will verify Server's Certificate
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslCommonNameToCheck=splunk-idx-01.example.com
09-22-2010 16:22:12.017 INFO  TcpOutputProc - Will check server's Common Name against splunk-idx-01.example.com
09-22-2010 16:22:12.017 DEBUG TcpOutputProc - prop sslAltNameToCheck=splunk-idx-01
09-22-2010 16:22:12.017 INFO  TcpOutputProc - ALL Connections will use SSL with sslCipher=
09-22-2010 16:22:12.018 INFO  TcpOutputProc - initializing single connection with retry strategy for 192.168.1.100:9997
09-22-2010 16:22:12.018 DEBUG TcpOutputProc - creating sender thread for group ssltest
09-22-2010 16:22:12.018 DEBUG TcpOutputProc - registering timer callback
09-22-2010 16:22:12.019 DEBUG TcpOutputProc - IndexAndForward explicitly set to : false
09-22-2010 16:22:12.019 DEBUG TcpOutputProc - returning from updateConfig
09-22-2010 16:22:12.020 INFO  TcpOutputProc - attempting to connect to 192.168.1.100:9997...
09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validating URI - 192.168.1.100:9997
09-22-2010 16:22:12.020 DEBUG TcpOutputProc - Validation complete
09-22-2010 16:22:12.029 INFO  TcpOutputProc - Connected to 192.168.1.100:9997 
09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Sending signature
09-22-2010 16:22:12.029 DEBUG TcpOutputProc - Signature sent

(...then, as the forwarder keeps the connection alive about every 30 seconds...)

09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Inserting timer token into the queue.
09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Processing timer event from the queue.
09-22-2010 16:27:21.953 DEBUG TcpOutputProc - Size of sockets is: 1
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Inserting timer token into the queue.
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Processing timer event from the queue.
09-22-2010 16:27:52.958 DEBUG TcpOutputProc - Size of sockets is: 1

7 - Troubleshooting : If things don't work out right!

a) First, check in $SPLUNK_HOME/var/log/splunk/splunkd.log on both ends for errors. On the indexer, check for the messages from the TCP input processor "TcpInputProc", and on the forwarder, check the messages from the TCP output processor "TcpOutputProc".

b) If necessary, increase the logging level of the appropriate processors on the indexer and the forwarder in $SPLUNK_HOME/etc/log.cfg.

On the forwarder, set "category.TcpOutputProc=DEBUG", on the indexer set "category.TcpInputProc=DEBUG". Restart Splunk for these to take effect.

c) Check the SSL configuration as it is seen by Splunk using btool.

• On the indexer :

$SPLUNK_HOME/bin/splunk cmd btool inputs list --debug

• On the forwarder :

$SPLUNK_HOME/bin/splunk cmd btool outputs list --debug

d) Make sure that the certificates are readable by the user that Splunk runs as. Indexer-side, two common problems are :

• The path to the server certificate file set as the value of "serverCert" in inputs.conf is wrong, or the file cannot be read. This will generate the following error :

12-16-2010 16:07:30.965 ERROR SSLCommon - Can't read certificate file /opt/splunk/etc/auth/server.pem errno=33558530 error:02001002:system library:fopen:No such file or directory

• The password to the RSA private key contained in the server certificate file is wrong. This password is set as the value of "password" in inputs.conf. This will generate the following error :

12-07-2010 07:56:45.663 ERROR SSLCommon - Can't read key file /opt/splunk/etc/auth/server.pem

On *nix, you can manually test the password of the RSA key contained in the file by running the following openssl command :

# openssl rsa -in /opt/splunk/etc/auth/server.pem -text

The same can be done on Windows with the openssl binary that ships with Splunk :

C:\Program Files\Splunk\bin>openssl.exe rsa -in "c:\Program Files\Splunk\etc\auth\server.pem" -text

e) More information regarding the configuration of splunk2splunk SSL connections can be found here in the online documentation :

http://www.splunk.com/base/Documentation/latest/Admin/EncryptandauthenticatedatawithSSL
http://www.splunk.com/base/Documentation/latest/Admin/SecureaccesstoyourSplunkserverwithSSL

The Splunk community wiki also has detailed tutorials on how to set up SSL for forwarding with 3 different scenarios :

• Configuring Splunk forwarding to use the default SSL server certificate
• Configuring Splunk forwarding to use SSL certificates self-signed by a newly generated root certificate
• Configuring Splunk forwarding to use SSL certificates signed by a third party Certificate Authority