Getting Data In

What kind of compression is used between forwarders and indexers?

bobwalden
Explorer

Just curious to know the format (gzip?) of the compression, and also the compression factor that can be expected?

Thanks,
bw

1 Solution

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.

View solution in original post

hexx
Splunk Employee
Splunk Employee

This mostly depends on one fact: Are you forwarding using SSL or not?

  • If you are not using SSL to forward, there is no compression enabled by default, so you'll have to set compressed = true in the appropriate stanzas of the indexer's inputs.conf:

compressed = [true|false]
* Specifies whether receiving compressed data.
* If set to true, the forwarder port(s) should also have compression turned on; otherwise, the receiver will reject the connection.
* Defaults to false.

... and in the forwarder's outputs.conf:

compressed = [true|false]
* Applies to non-SSL forwarding only. For SSL useClientSSLCompression setting is used.
* If true, forwarder sends compressed data.
* If set to true, the receiver port must also have compression turned on (in its inputs.conf file).
* Defaults to false.

In this configuration, splunkd performs what is sometimes referred to as "native compression". The zlib library will be used to compress outgoing data chunks individually. Please note that this is particularly inefficient when the sender is a heavy/regular forwarder, as each data chunk represents one event and therefore is too small to be heavily compressed. For universal/lightweight forwarders, compression is significantly more efficient due to the 64KB size of the data chunks processed and sent out.

  • If you are using SSL to forward, you will benefit from zlib compression by default, courtesy of OpenSSL. The zlib library is also used here but the compression ratio will typically be a lot higher than for "native" compression because zlib will operate on much larger chunks of data.
Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...