Co-locate the deployment server with other splunk HF

[VK18]

Hi All,

We have approximately 100 Splunk Universal Forwarders (UFs) installed at a remote site, and we're interested in setting up a Heavy Forwarder (HF) at that location to forward the data to the indexers from the UFs. Additionally, we plan to deploy the deployment server on the same virtual machine (VM).

Based on the documentation, it appears that a deployment server can be co-located with another Splunk Enterprise instance as long as the deployment client count remains at or below 50.

We would like to better understand the rationale behind this limitation of 50 clients and why it is not possible to manage more than 50 clients by adding another component of Splunk Enterprise ?

Regards
VK

[PickleRick]

It's a rough estimate. It's just an assumption that 50 clients should not generate such load on the DS that it would interfere with other functionalities.

I must say though that combining a HF and DS is a very unusual architecture choice.

Also - why do you want to set up a HF here? If you absolutely need an intermediate forwarder (with the traditional remarks about load-balancing or rather lack thereof), why not an UF? Remember that HF will parse your events and send the data as parsed which multiplies the needed bandwidth.

[gcusello]

Hi @VK18,

this is a precise requirement from Splunk related to the fact that the HF could have overloading managing more tham 50 clients and its normal job as HF.

Probably you see that CPUs and RAM aren't overloaded on this HF, but they could be with relevant impact on your log ingestion because DS and HF use very muche the network interface and managing 100 clients is heavy for that server's network interface.

In addition, you spoke of 100 clients not few more than 50, so I'd avoid to use both the roles in the same machine.

If it's mandatory for you this architecture, give more resources (CPUs and RAM to that server and analyze the network activity because this could be the bottleneck.

As last consideration, if you'll have problems on that server, this will be the first annotation from Splunk Support.

Ciao.

Giuseppe

[VK18]

Hi Giuseppe,

Thank you for your response.

Indeed, it has become necessary due to the customer services residing in an OT network with an air gap environment. They aim to manage both the HF and DS roles on a single virtual machine and are willing to allocate additional CPU and RAM resources to that server.

If I were to increase the CPU and RAM, would that suffice, or are there other challenges to consider when consolidating both roles on one server?

Additionally, I would like to clarify what you mean when you mention the possibility of them being a bottleneck. DS clients use port 8089, and Forwarders transmit data over port 9997. Could you explain the potential network bottleneck in this context?

[gcusello]

Hi @VK18 ,

As I said, it isn't a best practice, but you can locate bothe roles in the same machine, adding more resources to the VM.

About the bottleneck: yes they use different ports, but the network interface it's the same.

eventually, as you can read at https://docs.splunk.com/Documentation/Splunk/9.1.1/Admin/Deploymentclientconf you could reduce the update frequency in the DS, changing the phoneHomeIntervalInSecs parameter in deploymentclient.conf (on forwarders) from the default (60 seconds) to 120 or 180 seconds or more, as you can read at https://community.splunk.com/t5/Deployment-Architecture/When-managing-large-numbers-of-deployment-cl... 

Ciao.

Giuseppe