SplunkCloud - Heavy Forwarder Communication: Why is my heavy forwarder is now skipping data?

[herguzav]

Hi friends.

I've followed de path to use UniversarForwarder app from my splunk cloud enviromen. But i have the next message:

The TCP output processor has paused the data flow. Forwarding to host_dest=inputs1.XXXX.splunkcloud.com inside output group splunkcloud_ from host_src=YYYYYY has been blocked for blocked_seconds=10. This can stall the data flow towards indexing and other network outputs. Review the receiving system's health in the Splunk Monitoring Console. It is probably not accepting data. Learn more.

I've tested the communications to splunk cloud

splunkcloud.com:9997

splunkcloud.com:8000

splunkcloud.com:8089

And all are OK.

My heavy forwarder is now skipping data. Is there something else I clould check out?

[burwell]

Hi.  Isn't this saying  indexer inputs1.XXXX.splunkcloud.com has full queues?

Can you look at the queues there? Messages?

[herguzav]

Hi!!

All splunkcloud queues are ok. Even my other heavy forwarder is working fine.

Do you have another test to do?

[isoutamo]

Hi

Usually it's like @burwell said. This error/warning means that for some reason there are some queues full. Earlier You could try this 

index=_internal host=*.<your stack name>.splunkcloud.com source=*metrics.log sourcetype=splunkd TERM(group=queue) (TERM(name=parsingQueue) OR TERM(name=indexqueue) OR TERM(name=tcpin_queue) OR TERM(name=aggqueue))
| eval is_blocked=if(blocked=="true",1,0), host_queue=host." - ".name
| stats sparkline sum(is_blocked) as blocked,count by host_queue
| eval blocked_ratio=round(blocked/count*100,2)
| where blocked_ratio > 0
| sort 50 -blocked_ratio 
| eval requires_attention=case(blocked_ratio>50.0,"fix highly recommended!",blocked_ratio>40.0,"you better check..",blocked_ratio>20.0,"usually no need to worry but keep an eye on it",1=1,"not unusual")

to check  what is status of those input forwarders. But not I cannot see that information on SC, probably it's forwarder to somewhere else?

I suppose that you will create a ticket to SC support and ask situation about those input forwarders.

r. Ismo

[herguzav]

Hi .

I've run your search and no results were displayed, I've retired the condition and all results said 

"not unusual"

The problem persist. 😞

Regards

[isoutamo]

As I said, I cannot see that information (inputs1-15) on SC side anymore. Earlier this was stored to customer cloud stack, but now I expecting that this information is currently forwarder to Splunk Clouds' admin stack or somewhere else where Customers haven't access?

I think that you haven't any other option than create a support ticket to Splunk and ask that they solve this issue.