Azure Billing Add-On With Proxy

[chito]

Is anyone managed to setup the Microsoft Azure Billing Add-On with Proxy. The add-on does not have any option or configuration item to set or use proxy for http request. I set the os.environment http_proxy into the main and it managed to logged in to the azure but now I am getting 401 error with the following message:

AuthenticationError: Autologin succeeded, but there was an auth error on next request. Something is very wrong.

[chito]

In summary there are 3 request calls that need the proxies applied:

input_module_azure_billing.py: response = requests.post(endpoint, proxies=proxies, data=payload).json()
input_module_azure_billing.py: r = requests.get(usage_url, headers=header, proxies=proxies)
input_module_azure_billing.py: r = requests.get(billing_period_url, headers=header,proxies=proxies)

[chito]

I have tried this and get the 'Connection refused'

ConnectionError: HTTPSConnectionPool(host='login.windows.net', port=443): Max retries exceeded with url: /************/oauth2/token (Caused by NewConnectionError('<requests.packages.urllib3.connection.VerifiedHTTPSConnection object at 0x7f749994ad90>: Failed to establish a new connection: [Errno 111] Connection refused',))

What I did was use the OS environment var by adding this at the top of the azure_billing.py:

import os

proxy='http://10.10.1.10:3128'
sproxy='http://10.10.1.10:1080'

os.environ['HTTP_PROXY'] = proxy
os.environ['HTTPS_PROXY'] = sproxy

This allowed it to reach the https://management.azure.com, but now I am getting this error:

2019-04-16 16:10:21,171 INFO pid=8102 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-04-16 16:10:21,269 INFO pid=8102 tid=MainThread file=connectionpool.py:_new_conn:758 | Starting new HTTPS connection (1): 127.0.0.1
2019-04-16 16:10:21,375 ERROR pid=8102 tid=MainThread file=base_modinput.py:log_error:307 | Traceback (most recent call last):
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/modinput_wrapper/base_modinput.py", line 113, in stream_events
    self.parse_input_args(input_definition)
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/modinput_wrapper/base_modinput.py", line 152, in parse_input_args
    self._parse_input_args_from_global_config(inputs)
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/modinput_wrapper/base_modinput.py", line 171, in _parse_input_args_from_global_config
    ucc_inputs = global_config.inputs.load(input_type=self.input_type)
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/splunktaucclib/global_config/configuration.py", line 270, in load
    input_item['entity']
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/splunktaucclib/global_config/configuration.py", line 175, in _load_endpoint
    **query
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/solnlib/packages/splunklib/binding.py", line 298, in wrapper
    return request_fun(self, *args, **kwargs)
  File "/opt/splunk/lib/python2.7/contextlib.py", line 35, in __exit__
    self.gen.throw(type, value, traceback)
  File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/solnlib/packages/splunklib/binding.py", line 232, in _handle_auth_error
    raise AuthenticationError(msg, he)
AuthenticationError: Autologin succeeded, but there was an auth error on next request. Something is very wrong.

[jconger]

The http_proxy setting is used for splunkd traffic. The Azure billing add-on uses modular inputs to gather usage data from Azure, and modular inputs can implement any mechanism they want (meaning they don't go through splunkd). Therefore, modular inputs have to implement any proxy configurations internally. But, as you have found, this add-on doesn't currently implement any proxy settings. If you want to dig just a little into the code, here is how you could make it use a proxy:

• edit input_module_azure_billing.py
• look for the text r = requests.get(usage_url, headers=header) and r = requests.get(billing_period_url, headers=header)
• right before those lines, create the following code (substitute your proxy information):

proxies = {'http': 'http://10.10.1.10:3128','https': 'http://10.10.1.10:1080'}

• change r = requests.get(usage_url, headers=header) to r = requests.get(usage_url, headers=header, proxies=proxies)
• change r = requests.get(billing_period_url, headers=header) to r = requests.get(billing_period_url, headers=header, proxies=proxies)

[thobreit]

Hi Jason,

I made the changes you mentioned but still got these errors:

Get error when collecting events. Traceback (most recent call last):
File "/opt/splunk/etc/apps/TA-Azure-billing/bin/ta_azure_billing/modinput_wrapper/base_modinput.py", line 127, in stream_events self.collect_events(ew)
File "/opt/splunk/etc/apps/TA-Azure-billing/bin/azure_billing.py", line 80, in collect_events input_module.collect_events(self, ew) File "/opt/splunk/etc/apps/TA-Azure-billing/bin/input_module_azure_billing.py", line 133, in collect_events access_token = get_access_token(global_client_id, global_client_secret, tenant_id)
File "/opt/splunk/etc/apps/TA-Azure-billing/bin/input_module_azure_billing.py", line 32, in get_access_token raise e
ProxyError: HTTPSConnectionPool(host='login.windows.net', port=443): Max retries exceeded with url: /*****************/oauth2/token (Caused by ProxyError('Cannot connect to proxy.', NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',)))

So I also made the same changes you suggested in the get_access_token function:

from:

    response = requests.post(endpoint,data=payload).json()

to:

proxies = {'http':
'http://host.com:3128','https':
'http://host.com:1080'}
response = requests.post(endpoint, data=payload,
proxies=proxies).json()

After these changes the app is now able to use my proxy server and I am receiving data from Azure.

Regards
Thomas