All Apps and Add-ons

On SCK from my Kubernetes metrics, can I calculate the memory percentage usage per node ?

yannK
Splunk Employee
Splunk Employee

I have SCK setup, and collect my Kubernetes metrics. We have access out of the box to the node memory limit kube.node.memory.allocatable (in MB), and to the memory usage kube.node.memory.working_set_bytes (in bytes)

but we want to do some calculations to get the memory usage percentage per node.

Labels (1)
Tags (1)
0 Karma
1 Solution

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

View solution in original post

0 Karma

yannK
Splunk Employee
Splunk Employee

If you use the Graphs in the analytics UI are based on those searches

 

| mstats avg("kube.node.memory.allocatable") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.allocatable") AS Avg span=10s
| fields - _span*

 

and

 

 

| mstats avg("kube.node.memory.working_set_bytes") prestats=true WHERE "index"="k8s_metrics" span=10s
| timechart avg("kube.node.memory.working_set_bytes") AS Avg span=10s
| fields - _span*

 

 

To extract them on a single search and do the calculation, you can use that method.

 

| mstats avg("kube.node.memory.allocatable") As alloc_MB avg("kube.node.memory.working_set_bytes") AS working_B prestat=f WHERE "index"="k8s_metrics" span=10s by node
| timechart avg(eval(100*working_B/(alloc_MB*1024*1024))) AS mem_used_percent span=10s by node

 

 

 

Key points : need to use AS to cast the results into fields you can manipulate later. Use an EVAL in the function to do the calculation on the fly. For timecharts ensure that the span in the mstats is equal or smaller than the span in the timechart. Use a split BY the appropriate field, here by node. We also disabled the "prestat" from the mstats, otherwise the fields names  would be slightly different.

0 Karma
Get Updates on the Splunk Community!

Stay Connected: Your Guide to May Tech Talks, Office Hours, and Webinars!

Take a look below to explore our upcoming Community Office Hours, Tech Talks, and Webinars this month. This ...

They're back! Join the SplunkTrust and MVP at .conf24

With our highly anticipated annual conference, .conf, comes the fez-wearers you can trust! The SplunkTrust, as ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had two releases of new security content via the Enterprise ...