Splunk Search

Why is my search head sorting columns when displaying statistics/visualizations?

gf13579
Communicator

We have two search heads, one for general use and one for Enterprise Security.

Any table/stats searches on the ES search head appear with the columns sorted alphanumerically — resulting in _time as the last field if you're doing a timechart. This then messes up line charts etc. that expect _time to be the first column (i.e. the x axis).

The sorting happens as the search head finalizes search results — if you run a long search you'll see columns {_time, b, c, a} then as the search finalizes, you'll see the browser rearrange them into {a, b, c, _time}. Viewing that as a line chart is then a disaster.

Has anyone seen this before?

0 Karma
1 Solution

gf13579
Communicator

It turned out that limits.conf was the problem, weirdly.

There was a complete - and possibly modified - copy of the 6.6.2 defaults/limits.conf placed in local/. After removing this the upgraded environment (7.2.1) worked fine - columns were sorted as expected, preventing line charts of timechart data from appearing incorrectly.

View solution in original post

0 Karma

gf13579
Communicator

It turned out that limits.conf was the problem, weirdly.

There was a complete - and possibly modified - copy of the 6.6.2 defaults/limits.conf placed in local/. After removing this the upgraded environment (7.2.1) worked fine - columns were sorted as expected, preventing line charts of timechart data from appearing incorrectly.

0 Karma

inventsekar
SplunkTrust
SplunkTrust

sorting columns
Yep, i also faced that, at sometimes, splunk developers sorted some columns as per their coding practice(but end users may be facing some troubles).
Core Developers and Software Testing/End Users can never agree at some strange issues like this, often. It happens 😉

0 Karma

skalliger
Motivator

I simply don't want any developers to messs with standard Splunk command behaviours, that's quite simple.
So, we need to check our Apps installed. Guess I have to check all configs from the lesser known ones of our environment. Thanks for the hint.

Skalli

0 Karma

skalliger
Motivator

Oh, wow. I am not the only one who noticed this weird behaviour. For me it seems like a strange bug. You can even reproduce this when using

search ...
| table a, b,c

which should sort the columns in the mentioned order. Instead, it just doesn't.

What ES version are you running? I noticed this behaviour on version 5.1.0. After 5.2.0, the issue was gone. Try upgrading if you can.

Skalli

0 Karma

gf13579
Communicator

ES 5.2.x so sadly it looks unrelated.

0 Karma

skalliger
Motivator

And what Splunk Enterprise version are you using?

0 Karma

gf13579
Communicator

7.2.0 on both Search Heads though we've just upgraded both to 7.2.1 and the ES SH is still showing the same frustrating behaviour.

0 Karma
Get Updates on the Splunk Community!

Modern way of developing distributed application using OTel

Recently, I had the opportunity to work on a complex microservice using Spring boot and Quarkus to develop a ...

Enterprise Security Content Update (ESCU) | New Releases

Last month, the Splunk Threat Research Team had 3 releases of new security content via the Enterprise Security ...

Archived Metrics Now Available for APAC and EMEA realms

We’re excited to announce the launch of Archived Metrics in Splunk Infrastructure Monitoring for our customers ...