Monitoring Splunk

Splunk and Hyper-threading - Has anyone run benchmarks?

gfriedmann
Communicator

I'm wondering weather using hyper-threading would be a good idea or not. I am hoping somebody has tried this with 4.x. I'm mostly interested in search performance, not indexing.

sometimes typed as "hyperthreading"

Tags (1)
1 Solution

gfriedmann
Communicator

Quick Result Summary

You probably want to enable hyper-threading and disable power saving bios configs. The hyper-threading benefits can be notable but do not kick in until you have more search jobs than physical cores. But if the server is usually under utilized (in terms of jobs/physical core), then hyperthreading could hurt more than help if power saving options are enabled in the system BIOS.

Testing

With the help of the perftest application, i have gone through multiple rounds of performance testing.

My test was 1 - 30 parallel CPU bound search jobs over the same sample data. The search individually could complete in about 60 seconds.

The hardware is Intel Xeon X5680 @ 3.33GHz (Dual 6-core server)

I tested the following cases

A) Disable hyper-threading. Enable Power Saving BIOS Config
B) Enable hyper-threading. Enable Power Saving BIOS Config
C) Enable hyper-threading. Disable Power Saving BIOS Config
D) Disable hyper-threading. Disable Power Saving BIOS Config 

Test Results

Median search completion time by case by # of parallel searches.

                      CASE
               A     B     C     D
1 search      70    82    58    59
2 search      62   100    59    58
8 search      63    75    63    63
16 search     97    85    86    98
20 search    121    99    99   122 
30 search    180   144   144   182

Notable Observations

  • A single search load was most volatile. I suspect this is due to power throttling for partially used processor resources.
  • Hyper-threading on with power savings can increase search time by ~40% when load is low (less searches than physical cores). (see case B)


Definition

power savings BIOS config - Settings such as:

  • HP power Profile
  • HP Power Regulator
  • Turbo boost Optimization Dynamic
  • Power Savings Mode Response Minimum
  • Processor Idle Power States (C-States)

View solution in original post

gfriedmann
Communicator

Quick Result Summary

You probably want to enable hyper-threading and disable power saving bios configs. The hyper-threading benefits can be notable but do not kick in until you have more search jobs than physical cores. But if the server is usually under utilized (in terms of jobs/physical core), then hyperthreading could hurt more than help if power saving options are enabled in the system BIOS.

Testing

With the help of the perftest application, i have gone through multiple rounds of performance testing.

My test was 1 - 30 parallel CPU bound search jobs over the same sample data. The search individually could complete in about 60 seconds.

The hardware is Intel Xeon X5680 @ 3.33GHz (Dual 6-core server)

I tested the following cases

A) Disable hyper-threading. Enable Power Saving BIOS Config
B) Enable hyper-threading. Enable Power Saving BIOS Config
C) Enable hyper-threading. Disable Power Saving BIOS Config
D) Disable hyper-threading. Disable Power Saving BIOS Config 

Test Results

Median search completion time by case by # of parallel searches.

                      CASE
               A     B     C     D
1 search      70    82    58    59
2 search      62   100    59    58
8 search      63    75    63    63
16 search     97    85    86    98
20 search    121    99    99   122 
30 search    180   144   144   182

Notable Observations

  • A single search load was most volatile. I suspect this is due to power throttling for partially used processor resources.
  • Hyper-threading on with power savings can increase search time by ~40% when load is low (less searches than physical cores). (see case B)


Definition

power savings BIOS config - Settings such as:

  • HP power Profile
  • HP Power Regulator
  • Turbo boost Optimization Dynamic
  • Power Savings Mode Response Minimum
  • Processor Idle Power States (C-States)

jodros
Builder

To further support this answer, micron just finished very similar benchmarking, noting performance impacts from BIOS changes and how that affects IO. Other BIOS setting tweaks are mentioned that might also yield gains. As we utilize SSD's in all 4 of our indexers, CPU is our current bottleneck and any sort of tuning would be a positive impact.

http://www.micron.com/~/media/Documents/Products/White%20Paper/pcie_ssd_performance_hi.pdf

0 Karma

LCM
Contributor

This question could be similarish to the question I had recently. Since you are looking for the search performance & hyperthreading, I reckon hyperthreading is no option for that:

http://answers.splunk.com/questions/12027/single-multi-threading-cpu

Get Updates on the Splunk Community!

Index This | I am a number, but when you add ‘G’ to me, I go away. What number am I?

March 2024 Edition Hayyy Splunk Education Enthusiasts and the Eternally Curious!  We’re back with another ...

What’s New in Splunk App for PCI Compliance 5.3.1?

The Splunk App for PCI Compliance allows customers to extend the power of their existing Splunk solution with ...

Extending Observability Content to Splunk Cloud

Register to join us !   In this Extending Observability Content to Splunk Cloud Tech Talk, you'll see how to ...