-
KDB+ and hyperthreading
I’m looking for information regarding hyperthreading with kdb+, main questions below
1) Does it cause any issues when enabled?
2) Does it impact the license in any way.
3) Are there any types of queries or operations which benefit from it being enabled?
4) Is the -s flag for secondary threads impacted or operate differently
Thanks
-
1 Reply
-
Hyperthreading does not cause problems for kdb+ but you should be aware of your core allocations to get the most out of your kdb+ licence.
Example:
- I have a 4 core 8 thread machine.
- I have a 4 core kdb+ licence.
- How best to allocate my licence?
Your machine will list cores 0,1,2,3,4,5,6,7.
You want to make sure you use your 4 core licence only once on each logical CPU core for best performance.
Inspecting
/proc/cpuinfogives you the information you need.cpuinfo:.Q.id {{{(`$x[0])!x[1]}flip {ssr[;"t";""] each trim ":" vs x}each x y}[x] each {{x[0]+til 1+x[1]-x[0]}each flip (0^1+prev x;-1+x)}where x~:""}system"cat /proc/cpuinfo" select processor,physicalid,siblings,coreid,cpucores from cpuinfo processor physicalid siblings coreid cpucores --------------------------------------------- ,"0" ,"0" ,"8" ,"0" ,"4" ,"1" ,"0" ,"8" ,"0" ,"4" ,"2" ,"0" ,"8" ,"1" ,"4" ,"3" ,"0" ,"8" ,"1" ,"4" ,"4" ,"0" ,"8" ,"2" ,"4" ,"5" ,"0" ,"8" ,"2" ,"4" ,"6" ,"0" ,"8" ,"3" ,"4" ,"7" ,"0" ,"8" ,"3" ,"4"Here the output suggests I should use taskset 0,2,4,6 or 1,3,5,7 so that I only reuse each coreid once.
This way I am getting best usage from your kdb+ licence as each one is taxing a full core without overlap.
It’s possible on another OS/machine the layout may suggest 0-3 or 4-7.
On this example if only kdb+ is running on the machine you may see a fractional perf benefit from disabling hyperthreading. It can depend on various hardware and workload variables.
Lots of discussion available online on the topic:
https://unix.stackexchange.com/questions/57920/how-do-i-know-which-processors-are-physical-cores
Note: some extra checks and verifications should be done if using Virtual Machines as they may not pass through all correct detailed information through to the VM and you may need to verify at the hypervisor level.
If you had unlimited cores licence the answer is a little more nuanced and use case specific in how you may choose to taskset your processes. Or you may choose to let not pin individual processes to cores but instead let the OS move tasks around for best efficiency.
For machines heavily taxing IO hyperthreading can help here specifically with reading of data from disk.
Once on to multi CPU systems NUMA also comes in to play and should be paid attention to.
https://code.kx.com/q/kb/linux-production/#non-uniform-memory-access-numa-hardware
Log in to reply.