Re: Loading data for combinations of instruments and date ranges (rolled series)

[rocuinneagain]

kdb+ is already automatically using multithreaded map-reduce for each of your subqueries without any changes as long as you started your process with -s

https://code.kx.com/q/wp/multi-thread/#map-reduce-with-multi-threading

//I'll create a test HDB to test against - you can use real data instead 
.z.zd:17 2 6 //Turning compression on 
n:1000000 //Write some partitions 
{.Q.dd[`:HDB;x,`tab`] set .Q.en[`:HDB] 
([] sym:`p#asc n?`3;p:n?100.0;t:n?1000)} each 2022.01.01 + til 365 //Load the HDB l HDB //q was started with -s 4 for 4 secondary threads 
q)system"s" 4i //Run our sample query 100 times for each of 0-4 secondary threads available 
q)results:([] secondaryThreads:til 1+system"s"; 
timings:{system"s ",string x;value"\t:100 select from tab where date within 2022.01.01 2022.03.31,sym=`aaa"}each til 1+system"s") //Results show that the threads are being used and our query is speeding up with more threads available 
q)results secondaryThreads 
timings 
------------------------ 
0 12153 
1 12225 
2 7919 
3 6222 
4 5767

You can see the benefit of threads by running through some sample queries

 

 

If you wanted you could experiment with using the threads per subquery rather within each subquery.

This would only require you change each to peach on your existing code to have threads execute each subquery

result: raze loadContract peach spec

For your query though I expect the existing code is probably the better balance of compute and memory.