-
Thank you – the syntax for inspecting the thread timings is very useful.
If we had our data segmented over 2+ disks and were not using peach, surely we wouldn’t utilise the available I/O throughput?
Whereas if we were to use peach, with the thread number being equal to, or a multiple of, the number of disks which our data is segmented across, then we would hopefully max out our I/O capacity. However this requires us manually ensuring that the date order in which we invoke the sub-function with peach matches the order in which data is distributed across segments.
I believe KDB does this natively* if you have a segmented, partitioned HDB but it seems like we aren’t able to utilise that efficiently with the query we’re trying to run.
*
https://code.kx.com/q/database/segment/#multithreading
https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#1445-query-execution-against-segmented-tables
https://code.kx.com/q/wp/multi-thread/#parallel-access-and-segmentation