[erichards]

Many thanks for the reply and examples.

 

“in fact since `neg` has native multithreading and operates on vectors and vectors of vectors it is best of off left on it’s own”

This is what I was keen to understand, and it’s useful to know that there are cases when you may be better off without peach.

[erichards]

I guess a more succint version of my question is “what happens to native parallelisations when running queries inside an instance of peach?”

[erichards]

Thank you David, it is re-assuring to know that we are not following an anti-pattern or something too bespoke.

 

With a small demo we have found a sym-partitioned db to be much more performant for our access patterns.

 

I had not considered the partition being a combination of sym + time, that might come in useful.

[erichards]

Thank you – the syntax for inspecting the thread timings is very useful.

If we had our data segmented over 2+ disks and were not using peach, surely we wouldn’t utilise the available I/O throughput?

Whereas if we were to use peach, with the thread number being equal to, or a multiple of, the number of disks which our data is segmented across, then we would hopefully max out our I/O capacity. However this requires us manually ensuring that the date order in which we invoke the sub-function with peach matches the order in which data is distributed across segments.

I believe KDB does this natively* if you have a segmented, partitioned HDB but it seems like we aren’t able to utilise that efficiently with the query we’re trying to run.

*

https://code.kx.com/q/database/segment/#multithreading

https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#1445-query-execution-against-segmented-tables

https://code.kx.com/q/wp/multi-thread/#parallel-access-and-segmentation

[erichards]

Thank you for the prompt and thorough response. This is undoubtedly useful for loading various instruments with different ranges.

In our specific example of rolled series there would only be one day of overlap between contracts. I am wondering if there is a way of constructing this as a single-query, rather than looping by date ranges. This would allow KDB to decompose the query over different segments/threads as appropriate.

Perhaps this could be done by exploding the contract/date combinations as you suggest, and looking up into this table. However I’m not sure how to do this without losing the efficiency gained from filtering by date first. Could a join help?