-
Running user defined aggregation on partitioned tables
Hello
I want to run a user defined aggregating function getPercentile (to get say 99% or 50% or 1% percentile of a list of value) on a partitioned table. From this link map-reduce , only few types of aggregation can KDB decompose into map step and reduce step (Although med is listed, it will throw a part error if being executed across partitions). Since I got lots of data in each partition, I don’t want to load them all into memory and apply the aggregation, so is there any way to apply user specific map-reduce for aggregation on partitioned tables?
Thanks!
-
1 Reply
-
To my knowledge, the best way you can do is to fetch the data into memory and apply the custom aggregate function to the retrieved data, like the below
update percentile:getPercentile price from select sym, price from trade where date>=.z.d-7 // and other filters
Certain function like
sumandprdcan reduce in a memory-effective way as they don’t need to keep the original values from each partition. Howevermedor yourgetPercentilefunction cannot.
Log in to reply.