• kdb+

  Running user defined aggregation on partitioned tables

  Posted by lzl on June 6, 2023 at 12:00 am

  Hello

  I want to run a user defined aggregating function getPercentile (to get say 99% or 50% or 1% percentile of a list of value) on a partitioned table. From this link map-reduce , only few types of aggregation can KDB decompose into map step and reduce step (Although med is listed, it will throw a part error if being executed across partitions). Since I got lots of data in each partition, I don’t want to load them all into memory and apply the aggregation, so is there any way to apply user specific map-reduce for aggregation on partitioned tables?

  Thanks!

  lzl replied 8 months, 2 weeks ago 2 Members · 1 Reply
• 1 Reply

• 

  darrenwsun

  Member

  June 6, 2023 at 12:00 am

  To my knowledge, the best way you can do is to fetch the data into memory and apply the custom aggregate function to the retrieved data, like the below

   

  update percentile:getPercentile price from select sym, price from trade where date>=.z.d-7 // and other filters

   

  Certain function like sum and prd can reduce in a memory-effective way as they don’t need to keep the original values from each partition. However med or your getPercentile function cannot.

Log In to Reply