Re: Partitioning Tables Intraday by Custom Fields?

[rocuinneagain]

That would require for deep level changes to the codebase.

For example when a HDB is written to a new:

1. A new date folder is created.
2. Tables are written.
3. The HDB process is reloaded

If you reloaded the HDB before all tables were written you would have errors.

q)mkdir HDB 
q)cd HDB 
q)`:2021.01.01/tab1/ set ([] a:1 2 3) 
`:2021.01.01/tab1/ 
q)`:2021.01.01/tab2/ set ([] b:1 2 3) 
`:2021.01.01/tab2/ 
q)\l . 
q)select from tab1 
date       a 
------------ 
2021.01.01 1 
2021.01.01 2 
2021.01.01 3 

q)select from tab2 
date       b 
------------ 
2021.01.01 1 
2021.01.01 2 
2021.01.01 3 

q)`:2021.01.02/tab1/ set ([] a:4 5 6) 
`:2021.01.02/tab1/ 
q)\l . 
q)select from tab1 
date       a 
------------ 
2021.01.01 1 
2021.01.01 2 
2021.01.01 3 
2021.01.02 4 
2021.01.02 5 
2021.01.02 6 

q)select from tab2 
'./2021.01.02/tab2/b. 
OS reports: No such file or directory 
[0] select from tab2 ^

.Q.bv – would be one possible helpful extension which can in memory fill tables missing from partitions (using `  as argument to fill using first partition as a template)

q)\l HDB 
q)select from tab1 
date       a 
------------ 
2021.01.01 1 
2021.01.01 2 
2021.01.01 3 
2021.01.02 4 
2021.01.02 5 
2021.01.02 6 

q)select from tab2 //Table missing from latest partition is not found 
'tab2 
[0] select from tab2 ^ 
q).Q.bv` //Using ` first partition used as prototype and table is found 
q)select from tab2 
date       b 
------------ 
2021.01.01 1 
2021.01.01 2 
2021.01.01 3

(.Q.chk is unsuitable as it uses most recent partition as template to fills tables missing from partitions)

 

This is only one item that would be needed to implement what you asked.

Currently EOD is a single action for all tables, all code and processes involved would need updates to operate table by table. Meaning changes in RDB/IDB/HDB and others.