-
How to efficiently store historical data registered in memory(RDB).
Hello,
I am using Tickplant, but I wonder if there is an efficient way to store historical data in memory in the correct partition as historical data should be registered in real-time memory.
You are using a non-date time partition, and after you go down to the HDB, you are distributing historical data in that partition to each partition in a separate process.Is there a way to do it efficiently and quickly?…
-
1 Reply
-
Hi k-mambo,
Do you mean if the data in the RDB contains data that has e.g. timestamps from multiple different dates, you’d want to save those down to their respective date partitions?
If yes, you’d need to customize your RDB’s .u.end function .
For example, instead of calling .Q.hdft (as default in r.q), you could instead write some logic to determine which dates you have for each table, then upsert on disk for each pair.
You’d also need to handle enumeration, sorting, compression (where applicable) and adding attributes on disk post operation.
To get a bit of an idea, check out some of the logic in w.q as mentioned here
Log in to reply.