-
Hi Nick,
Here are the steps I did to attempt reproducing your issue:
Host Machine (Port 5000):
q)n:50000000 q)position:([]time:n?.z.p;sym:n?`ABC`APPL`WOW;x:n?10f)
Client Machine:
q)h:hopen`::5000 q).Q.w[] used| 357632 heap| 67108864 peak| 67108864 wmax| 0 mmap| 0 mphy| 8335175680 syms| 668 symw| 28560 q)position:h"position" q).Q.w[] used| 1610970544 heap| 2751463424 peak| 2751463424 wmax| 0 mmap| 0 mphy| 8335175680 syms| 672 symw| 28678 q).Q.gc[] 1073741824 q).Q.w[] used| 1610969232 heap| 1677721600 peak| 2751463424 wmax| 0 mmap| 0 mphy| 8335175680 syms| 673 symw| 28708 q)position:h"position" q).Q.w[] used| 1610969232 heap| 4362076160 peak| 4362076160 wmax| 0 mmap| 0 mphy| 8335175680 syms| 673 symw| 28708 q).Q.gc[] 2684354560 q).Q.w[] used| 1610969232 heap| 1677721600 peak| 4362076160 wmax| 0 mmap| 0 mphy| 8335175680 syms| 673 symw| 28708
As you can see in trying to replicate your issue, my example releases the expected amount of memory back to OS. Due to the number of records you have and the relative size of the table after, the issue I think you’re encountering is due to the data structure of position leading to memory fragmentation. As per my other reply the reference on code kx gives an example of this stating “nested data, e.g. columns of char vectors, or much grouping” will lead to fragmenting memory heavily, does this reflect your data?
To fix this I’d suggest the approach on the reference of serialise, release, deserialise. Or to extend further to your case: serialise, release, deserialise, release, IPC reassign, release. This will maintain a low memory footprint and try to remedy the memory fragmentation but you may still unavoidably have heap greater than used purely due to the data structure (however to a lesser extent to what you’re experiencing).
If memory fragmentation isn’t the cause can you give a bit more insight on the data structure of position as my attempt to replicate shows this problem is data specific.