-
Hi there,
Firstly, what you’re seeing here isn’t memory fragmentation due to nested columns. Looking at the documentation the memory fragmentation issue in relation to garbage collection is when KDB/Q can’t find contiguous blocks of memory to release back to the OS due to the memory being fragmented. Noting specifically here that although the used memory is low, even after garbage collection the heap remains large (see attached snippet). The solution of this fragmentation is to serialise your data, release then de-serialise (see the comments within the coded example)
Secondly, the coded example you provided doesn’t contain nested columns. An example of a nested column, continuing from your definition of trades:
select nested from update nested:n#enlist (.z.p;3i) from trades
Where the column type of nested is a mixed list with datatypes of 2-tuples at every index of the nested array
In relation to the time taken to garbage collect, you’ll find your second query requires more memory allocated to the process to complete, thus more time to release the memory.
To solve the issue of time for you, I’d advise using Immediate Garbage Collection which can be started within the Q session using the commandline argument of -g 1 or dynamically on the process using g 1. This will mean you don’t have to manually invoke .Q.gc[] as the system will automatically release memory blocks back to the OS upon availability.
More reading on Memory Fragmentation for interest here
Regards,
Sam B