nick_mospan
Forum Replies Created
-
To replicate the issue please copy position table twice, like you did with the cloud edition. It’s the second copy that takes and not releases the memory. I’m not running a cloud edition but the windows version:
KDB+ 4.0 2021.04.26 Copyright (C) 1993-2021 Kx Systems w64/ 8()core 32767MB
My theory is that the first copy creates the object in the first 64Mb block. For the second invocation of h”position” it had to create the second block and then assignment repoints the columns from the first to the second block. But because the first block has other objects already it cannot be freed. When the process is constantly updating this position table and at the same time serves other queries this situation repeats over and over slowly leading to a memory fragmentation that appears as a memory leak.
Is it possible to control the minimum block from command line? So knowing that a process is frequently creating “small” objects I could start it with 1Mb minimum block size instead of 64Mb?
-
My table has 54 columns of various simple types, mainly floats, symbols, ints and timestamps. Each column is of around 2Mb in size.
I can reproduce it with your code by dropping n to 2000000, which makes columns similar in size to my case. .Q.gc[] does not help releasing the excess heap to the OS:
q).Q.w[] used| 50694464 heap| 134217728 peak| 201326592 wmax| 0 mmap| 0 mphy| 34359267328 syms| 696 symw| 37613
Each column with n:2000000 should be allocated 16777216 bytes of heap.
q)(-22!) each value flip position 16000014 8667837 16000014
What is the reason for such behaviour? Are these columns small enough to lead to memory fragmentation or there’s something else going on?
-
Thanks, I found one of the causes – code that brings and refreshes a large table from another process.
I’m starting a fresh process and bringing in a table of 107Mb. The heap settles to 268Mb after .Q.gc[].
However after updating this table the heap jumps up to 469Mb and stays there.
What’s different between the first and second call to position:h”position” ? Why the heap does not go back to the initial 268Mb?
Here’s the console output:
q).Q.w[] used| 360512 heap| 67108864 peak| 67108864 wmax| 0 mmap| 0 mphy| 34359267328 syms| 686 symw| 37328 q)position:h"position" q).Q.w[] used| 226930848 heap| 402653184 peak| 402653184 wmax| 0 mmap| 0 mphy| 34359267328 syms| 1833 symw| 95932 q).Q.gc[] 134217728 q).Q.w[] used| 226930848 heap| 268435456 peak| 402653184 wmax| 0 mmap| 0 mphy| 34359267328 syms| 1834 symw| 95962 q)position:h"position" q).Q.gc[] 134217728 q).Q.w[] used| 226933216 heap| 469762048 peak| 603979776 wmax| 0 mmap| 0 mphy| 34359267328 syms| 1834 symw| 95962 q).Q.gc[] 0 q)count position 276765 q)-22!position 107637762
-
Thanks David, I got it working now. I was confused at first as there’s no way to set a view state to a value in an action for a DataGrid, it can only copy from a cell. I adapted your solution and used the Row Id column instead of ‘true’, so in my case triggerEnabled is a string, not a boolean.
-
Hi David,
Thanks for reply. However I don’t think this solution will work as the fact of setting triggerEnabled to true will trigger the query, giving similar side effect.
-
How do I set triggerEnabled from an action (e.g. click on a cell in a data grid)? Isnt that the same problem as setting the original view state to null?