-
Orphan memory in KDB process using Rserve
Hi Everyone,
our team is one of the users of Rserve for integrating KDB with R. Recently we noticed that one of our processes working with a R server was bloating in memory. Oddly, the .Q.w did not show any bloating in memory and .Q.gc didnt release any memory either. However the OS view of the memory consumed by the process was high and different from KDB view of its own memory.
We dug bit deeper and found that this was specifically happening in RSet function of the library.
While we are unable to view the implementation of Rserve at https://code.kx.com/q/interfaces/r/#remote-r-rserve I believe we are missing de-allocating memory in Rset function. Can someone help in fixing this issue or provide more insight.
regards,
Muneish
-
8 Replies
-
- Is it the embedR project that your are using?
- Are you on the latest release of the code? https://github.com/KxSystems/embedR/releases
- Is it possible share a small generic piece of code to reproduce the issue?
-
Hi Rocuinneagain ,
1. I believe it is embedR project indeed.
2. yes we are using the latest code.
3.
Rset:`rserve 2:(`rsset;2);meminfo:{(5#system”w”), 1024*”J”$system “ps -eo size -h -q “,string .z.i};
m1:meminfo[];
RSet[“temp”;100000?100];
m2:meminfo[];
-
The src is available at https://github.com/KxSystems/embedR/blob/fa5101b64e15f9ba0aa5c20affc0cd041fb41bc0/src/rserver.c#L458
Have you tried calling
Rcmd “rm(temp)” Rcmd “gc()”
q does not manage the memory for R – you must still delete variables and call garbage collection
-
You can delete R variables and run R garbage collection using
Rcmdas above. -
Currently we reassign the same variable with new data, will just calling gc resolve the bloating in that case?
-
Also, need to add that we are using a file called as rserve.so , I believe its not part of embedR project.
-
https://github.com/KxSystems/cookbook/tree/master/r/rserve
You are only looking at the memory usage of the q process.
With Rserve R is a separate process. You need to monitor it’s memory usage also.
To free memory in q after creating a large object use
.Q.gc[]. You can also start q with-g 1. -
Hi there
the memory bloating problem is with the kdb process and not the R process.
whenever the q process calls Rset there is orphan memory on the q process.
Also, .Q.gc does not release any memory. The q process and OS have different view on the amount of memory being consumed by the q process .in the end the q process shuts down even though it thinks it has low memory foot print.
Log in to reply.