rocuinneagain
Forum Replies Created
-
The file is there:
kx-insights-packages > kdbInsightsCore > 4.0.2
Are you signed in with username and password?
-
https://github.com/KxSystems/cookbook/tree/master/r/rserve
You are only looking at the memory usage of the q process.
With Rserve R is a separate process. You need to monitor it’s memory usage also.
To free memory in q after creating a large object use
.Q.gc[]. You can also start q with-g 1. -
You can delete R variables and run R garbage collection using
Rcmdas above. -
The src is available at https://github.com/KxSystems/embedR/blob/fa5101b64e15f9ba0aa5c20affc0cd041fb41bc0/src/rserver.c#L458
Have you tried calling
Rcmd “rm(temp)” Rcmd “gc()”
q does not manage the memory for R – you must still delete variables and call garbage collection
-
-25! is for use with IPC handles only. Not websocket handles.
The reason is for IPC handles there is a serialization step and here -25! is efficient in allowing this to only be run once for many handles.
For websocket handles data is sent directly without any serialization step, -25! would offer no benefit here.
- https://code.kx.com/q/basics/internal/#-25x-async-broadcast
- https://code.kx.com/q/basics/internal/#-38x-socket-table
q){([]h)!-38!h:.z.H}[] h | p f ---| --- 612| w t 580| q t q)-25!(enlist 612i;"test") '612 is not an ipc handle [0] -25!(enlist 612i;"test") ^ q)-25!(enlist 580i;"test")For websockets if there is a large operation like converting a table to JSON you can control this being done once and then sent to multiple websockets:
neg[webSockethandles]@:.j.j bigTable
Wrapping the same in a helper:
wsBroadcast:{[handles;data] neg[(),handles]@:data} wsBroadcast[myWebsocketHandles] .j.j bigTable
-
All columns in the splayed table should have the same number of rows so there was some issue with the writedown of this data. This is most likely the source of the issue.
Can you recreate the data from source/backup/TP-logs?
When you read/write you are losing 33199-22210=10989 rows of data from the “good” columns.
-
q makes a balance when displaying about levels of detail.
Always giving full exact information would make all output very messy and more like code rather than nicely displayed information for human to view.
There are some tools and tips you can use to inspect items in more detail during development and debugging.
Using
.Q.s1is most useful I find to drill down. (Along with usualtype,countetc)https://code.kx.com/q/ref/dotq/#s1-string-representation
-1 .Q.s1 itemToInspect
A script designed to display items in detail nicely:
https://github.com/LeslieGoldsmith/dpy
-
The locking is done by the system call lockf (see link in previous message), this locks the file at the system level. No other process can get a lock on the file until the first process releases the lock.
-
- Do you know what version of kdb+ wrote the files?
- Is the data compressed?
- Are there attributes on any of the effected columns?
- Are any of the columns linked columns?
- If you go through the columns one at a time do their counts all match?
- Can you read the bad partitions in q and write them to a temp HDB location. Do these rewritten files still show the same memory behaviour when you load the temp HDB?
-
First suggestion would be to test against the latest version of 3.5.
Several fixes were released after the version you are using.
-
.Q.enuses?which does locking- https://code.kx.com/q/ref/dotq/#en-enumerate-varchar-cols
- https://code.kx.com/q/ref/enum-extend/#filepath
“The file is locked at a process level for writing during
.Q.enonly. Avoid reading from any file which may be being written to.”“The system call used is https://linux.die.net/man/3/lockf.”
-
Kdb+ can parse binary files as nicely shown at the recent KX meetup by
https://community.kx.com/t5/Events/KX-Community-Meetup-New-York/ba-p/13880
File formats can get complicated though so this can be a lot of work.
If I wanted to do this task quickly I would either:
a) Use a system call to a command line tool to extract the files on disk and then read them in from there.
Writing them to current directory or using mktemp command to write in /var/tmp
b) Wrap some exiting python code using EmbedPy to extract the email and attachments to JSON and read in to kdb+ this way.
Similar to how I did for XML with https://github.com/rianoc/qXML
Some discussions on the topic in Python world:
(Note: I have not tested these for functionality or safety)
-
For KX Platform Dashboards on the backend follow:
The UI options** are covered on:
https://code.kx.com/dashboards/datasources/#streaming
(**as well as backend if instead using Dashboards Direct)
-
Most modern shells will be UTF-8 encoded rather than ASCII.
kdb+ is storing the information correctly – it is just being displayed differently than you expect.
On my machine:
$ locale LANG=C.UTF-8 LANGUAGE= LC_CTYPE=”C.UTF-8″ LC_NUMERIC=”C.UTF-8″ LC_TIME=”C.UTF-8″ LC_COLLATE=”C.UTF-8″ LC_MONETARY=”C.UTF-8″ LC_MESSAGES=”C.UTF-8″ LC_PAPER=”C.UTF-8″ LC_NAME=”C.UTF-8″ LC_ADDRESS=”C.UTF-8″ LC_TELEPHONE=”C.UTF-8″ LC_MEASUREMENT=”C.UTF-8″ LC_IDENTIFICATION=”C.UTF-8″ LC_ALL=
UTF-8 encoding for �� 0xC3 0xB4
Testing this in kdb+
q)`char$0xC3B4
“ô”
q)`$`char$0xC3B4
`��i-code>
For your shell to print the extended ASCII table characters you will need to make some changes outside of kdb+.
-
- Is it the embedR project that your are using?
- Are you on the latest release of the code? https://github.com/KxSystems/embedR/releases
- Is it possible share a small generic piece of code to reproduce the issue?