[rocuinneagain]

https://code.kx.com/q/ref/accumulators/#ternary-values

You can use a sample to see how it progresses:

 

({-1 .Q.s1 `x`y`z!(x;y;z);x+y+z}/)[0;1 2 3;4 5 6] 
`x`y`z!0 1 4 
`x`y`z!5 2 5 
`x`y`z!12 3 6 21

 

And on the ssr example:

 

({-1 .Q.s1 `x`y`z!(x;y;z);ssr[x;y;z]}/)["results_%div_%dept.csv"; ("%div";"%dept"); ("banking";"m&a")] 
`x`y`z!("results_%div_%dept.csv";"%div";"banking") 
`x`y`z!("results_banking_%dept.csv";"%dept";"m&a") 
"results_banking_m&a.csv"

 

 

[rocuinneagain]

1. Does the file definitely exist?

 

ls -l /home/user/anaconda3/envs/pykx/lib/python3.9/site-packages/pykx/lib/l64/pykx.so

 

2. Does forcing the package to reinstall help?

 

pip install –upgrade –force-reinstall pykx

 

3. Is the conda environment pykx activated when a) You ran pip install? b) You started python?

4. When exactly do you get the error? Are you just starting python and running import pykx?

 

 

[rocuinneagain]

The file is there:

kx-insights-packages > kdbInsightsCore > 4.0.2

Are you signed in with username and password?

[rocuinneagain]

Listed on https://code.kx.com/insights/1.5/core/install.html#license

A valid license is needed to run kdb Insights Core. For more information on licensing, please contact licadmin@kx.com.

[rocuinneagain]

Yes this can be solved by ensuring gcloud is up to date and initialised:

 

gcloud components update gcloud init

 

 

[rocuinneagain]

https://github.com/KxSystems/cookbook/tree/master/r/rserve

You are only looking at the memory usage of the q process.

With Rserve R is a separate process. You need to monitor it’s memory usage also.

 

To free memory in q after creating a large object use .Q.gc[]. You can also start q with -g 1.

• https://code.kx.com/q/basics/cmdline/#-g-garbage-collection
• https://code.kx.com/q/ref/dotq/#gc-garbage-collect

[rocuinneagain]

You can delete R variables and run R garbage collection using Rcmd as above.

[rocuinneagain]

The src is available at https://github.com/KxSystems/embedR/blob/fa5101b64e15f9ba0aa5c20affc0cd041fb41bc0/src/rserver.c#L458

 

Have you tried calling

Rcmd “rm(temp)” Rcmd “gc()”

 

 

q does not manage the memory for R – you must still delete variables and call garbage collection

[rocuinneagain]

-25! is for use with IPC handles only. Not websocket handles.

The reason is for IPC handles there is a serialization step and here -25! is efficient in allowing this to only be run once for many handles.

For websocket handles data is sent directly without any serialization step, -25! would offer no benefit here.

• https://code.kx.com/q/basics/internal/#-25x-async-broadcast
• https://code.kx.com/q/basics/internal/#-38x-socket-table

q){([]h)!-38!h:.z.H}[] 
h  | p f 
---| --- 
612| w t 
580| q t 

q)-25!(enlist 612i;"test") 
'612 is not an ipc handle [0] 
-25!(enlist 612i;"test") ^ 

q)-25!(enlist 580i;"test")

 

For websockets if there is a large operation like converting a table to JSON you can control this being done once and then sent to multiple websockets:

neg[webSockethandles]@:.j.j bigTable

 

Wrapping the same in a helper:

wsBroadcast:{[handles;data] neg[(),handles]@:data} wsBroadcast[myWebsocketHandles] .j.j bigTable

 

[rocuinneagain]

All columns in the splayed table should have the same number of rows so there was some issue with the writedown of this data. This is most likely the source of the issue.

 

Can you recreate the data from source/backup/TP-logs?

When you read/write you are losing 33199-22210=10989 rows of data from the “good” columns.

[rocuinneagain]

q makes a balance when displaying about levels of detail.

Always giving full exact information would make all output very messy and more like code rather than nicely displayed information for human to view.

 

There are some tools and tips you can use to inspect items in more detail during development and debugging.

Using .Q.s1 is most useful I find to drill down. (Along with usual type, count etc)

https://code.kx.com/q/ref/dotq/#s1-string-representation

 

 

-1 .Q.s1 itemToInspect

 

 

 

A script designed to display items in detail nicely:

https://github.com/LeslieGoldsmith/dpy

 

[rocuinneagain]

The locking is done by the system call lockf (see link in previous message), this locks the file at the  system level. No other process can get a lock on the file until the first process releases the lock.

[rocuinneagain]

1. Do you know what version of kdb+ wrote the files?
2. Is the data compressed?
3. Are there attributes on any of the effected columns?
4. Are any of the columns linked columns?
5. If you go through the columns one at a time do their counts all match?
6. Can you read the bad partitions in q and  write them to a temp HDB location. Do these rewritten files still show the same memory behaviour when you load the temp HDB?

[rocuinneagain]

First suggestion would be to test against the latest version of 3.5.

Several fixes were released after the version you are using.

[rocuinneagain]

.Q.en uses ? which does locking

• https://code.kx.com/q/ref/dotq/#en-enumerate-varchar-cols
• https://code.kx.com/q/ref/enum-extend/#filepath

     “The file is locked at a process level for writing during .Q.en only. Avoid reading from any file which may be being written to.”

    “The system call used is https://linux.die.net/man/3/lockf.”