[rocuinneagain]

The issue you are having is unrelated to peach/multithreading on the kdb+ side. The issue is the reuse of the same connection among@taskblocks. The different @tasks are attempting to read data from the same connection concurrently leading to junk data being passed through. kdb+ processes incoming queries sequentially. This means even through you see the log messages from the tasks as if they run in parallel in fact once these arrive to kdb+ they will be processed one after the other always. For this reason switching to @flow will not be slower and will result is safe consistent results.

[rocuinneagain]

Only PyKX has been released on open PyPi in that way.

The correction to the documentation for kxi will be online soon.

[rocuinneagain]

The issue you are having is unrelated to peach/multithreading on the kdb+ side.

 

The issue is the reuse of the same connection among @task blocks.

The different @tasks are attempting to read data from the same connection concurrently leading to junk data being passed through.

 

kdb+ processes incoming queries sequentially. This means even through you see the log messages from the tasks as if they run in parallel in fact once these arrive to kdb+ they will be processed one after the other always. For this reason switching to @flow will not be slower and will result is safe consistent results.

 

 

[rocuinneagain]

1. There is no need for raze to be included.

 

q)select type close by sym from daily

sym | close 
----| ------ 
AAPL| 9h 
AIG | 9h 
AMD | 9h ..

 

Float vectors are being passed to your function (type 9h). raze has no effect on these.

 

2. select size by sym from daily only does grouping as no aggregate function is included.

It is the same as select {x} size by sym from daily

Including a real aggregate would be select last5 size by sym from daily

[rocuinneagain]

Hi,

1. Are you running PyKX under Prefect? http://www.prefect.io
2. Are you connecting and querying using PyKX in @task or @flow ?
3. If using @task does switching to @flow (.i.e subflows) stop the issue happening?
4. Do you see the same issue if you query in a standalone python process outside of Prefect?
5. Are you making a single query when the issue happens or are multiple queries being run?
6. What version of PyKX are you running? pykx.__version__
7. Are you running in licensed or unlicensed mode? pykx.licensed
8. How did you create the connection conn?

 

• Adding inline code to your questions & messages
• Creating q code block with syntax highlighting

[rocuinneagain]

https://code.kx.com/insights/1.5/microservices/service-discovery/prerequisites.html#running-in-docker

To gain access please contact KX at licadmin@kx.com

[rocuinneagain]

If you went directly in to writing what you ask:

 

q)t:flip `date`data!(2023.05.20 2023.05.20 2023.05.20 2023.05.19 2023.05.19 2023.05.19;`a`c`b`b`b`c) 
q)select from t where any (and2024;and2024)
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b 

q)parse "select from t where any (and2024;and2024)" 
? 
`t 
,,(max$["b"];(enlist;(&;(=;`date;2023.05.20);(in;`data;,`a`b));(&;(=;`date;2023.05.19);(in;`data;(enlist;,`b))))) 
0b 
() 
q)?[`t;enlist(any;(enlist;(and;(=;`date;2023.05.20);(in;`data;enlist `a`b));(and;(=;`date;2023.05.19);(in;`data;(enlist;enlist`b)))));0b;()] 
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b 

q)f:((2023.05.20;`a`b);(2023.05.19;enlist `b);(2023.05.18;`c`d`a);(2023.05.17;`d`a)) 
q)?[`t;(enlist(any;enlist,{(and;(=;`date;x 0);(in;`data;enlist x 1))}each f));0b;()] 
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b

 

Much better if the data is on disk and partitioned by date would be to iterate over each date and not each filter:

 

q)f:flip `date`syms!flip f 
date syms 
----------------- 
2023.05.20 `a`b 
2023.05.19 ,`b 
2023.05.18 `c`d`a 
2023.05.17 `d`a 

q)raze {select from t where date=x`date,data in x`data} peach 0!select distinct raze data by date from f 
date data 
--------------- 
2023.05.19 b 
2023.05.19 b 
2023.05.20 a 
2023.05.20 b

 

[rocuinneagain]

Is this what you mean;

 

You have a q server with a function which logs some information:

 

q)p 5000 p)func:{-1 “Logging some info on params:”,.Q.s1 (x;y);x+y}

 

You call that function from PyKX:

 

>>> conn = pykx.QConnection(‘127.0.0.1’, 5000) >>> conn(“func”,4,5) pykx.LongAtom(pykx.q(‘9’))

 

In the q console the log is printed:

 

q) Logging some info on params:4 5

 

But you would like this message to be logged in your PyKX console?

The standard functionality will not print those messages logged to SDTOUT back to your python process.

[rocuinneagain]

https://code.kx.com/q/ref/accumulators/#ternary-values

You can use a sample to see how it progresses:

 

({-1 .Q.s1 `x`y`z!(x;y;z);x+y+z}/)[0;1 2 3;4 5 6] 
`x`y`z!0 1 4 
`x`y`z!5 2 5 
`x`y`z!12 3 6 21

 

And on the ssr example:

 

({-1 .Q.s1 `x`y`z!(x;y;z);ssr[x;y;z]}/)["results_%div_%dept.csv"; ("%div";"%dept"); ("banking";"m&a")] 
`x`y`z!("results_%div_%dept.csv";"%div";"banking") 
`x`y`z!("results_banking_%dept.csv";"%dept";"m&a") 
"results_banking_m&a.csv"

 

 

[rocuinneagain]

1. Does the file definitely exist?

 

ls -l /home/user/anaconda3/envs/pykx/lib/python3.9/site-packages/pykx/lib/l64/pykx.so

 

2. Does forcing the package to reinstall help?

 

pip install –upgrade –force-reinstall pykx

 

3. Is the conda environment pykx activated when a) You ran pip install? b) You started python?

4. When exactly do you get the error? Are you just starting python and running import pykx?

 

 

[rocuinneagain]

Yes this can be solved by ensuring gcloud is up to date and initialised:

 

gcloud components update gcloud init

 

 

[rocuinneagain]

The file is there:

kx-insights-packages > kdbInsightsCore > 4.0.2

Are you signed in with username and password?

[rocuinneagain]

Listed on https://code.kx.com/insights/1.5/core/install.html#license

A valid license is needed to run kdb Insights Core. For more information on licensing, please contact licadmin@kx.com.

[rocuinneagain]

https://github.com/KxSystems/cookbook/tree/master/r/rserve

You are only looking at the memory usage of the q process.

With Rserve R is a separate process. You need to monitor it’s memory usage also.

 

To free memory in q after creating a large object use .Q.gc[]. You can also start q with -g 1.

• https://code.kx.com/q/basics/cmdline/#-g-garbage-collection
• https://code.kx.com/q/ref/dotq/#gc-garbage-collect

[rocuinneagain]

You can delete R variables and run R garbage collection using Rcmd as above.