[rocuinneagain]

q)b@'(0;2;1)

1 76 12

[rocuinneagain]

Link to your original post for reference:

•  https://community.kx.com/t5/KX-Academy/Issues-with-Queries-and-qSQL/td-p/14099

You can test yourself and compare time and memory usage:

• https://code.kx.com/q/basics/syscmds/#ts-time-and-space
• https://code.kx.com/q/wp/columnar-database/
• https://code.kx.com/q/ref/set-attribute/

 

//Try and time the queries to compare 
\ts select last price by hour:60 xbar time.minute, sym from trade 
\ts select last price by sym,hour:60 xbar time.minute from trade 

//If they run very fast you can run them N number of times to get more accurate result 
//Running 100 times here 
\ts:100 select last price by hour:60 xbar time.minute, sym from trade 
\ts:100 select last price by sym,hour:60 xbar time.minute from trade 

//Attributes are useful in kdb+ 
//Apply grouped on sym and test 
update `g#sym from `trade 
\ts:100 select last price by hour:60 xbar time.minute, sym from trade 
\ts:100 select last price by sym,hour:60 xbar time.minute from trade 

//Removed grouped and test again 
update `#sym from `trade 
\ts:100 select last price by hour:60 xbar time.minute, sym from trade 
\ts:100 select last price by sym,hour:60 xbar time.minute from trade

 

 

[rocuinneagain]

The issue you are having is unrelated to peach/multithreading on the kdb+ side.

 

The issue is the reuse of the same connection among @task blocks.

The different @tasks are attempting to read data from the same connection concurrently leading to junk data being passed through.

 

kdb+ processes incoming queries sequentially. This means even through you see the log messages from the tasks as if they run in parallel in fact once these arrive to kdb+ they will be processed one after the other always. For this reason switching to @flow will not be slower and will result is safe consistent results.

 

 

[rocuinneagain]

The issue you are having is unrelated to peach/multithreading on the kdb+ side. The issue is the reuse of the same connection among@taskblocks. The different @tasks are attempting to read data from the same connection concurrently leading to junk data being passed through. kdb+ processes incoming queries sequentially. This means even through you see the log messages from the tasks as if they run in parallel in fact once these arrive to kdb+ they will be processed one after the other always. For this reason switching to @flow will not be slower and will result is safe consistent results.

[rocuinneagain]

Only PyKX has been released on open PyPi in that way.

The correction to the documentation for kxi will be online soon.

[rocuinneagain]

Pass your credentials and the URL to the KX repository:

pip install --extra-index-url=https://$KX_PYPI_USER:$KX_PYPI_PASS@nexus.dl.kx.com/repository/kxi/simple/ kxi

The documentation will be updated to correct this.

[rocuinneagain]

Hi,

1. Are you running PyKX under Prefect? http://www.prefect.io
2. Are you connecting and querying using PyKX in @task or @flow ?
3. If using @task does switching to @flow (.i.e subflows) stop the issue happening?
4. Do you see the same issue if you query in a standalone python process outside of Prefect?
5. Are you making a single query when the issue happens or are multiple queries being run?
6. What version of PyKX are you running? pykx.__version__
7. Are you running in licensed or unlicensed mode? pykx.licensed
8. How did you create the connection conn?

 

• Adding inline code to your questions & messages
• Creating q code block with syntax highlighting

[rocuinneagain]

1. There is no need for raze to be included.

 

q)select type close by sym from daily

sym | close 
----| ------ 
AAPL| 9h 
AIG | 9h 
AMD | 9h ..

 

Float vectors are being passed to your function (type 9h). raze has no effect on these.

 

2. select size by sym from daily only does grouping as no aggregate function is included.

It is the same as select {x} size by sym from daily

Including a real aggregate would be select last5 size by sym from daily

[rocuinneagain]

https://code.kx.com/insights/1.5/microservices/service-discovery/prerequisites.html#running-in-docker

To gain access please contact KX at licadmin@kx.com

[rocuinneagain]

If you went directly in to writing what you ask:

 

q)t:flip `date`data!(2023.05.20 2023.05.20 2023.05.20 2023.05.19 2023.05.19 2023.05.19;`a`c`b`b`b`c) 
q)select from t where any (and2024;and2024)
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b 

q)parse "select from t where any (and2024;and2024)" 
? 
`t 
,,(max$["b"];(enlist;(&;(=;`date;2023.05.20);(in;`data;,`a`b));(&;(=;`date;2023.05.19);(in;`data;(enlist;,`b))))) 
0b 
() 
q)?[`t;enlist(any;(enlist;(and;(=;`date;2023.05.20);(in;`data;enlist `a`b));(and;(=;`date;2023.05.19);(in;`data;(enlist;enlist`b)))));0b;()] 
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b 

q)f:((2023.05.20;`a`b);(2023.05.19;enlist `b);(2023.05.18;`c`d`a);(2023.05.17;`d`a)) 
q)?[`t;(enlist(any;enlist,{(and;(=;`date;x 0);(in;`data;enlist x 1))}each f));0b;()] 
date data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b

 

Much better if the data is on disk and partitioned by date would be to iterate over each date and not each filter:

 

q)f:flip `date`syms!flip f 
date syms 
----------------- 
2023.05.20 `a`b 
2023.05.19 ,`b 
2023.05.18 `c`d`a 
2023.05.17 `d`a 

q)raze {select from t where date=x`date,data in x`data} peach 0!select distinct raze data by date from f 
date data 
--------------- 
2023.05.19 b 
2023.05.19 b 
2023.05.20 a 
2023.05.20 b

 

[rocuinneagain]

Is this what you mean;

 

You have a q server with a function which logs some information:

 

q)p 5000 p)func:{-1 “Logging some info on params:”,.Q.s1 (x;y);x+y}

 

You call that function from PyKX:

 

>>> conn = pykx.QConnection(‘127.0.0.1’, 5000) >>> conn(“func”,4,5) pykx.LongAtom(pykx.q(‘9’))

 

In the q console the log is printed:

 

q) Logging some info on params:4 5

 

But you would like this message to be logged in your PyKX console?

The standard functionality will not print those messages logged to SDTOUT back to your python process.

[rocuinneagain]

https://code.kx.com/q/ref/accumulators/#ternary-values

You can use a sample to see how it progresses:

 

({-1 .Q.s1 `x`y`z!(x;y;z);x+y+z}/)[0;1 2 3;4 5 6] 
`x`y`z!0 1 4 
`x`y`z!5 2 5 
`x`y`z!12 3 6 21

 

And on the ssr example:

 

({-1 .Q.s1 `x`y`z!(x;y;z);ssr[x;y;z]}/)["results_%div_%dept.csv"; ("%div";"%dept"); ("banking";"m&a")] 
`x`y`z!("results_%div_%dept.csv";"%div";"banking") 
`x`y`z!("results_banking_%dept.csv";"%dept";"m&a") 
"results_banking_m&a.csv"

 

 

[rocuinneagain]

1. Does the file definitely exist?

 

ls -l /home/user/anaconda3/envs/pykx/lib/python3.9/site-packages/pykx/lib/l64/pykx.so

 

2. Does forcing the package to reinstall help?

 

pip install –upgrade –force-reinstall pykx

 

3. Is the conda environment pykx activated when a) You ran pip install? b) You started python?

4. When exactly do you get the error? Are you just starting python and running import pykx?

 

 

[rocuinneagain]

Yes this can be solved by ensuring gcloud is up to date and initialised:

 

gcloud components update gcloud init

 

 

[rocuinneagain]

Listed on https://code.kx.com/insights/1.5/core/install.html#license

A valid license is needed to run kdb Insights Core. For more information on licensing, please contact licadmin@kx.com.