[rocuinneagain]

PyKX under q is replacing embedPy and is recommended if you are starting a new project (*Except for on Windows as support is still in development for this)

https://code.kx.com/pykx/2.2/pykx-under-q/intro.html

Install it with:

pip install pykx 
python -c "import pykx;pykx.install_into_QHOME()"

Then you can load in your python code (I placed it in a file called functions.py):

q)\l pykx.q 
q)loadPy:{.pykx.pyexec"exec(open('",(1_ string x),"').read())"} 
q)loadPy`:functions.py 
q)readJSONFile:.pykx.get[`readJsonFile;<] 
q)readJSONFile`sampleJSONFile.json 
Employees 
------------------------------------------------------------------------------- 
+`userId`jobTitleName`firstName`lastName`preferredFullName`employeeCode`regio.. 

q)(readJSONFile`sampleJSONFile.json)[0;`Employees] 

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
--------------------------------------------------------------------------------------------------------------------- 
rirani Developer    Romin     Irani    Romin Irani       E1           CA     408-1234567 romin.k.irani@gmail.com 
nirani Developer    Neil      Irani    Neil Irani        E2           CA     408-1111111 neilrirani@gmail.com 
thanks Program Directory Tom  Hanks    Tom Hanks         E3           CA     408-2222222 tomhanks@gmail.com

 

One  issue to watch for here though is all the data being brought back as symbols to q. For the lifetime of a q process all unique symbols are interned in memory and cannot be garage collected. You would need to update your code to convert columns to suitable datatypes.  Text data should mostly be passed to q as bytes so it is converted to type C in q.

meta (readJSONFile`sampleJSONFile.json)[0;`Employees] 
c                | t f a 
-----------------| ----- 
userId           | s 
jobTitleName     | s 
firstName        | s 
lastName         | s 
preferredFullName| s 
employeeCode     | s 
region           | s
phoneNumber      | s 
emailAddress     | s

 

Using q’s inbuilt JSON parser is simple and default textual data to type C which prevents any issues with unique symbols.

q).j.k raze read0`:sampleJSONFile.json

Employees| +`userId`jobTitleName`firstName`lastName`preferredFullName`employe..

q)(.j.k raze read0`:sampleJSONFile.json)`Employees

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
----------------------------------------------------------------------------------------------------------------------------- 
"rirani" "Developer" "Romin" "Irani" "Romin Irani" "E1" "CA" "408-1234567" "romin.k.irani@gmail.com" 
"nirani" "Developer" "Neil" "Irani" "Neil Irani" "E2" "CA" "408-1111111" "neilrirani@gmail.com" 
"thanks" "Program Directory" "Tom" "Hanks" "Tom Hanks" "E3" "CA" "408-2222222" "tomhanks@gmail.com" 

meta (.j.k raze read0`:sampleJSONFile.json)`Employees 
c                | t f a 
-----------------| ----- 
userId           | C 
jobTitleName     | C 
firstName        | C 
lastName         | C 
preferredFullName| C 
employeeCode     | C 
region           | C 
phoneNumber      | C 
emailAddress     | C

 

[rocuinneagain]

See: PYKX_SKIP_SIGNAL_OVERWRITE on

https://code.kx.com/pykx/2.2/user-guide/configuration.html

*New in 2.2.1

[rocuinneagain]

//Sample table

q)t:([] col1:("abc";"def");col2:("ghi";"jkl")) 
q)t col1 col2 ----------- "abc" "ghi" "def" "jkl" 
//qsql update 
update {"col1#",x} each col1,{"col2#",x} each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" 
//qsql improved using column name as variable 
update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" \parse the query to see functional form 
parse"update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t" ! `t () 0b `col1`col2!((k){x'y};({string[x],"#",y};,`col1);`col1);(k){x'y};({string[x],"#",y};,`col2);`col2)) 
//Simplify functional form 
![t;();0b;{x!{(each;{string[x],(enlist "#"),y}[x];x)}each x}`col1`col2] col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl"

[rocuinneagain]

The parallelism can only go one layer deep.

.i.ie These 2 statements end up executing the same path. In the first one the inner “peach“  can only run like an `each` as it is already in a thread:

data:8#enlist til 1000000 ts {{neg x} peach x} peach data 553 1968 ts {{neg x} each x} peach data 551 1936

For queries map-reduce still will be used to reduce the memory load of your nested queries even if run inside a “peach` even if not running the sub parts in parallel.

https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#1437-map-reduce

Where you choose to put your `peach` can be important and change the performance of your execution.

My example actually runs better without peach due to the overhead of passing data around versus `neg` being a simple operation

ts {{neg x} each x} each data 348 91498576

.Q.fc exists to help in these cases

ts {.Q.fc[{neg x};x]} each data 19 67110432

https://code.kx.com/q/ref/dotq/#fc-parallel-on-cut

And in fact since `neg` has native multithreading and operates on vectors and vectors of vectors it is best of off left on it’s own:

ts neg each data 5 67109216 
ts neg data 5 67109104 
neg data

This example of course is extreme but does show that thought and optimisation can go in to each use-case on where to choose to iterate and place `peach“

[rocuinneagain]

For normal sorting operations see:

• https://code.kx.com/q/ref/asc/
• https://code.kx.com/q/ref/desc/

q)tab:([] sym:`a`b`c`d`e; x:5?10) q)tab sym x ----- a 8 b 1 c 9 d 5 e 4 q)`sym xdesc tab sym x ----- e 4 d 5 c 9 b 1 a 8

For your request perhaps rotate is what you are looking for

• https://code.kx.com/q/ref/rotate/

q)2 rotate tab sym x ----- c 9 d 5 e 4 a 8 b 1

 

 

[rocuinneagain]

Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

{  "argv": [   "taskset",   "-c",   "0,1",   "q",   "jupyterq_kernel.q",   "-cds",   "{connection_file}"  ],  "display_name": "Q (kdb+)",  "language": "q",  "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"} } 

If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec 

And you can validate in a notebook with system"taskset -pc ",string .z.i:

 

?

[rocuinneagain]

Hi, What version of PyKX are you using?

.pykx.i.isw has been renamed to .pykx.util.isw since 2.0.0

[rocuinneagain]

You have tried to use the function .arrowkdb.tb.prettyPrintTableFromTable and received the error '.arrowkdb.tb.prettyPrintTableFromTable.

In kdb this means the variable does not exist:

 

q)doesNotExist[1] 'doesNotExist [0] doesNotExist[1]

 

This suggests you have yet to load the library.

Either pass it to the command line q arrowkdb.q or load it in your running session q)l arrowkdb.q

 

[rocuinneagain]

When you download q it is included. Folder structure looks like:

kc.lic 
w64
q 
q.k

As shown on https://code.kx.com/q/learn/install/#step-3-install-the-license-file

[rocuinneagain]

Use this link: https://kx.com/download-kdb/

[rocuinneagain]

This is due to an issue with the QHOME environment variable

See: https://code.kx.com/q/learn/install/

Specifically: https://code.kx.com/q/learn/install/#step-5-edit-your-profile

setx QHOME “C:q”

Note that this points to the folder containing q.k and not to the file itself

 

[rocuinneagain]

https://code.kx.com/q/basics/errors/#:~:text=formed%20select%20query-,hop,-Request%20to%20hopen

hop

Request to hopen a handle fails; includes message from OS

Your code is trying to open a handle to a process listening on port 5001 but one is not available.

 

 

[rocuinneagain]

Yes it could be used.

 

To test you could look at PyKX for an easy Python interface.

A 2 minute example of passing in a Dataset to q in shown below.

PyKX allows Registering Custom Conversions so you could create a function to pass the Dataset in exactly the form you wish to q instead of passing it all as a dictionary in my example.

 

import pykx as kx 
import xarray as xr 
import numpy as np 
import pandas as pd 

ds = xr.Dataset(     
      {"foo": (("x", "y"), np.random.rand(5, 5))},     
      coords={         "x": [10, 20, 30, 40, 50],         
                       "y": pd.date_range("2000-01-01", periods=5),         
                       "z": ("x", list("abcde")),     }, ) 

kx.q['ds'] = kx.toq(ds.to_dict()) 

kx.q('ds') 

pykx.Dictionary(pykx.q(' 
coords   | `x`y`z!+`dims`attrs`data!((,`x;,`y;,`x);(()!();()!();()!());(10 20.. attrs    | ()!() dims     | `x`y!5 5 data_vars| (,`foo)!+`dims`attrs`data!(,`x`y;,()!();,(0.7412575 0.2054306 0.10.. ')) 

kx.q('flip ds[`coords;;`data]') 

pykx.Table(pykx.q(' 
x  y                             z 
---------------------------------- 
10 2000.01.01D00:00:00.000000000 a 
20 2000.01.02D00:00:00.000000000 b 
30 2000.01.03D00:00:00.000000000 c 
40 2000.01.04D00:00:00.000000000 d 
50 2000.01.05D00:00:00.000000000 e ')) 

kx.q('ds[`data_vars;`foo;`data]') 

pykx.List(pykx.q(' 0.7412575 0.2054306   0.1009393 0.8792678 0.04105999 0.1811459 0.01659637  0.2406029 0.4900055 0.551788   0.6303767 0.0702013   0.6831359 0.5961667 0.3722388  0.9255059 0.9202499   0.5055902 0.9767793 0.7440498  0.7331576 0.003197568 0.4939932 0.5433492 0.01175784 ')) 

 

[rocuinneagain]

From 3.6 Readme

2018.09.26 NEW added -27! as a more precise, builtin version of .Q.f. n.b. It is atomic and doesn't take P into account. e.g.  q)("123456789.457";"123456790.457")~-27!(3i;0 1+2#123456789.4567)

The definition of .Q.f changed at this time also – when comparing q.k file definitions:

3.6.0 2018.09.10

f:{$[^y;"";y<0;"-",f[x;-y];y<1;1_f[x;10+y];9e15>j:"j"$y*/x#10;(x_j),".",(x:-x)#j:$j;$y]}

3.6.0 2018.10.03

f:{$[^y;"";y<0;"-",f[x;-y];y<1;1_f[x;10+y];9e15>j:"j"$y*prd x#10f;(x_j),".",(x:-x)#j:$j;$y]}

 

Floating point numbers are not exact and such behaviours should be expected in certain cases:

https://code.kx.com/q/basics/precision/

 

They are approximations:

q)P 
0 
q)4194303.975 
4194303.9750000001 
q)4194304.975 
4194304.9749999996

 

The same in C:

int main() {         double x = 4194303.975;         printf("%10.10fn",x); }

Result:

4194303.9750000001

 

If you find that you need exact decimal rounding, consider whether you actually need to operate in fixed-point, rather than floating-point.
e.g. keep monetary values in integral millicents etc.

 

 

 

[rocuinneagain]

You can take the old definition from the q.k file from the 2.8 version

And define it in your process with a new name:

k)wj1_28:{…

 

https://nexus.firstderivatives.com client login has downloads for versions of kdb+ including 2.8:

 KxReleases --> KDBPlus -->Releases

Some instead use: https://downloads.kx.com/