[rocuinneagain]

You can index at depth:

records[;`SourceSystemName]

https://kx.com/blog/kdb-q-insights-parsing-json-files/

[rocuinneagain]

Copying my reply from under your request on a different thread:

Re: Getting License error – KX Community – 13316

 

First image looks fine for QHOME=/voiddump/q/m64

 

More traditional structure would be to not have that extra level of m64

.i.e

• voiddump/q/kc.lic
• voiddump/q/q.k
• voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

Your Windows VM screenshots are incorrect:

You have q/w64/w64/q it should be q/w64/q

[rocuinneagain]

First image looks fine for QHOME=/voiddump/q/m64

More traditional structure would be to not have that extra level of m64

.i.e

• voiddump/q/kc.lic
• voiddump/q/q.k
• voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

[rocuinneagain]

differ is not one of the aggregations which will auto perform map-reduce

https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#1437-map-reduce

In this case the operation is applied once per partition.

 

To have it operate on your full result set you can change your query to query the data from disk untouched and then perform the aggregation in memory:

select diffDate:differ startDate from select startDate from tab where date within(.z.d-2;.z.d-1)

 

[rocuinneagain]

Here is a basic example if you wanted to send some data from a Pandas Dataframe

>>> import pykx as kx 
>>> import pandas as pd 
>>> import numpy as np 
>>> conn = kx.SyncQConnection('localhost', 5000) 
>>> df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]), columns=['a', 'b', 'c']) 
>>> conn('tab:([]a:`long$();b:`long$();c:`long$())') pykx.Identity(pykx.q('::')) 
>>> conn('insert','tab',df) pykx.LongVector(pykx.q('0 1 2')) 
>>> conn('tab') pykx.Table(pykx.q(' a b c ----- 1 2 3 4 5 6 7 8 9 '))

The documentation website has more examples

https://code.kx.com/pykx

 

 

[rocuinneagain]

A splayed table on disk with an attribute on a column would be worth testing as these can be mapped rather than requiring to be all in memory

https://code.kx.com/q/ref/set-attribute/#unique

 

Using 1: to write an Anymap file also creates a mappable object worth exploring

https://code.kx.com/q/releases/ChangesIn3.6/#anymap

 

If a single splay/anymap would be too large a fixed size int partitioned DB on a fixed range hash of the alphanumeric string could be used

[rocuinneagain]

If you want to complete this with embedPy you will need the df2tab utility function

ml/util/utilities.q at master KxSystems/ml (github.com)

 

You can install the library following:

https://github.com/KxSystems/ml/tree/master?tab=readme-ov-file#installation

 

Or the function can also be defined stand alone if you prefer:

https://stackoverflow.com/a/51612291/4256419

 

 

 

//You cannot pass < as there is no automatic way to convert a dataframe to q readJSONFile:.p.get[`readJsonFile];

//Pass the embedPy foreign to df2tab to do the conversion out:df2tab[readJSONFile[fp]];

 

 

 

[rocuinneagain]

PyKX under q is replacing embedPy and is recommended if you are starting a new project (*Except for on Windows as support is still in development for this)

https://code.kx.com/pykx/2.2/pykx-under-q/intro.html

Install it with:

pip install pykx 
python -c "import pykx;pykx.install_into_QHOME()"

Then you can load in your python code (I placed it in a file called functions.py):

q)\l pykx.q 
q)loadPy:{.pykx.pyexec"exec(open('",(1_ string x),"').read())"} 
q)loadPy`:functions.py 
q)readJSONFile:.pykx.get[`readJsonFile;<] 
q)readJSONFile`sampleJSONFile.json 
Employees 
------------------------------------------------------------------------------- 
+`userId`jobTitleName`firstName`lastName`preferredFullName`employeeCode`regio.. 

q)(readJSONFile`sampleJSONFile.json)[0;`Employees] 

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
--------------------------------------------------------------------------------------------------------------------- 
rirani Developer    Romin     Irani    Romin Irani       E1           CA     408-1234567 romin.k.irani@gmail.com 
nirani Developer    Neil      Irani    Neil Irani        E2           CA     408-1111111 neilrirani@gmail.com 
thanks Program Directory Tom  Hanks    Tom Hanks         E3           CA     408-2222222 tomhanks@gmail.com

 

One  issue to watch for here though is all the data being brought back as symbols to q. For the lifetime of a q process all unique symbols are interned in memory and cannot be garage collected. You would need to update your code to convert columns to suitable datatypes.  Text data should mostly be passed to q as bytes so it is converted to type C in q.

meta (readJSONFile`sampleJSONFile.json)[0;`Employees] 
c                | t f a 
-----------------| ----- 
userId           | s 
jobTitleName     | s 
firstName        | s 
lastName         | s 
preferredFullName| s 
employeeCode     | s 
region           | s
phoneNumber      | s 
emailAddress     | s

 

Using q’s inbuilt JSON parser is simple and default textual data to type C which prevents any issues with unique symbols.

q).j.k raze read0`:sampleJSONFile.json

Employees| +`userId`jobTitleName`firstName`lastName`preferredFullName`employe..

q)(.j.k raze read0`:sampleJSONFile.json)`Employees

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
----------------------------------------------------------------------------------------------------------------------------- 
"rirani" "Developer" "Romin" "Irani" "Romin Irani" "E1" "CA" "408-1234567" "romin.k.irani@gmail.com" 
"nirani" "Developer" "Neil" "Irani" "Neil Irani" "E2" "CA" "408-1111111" "neilrirani@gmail.com" 
"thanks" "Program Directory" "Tom" "Hanks" "Tom Hanks" "E3" "CA" "408-2222222" "tomhanks@gmail.com" 

meta (.j.k raze read0`:sampleJSONFile.json)`Employees 
c                | t f a 
-----------------| ----- 
userId           | C 
jobTitleName     | C 
firstName        | C 
lastName         | C 
preferredFullName| C 
employeeCode     | C 
region           | C 
phoneNumber      | C 
emailAddress     | C

 

[rocuinneagain]

See: PYKX_SKIP_SIGNAL_OVERWRITE on

https://code.kx.com/pykx/2.2/user-guide/configuration.html

*New in 2.2.1

[rocuinneagain]

//Sample table

q)t:([] col1:("abc";"def");col2:("ghi";"jkl")) 
q)t col1 col2 ----------- "abc" "ghi" "def" "jkl" 
//qsql update 
update {"col1#",x} each col1,{"col2#",x} each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" 
//qsql improved using column name as variable 
update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" \parse the query to see functional form 
parse"update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t" ! `t () 0b `col1`col2!((k){x'y};({string[x],"#",y};,`col1);`col1);(k){x'y};({string[x],"#",y};,`col2);`col2)) 
//Simplify functional form 
![t;();0b;{x!{(each;{string[x],(enlist "#"),y}[x];x)}each x}`col1`col2] col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl"

[rocuinneagain]

The parallelism can only go one layer deep.

.i.ie These 2 statements end up executing the same path. In the first one the inner “peach“  can only run like an `each` as it is already in a thread:

data:8#enlist til 1000000 ts {{neg x} peach x} peach data 553 1968 ts {{neg x} each x} peach data 551 1936

For queries map-reduce still will be used to reduce the memory load of your nested queries even if run inside a “peach` even if not running the sub parts in parallel.

https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#1437-map-reduce

Where you choose to put your `peach` can be important and change the performance of your execution.

My example actually runs better without peach due to the overhead of passing data around versus `neg` being a simple operation

ts {{neg x} each x} each data 348 91498576

.Q.fc exists to help in these cases

ts {.Q.fc[{neg x};x]} each data 19 67110432

https://code.kx.com/q/ref/dotq/#fc-parallel-on-cut

And in fact since `neg` has native multithreading and operates on vectors and vectors of vectors it is best of off left on it’s own:

ts neg each data 5 67109216 
ts neg data 5 67109104 
neg data

This example of course is extreme but does show that thought and optimisation can go in to each use-case on where to choose to iterate and place `peach“

[rocuinneagain]

For normal sorting operations see:

• https://code.kx.com/q/ref/asc/
• https://code.kx.com/q/ref/desc/

q)tab:([] sym:`a`b`c`d`e; x:5?10) q)tab sym x ----- a 8 b 1 c 9 d 5 e 4 q)`sym xdesc tab sym x ----- e 4 d 5 c 9 b 1 a 8

For your request perhaps rotate is what you are looking for

• https://code.kx.com/q/ref/rotate/

q)2 rotate tab sym x ----- c 9 d 5 e 4 a 8 b 1

 

 

[rocuinneagain]

Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

{  "argv": [   "taskset",   "-c",   "0,1",   "q",   "jupyterq_kernel.q",   "-cds",   "{connection_file}"  ],  "display_name": "Q (kdb+)",  "language": "q",  "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"} } 

If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec 

And you can validate in a notebook with system"taskset -pc ",string .z.i:

 

?

[rocuinneagain]

Hi, What version of PyKX are you using?

.pykx.i.isw has been renamed to .pykx.util.isw since 2.0.0

[rocuinneagain]

You have tried to use the function .arrowkdb.tb.prettyPrintTableFromTable and received the error '.arrowkdb.tb.prettyPrintTableFromTable.

In kdb this means the variable does not exist:

 

q)doesNotExist[1] 'doesNotExist [0] doesNotExist[1]

 

This suggests you have yet to load the library.

Either pass it to the command line q arrowkdb.q or load it in your running session q)l arrowkdb.q