[rocuinneagain]

You example will run if you wait and only import pykx within the app

from flask import Flask 
#import pykx as kx <-- remove this 
import json as j 
app = Flask(__name__)

@app.route("/") 
def hello_kx(): 
    import pykx as kx 
    qcon=kx.QConnection(host='localhost', port=5000) 
    tables=qcon("tables[]").py() 
    import json as j 
    print(j.dumps(ports)) 
    return tables 

if __name__ == '__main__': 
    app.run(host='127.0.0.1', debug=True, port=8100)

[rocuinneagain]

Or if you only use PyKX for querying then you can use in unlicensed mode

https://code.kx.com/pykx/2.3/user-guide/advanced/modes.html#operating-in-the-absence-of-a-kx-license

from flask import Flask 
import os 
os.environ['PYKX_UNLICENSED']='true' 

import pykx as kx 
import json as j 
app = Flask(__name__) 

@app.route("/") 
def hello_kx(): 
    #import pykx as kx 
    qcon=kx.QConnection(host='localhost', port=5000) 
    tables=qcon("tables[]").py() 
    #import json as j 
    #print(j.dumps(ports)) 
    return tables 

if __name__ == '__main__': 
    app.run(host='127.0.0.1', debug=True, port=8100)

[rocuinneagain]

Pass your license through and this should run:

docker run -it –name myjupyterq -v $QHOME/kc.lic:/home/kx/kc.lic -p 8888:8888 kxsys/jupyterq

• Replace kc.lic with k4.lic if that is your license type
• You can use /a/path/to/kc.lic if you have not set QHOME

[rocuinneagain]

The first example line you show is creating parse trees and then evaluating them with value

https://code.kx.com/q/basics/parsetrees/

You then replace the parse tree with a qsql statement which evaluates, this means you no longer need value:

tables[]!{count select from x} each tables[]

The debugger uses ^ to show you exactly where the error is thrown

// ^ tells us the error happened when value was run

q)tables[] ! ( { count value ( select from x ) } each tables [] ) 'type [2] { count value ( select from x ) } ^ 

q))select from x // select from x returns a table a - 1 2 3 q))value select from x //Running value on a table is not a valid command 'type [4] value select from x ^ 

q))count select from x //We should run count directly on the returned table 3

See: https://code.kx.com/q/basics/debug/

[rocuinneagain]

You can index at depth:

records[;`SourceSystemName]

https://kx.com/blog/kdb-q-insights-parsing-json-files/

[rocuinneagain]

First image looks fine for QHOME=/voiddump/q/m64

More traditional structure would be to not have that extra level of m64

.i.e

• voiddump/q/kc.lic
• voiddump/q/q.k
• voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

[rocuinneagain]

Copying my reply from under your request on a different thread:

Re: Getting License error – KX Community – 13316

 

First image looks fine for QHOME=/voiddump/q/m64

 

More traditional structure would be to not have that extra level of m64

.i.e

• voiddump/q/kc.lic
• voiddump/q/q.k
• voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

Your Windows VM screenshots are incorrect:

You have q/w64/w64/q it should be q/w64/q

[rocuinneagain]

differ is not one of the aggregations which will auto perform map-reduce

https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#1437-map-reduce

In this case the operation is applied once per partition.

 

To have it operate on your full result set you can change your query to query the data from disk untouched and then perform the aggregation in memory:

select diffDate:differ startDate from select startDate from tab where date within(.z.d-2;.z.d-1)

 

[rocuinneagain]

Here is a basic example if you wanted to send some data from a Pandas Dataframe

>>> import pykx as kx 
>>> import pandas as pd 
>>> import numpy as np 
>>> conn = kx.SyncQConnection('localhost', 5000) 
>>> df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]), columns=['a', 'b', 'c']) 
>>> conn('tab:([]a:`long$();b:`long$();c:`long$())') pykx.Identity(pykx.q('::')) 
>>> conn('insert','tab',df) pykx.LongVector(pykx.q('0 1 2')) 
>>> conn('tab') pykx.Table(pykx.q(' a b c ----- 1 2 3 4 5 6 7 8 9 '))

The documentation website has more examples

https://code.kx.com/pykx

 

 

[rocuinneagain]

A splayed table on disk with an attribute on a column would be worth testing as these can be mapped rather than requiring to be all in memory

https://code.kx.com/q/ref/set-attribute/#unique

 

Using 1: to write an Anymap file also creates a mappable object worth exploring

https://code.kx.com/q/releases/ChangesIn3.6/#anymap

 

If a single splay/anymap would be too large a fixed size int partitioned DB on a fixed range hash of the alphanumeric string could be used

[rocuinneagain]

If you want to complete this with embedPy you will need the df2tab utility function

ml/util/utilities.q at master KxSystems/ml (github.com)

 

You can install the library following:

https://github.com/KxSystems/ml/tree/master?tab=readme-ov-file#installation

 

Or the function can also be defined stand alone if you prefer:

https://stackoverflow.com/a/51612291/4256419

 

 

 

//You cannot pass < as there is no automatic way to convert a dataframe to q readJSONFile:.p.get[`readJsonFile];

//Pass the embedPy foreign to df2tab to do the conversion out:df2tab[readJSONFile[fp]];

 

 

 

[rocuinneagain]

PyKX under q is replacing embedPy and is recommended if you are starting a new project (*Except for on Windows as support is still in development for this)

https://code.kx.com/pykx/2.2/pykx-under-q/intro.html

Install it with:

pip install pykx 
python -c "import pykx;pykx.install_into_QHOME()"

Then you can load in your python code (I placed it in a file called functions.py):

q)\l pykx.q 
q)loadPy:{.pykx.pyexec"exec(open('",(1_ string x),"').read())"} 
q)loadPy`:functions.py 
q)readJSONFile:.pykx.get[`readJsonFile;<] 
q)readJSONFile`sampleJSONFile.json 
Employees 
------------------------------------------------------------------------------- 
+`userId`jobTitleName`firstName`lastName`preferredFullName`employeeCode`regio.. 

q)(readJSONFile`sampleJSONFile.json)[0;`Employees] 

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
--------------------------------------------------------------------------------------------------------------------- 
rirani Developer    Romin     Irani    Romin Irani       E1           CA     408-1234567 romin.k.irani@gmail.com 
nirani Developer    Neil      Irani    Neil Irani        E2           CA     408-1111111 neilrirani@gmail.com 
thanks Program Directory Tom  Hanks    Tom Hanks         E3           CA     408-2222222 tomhanks@gmail.com

 

One  issue to watch for here though is all the data being brought back as symbols to q. For the lifetime of a q process all unique symbols are interned in memory and cannot be garage collected. You would need to update your code to convert columns to suitable datatypes.  Text data should mostly be passed to q as bytes so it is converted to type C in q.

meta (readJSONFile`sampleJSONFile.json)[0;`Employees] 
c                | t f a 
-----------------| ----- 
userId           | s 
jobTitleName     | s 
firstName        | s 
lastName         | s 
preferredFullName| s 
employeeCode     | s 
region           | s
phoneNumber      | s 
emailAddress     | s

 

Using q’s inbuilt JSON parser is simple and default textual data to type C which prevents any issues with unique symbols.

q).j.k raze read0`:sampleJSONFile.json

Employees| +`userId`jobTitleName`firstName`lastName`preferredFullName`employe..

q)(.j.k raze read0`:sampleJSONFile.json)`Employees

userId jobTitleName firstName lastName preferredFullName employeeCode region phoneNumber emailAddress 
----------------------------------------------------------------------------------------------------------------------------- 
"rirani" "Developer" "Romin" "Irani" "Romin Irani" "E1" "CA" "408-1234567" "romin.k.irani@gmail.com" 
"nirani" "Developer" "Neil" "Irani" "Neil Irani" "E2" "CA" "408-1111111" "neilrirani@gmail.com" 
"thanks" "Program Directory" "Tom" "Hanks" "Tom Hanks" "E3" "CA" "408-2222222" "tomhanks@gmail.com" 

meta (.j.k raze read0`:sampleJSONFile.json)`Employees 
c                | t f a 
-----------------| ----- 
userId           | C 
jobTitleName     | C 
firstName        | C 
lastName         | C 
preferredFullName| C 
employeeCode     | C 
region           | C 
phoneNumber      | C 
emailAddress     | C

 

[rocuinneagain]

See: PYKX_SKIP_SIGNAL_OVERWRITE on

https://code.kx.com/pykx/2.2/user-guide/configuration.html

*New in 2.2.1

[rocuinneagain]

//Sample table

q)t:([] col1:("abc";"def");col2:("ghi";"jkl")) 
q)t col1 col2 ----------- "abc" "ghi" "def" "jkl" 
//qsql update 
update {"col1#",x} each col1,{"col2#",x} each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" 
//qsql improved using column name as variable 
update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl" \parse the query to see functional form 
parse"update {string[x],"#",y}[`col1] each col1,{string[x],"#",y}[`col2] each col2 from t" ! `t () 0b `col1`col2!((k){x'y};({string[x],"#",y};,`col1);`col1);(k){x'y};({string[x],"#",y};,`col2);`col2)) 
//Simplify functional form 
![t;();0b;{x!{(each;{string[x],(enlist "#"),y}[x];x)}each x}`col1`col2] col1 col2 
--------------------- 
"col1#abc" "col2#ghi" "col1#def" "col2#jkl"

[rocuinneagain]

The parallelism can only go one layer deep.

.i.ie These 2 statements end up executing the same path. In the first one the inner “peach“  can only run like an `each` as it is already in a thread:

data:8#enlist til 1000000 ts {{neg x} peach x} peach data 553 1968 ts {{neg x} each x} peach data 551 1936

For queries map-reduce still will be used to reduce the memory load of your nested queries even if run inside a “peach` even if not running the sub parts in parallel.

https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#1437-map-reduce

Where you choose to put your `peach` can be important and change the performance of your execution.

My example actually runs better without peach due to the overhead of passing data around versus `neg` being a simple operation

ts {{neg x} each x} each data 348 91498576

.Q.fc exists to help in these cases

ts {.Q.fc[{neg x};x]} each data 19 67110432

https://code.kx.com/q/ref/dotq/#fc-parallel-on-cut

And in fact since `neg` has native multithreading and operates on vectors and vectors of vectors it is best of off left on it’s own:

ts neg each data 5 67109216 
ts neg data 5 67109104 
neg data

This example of course is extreme but does show that thought and optimisation can go in to each use-case on where to choose to iterate and place `peach“