Replies – Forums – rocuinneagain – Page 6

rocuinneagain

Member

March 19, 2024 at 11:29 am in reply to: How to convert sym columns to guid or to original data

Can you include some more code to explain what the issue is?

To create a symbol vector you are using:

`symbol$()

but you should use:

`$()

.i.e

q)order_histories:([]time:`timespan$();sym:`$();user_uuid:`$();order_id:`long$();instrument:`$())
q)order_histories
time sym user_uuid order_id instrument
--------------------------------------

Insert some data and set a port:

q)order_histories insert (2?.z.N;2?`6;2?`6;2?100000;2?`6)
0 1
q)order_histories
time                 sym    user_uuid order_id instrument
---------------------------------------------------------
0D09:21:01.649029855 kfmohp kfhogj    81418    milgli
0D05:24:41.981647398 lkklco jecpae    75009    igfbag
q)\p 5000

Query from PyKX

>>> import pykx as kx
>>> conn('order_histories')
pykx.Table(pykx.q('
time                 sym    user_uuid order_id instrument
---------------------------------------------------------
0D09:21:01.649029855 kfmohp kfhogj    81418    milgli
0D05:24:41.981647398 lkklco jecpae    75009    igfbag
'))
>>> conn('order_histories').pd()
                       time     sym user_uuid  order_id instrument
0 0 days 09:21:01.649029855  kfmohp    kfhogj     81418     milgli
1 0 days 05:24:41.981647398  lkklco    jecpae     75009     igfbag

rocuinneagain

Member

March 19, 2024 at 11:01 am in reply to: Five easy pieces #3: Mark this

A few different variations:

-1 {i:where ""~/:x;raze "<html><body><p>",@[x;i;:;count[i]#enlist "</p><p>"],"</p></body></html>"}
read0 `:smile.txt 
-1"<html><body><p>",,[;"</p></body></html>"]
ssr[;"n";""]
"</p><p>" sv "nn" vs`char$read1`:smile.txt 
-1"<html><body><p>",ssr["</p><p>"sv"nn"vs`char$read1`:smile.txt;"n";""],"</p></body></html>" 
-1{.h.htc[y;x]}over(ssr["</p><p>"sv"nn"vs`char$read1`:smile.txt;"n";""];`p;`body;`html) 
-1({.h.htc[y;x]})(ssr["</p><p>"sv"nn"vs`char$read1`:smile.txt;"n";""];`p;`body;`html)

rocuinneagain

Member

February 23, 2024 at 12:00 am in reply to: Interaction between peach and other optimisations

kdb+ 4.1 has been released with some interesting improvements for peach which changes some of my answers as nesting is now supported

https://code.kx.com/q//releases/ChangesIn4.1/#peachparallel-processing-enhancements

rocuinneagain

Member

February 9, 2024 at 12:00 am in reply to: PyKX could not establish an IPC connection in the Flask app

You example will run if you wait and only import pykx within the app

from flask import Flask 
#import pykx as kx <-- remove this 
import json as j 
app = Flask(__name__)

@app.route("/") 
def hello_kx(): 
    import pykx as kx 
    qcon=kx.QConnection(host='localhost', port=5000) 
    tables=qcon("tables[]").py() 
    import json as j 
    print(j.dumps(ports)) 
    return tables 

if __name__ == '__main__': 
    app.run(host='127.0.0.1', debug=True, port=8100)

rocuinneagain

Member

February 9, 2024 at 12:00 am in reply to: PyKX could not establish an IPC connection in the Flask app

Or if you only use PyKX for querying then you can use in unlicensed mode

https://code.kx.com/pykx/2.3/user-guide/advanced/modes.html#operating-in-the-absence-of-a-kx-license

from flask import Flask 
import os 
os.environ['PYKX_UNLICENSED']='true' 

import pykx as kx 
import json as j 
app = Flask(__name__) 

@app.route("/") 
def hello_kx(): 
    #import pykx as kx 
    qcon=kx.QConnection(host='localhost', port=5000) 
    tables=qcon("tables[]").py() 
    #import json as j 
    #print(j.dumps(ports)) 
    return tables 

if __name__ == '__main__': 
    app.run(host='127.0.0.1', debug=True, port=8100)

rocuinneagain

Member

February 2, 2024 at 12:00 am in reply to: JupyterQ(Docker) fails to get license

Pass your license through and this should run:

docker run -it –name myjupyterq -v $QHOME/kc.lic:/home/kx/kc.lic -p 8888:8888 kxsys/jupyterq

Replace kc.lic with k4.lic if that is your license type
You can use /a/path/to/kc.lic if you have not set QHOME

rocuinneagain

Member

February 2, 2024 at 12:00 am in reply to: How to convert below functional select query to a QSQL format.

The first example line you show is creating parse trees and then evaluating them with value

https://code.kx.com/q/basics/parsetrees/

You then replace the parse tree with a qsql statement which evaluates, this means you no longer need value:

tables[]!{count select from x} each tables[]

The debugger uses ^ to show you exactly where the error is thrown

// ^ tells us the error happened when value was run

q)tables[] ! ( { count value ( select from x ) } each tables [] ) 'type [2] { count value ( select from x ) } ^ 

q))select from x // select from x returns a table a - 1 2 3 q))value select from x //Running value on a table is not a valid command 'type [4] value select from x ^ 

q))count select from x //We should run count directly on the returned table 3

See: https://code.kx.com/q/basics/debug/

rocuinneagain

Member

January 22, 2024 at 12:00 am in reply to: Syntax to extract a particular key-value from BSON message

You can index at depth:

records[;`SourceSystemName]

https://kx.com/blog/kdb-q-insights-parsing-json-files/

rocuinneagain

Member

January 15, 2024 at 12:00 am in reply to: Getting License error

First image looks fine for QHOME=/voiddump/q/m64

More traditional structure would be to not have that extra level of m64

.i.e

voiddump/q/kc.lic
voiddump/q/q.k
voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

rocuinneagain

Member

January 15, 2024 at 12:00 am in reply to: How to address ‘License error: k4.lic’ during installation?

Copying my reply from under your request on a different thread:

Re: Getting License error – KX Community – 13316

First image looks fine for QHOME=/voiddump/q/m64

More traditional structure would be to not have that extra level of m64

.i.e

voiddump/q/kc.lic
voiddump/q/q.k
voiddump/q/m64/q

In that case you’d use QHOME=/voiddump/q

Your Windows VM screenshots are incorrect:

You have q/w64/w64/q it should be q/w64/q

rocuinneagain

Member

January 12, 2024 at 12:00 am in reply to: differ applied on each day rather then the entire dataset

differ is not one of the aggregations which will auto perform map-reduce

https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#1437-map-reduce

In this case the operation is applied once per partition.

To have it operate on your full result set you can change your query to query the data from disk untouched and then perform the aggregation in memory:

select diffDate:differ startDate from select startDate from tab where date within(.z.d-2;.z.d-1)

rocuinneagain

Member

January 9, 2024 at 12:00 am in reply to: how to push data to RDB/HDB using pykx

Here is a basic example if you wanted to send some data from a Pandas Dataframe

>>> import pykx as kx 
>>> import pandas as pd 
>>> import numpy as np 
>>> conn = kx.SyncQConnection('localhost', 5000) 
>>> df = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9]]), columns=['a', 'b', 'c']) 
>>> conn('tab:([]a:`long$();b:`long$();c:`long$())') pykx.Identity(pykx.q('::')) 
>>> conn('insert','tab',df) pykx.LongVector(pykx.q('0 1 2')) 
>>> conn('tab') pykx.Table(pykx.q(' a b c ----- 1 2 3 4 5 6 7 8 9 '))

The documentation website has more examples

https://code.kx.com/pykx

rocuinneagain

Member

January 4, 2024 at 12:00 am in reply to: Key Value Store

A splayed table on disk with an attribute on a column would be worth testing as these can be mapped rather than requiring to be all in memory

https://code.kx.com/q/ref/set-attribute/#unique

Using 1: to write an Anymap file also creates a mappable object worth exploring

https://code.kx.com/q/releases/ChangesIn3.6/#anymap

If a single splay/anymap would be too large a fixed size int partitioned DB on a fixed range hash of the alphanumeric string could be used

rocuinneagain

Member

December 21, 2023 at 12:00 am in reply to: Read JSON file output in KDB

If you want to complete this with embedPy you will need the df2tab utility function

ml/util/utilities.q at master KxSystems/ml (github.com)

You can install the library following:

https://github.com/KxSystems/ml/tree/master?tab=readme-ov-file#installation

Or the function can also be defined stand alone if you prefer:

https://stackoverflow.com/a/51612291/4256419

//You cannot pass < as there is no automatic way to convert a dataframe to q readJSONFile:.p.get[`readJsonFile];

//Pass the embedPy foreign to df2tab to do the conversion out:df2tab[readJSONFile[fp]];

rocuinneagain

Member

March 20, 2024 at 4:37 pm in reply to: How to convert sym columns to guid or to original data

Can you share a small sample of your PyKX code which shows the issue?

rocuinneagain

Forum Replies Created