rocuinneagain
Forum Replies Created
-
t:([] payment_type:10#
CARDCASH;vendor:10#abcd`e;fare:10?50;tip:10?50)
t
payment_type vendor fare tip
----------------------------
CARD a 30 39
CASH b 17 0
CARD c 23 49
CASH d 44 46
CARD e 12 41
CASH a 2 18
CARD b 36 49
CASH c 37 40
CARD d 44 24
CASH e 28 13
select from t where ({exec (tip=max tip) and (fare>avg fare) from x};([] tip;fare)) fby vendor
payment_type vendor fare tip
----------------------------
CARD a 30 39
CARD b 36 49You can pass in sub tables to aggregate on more than one column
fby – Reference – kdb+ and q documentation – kdb+ and q documentation (kx.com)
-
Sample table:
q)t:([] stock:10000?`1;AskOrder:10000?100;BidOrder:10000?100)
q)t
stock AskOrder BidOrder
-----------------------
n 85 43
p 40 15
n 9 47
j 10 64
i 87 88
i 79 87
m 46 41
h 92 63
g 86 39
......Function which returns a dictionary of percentiles:
q)f:{(`$x,/:string 1+til y)!az -1+(where deltas y xrank az:asc z),count z}Running the function returns a dictionary:
q)r:exec f["Ask_";4;AskOrder],f["Bid_";4;BidOrder] by stock from t
q)r
| AskOrder BidOrder
-| -------------------------------------------------------------------------
a| `Ask_1`Ask_2`Ask_3`Ask_4!25 50 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 51 74 99
b| `Ask_1`Ask_2`Ask_3`Ask_4!26 49 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 47 74 99
c| `Ask_1`Ask_2`Ask_3`Ask_4!23 49 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!26 49 75 99
d| `Ask_1`Ask_2`Ask_3`Ask_4!29 53 78 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 74 99
e| `Ask_1`Ask_2`Ask_3`Ask_4!22 46 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!26 54 79 99
f| `Ask_1`Ask_2`Ask_3`Ask_4!25 49 75 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 48 74 99
g| `Ask_1`Ask_2`Ask_3`Ask_4!26 49 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 76 99
h| `Ask_1`Ask_2`Ask_3`Ask_4!24 51 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 52 76 99
i| `Ask_1`Ask_2`Ask_3`Ask_4!24 48 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 49 72 99
j| `Ask_1`Ask_2`Ask_3`Ask_4!23 46 72 99 `Bid_1`Bid_2`Bid_3`Bid_4!31 53 75 99
k| `Ask_1`Ask_2`Ask_3`Ask_4!21 44 72 99 `Bid_1`Bid_2`Bid_3`Bid_4!22 50 75 99
l| `Ask_1`Ask_2`Ask_3`Ask_4!24 49 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 73 99
m| `Ask_1`Ask_2`Ask_3`Ask_4!23 45 70 99 `Bid_1`Bid_2`Bid_3`Bid_4!25 52 78 99
n| `Ask_1`Ask_2`Ask_3`Ask_4!26 52 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!25 53 76 99
o| `Ask_1`Ask_2`Ask_3`Ask_4!22 50 75 99 `Bid_1`Bid_2`Bid_3`Bid_4!24 47 74 99
p| `Ask_1`Ask_2`Ask_3`Ask_4!24 51 76 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 48 74 99Transforming the dictionary in to a table:
q){`stock xcols update stock:key x from (value x)[`AskOrder],'(value x)[`BidOrder]} r
stock Ask_1 Ask_2 Ask_3 Ask_4 Bid_1 Bid_2 Bid_3 Bid_4
-----------------------------------------------------
a 25 50 74 99 23 51 74 99
b 26 49 77 99 23 47 74 99
c 23 49 74 99 26 49 75 99
d 29 53 78 99 27 50 74 99
e 22 46 73 99 26 54 79 99
f 25 49 75 99 23 48 74 99
g 26 49 73 99 27 50 76 99
h 24 51 77 99 27 52 76 99
i 24 48 74 99 23 49 72 99
j 23 46 72 99 31 53 75 99
k 21 44 72 99 22 50 75 99
l 24 49 73 99 27 50 73 99
m 23 45 70 99 25 52 78 99
n 26 52 77 99 25 53 76 99
o 22 50 75 99 24 47 74 99
p 24 51 76 99 23 48 74 99 -
It is advised to avoid using name of column or globals as function param names as much as possible. Here you are seeing issues as a result of this.
In a HDB the issues caused by this scoping confusion can differ from in memory tables.
This is due to map-reduce in the HDB, which means the query execution path is more complicated.
1. type error:
In the query path date would be expected to be a list but if the function local date atom is used a type error is being raised
q){2025 select cnt:count i from Orders where date=2024.07.15}[.z.d] //Errorq){[] select cnt:count i from Orders where date=2024.07.15}[.z.d] //Okay2. count = 0:
You enlist the data param in one of the uses which means = returns a list and does not get the type error of query 1. However this list value does not make sense so an unexpected value is returned.
Returns the count of the table in the first partition of the HDB
q){2025 select cnt:count i from Orders where date=enlist 2024.07.15}[2024.07.15]Same result
q){2025 select cnt:count i from Orders where date=2024.07.15}[enlist 2024.07.15]Returns the counts of the first 2 partitions
q){2025 select cnt:{enlist count x}i from Orders where date=.z.d}[2#.z.d]You don’t even need to put in dates as the date param is being compared to itself you can put anything in:
q){2025 select cnt:{enlist count x}i from Orders where date=1b}[11b]3. Works: Not using date as a param name – no issues
-
PyKX also will convert a hsym directly to a Path type object
q).pykx.print[.pykx.eval["lambda x:x"]`:/path/to/file]
/path/to/file
q).pykx.print[.pykx.eval["lambda x:type(x)"]`:/path/to/file]
<class 'pathlib.PosixPath'>If you want a string of the Path you can call:
streamer:.pylibs.security[`:smStreamer] .pykx.eval["lambda x:str(x)"] file
-
See: Upgrading from embedPy – PyKX
PyKX converts Python strings to q symbols which differs from embedPy
q).pykx.eval["'hello'"]`
`helloPass your data back as Bytes for it to become q character vectors
q)string_to_bytes:.pykx.eval["lambda x: bytes(x, 'utf-8')"]
q)string_to_bytes[.pykx.eval["'hello'"]]`
"hello" -
8 is a documented limit
https://code.kx.com/q/ref/deal/#generate
symbols, each of n chars (n≤8)
To create 3 random symbols of length 20:
q){`$y cut (x*y)?16#.Q.a}[3;20]
`hgpfbbonnjlpppnpjago`mebadmhpkcjgoheikpmh`pbkbabaofjhmacenfabc
-
I expect this entry in 4.0 README for 2022.04.15 is the version from where you will see the improvement:
2022.04.15
NEW
anymap write now detects consecutive deduplicated (address matching) toplevel objects, skipping them to save space
q)a:("hi";"there";"world");`:a0 set a;`:a1 set a@where 1000 2000 3000;(hcount`$":a0#")=hcount`$":a1#"
improved memory efficiency of writing nested data sourced from a type 77 file, commonly encountered during compression of files. e.g.
q)`:a set 500000 100#"abc";system"ts `:b set get`:a" / was 76584400 bytes, now 8390720 -
<div>Can you test against a newer version of 4.0? </div>
My 4.1 gets much improved numbers:
q)(.z.K;.z.k)
4.1
2024.04.29
q)n:10000000;tab:([]time:n#.z.p;val:n?1000;str:n#enlist "");(`:tab/;17;2;5) set tab
`:tab/
q)-21!`:tab/str
compressedLength | 136807
uncompressedLength| 80004096
algorithm | 2i
logicalBlockSize | 17i
zipLevel | 5i
//Your compression 5.6x
q)80004096%14074225
5.684441
//Compression now 584x
80004096%136807
584.7953
q)-21!`$":tab/str#"
compressedLength | 93
uncompressedLength| 4098
algorithm | 2i
logicalBlockSize | 17i
zipLevel | 5i -
Yes. The website changelogs cover main/highlighted features only.
For full changelog reference the README.txt for your version.
-
Sample table for testing:
q)data:([] timestamp:til 10;bq0:til 10;bq1:til 10;bq2:til 10;bp0:til 10;bp1:til 10;bp2:til 10;aq0:til 10;aq1:til 10;aq2:til 10;ap0:til 10;ap1:til 10;ap2:til 10) q)data timestamp bq0 bq1 bq2 bp0 bp1 bp2 aq0 aq1 aq2 ap0 ap1 ap2 --------------------------------------------------------- 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 3 3 3 3 3 3 3 3 3 3 4 4 4 4 4 4 4 4 4 4 4 4 4 5 5 5 5 5 5 5 5 5 5 5 5 5 6 6 6 6 6 6 6 6 6 6 6 6 6 7 7 7 7 7 7 7 7 7 7 7 7 7 8 8 8 8 8 8 8 8 8 8 8 8 8 9 9 9 9 9 9 9 9 9 9 9 9 9
Test provided query:q)select timestamp, depth_vwap1: (bq0; bq1; aq0; aq1) wavg (bp0; bp1; ap0; ap1) from data timestamp depth_vwap1 --------------------- 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9
parse query to functional form:q)parse"select timestamp, depth_vwap1: (bq0; bq1; aq0; aq1) wavg (bp0; bp1; ap0; ap1) from data" ? `data () 0b `timestamp`depth_vwap1!(`timestamp;(wavg; (enlist;`bq0;`bq1;`aq0;`aq1); (enlist;`bp0;`bp1;`ap0;`ap1)))
Test functional query:q)?[data;();0b;`timestamp`depth_vwap1!(`timestamp;(wavg; (enlist;`bq0;`bq1;`aq0;`aq1); (enlist;`bp0;`bp1;`ap0;`ap1)))] timestamp depth_vwap1 --------------------- 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9
Set up variables:q)maxDepth:2 q)quantities: `$raze(("bq"; "aq"),/:\:string til maxDepth) // Note `$ to convert strings to symbols q)prices : `$raze(("bp"; "ap"),/:\:string til maxDepth) // Note `$ to convert strings to symbolsCreate needed list to pass to functional query:q)enlist,quantities enlist `bq0 `bq1 `aq0 `aq1 q)-1 .Q.s1 enlist,quantities; //.Q.s1 very useful to show full structure of object (enlist;`bq0;`bq1;`aq0;`aq1)
Test new dynamic functional select:q)?[data;();0b;`timestamp`depth_vwap1!(`timestamp; (wavg;enlist,quantities;enlist,prices))] timestamp depth_vwap1 --------------------- 0 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9
-
https://code.kx.com/q/ref/dotz/#zpo-open
https://code.kx.com/q/ref/dotz/#zpc-close
You can use callbacks to see when handles open and close
.z.po:{string[.z.p]," opened ",string x;x}
.z.pc:{string[.z.p]," closed ",string x;x}Test this against your c.Close() calls to confirm it is closing the handles.
If still seeing a problem raising an issue can be done at: https://github.com/KxSystems/csharpkdb/issues
-
This limitation was removed in 4.1 release
https://code.kx.com/q/basics/errors/#runtime-errors
conn
Too many connections. Max connections was 1022 prior to 4.1t 2023.09.15, otherwise the limit imposed by the operating system (operating system configurable for system/protocol).
https://code.kx.com/q/releases/ChangesIn4.1/#unlimited-ipcwebsocket-connections
The number of connections is now limited only by the operating system and protocol settings (system configurable).
-
In my answer I use 0N which is the long null. If your table column type for AskOrder is float then the resulting column will be of mixed type. Check by calling meta on the result. save cannot write mixed type columns like this.
A good solution is to update f to ensure it uses a null of the correct datatype:
f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#z count z}The added bit is z count z in place of 0N
Indexing out of bounds on a q vector returns nulls of the dataype of the vector:
q)(0 1)2 / Long null returned
0N
q)(0 1f)2 / Float null returned
0n -
Best would be to update the function to pad out nulls when needed:
q)f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#0N}
q)r:exec f["Ask_";6;AskOrder],f["Bid_";6;BidOrder] by stock from st
q){`stock xcols update stock:key x from (value x)[`AskOrder],'(value x)[`BidOrder]} r
stock Ask_1 Ask_2 Ask_3 Ask_4 Ask_5 Ask_6 Bid_1 Bid_2 Bid_3 Bid_4 Bid_5 Bid_6
-----------------------------------------------------------------------------
a 17 28 59 61 64 34 49 52 90 92
b 41 46 74 92 94 97 26 30 70 82 88 93
c 6 35 35 37 96 2 30 30 38 58
d 4 4 7 10 66 96 12 23 38 39 45 61
e 4 21 49 68 91 92 28 59 66 84 97 97
f 10 18 45 65 83 91 8 39 43 64 77 78
g 5 16 16 30 59 62 84 84 90 97
h 48 51 56 59 71 12 45 47 82 95
i 10 15 37 48 59 91 24 37 52 68 69 84
j 0 57 57 66 74 36 63 63 73 83
k 14 19 43 46 58 60 30 37 50 53 90 99
l 25 29 33 49 59 98 6 26 59 60 68 93
m 9 65 65 74 88 12 64 64 66 77
n 14 14 40 40 98 44 44 73 73 93
o 32 46 50 63 70 80 20 32 54 89 93 94
p 4 48 48 51 92 23 40 41 74 88 -
select from t where 16<=(count;AskOrder) fby stock
You’d want an fby for that filtering