rocuinneagain
Forum Replies Created
-
These are examples of using the C interface to extend the Kdb+ database.
Documentation is here:
https://code.kx.com/q/interfaces/using-c-functions/
It shows an example of writing a new ‘add’ function in C:
#include"k.h" #ifdef __cplusplus extern "C"{ #endif K add(K x,K y) { if(x->t!=-KJ||y->t!=-KJ) return krr("type"); return kj(x->j+y->j); } #ifdef __cplusplus } #endifThen this can be used from inside a Kdb+ process:
q)add:(`add 2:(`add;2)) q)add[3;4] 7
This is a possible method to add math functions for 128-bit floating point numbers inside Kdb+.
-
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable ‘x’ inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on ‘x`b’ as a vector rather than inside ‘each’.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t 16 4746672 q)\ts {((first;last) x`b)@'x`a}t 7 4746640It’s goal is to avoid calculating ‘last each’ and ‘first each’ for every row.
Instead it uses each both (‘) to apply first or last after it is known which function is needed.
-
A single character .i.e “1” is a type -10h
A list of characters .i.e “10” is a type 10h
You can create a single character list using ‘enlist’ .i.eq)type each ("1";"0";"11-15") /Some are lists some are not -10 -10 10h q)type each (enlist "1";enlist "0";"11-15") /All are lists 10 10 10h q)("1";"0") "10" q)type each ("1";"0") -10 -10h q)(enlist "1";enlist "0") //Using enlist to make single character lists ,"1" ,"0" q)type each (enlist "1";enlist "0") 10 10hUsing ‘enlist’ will help prevent unwanted concatenation for you.
https://code.kx.com/q/ref/enlist/ -
You can use each-right to cast each character one at a time:
https://code.kx.com/q/ref/maps/#each-left-and-each-right
q) `$”10″
`10
q) `$/:”10″
`1`0
-
One option would be to unpivot the table using a helper function
/tab : the table to operate on /baseCols : the columns not to unpivot /pivotCols : the columns which you wish to unpivot /kCol : the key name for unpivoted data /vCol : the value name for unpivoted data unpivot:{[tab;baseCols;pivotCols;kCol;vCol] base:?[tab;();0b;{x!x}(),baseCols]; newCols:{[k;v;t;p] flip (k;v)!(count[t]#p;t p)}[kCol;vCol;tab] each pivotCols; baseCols xasc raze {[b;n] b,'n}[base] each newCols } unpivot[;`time;`price`pricev2;`priceType;`price] table time priceType price ---------------------------- 00:00:00.002 price 6.33 00:00:00.002 pricev2 6.32 00:00:01.001 price 4.05 00:00:01.001 pricev2 4.05 00:00:26.808 price 4.05 00:00:26.808 pricev2 5.07 00:00:27.002 price 5.12 00:00:27.002 pricev2 5.12 00:00:28.002 price 2.61 00:00:28.002 pricev2 2.61 00:00:29.002 price 4.61 00:00:29.002 pricev2 4.47 00:00:30.001 price 4.9 00:00:30.001 pricev2 4.47 00:00:31.000 price 4.64 00:00:31.000 pricev2 4.77 00:00:32.000 price 2.37 00:00:32.000 pricev2 4.87 00:00:33.000 price 3.75 00:00:33.000 pricev2 2.7Then you can set the options to graph the lines
-
All of these items are equivalent:
q)"10" "10" q)("1";"0") "10" q)("10") "10"They all resolve to 2 item lists containing characters
https://code.kx.com/q/basics/datatypes/
Here are some more example queries which may help:
q)select from t where Vals in `$/:("1";"10") Vals ---- 1 10 q)select from t where Vals in `$/:("1";"0") Vals ---- 1 0 q)select from t where Vals in `$"10" Vals ---- 10 q)select from t where Vals in `$"1" Vals ---- 1You can use ‘=’ rather than ‘in’ if you are searching for only one value:
q)select from t where Vals=`$"1" Vals ---- 1
You can input your symbols directly rather than casting:
q)`0`1`10 `0`1`10
-
In the visual inspector you can enter qsql queries.
Here are some example which may help. Firstly I recreated your table:
table:flip `time`price`pricev2!( (00:00:00.002 00:00:01.001 00:00:26.808 00:00:27.002 00:00:28.002 00:00:29.002 00:00:30.001 00:00:31 00:00:32 00:00:33); (6.33 4.05 4.05 5.12 2.61 4.61 4.9 4.64 2.37 3.75); (6.32 4.05 5.07 5.12 2.61 4.47 4.47 4.77 4.87 2.7));
Uses flip to create a table from a dictionary.
You can compare the columns using subtract:
update priceDiff:price-pricev2 from table time price pricev2 priceDiff ------------------------------------ 00:00:00.002 6.33 6.32 0.01 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14 00:00:30.001 4.9 4.47 0.43 00:00:31.000 4.64 4.77 -0.13 00:00:32.000 2.37 4.87 -2.5 00:00:33.000 3.75 2.7 1.05
You can use within to query within a time window:
select from table where time within 00:00:01 00:00:30 time price pricev2 -------------------------- 00:00:01.001 4.05 4.05 00:00:26.808 4.05 5.07 00:00:27.002 5.12 5.12 00:00:28.002 2.61 2.61 00:00:29.002 4.61 4.47
Then you can combine the 2 statements in to one:
update priceDiff:price-pricev2 from select from table where time within 00:00:01 00:00:30 time price pricev2 priceDiff ------------------------------------ 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14
This page has lots more examples: qsql
Hopefully this helps you.
-
You can see an example of mapping a decimal to either a list of bytes or scaling to a double in this interface.
-
You cannot control the use of /tmp but you could possibly use a redirect to send the bulk of output to a different location.
In a basic form:
q)system"ls > /my/chosen/path/out.txt 2>&1;" q)result:read0`:/my/chosen/path/out.txt q)hdel`:/my/chosen/path/out.txt `:/my/chosen/path/out.txt q)result "file1" "file2"
You could aim to make a more reusable function.
For familiarity you could use the TMPDIR environment variable:
q)setenv[`TMPDIR] "/my/chosen/path"
Then create a function to run system commands
systemTMPDIR:{[c] f:first system"mktemp"; //Make a temp file respecting TMPDIR c:c," > ",f," 2>&1;echo $?"; //Add redirect to tmp file and capture of exit code e:"J"$first system c; //Execute the command f:hsym `$f; r:read0 f; //Read the result of the command hdel f; //Delete the tmp file $[not 0=e; //Check if the exit code was an error (not 0) [-1 last r;'`os]; //If an error print the last line and signal with 'os r] //If success return the data }On success:
q)systemTMPDIR"ls" "file1" "file2"
On failure:
q)systemTMPDIR"blah" sh: 1: blah: not found 'os [0] systemTMPDIR"blah" ^
*Note: This is just a small example and likely will not behave the exact same as the native ‘system’ in all cases.
-
Todays release kdb+4.0 2021.10.01 contains a fix for the issue in your first example:
q)0 wavg 5 0n
-
No, it is not possible to update a global variable from any thread other the main thread.
The ‘noupdate’ error is detailed on: https://code.kx.com/q/basics/errors/
Full list of blocked operations within peach: https://code.kx.com/q/ref/each/#blocked-within-peach
-
1. I might suggest to think about it the other way. Dictionaries are more like special paired lists.
q)dict:`a`b!1 2 q)lists:{(key x;value x)} dict q)dict a| 1 b| 2 q)lists a b 1 2 q)dict `a 1 q)lists[0]?`a 0 q)lists[1] lists[0]?`a / The same as: dict `a 1https://code.kx.com/q/ref/find/
2. Yes a list of conforming dictionaries is promoted a table
q)(`a`b!1 2;`a`b!1 2) a b --- 1 2 1 2
Importantly in memory the way it is actually stored is ‘flipped’ so it is a dictionary of lists. (no longer a list of dictionaries)
q).Q.s1 (`a`b!1 2;`a`b!1 2) "+`a`b!(1 1;2 2)"
This way the keys/column-names only need to be stored once for the whole table and not for each row.
The columns then are vectors which is more efficient and performant.
There are more details on indexing at depth here :
https://code.kx.com/q4m3/3_Lists/#38-iterated-indexing-and-indexing-at-depth
The “querying unstructured data” section of this blog may be of interest:
https://kx.com/blog/kdb-q-insights-parsing-json-files/The code in it focuses on tables but can be adapted to lists/dictionaries as well:
q)asLists:sample cols sample q)asLists[0;;`expiry] 17682D19:58:45.000000000 ` `long$() ,"" `long$() 0N ,""
q)@[`asLists;0;{(enlist[`]!enlist (::))(,)/:x}]
`asLists
q)asLists[0;;`expiry]
17682D19:58:45.000000000
::
::
::
::
::
::
q)fill:{n:count i:where (::)~/:y;@[y;i;:;n#x]}
q)fill[0Wn]asLists[0;;`expiry]
17682D19:58:45.000000000 0W 0W 0W 0W 0W 0W -
ssr is a much slower operation than the above examples.
Using wildcards such as “*#” is also difficult as it matches more than one suffix (.e.g “#” and “^#”).
To work around this you can choose the longest matching suffix.Using ‘like’ and ‘@’ where possible rather than ‘ssr’ as below is a big speed improvement.
Searching for “*” is also difficult as it is a wildcard. Instead, I use tab “t” in it’s place.
If “t” is possibly in your data you would need to change this for another character.
symbology:.Q.id ("****";enlist ",")0:`:symbology.csv update searchNASDAQ:{"*",@[x;where x="*";:;"t"]} each NASDAQ from `symbology func:{s:string x; m:select from symbology where @[s;where s="*";:;"t"] like/:searchNASDAQ; l:max count each m`NASDAQ; c:first exec CMS from m where l=count each NASDAQ; `$$[c~();s;(neg[l]_s),c] };In a test it does seem to operate correctly.
Overall it still has risks as if bad data is sent in the function cannot truly validate what it is doing. Ideally you would have root and suffix separated by a known delimiter such as a space ” ” in the source data.
-6 sublist {([] symNASDAQ:n;symCMS:func each n:`$"AAPL",/:x)}symbology`NASDAQ symNASDAQ symCMS ------------------ AAPL# AAPLWI AAPL^# AAPLRTWI AAPL-# AAPLPRWI AAPL.A# AAPLAWI AAPL+# AAPLWSWI AAPL~ AAPLTESTThe extra complexity does come at a cost of speed.
\ts func each 10000#`$"AAPL+#" 41 553776
symbologyOld:`NASDAQ xkey .Q.id (“****”;enlist “,”)0:`:symbology.csv
\ts {s:string x;`$(4#s),symbologyOld[4 _ s]`CQSSuffix} each 10000#`$”AAPL+#”
12 554448
\ts {s:string x;r:first where not s in .Q.A;`$(r#s),symbologyOld[r _ s]`CQSSuffix} each 10000#`$”AAPL+#”
16 554464As you used in your original example .Q.fu is a great tool when performing an intensive task repeatedly.
Extreme example with only one unique input:
\ts .Q.fu[func each] 10000#`$"AAPL+#" 0 394032
One limitation of .Q.fu is that it has no memory between executions.
One example of a way to bypass this would be a memory cache.
The library https://github.com/gitrj95/q-memo could be used.
In this use-case your function is fast enough that the cache is overkill and should not actually be used but would be useful for very slow operations you may have to run many times.
l memo.k .memo.init[`.;10000h] //Create a cache with 10k limit .memo.mk[`func;`memoFunc;`cache.0] //Create a wrapped function to use the cache
ts memoFunc each 10000#`$(“AAPL~”;”AAPL+#”)
126 553936//The cache pre stores the results cache.0 f a | r -------------------| --------- :: | :: ..memoFunc ,`AAPL~ | `AAPLTEST ..memoFunc ,`AAPL+#| `AAPLWSWI
-
Without knowing the full rules and logic here are some thoughts.
//Table taken from https://www.nasdaqtrader.com/trader.aspx?id=CQSsymbolconvention q)symbology:`NASDAQIntegratedPlatformSuffix xkey .Q.id ("****";enlist ",")0:`:symbology.csv q)5 sublist symbology NASDAQIntegratedPlatformSuffix| Security CQSSuffix CMSS.. ------------------------------| ---------------------------------------------.. ,"-" | "Preferred" ,"p" "PR".. "-A" | "Preferred Class "A"*" "pA" "PRA.. "-B" | "Preferred Class "B"*" "pB" "PRB.. ".A" | "Class "A"*" ".A" ,"A".. ".B" | "Class "B"*" ".B" ,"B"..
//Assuming 4 character root q){s:string x;`$(4#s),symbology[4 _ s]`CQSSuffix} each (`$"AAPL.B*";`$"AAPL^#") `AAPL`AAPLrw//Assuming there is always a suffix and first non capital letter is beginning of suffix {s:string x;r:first where not s in .Q.A;`$(r#s),symbology[r _ s]`CQSSuffix} each (`$"AAPL.B*";`$"AAPL^#") `AAPL`AAPLrwNote: ‘.B*’ does not appear in that mapping table so ‘AAPL.B*’ does not receive a new CQS Suffix.
-
You can use .p.wrap https://code.kx.com/q/ml/embedpy/userguide/#embedpy-objects_1
Small example giving each subplot a title:
plt:.p.import[`matplotlib;`:pyplot] plts:plt[`:subplots;<;pykwargs `ncols`figsize`dpi!(2;24 8;100)] .p.wrap[plts[1;0]][`:set_title]"Plot1"; .p.wrap[plts[1;1]][`:set_title]"Plot2"; plt[`:show][];
Results in: