rocuinneagain
Forum Replies Created
-
Command prompt is the default, but the new Windows Terminal app is much nicer. I also use it to open sessions to WSL (Windows Subsystem for Linux).
Visual Studio Code is a great option if building up bigger projects. An official kdb plugin is available for it.
-
Testing speed and equivalence:
fun_ema:{[lambda; liste];({[lambda; x; y]; (lambda*y)+ (x*1-lambda)}[lambda]\) liste}
expma1:{[lambda;vector] {[x;y;z] (x*y)+z}\[ first vector; 1-lambda; vector * lambda]};
q)a:10
q)b:til 1000000
q)\ts r1:fun_ema[a;b]
174 32777680
q)\ts r2:expma1[a;b]
91 41166288
q)r1~r2
1bYour fun_ema performs *, +, *, – a total of count[b] times. The operations are all on atom type data.
expma1 performs *, + a total of count[b] times and –, * once. The single use of * is on vector type data.
q uses vector instructions for speed. These take advantage of CPU instructions to be more efficient
q)a:til 1000000
q)b: til 1000000
q)\ts a+b // + operates on vectors and is fast
1 8388800
q)\ts a+'b // Using ' to force + to loop through each pair of atoms is much slower
26 32777488
q)\ts {x+y}'[a;b] //Wrapping in an unnecessary lambda is slower again
62 32777792
Since 4.0, kdb further sped up large vector operations by adding Multithreaded primitives which will spread the vector operations across multiple threads when available.
-
https://code.kx.com/q/ref/each/
Use each:
q)select {[x]ssr[x;".C";""]} each orderID, cancelTime from thttps://code.kx.com/q/basics/application/#projection
Neater with projection:
select ssr[;".C";""] each orderID, cancelTime from t
https://code.kx.com/q/ref/drop/
If you know .C is always at the end it’s much better to use drop _ rather than ssr as ssr is compute intensive
select {-2_ x} each orderID, cancelTime from thttps://code.kx.com/q/ref/maps/#each-left-and-each-right
each-right /: is neater
q)select -2_/:orderID, cancelTime from t
-
This is a qSQL statement:
https://code.kx.com/q/ref/delete/
q)delete ps from x
qSQL has some differences to q.
“delete is a qSQL query template and varies from regular q syntax.”
In qSQL statements backticks are not needed for looking up columns in a table or keys in a dictionary.
These are normal q statements:
q)load `:flatT //Uses ` to declare a symbol, this case is known as a hsym, denoting a filepath as it starts with :
q)`flatT in key `. //` used to declare a symbol `flatT and also to refer to . the default namespacecode.kx.com
delete query | Reference | kdb+ and q documentation - kdb+ and q documentation
delete is a qSQL template that removes rows or columns from a table, entries from a dictionary, or objects from a namespace.
-
‘ form of each https://code.kx.com/q/ref/maps/#each
q){x[y]:99;x}'[(1 2 3;4 5 6;7 8 9);0 1 2]
99 2 3
4 99 6
7 8 99Amend at @ version https://code.kx.com/q/ref/apply/#amend-amend-at
q){@[x;y;:;99]}'[(1 2 3;4 5 6;7 8 9);0 1 2]
99 2 3
4 99 6
7 8 99
q)@[;;:;99]'[(1 2 3;4 5 6;7 8 9);0 1 2]
99 2 3
4 99 6
7 8 99 -
- except https://code.kx.com/q/ref/except/
- ‘ form of each https://code.kx.com/q/ref/maps/#each
q)tab:([] timestamp:1 2 3;timeWin:(1 2 3;1 2 4 5;3 1 1 3 5))
q)tab
timestamp timeWin
-------------------
1 1 2 3
2 1 2 4 5
3 3 1 1 3 5
q)update timeWin:except'[timeWin;timestamp] from tab
timestamp timeWin
-----------------
1 2 3
2 1 4 5
3 1 1 5 -
Yes with right to left evaluation it is being saved then sorted.
The reason this may have been done is that sorting in memory will temporarily use extra memory to perform the operation.
https://code.kx.com/q/ref/asc/#sorting-data-on-disk
Sorting on disk is very memory efficient as it does not bring the full table in to memory.
The trade off is it is slower as each column gets written twice.
q)tab:flip (`$/:.Q.an)!{1000000?10000.0}each .Q.an q)\ts `a xasc `:tab/ set tab q) 1824 25168800 q)\ts `:tab/ set `a xasc tab 1165 536874192 q)536874192%25168800 21.33094 //Sorting in memory uses 21x memory than sorting on disk -
1. qSQL has some differences to q.
https://code.kx.com/q/ref/select/
“select is a qSQL query template and varies from regular q syntax.”
https://code.kx.com/q/basics/qsql/#where-phrase
“Efficient Where phrases start with their most stringent tests.”
Each where clause is it’s own expression. You can see inside [] the expression are executed left to right but within each expression the code is executed right to left.
q)[(show 2;show 1);(show 4;show 3);(show 6;show 5)];
1
2
3
4
5
62.
https://code.kx.com/q/basics/funsql/#group-by-column
“b is a column name. The result is a dictionary.”
https://code.kx.com/q/basics/funsql/#group-by-columns
“b is a list of column names. … returns a dictionary”
https://code.kx.com/q/basics/funsql/#group-by-a-dictionary
“b is a dictionary. Result is a dictionary in which the key is a table with columns as specified by b …”
If you parse the queries you can see that sym:sym is group by dictionary
q)parse"exec first open, last close by sym from daily where sym like \"*L\""
?
`daily
,,(like;`sym;"*L")
,`sym
`open`close!((*:;`open);(last;`close))vs:
q)parse"exec first open, last close by sym:sym from daily where sym like \"*L\""
?
`daily
,,(like;`sym;"*L")
(,`sym)!,`sym
`open`close!((*:;`open);(last;`close)) -
PyKX converts a q CharVector to Python NumPy array of bytes by default.
https://code.kx.com/pykx/2.5/pykx-under-q/upgrade.html#functional-differences
q).pykx.eval["lambda x: print(type(x))"][test]; <class 'numpy.ndarray'> q).pykx.eval["lambda x: print(x)"][test]; [b'<' b't' b'a' b'b' b'l' b'e' b'>' b'<' b't' b'r' b'>' b'<' b't' b'h' ...You can create a helper to convert to strings as you require:
q)b2s:.pykx.eval["lambda x: x.tobytes().decode('UTF-8')"] q).pykx.print pd[`:read_html]b2s[test] [ First Name Last Name Age 0 John Doe 30 1 Jane Smith 25 2 Emily Jones 22]More info on default conversions:
https://code.kx.com/pykx/2.5/pykx-under-q/intro.html#function-argument-types
And:
https://code.kx.com/pykx/2.5/user-guide/fundamentals/text.html
-
@laura – This was a format testing reply – can be deleted. Thanks
-
<div>See these sections of docs on:</div>
Order of evaluation also matters: Left-of-Right Evaluation
abs(price)=({abs max x};price)Will run in order:
- ({abs max x};price)
- (price)=({abs max x};price)
- abs(price)=({abs max x};price)
whereas:
abs[price]=({abs max x};price)Will run in order:
- ({abs max x};price)
- abs[price]
- abs[price]=({abs max x};price)
This tries to pass 2 arguments to all which only takes one argument so you get a rank error
https://code.kx.com/q/ref/all-any/#all
all[col=a;col2=b]
-
Full qSQL docs on: https://code.kx.com/q/basics/qsql/
The 4 keywords for qSQL are:
-
https://code.kx.com/q/ref/asc/#sorting-data-on-disk
“xasc can sort data on disk directly, without loading the entire table into memory.”
If you interrupt an on disk xasc you could be in a state where some columns were sorted but not others.
-
There is a docs page on PyKX modes of operation.
The 3 modes are:
- Unlicensed mode – used for IPC query and publish. No license needed.
- Licensed mode – import pykx in Python session.
- PyKX under q – \l pykx.q in a q session.
Both 2 & 3 require a KX license which includes insights.lib.pykx and insights.lib.embedq licensing flags.
- The kdb Insights Personal Edition free trial license includes these flags.
- For commercial licences with PyKX enabled contact your KX sales representative or sales@kx.com. Alternately apply through https://kx.com/book-demo.
-
Node.js is required for the upload server.
If node is not installed, dashboards will work as normal without the upload component.
Documentation will be updated to detail this.