rocuinneagain
Forum Replies Created
-
q may have been quarantined by the OS. Can you try to run :
xattr -d com.apple.quarantine path/to/m64/q
-
There is a note on this page which will be of help
https://code.kx.com/q/learn/install/#step-4-confirm-success
“If the
spctlandxattrcommands above have not authorized the OS to run q, open System Preferences > Security & Privacy.You should see a notification that q has been blocked and a button to override the block.
-
It depends on the data/access-patterns/hardware. May make sense in your use case but not in others.
If you had a DB with a very large number of unique values in the
`p#field then it can make sense to group them together to make the attribute more efficient and have the disk doing more sequential reads which are faster.There are some notes on estimating attribute overheads on:
https://aquaq.co.uk/adventure-in-retrieving-memory-size-of-kdb-object/
In some IoT schemas where there can be millions of unique identifiers even these approaches would not be enough and introducing a hash column/lookup can be needed to not have the DB be slowed by larger attributes and small random read reads.
-
A more direct path to that dictionary in this case would be:
.[metaTbl;(`ref;`m)]
-
The syntax you need is:
refServiceHandle(`getOptionRef;uniqueOpts)
This will pass the local variable down to the remote function as a parameter.
I’ve added a pull request to have this documented better:
https://github.com/KxSystems/docs/pull/85
Q for Mortals has some notes already:
https://code.kx.com/q4m3/1_Q_Shock_and_Awe/#119-interprocess-communication-101
-
The sym file contains enumerated symbols for an symbol column in any table.
The reason you can select it is that if q fails to find a column of that name it will try to use a global variable , which sym will be.
q)`:tab/ set .Q.en[`:.] ([] a:1 2 3;b:`a`b`c) `:tab/ q)tab:get `:tab/ q)tab a b --- 1 a 2 b 3 c q)select sym from tab sym --- a b c q)sym //sym is in memory `a`b`c 2 q)nonsense:`one`two`three q)select nonsense from tab //nonsense is not a column but a global variable nonsense -------- one two three q)delete sym from `. /delete sym from memory `. q)tab / now any sym columns show indexes as they cannot resolve their text a b --- 1 0 2 1 3 2
Review some documentation on enumeration and on disk data:
- https://code.kx.com/q/basics/enumerations/
- https://code.kx.com/q4m3/14_Introduction_to_Kdb+/#1428-working-with-sym-files
-
metaon an empty table will never showC. This is because kdb+ does not have a dedicated datatype for lists of lists. The empty list will be()which is of type0h. On a populated table meta inspects the first item in a list of lists and uses it’s type to populate it’s result. It cannot do this on an empty table.q)tb:([] a:1 2;b:("wo";"rd")) q)tb a b ------ 1 "wo" 2 "rd" q)meta tb c| t f a -| ----- a| j b| C q)delete from `tb `tb q)tb a b --- q)meta tb c| t f a -| ----- a| j b| q).Q.s1 tb "+`a`b!(`long$();())" -
kdb+ does not implement full regex.
https://code.kx.com/q/basics/regex/
It supports
?*[]^If you want full support you can use the C interface to bring in external libraries
- https://code.kx.com/q/basics/regex/#regex-libraries
- https://code.kx.com/developer/libraries/regular-expression-pcre2/
- https://github.com/KxSystems/ffi/blob/master/examples/pcre.q
-
Try this to confirm the file is valid:
//Bytes per record q)sum 3 3 2 2 4 14 //Bytes in file q)hcount `:file1.txt ?? //Confirm bytes in file has no remainder if divided by bytes in record q)0~hcount[`:file1.txt] mod sum 3 3 2 2 4 ??
If last line returns
0bthen your file is not a valid size for those fixed length messages.You can see quickly at the end of the file if the last record looks valid.
//Inspect the last records in the file to see visually if records look correct
q)-2#(sum 3 3 2 2 4) cut `char$read1
The issue could start anywhere in the file so you may need to do further investigation if it is still not obvious after checking the above.
-
One item to note is that .j.j will put processing load on to the kdb+ process.
JSON also only has a limited set of datatypes which means you will need to write more q code to perform parsing/casting
- https://www.w3schools.com/js/js_json_datatypes.asp
- https://kx.com/blog/kdb-q-insights-parsing-json-files/
Using the c.flip distributes the serialisation work across your java processes/threads and minimizes the work kdb+ will have to do to deserialise/parse/cast.
kdb+ does have a multithreaded input queue mode (but it does have restrictions)
-
I added extra columns and it did not slow down the operation for me.
Is your data in memory or on disk?data:([] dt:`s#.z.p+0D00:00:01*til 800000;mmm3:800000?1000.0) data:data,'flip (`$"col",/:string til 30)!30#enlist til 800000 f:`dt; w:(-1000000; 0)+:data[`dt]; result: wj[w;f;data;(data;(max;`mmm3);(min;`mmm3))]
-
On my laptop this took 765 milliseconds – what runtime are you seeing?
data:([] dt:`s#.z.p+0D00:00:01*til 800000;mmm3:800000?1000.0) f:`dt; w:(-1000000; 0)+:data[`dt]; \ts result: wj[w;f;data;(data;(max;`mmm3);(min;`mmm3))]
-
The documentation does have a note on attributes to apply:
t and q are simple tables to be joined (q should be sorted `sym`time with `p# on sym)
Have you applied this?
-
So you are only joining on time windows? no other column?
You could apply the sorted attribute.
https://code.kx.com/q/ref/set-attribute/#sorted
The ‘datetime’ datatype is deprecated. ‘timestamp’ should be used instead.
https://code.kx.com/q/basics/datatypes/#temporal
If you can share a minimal reproducible example in q that will help us answer your questions.