rocuinneagain
Forum Replies Created
-
That would require for deep level changes to the codebase.
For example when a HDB is written to a new:
- A new date folder is created.
- Tables are written.
- The HDB process is reloaded
If you reloaded the HDB before all tables were written you would have errors.
q)mkdir HDB q)cd HDB q)`:2021.01.01/tab1/ set ([] a:1 2 3) `:2021.01.01/tab1/ q)`:2021.01.01/tab2/ set ([] b:1 2 3) `:2021.01.01/tab2/ q)\l . q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 q)select from tab2 date b ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 q)`:2021.01.02/tab1/ set ([] a:4 5 6) `:2021.01.02/tab1/ q)\l . q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 2021.01.02 4 2021.01.02 5 2021.01.02 6 q)select from tab2 './2021.01.02/tab2/b. OS reports: No such file or directory [0] select from tab2 ^
.Q.bv – would be one possible helpful extension which can in memory fill tables missing from partitions (using ` as argument to fill using first partition as a template)
q)\l HDB q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 2021.01.02 4 2021.01.02 5 2021.01.02 6 q)select from tab2 //Table missing from latest partition is not found 'tab2 [0] select from tab2 ^ q).Q.bv` //Using ` first partition used as prototype and table is found q)select from tab2 date b ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3
(.Q.chk is unsuitable as it uses most recent partition as template to fills tables missing from partitions)
This is only one item that would be needed to implement what you asked.
Currently EOD is a single action for all tables, all code and processes involved would need updates to operate table by table. Meaning changes in RDB/IDB/HDB and others.
-
Usually when you see this error it is due to one of:
- A fault in the network infrastructure between hosts
- One of the involved processes has died unexpectedly
- Some code in either your process or the remote has purposefully closed the connection (hclose)
There are handlers available in the .z namespace which are helpful to detect events on IPC handles:
- .z.pc is called after a connection has been closed
- .z.po is called after a connection has been opened
These could be implemented to track and attempt to reconnect dropped connections.
(.z.W is a useful dictionary or current open handles)
There are some examples such as dotz which is a library building on these features to trace, monitor and control execution.
-
The Platform codebase is designed to write date partitioned HDB’s only.
The Intraday database – KX Platform exists to so that 24hrs of data does not need to be kept in memory, but it will again store to the HDB on a date partition only.
-
Hi Leah,
This blog post will likely be of interest: https://kx.com/blog/partitioning-data-in-kdb/
It covers the basics of looking at hourly partitions and also fixed size partitions.
Regards,
Rian
-
These are examples of using the C interface to extend the Kdb+ database.
Documentation is here:
https://code.kx.com/q/interfaces/using-c-functions/
It shows an example of writing a new ‘add’ function in C:
#include"k.h" #ifdef __cplusplus extern "C"{ #endif K add(K x,K y) { if(x->t!=-KJ||y->t!=-KJ) return krr("type"); return kj(x->j+y->j); } #ifdef __cplusplus } #endifThen this can be used from inside a Kdb+ process:
q)add:(`add 2:(`add;2)) q)add[3;4] 7
This is a possible method to add math functions for 128-bit floating point numbers inside Kdb+.
-
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable ‘x’ inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on ‘x`b’ as a vector rather than inside ‘each’.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t 16 4746672 q)\ts {((first;last) x`b)@'x`a}t 7 4746640It’s goal is to avoid calculating ‘last each’ and ‘first each’ for every row.
Instead it uses each both (‘) to apply first or last after it is known which function is needed.
-
A single character .i.e “1” is a type -10h
A list of characters .i.e “10” is a type 10h
You can create a single character list using ‘enlist’ .i.eq)type each ("1";"0";"11-15") /Some are lists some are not -10 -10 10h q)type each (enlist "1";enlist "0";"11-15") /All are lists 10 10 10h q)("1";"0") "10" q)type each ("1";"0") -10 -10h q)(enlist "1";enlist "0") //Using enlist to make single character lists ,"1" ,"0" q)type each (enlist "1";enlist "0") 10 10hUsing ‘enlist’ will help prevent unwanted concatenation for you.
https://code.kx.com/q/ref/enlist/ -
One option would be to unpivot the table using a helper function
/tab : the table to operate on /baseCols : the columns not to unpivot /pivotCols : the columns which you wish to unpivot /kCol : the key name for unpivoted data /vCol : the value name for unpivoted data unpivot:{[tab;baseCols;pivotCols;kCol;vCol] base:?[tab;();0b;{x!x}(),baseCols]; newCols:{[k;v;t;p] flip (k;v)!(count[t]#p;t p)}[kCol;vCol;tab] each pivotCols; baseCols xasc raze {[b;n] b,'n}[base] each newCols } unpivot[;`time;`price`pricev2;`priceType;`price] table time priceType price ---------------------------- 00:00:00.002 price 6.33 00:00:00.002 pricev2 6.32 00:00:01.001 price 4.05 00:00:01.001 pricev2 4.05 00:00:26.808 price 4.05 00:00:26.808 pricev2 5.07 00:00:27.002 price 5.12 00:00:27.002 pricev2 5.12 00:00:28.002 price 2.61 00:00:28.002 pricev2 2.61 00:00:29.002 price 4.61 00:00:29.002 pricev2 4.47 00:00:30.001 price 4.9 00:00:30.001 pricev2 4.47 00:00:31.000 price 4.64 00:00:31.000 pricev2 4.77 00:00:32.000 price 2.37 00:00:32.000 pricev2 4.87 00:00:33.000 price 3.75 00:00:33.000 pricev2 2.7Then you can set the options to graph the lines
-
All of these items are equivalent:
q)"10" "10" q)("1";"0") "10" q)("10") "10"They all resolve to 2 item lists containing characters
https://code.kx.com/q/basics/datatypes/
Here are some more example queries which may help:
q)select from t where Vals in `$/:("1";"10") Vals ---- 1 10 q)select from t where Vals in `$/:("1";"0") Vals ---- 1 0 q)select from t where Vals in `$"10" Vals ---- 10 q)select from t where Vals in `$"1" Vals ---- 1You can use ‘=’ rather than ‘in’ if you are searching for only one value:
q)select from t where Vals=`$"1" Vals ---- 1
You can input your symbols directly rather than casting:
q)`0`1`10 `0`1`10
-
You can use each-right to cast each character one at a time:
https://code.kx.com/q/ref/maps/#each-left-and-each-right
q) `$”10″
`10
q) `$/:”10″
`1`0
-
You can see an example of mapping a decimal to either a list of bytes or scaling to a double in this interface.
-
In the visual inspector you can enter qsql queries.
Here are some example which may help. Firstly I recreated your table:
table:flip `time`price`pricev2!( (00:00:00.002 00:00:01.001 00:00:26.808 00:00:27.002 00:00:28.002 00:00:29.002 00:00:30.001 00:00:31 00:00:32 00:00:33); (6.33 4.05 4.05 5.12 2.61 4.61 4.9 4.64 2.37 3.75); (6.32 4.05 5.07 5.12 2.61 4.47 4.47 4.77 4.87 2.7));
Uses flip to create a table from a dictionary.
You can compare the columns using subtract:
update priceDiff:price-pricev2 from table time price pricev2 priceDiff ------------------------------------ 00:00:00.002 6.33 6.32 0.01 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14 00:00:30.001 4.9 4.47 0.43 00:00:31.000 4.64 4.77 -0.13 00:00:32.000 2.37 4.87 -2.5 00:00:33.000 3.75 2.7 1.05
You can use within to query within a time window:
select from table where time within 00:00:01 00:00:30 time price pricev2 -------------------------- 00:00:01.001 4.05 4.05 00:00:26.808 4.05 5.07 00:00:27.002 5.12 5.12 00:00:28.002 2.61 2.61 00:00:29.002 4.61 4.47
Then you can combine the 2 statements in to one:
update priceDiff:price-pricev2 from select from table where time within 00:00:01 00:00:30 time price pricev2 priceDiff ------------------------------------ 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14
This page has lots more examples: qsql
Hopefully this helps you.
-
Todays release kdb+4.0 2021.10.01 contains a fix for the issue in your first example:
q)0 wavg 5 0n
-
You cannot control the use of /tmp but you could possibly use a redirect to send the bulk of output to a different location.
In a basic form:
q)system"ls > /my/chosen/path/out.txt 2>&1;" q)result:read0`:/my/chosen/path/out.txt q)hdel`:/my/chosen/path/out.txt `:/my/chosen/path/out.txt q)result "file1" "file2"
You could aim to make a more reusable function.
For familiarity you could use the TMPDIR environment variable:
q)setenv[`TMPDIR] "/my/chosen/path"
Then create a function to run system commands
systemTMPDIR:{[c] f:first system"mktemp"; //Make a temp file respecting TMPDIR c:c," > ",f," 2>&1;echo $?"; //Add redirect to tmp file and capture of exit code e:"J"$first system c; //Execute the command f:hsym `$f; r:read0 f; //Read the result of the command hdel f; //Delete the tmp file $[not 0=e; //Check if the exit code was an error (not 0) [-1 last r;'`os]; //If an error print the last line and signal with 'os r] //If success return the data }On success:
q)systemTMPDIR"ls" "file1" "file2"
On failure:
q)systemTMPDIR"blah" sh: 1: blah: not found 'os [0] systemTMPDIR"blah" ^
*Note: This is just a small example and likely will not behave the exact same as the native ‘system’ in all cases.
-
No, it is not possible to update a global variable from any thread other the main thread.
The ‘noupdate’ error is detailed on: https://code.kx.com/q/basics/errors/
Full list of blocked operations within peach: https://code.kx.com/q/ref/each/#blocked-within-peach