rocuinneagain
Forum Replies Created
-
TP = Tickerplant (Relays data to subscribers + recovers from logfile after crashes)
RDB = Realtime Database (Stores data for query)
RTE = Realtime engine (Performs streaming calculations and stores caches or publishes results)
(Any process can be customised)
1. Yes
2. It is acting more like a mixture between a TP,RDB,RTE
a) It does not store a logfile to recover in case of a crash (tp-logfile – a TP normally does this)
b) It stores data indefinitely instead of acting only as a relay. (Unlike a TP, more like and RDB, although an RDB will clear once every 24hrs)
c) It does not relay data untouched instead only specific data is forwarded (similar to an RTE)
getSyms – sends like of unique symbols across tables
getQuotes – sends last row by sym from quote table
getTrades – sends last row by sym from trade table
3. Yes
The execution flow is:
1. FH sends messages to PubSub (lines 18/19) every 100ms
2. The messages arrive to PubSub and .z.pg evaluates them. This mean upd/insert (pubsub.q line 8 will save the incoming data to quote/trade. PubSub now has some data cached.
3. The next time the PubSub timer (.z.ts) is triggered (every second) the ‘pub’ function will trigger and send data to subscriptions.
This code is a basic demo so it may have some holes in it’s logic (like never clearing data in PubSub so eventually memory will run out)
-
.z.ts is the timer function which is evaluated on intervals of the timer variable set by system command t
In your example ‘pub’ is called for each subscriber in ‘subs’ once per second.
To start subs has count 0 but when a new subscriber connects and subscribes they will be added.
How the subscribers are added to that table in that demo is:
- websockets.html calls connect() on load of the page (code)
- websockets.js holds the connect() definition which shows it sends loadPage[] to the q process (code)
- pubsub.q holds the definition of loadPage which shows that it calls ‘sub’ (code)
- ‘sub’ then adds your subscriber to ‘subs’
-
Yes defining upd in this way means it behaves the same as insert (mostly)
q)upd:insert q)tab:([] a:1 2) q)insert[`tab;enlist 3] ,2 q)tab a - 1 2 3 q)upd[`tab;enlist 4] ,3 q)tab a - 1 2 3 4
But there are differences. ‘insert’ is a built in operator which cannot be passed as the first item by reference over a handle.
(This is causing the issue you are seeing)
q)value(`upd;`tab;enlist 5) //Pass by reference succeeds for user defined function ,4 q)value(`insert;`tab;enlist 6) //Pass by reference fails for operator 'insert [0] value(`insert;`tab;enlist 5) ^ q)value("insert";`tab;enlist 6) //Pass as parse string succeeds ,5 q)value(insert;`tab;enlist 6) //Pass by value succeeds ,6User defined functions can only use prefix notation whereas operators can be used prefix or infix.
q)`tab insert enlist 7 //Infix with operator succeeds ,7 q)`tab upd enlist 8 //Infix with user defined function fails 'type [0] `tab upd enlist 8 ^ q)insert[`tab;enlist 8] //Prefix with operator succeeds ,8 q)upd[`tab;enlist 9] //Prefix with user defined function succeeds ,9
-
That would require for deep level changes to the codebase.
For example when a HDB is written to a new:
- A new date folder is created.
- Tables are written.
- The HDB process is reloaded
If you reloaded the HDB before all tables were written you would have errors.
q)mkdir HDB q)cd HDB q)`:2021.01.01/tab1/ set ([] a:1 2 3) `:2021.01.01/tab1/ q)`:2021.01.01/tab2/ set ([] b:1 2 3) `:2021.01.01/tab2/ q)\l . q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 q)select from tab2 date b ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 q)`:2021.01.02/tab1/ set ([] a:4 5 6) `:2021.01.02/tab1/ q)\l . q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 2021.01.02 4 2021.01.02 5 2021.01.02 6 q)select from tab2 './2021.01.02/tab2/b. OS reports: No such file or directory [0] select from tab2 ^
.Q.bv – would be one possible helpful extension which can in memory fill tables missing from partitions (using ` as argument to fill using first partition as a template)
q)\l HDB q)select from tab1 date a ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3 2021.01.02 4 2021.01.02 5 2021.01.02 6 q)select from tab2 //Table missing from latest partition is not found 'tab2 [0] select from tab2 ^ q).Q.bv` //Using ` first partition used as prototype and table is found q)select from tab2 date b ------------ 2021.01.01 1 2021.01.01 2 2021.01.01 3
(.Q.chk is unsuitable as it uses most recent partition as template to fills tables missing from partitions)
This is only one item that would be needed to implement what you asked.
Currently EOD is a single action for all tables, all code and processes involved would need updates to operate table by table. Meaning changes in RDB/IDB/HDB and others.
-
Usually when you see this error it is due to one of:
- A fault in the network infrastructure between hosts
- One of the involved processes has died unexpectedly
- Some code in either your process or the remote has purposefully closed the connection (hclose)
There are handlers available in the .z namespace which are helpful to detect events on IPC handles:
- .z.pc is called after a connection has been closed
- .z.po is called after a connection has been opened
These could be implemented to track and attempt to reconnect dropped connections.
(.z.W is a useful dictionary or current open handles)
There are some examples such as dotz which is a library building on these features to trace, monitor and control execution.
-
The Platform codebase is designed to write date partitioned HDB’s only.
The Intraday database – KX Platform exists to so that 24hrs of data does not need to be kept in memory, but it will again store to the HDB on a date partition only.
-
Hi Leah,
This blog post will likely be of interest: https://kx.com/blog/partitioning-data-in-kdb/
It covers the basics of looking at hourly partitions and also fixed size partitions.
Regards,
Rian
-
These are examples of using the C interface to extend the Kdb+ database.
Documentation is here:
https://code.kx.com/q/interfaces/using-c-functions/
It shows an example of writing a new ‘add’ function in C:
#include"k.h" #ifdef __cplusplus extern "C"{ #endif K add(K x,K y) { if(x->t!=-KJ||y->t!=-KJ) return krr("type"); return kj(x->j+y->j); } #ifdef __cplusplus } #endifThen this can be used from inside a Kdb+ process:
q)add:(`add 2:(`add;2)) q)add[3;4] 7
This is a possible method to add math functions for 128-bit floating point numbers inside Kdb+.
-
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable ‘x’ inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on ‘x`b’ as a vector rather than inside ‘each’.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t 16 4746672 q)\ts {((first;last) x`b)@'x`a}t 7 4746640It’s goal is to avoid calculating ‘last each’ and ‘first each’ for every row.
Instead it uses each both (‘) to apply first or last after it is known which function is needed.
-
A single character .i.e “1” is a type -10h
A list of characters .i.e “10” is a type 10h
You can create a single character list using ‘enlist’ .i.eq)type each ("1";"0";"11-15") /Some are lists some are not -10 -10 10h q)type each (enlist "1";enlist "0";"11-15") /All are lists 10 10 10h q)("1";"0") "10" q)type each ("1";"0") -10 -10h q)(enlist "1";enlist "0") //Using enlist to make single character lists ,"1" ,"0" q)type each (enlist "1";enlist "0") 10 10hUsing ‘enlist’ will help prevent unwanted concatenation for you.
https://code.kx.com/q/ref/enlist/ -
All of these items are equivalent:
q)"10" "10" q)("1";"0") "10" q)("10") "10"They all resolve to 2 item lists containing characters
https://code.kx.com/q/basics/datatypes/
Here are some more example queries which may help:
q)select from t where Vals in `$/:("1";"10") Vals ---- 1 10 q)select from t where Vals in `$/:("1";"0") Vals ---- 1 0 q)select from t where Vals in `$"10" Vals ---- 10 q)select from t where Vals in `$"1" Vals ---- 1You can use ‘=’ rather than ‘in’ if you are searching for only one value:
q)select from t where Vals=`$"1" Vals ---- 1
You can input your symbols directly rather than casting:
q)`0`1`10 `0`1`10
-
You can use each-right to cast each character one at a time:
https://code.kx.com/q/ref/maps/#each-left-and-each-right
q) `$”10″
`10
q) `$/:”10″
`1`0
-
One option would be to unpivot the table using a helper function
/tab : the table to operate on /baseCols : the columns not to unpivot /pivotCols : the columns which you wish to unpivot /kCol : the key name for unpivoted data /vCol : the value name for unpivoted data unpivot:{[tab;baseCols;pivotCols;kCol;vCol] base:?[tab;();0b;{x!x}(),baseCols]; newCols:{[k;v;t;p] flip (k;v)!(count[t]#p;t p)}[kCol;vCol;tab] each pivotCols; baseCols xasc raze {[b;n] b,'n}[base] each newCols } unpivot[;`time;`price`pricev2;`priceType;`price] table time priceType price ---------------------------- 00:00:00.002 price 6.33 00:00:00.002 pricev2 6.32 00:00:01.001 price 4.05 00:00:01.001 pricev2 4.05 00:00:26.808 price 4.05 00:00:26.808 pricev2 5.07 00:00:27.002 price 5.12 00:00:27.002 pricev2 5.12 00:00:28.002 price 2.61 00:00:28.002 pricev2 2.61 00:00:29.002 price 4.61 00:00:29.002 pricev2 4.47 00:00:30.001 price 4.9 00:00:30.001 pricev2 4.47 00:00:31.000 price 4.64 00:00:31.000 pricev2 4.77 00:00:32.000 price 2.37 00:00:32.000 pricev2 4.87 00:00:33.000 price 3.75 00:00:33.000 pricev2 2.7Then you can set the options to graph the lines
-
In the visual inspector you can enter qsql queries.
Here are some example which may help. Firstly I recreated your table:
table:flip `time`price`pricev2!( (00:00:00.002 00:00:01.001 00:00:26.808 00:00:27.002 00:00:28.002 00:00:29.002 00:00:30.001 00:00:31 00:00:32 00:00:33); (6.33 4.05 4.05 5.12 2.61 4.61 4.9 4.64 2.37 3.75); (6.32 4.05 5.07 5.12 2.61 4.47 4.47 4.77 4.87 2.7));
Uses flip to create a table from a dictionary.
You can compare the columns using subtract:
update priceDiff:price-pricev2 from table time price pricev2 priceDiff ------------------------------------ 00:00:00.002 6.33 6.32 0.01 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14 00:00:30.001 4.9 4.47 0.43 00:00:31.000 4.64 4.77 -0.13 00:00:32.000 2.37 4.87 -2.5 00:00:33.000 3.75 2.7 1.05
You can use within to query within a time window:
select from table where time within 00:00:01 00:00:30 time price pricev2 -------------------------- 00:00:01.001 4.05 4.05 00:00:26.808 4.05 5.07 00:00:27.002 5.12 5.12 00:00:28.002 2.61 2.61 00:00:29.002 4.61 4.47
Then you can combine the 2 statements in to one:
update priceDiff:price-pricev2 from select from table where time within 00:00:01 00:00:30 time price pricev2 priceDiff ------------------------------------ 00:00:01.001 4.05 4.05 0 00:00:26.808 4.05 5.07 -1.02 00:00:27.002 5.12 5.12 0 00:00:28.002 2.61 2.61 0 00:00:29.002 4.61 4.47 0.14
This page has lots more examples: qsql
Hopefully this helps you.
-
You can see an example of mapping a decimal to either a list of bytes or scaling to a double in this interface.