rocuinneagain
Forum Replies Created
-
Another kdb implementation of the game: https://github.com/psaris/wordle
-
I didn’t fully read the question, got distracted by the idea of a q implementation of the game.
-
The day part is an integral portion of the datatype and will display by default.
In a UI/display time it can be dropped if required:
q)2_string .z.n /Atom "14:41:40.125906000" q)2_/:string 2#.z.n /List "14:42:00.701751000" "14:42:00.701751000" q)update 2_/:string time from ([] time:2#.z.n) /Column in table time -------------------- "14:42:38.625329000" "14:42:38.625329000" // All columns of timespan type in a table q)dropDays:{c:where -16h=type each first x;$[count c;![x;();0b;c!{((/:;_);2;($:;x))}each c];x]} q)dropDays ([] time:2#.z.n;b:1.1 1.2) time b ------------------------ "14:47:37.376270000" 1.1 "14:47:37.376270000" 1.2 -
You could use the python statsmodels.tsa.vector_ar.vecm.JohansenTestResult by importing it through EmbedPy and passing the data as a dataframe using mltab2df.
Below is an example based of a similar python version: http://web.pdx.edu/~crkl/ceR/Python/example14_3.py
$pip install statsmodels q)l p.q q)l ml/ml.q q).ml.loadfile`:init.q q)data:flip `YEAR`Y`C!"IFF"$flip 1_-12_{{x where not ""~/:x}" " vs x} each "rn" vs .Q.hg "http://web.pdx.edu/~crkl/ceR/data/usyc87.txt" q)coint_johansen:.p.import[`statsmodels.tsa.vector_ar.vecm;`:coint_johansen] q)pd:.ml.tab2df[data][`:set_index;"YEAR"] q)res:coint_johansen[pd;0;2] q)res[`:lr1]` 31.78169 12.17119 -1.566747e-012 q)res[`:lr2]` 19.6105 12.17119 -1.566747e-012 q)res[`:cvm]` 18.8928 21.1314 25.865 12.2971 14.2639 18.52 2.7055 3.8415 6.6349 q)res[`:cvt]` 27.0669 29.7961 35.4628 13.4294 15.4943 19.9349 2.7055 3.8415 6.6349 q){flip y!(x@/:hsym y)@:`}[res;`lr1`lr2`cvm`cvt] lr1 lr2 cvm cvt ---------------------------------------------------------------------------- 31.78169 19.6105 18.8928 21.1314 25.865 27.0669 29.7961 35.4628 12.17119 12.17119 12.2971 14.2639 18.52 13.4294 15.4943 19.9349 -1.566747e-012 -1.566747e-012 2.7055 3.8415 6.6349 2.7055 3.8415 6.6349 -
Is there a reason you cannot use -g 1 ?
Running .Q.gc[] so often is not the best approach but if you needed to a timer could check and run if after .z.pg for you:
runGC:0b .z.pg:{ bigresult: value x;runGC::1b; bigresult }; .z.ts:{if[runGC;.Q.gc[];runGC::0b]} t 1https://code.kx.com/q/ref/dotz/#zts-timer
To run it less often you could set a threshold and in the if statement check memory usage
-
As in the example it is a nested generic list the items need to be dealt with one at a time. As the list could have many different tables or even different datatypes within it.
q)cols each .[dsEg;(`html;`body)] a b q).[dsEg;(`html;`body);{cols each x}] doctype| ,"html" html | `text`body!(,"test";,`a`b)The use of :: may be useful to you if you have not been using it
https://code.kx.com/q/ref/apply/#nulls-in-i
It allows you to skips levels
q).[dsEg;(`html;`body;::;`a)] d f g //Better shown on an item with multiple entries in the list q)dsEg2:(`doctype`html)!(enlist "html";`text`body!(enlist"test";2#enlist ([]a: `d`f`g;b: 23 43 777))); q).[dsEg2;(`html;`body;::;`a)] d f g d f g
.Q.s1 may also be useful to you as it can help show the underlying structure of an item better than the console at times.
https://code.kx.com/q/ref/dotq/#qs1-string-representation
q).[dsEg;(`html;`body;::;`a)] d f g //Looks like a symbol list type 11h but is in fact a single item egeneric list type 0h q){-1 .Q.s1 x;} .[dsEg;(`html;`body;::;`a)] ,`d`f`g //.Q.s1 output can be ugly but always shows exact structure -
They can be on separate machines/environments.
You can see in the White Paper that they use Kdb+ running on an ec2 instance ‘ip-172-31-70-197.ec2.internal’ and it connects to Solace broker running on ‘mr2ko4me0p6h2f.messaging.solace.cloud’
As long as your network settings are setup so that the 2 hosts can communicate over the needed ports then you will be successful.
-
You can
1. Upload the .tgz to the server (SCP/FTP etc)
2. Extract it
3. Import the package through the UI
https://code.kx.com/platform/release_management/#import-from-a-different-location
-
The traditional tick.q is a single threaded process but there are many areas you can investigate to ensure performance is at it’s best:
- Use taskset to ensure other processes are not using the same core/recourses as the TP
- As the TP persists data to a log file ensuring that you have evaluated your hardware configuration is important to validate that your storage disk is not bottlenecking your system
- Chained tickerplants can be used to balance the flow of data in a system, particularly when there are many subscribers
- Async broadcast can be used to optimise publishing the same data to multiple subscribers
- Ensure the system is setup to follow best practices outlined in Linux production notes
- Use Unix domain sockets when opening connections on localhost to reduce CPU usage
- Unix domain sockets are also available from C feedhandlers
- Whitepaper on Kdb+ tick profiling
KX Platform:
- Messaging performance tuning
- Tickerplant template
- Process instances core affinity
-
Yes your understanding is correct for those 4 items.
These tutorial videos may be useful:
-
q)tab:([] a:`a`b`c;b:1 2 3) //Create a table q)tab 0 //Index in to the table to row 0. A dictionary is returned a| `a b| 1 q)til 3 0 1 2 q)count tab // tab contains 3 rows 3 q)til count tab // 'til' returns numbers up to entered value 0 1 2 q){show tab x}each til count tab //Using 'each' and 'show' in a lambda to display each row a| `a b| 1 a| `b b| 2 a| `c b| 3- “A simple table is a list of dictionaries” https://code.kx.com/q/learn/dicts-tables/
- https://code.kx.com/q/kb/faq/#row-indexing
- https://code.kx.com/q4m3/6_Functions/#618-anonymous-functions-and-lambda-expressions
-
Kdb+ does have inbuilt HTTP functions for GET (.Q.hg) and POST (.Q.hp)
These blog posts may be of interest:
Another option would be to use embedPy.
You can use it to expose Python functions to q
- Web Scraping – A Kdb+ Use case – KX – (Related code: abin-saju/qblog)
When data is returned as JSON you can use .j.k to deserialize
-
TP = Tickerplant (Relays data to subscribers + recovers from logfile after crashes)
RDB = Realtime Database (Stores data for query)
RTE = Realtime engine (Performs streaming calculations and stores caches or publishes results)
(Any process can be customised)
1. Yes
2. It is acting more like a mixture between a TP,RDB,RTE
a) It does not store a logfile to recover in case of a crash (tp-logfile – a TP normally does this)
b) It stores data indefinitely instead of acting only as a relay. (Unlike a TP, more like and RDB, although an RDB will clear once every 24hrs)
c) It does not relay data untouched instead only specific data is forwarded (similar to an RTE)
getSyms – sends like of unique symbols across tables
getQuotes – sends last row by sym from quote table
getTrades – sends last row by sym from trade table
3. Yes
The execution flow is:
1. FH sends messages to PubSub (lines 18/19) every 100ms
2. The messages arrive to PubSub and .z.pg evaluates them. This mean upd/insert (pubsub.q line 8 will save the incoming data to quote/trade. PubSub now has some data cached.
3. The next time the PubSub timer (.z.ts) is triggered (every second) the ‘pub’ function will trigger and send data to subscriptions.
This code is a basic demo so it may have some holes in it’s logic (like never clearing data in PubSub so eventually memory will run out)
-
.z.ts is the timer function which is evaluated on intervals of the timer variable set by system command t
In your example ‘pub’ is called for each subscriber in ‘subs’ once per second.
To start subs has count 0 but when a new subscriber connects and subscribes they will be added.
How the subscribers are added to that table in that demo is:
- websockets.html calls connect() on load of the page (code)
- websockets.js holds the connect() definition which shows it sends loadPage[] to the q process (code)
- pubsub.q holds the definition of loadPage which shows that it calls ‘sub’ (code)
- ‘sub’ then adds your subscriber to ‘subs’
-
Yes defining upd in this way means it behaves the same as insert (mostly)
q)upd:insert q)tab:([] a:1 2) q)insert[`tab;enlist 3] ,2 q)tab a - 1 2 3 q)upd[`tab;enlist 4] ,3 q)tab a - 1 2 3 4
But there are differences. ‘insert’ is a built in operator which cannot be passed as the first item by reference over a handle.
(This is causing the issue you are seeing)
q)value(`upd;`tab;enlist 5) //Pass by reference succeeds for user defined function ,4 q)value(`insert;`tab;enlist 6) //Pass by reference fails for operator 'insert [0] value(`insert;`tab;enlist 5) ^ q)value("insert";`tab;enlist 6) //Pass as parse string succeeds ,5 q)value(insert;`tab;enlist 6) //Pass by value succeeds ,6User defined functions can only use prefix notation whereas operators can be used prefix or infix.
q)`tab insert enlist 7 //Infix with operator succeeds ,7 q)`tab upd enlist 8 //Infix with user defined function fails 'type [0] `tab upd enlist 8 ^ q)insert[`tab;enlist 8] //Prefix with operator succeeds ,8 q)upd[`tab;enlist 9] //Prefix with user defined function succeeds ,9