Laura
Forum Replies Created
-
I’ve tested it down to only 1 column, the timestamp, being wj to.
Is there a way to open threading or something? It’s like 12,000 entries per timestamp being read. How long should that take?
-
I have 800,000 rows of data in a table with the following meta:
c | t f a ---------------| ----- col1 | n col2 | s col3 | d mmm3 | f col4 | f col5 | s col6 | i col7 | i col8 | j col9 | f col10 | j dt | p s col11 | f col12 | f col13 | j col14 | j col15 | f col16 | f col17 | f col18 | f col19 | f col20 | f col21 | f col22 | f col23 | j col24 | j
Im attempting to get a rolling 5minute min/max using wj. (i recognize the window isn’t such for 5minutes, im just writing up an example)
f:`dt; w:(-1000000; 0)+:data[`dt];
data: wj[w;f;data;(data;(max;`mmm3);(min;`mmm3))];The wj works just fine, but takes insanely long to complete.
-
Q doesnt really offer built-in combinators as liberally as its ancestor language APL does. An elegant way to compose the projections of
f1andf2? You can lose the parensq)c: f1[;a] f2[;a]@ q)c 3 0n 0.5 0.5 0.3333333 0.25 0.2 0.1666667 0.1428571 0.125 0.1111111
Of course, if you have to do this often, you can write your own combinator. Call itcrfor curry right:q)cr:{x[;y]} q)c:('[;])over(f1;f2)cr:a / Compose over (f1[;a];f2[;a]) q)c 3 0n 0.5 0.5 0.3333333 0.25 0.2 0.1666667 0.1428571 0.125 0.1111111Compose over a list of functions as many as you need. -
Huh now I see what youre up to: implementing
rotatein q.This reminds me of a category of puzzle that used to be popular in qs ancestor language APL, called dead key questions. The premise was always that a certain key on your keyboard was not working; how do you code around it?
APL primitives are mostly variadic: they can be applied as either unaries or binaries, usually with related semantics. And they are (almost) all single characters, which is what gives the dead key problem its bite. For example,
rotateandreversein q are in APL the binary and unary forms of?l?2 3 5 7 11 ?l 11 7 5 3 2 2?l 5 7 11 2 3
In k, operators are similarly variadic; q replaces the unary forms with keywords. Like k itself, the unary forms of the operators are considered exposed infrastructure: use the q keywords instead. The
(#)you asked about is, as P鴥r advises, in q written ascount. -
Watch out The q language is implemented as a DSL embedded in the k4 programming language. The latter is exposed infrastructure. It is neither supported nor documented, and its use is deprecated and in production code, strongly deprecated.
-
The q language overloads many symbols, and yes, quote is one of them. It most commonly denotes the Each iterator, but in your examples it denotes the Compose operator.
This overloading makes precise terminology particularly important when discussing q syntax, which is why we revised the terminology and stopped using terms such as adverb.
In your first example,
'[f;ff]returns a derived function, equivalent to{f ff x}.The extra execution overhead of using a lambda is tiny, which is probably why you are more likely to find the lambda in code. But because Compose is a binary operator, you can use
'[;]to reduce a list of (mostly) unary functions or operators. (You need to use'[;]rather than'in order to resolve the overloading.)So, for example, if
f,g, andhare unaries, andffis a binary, then('[;])over(f;g;h;ff)is equivalent to{f g h ff x}. In most cases you would probably choose to write the lambda. But if the choice (or even the number) of functions is to be made at runtime, then the reduction could be just the way to put it all together.It is worth remembering here that lists and dictionaries are also unaries. If you had a sequence of mappings to apply, with the choice of mappings determined at runtime (perhaps columns picked from a table) then Compose Over
('[;])overmight be your best way to assemble it. -
Assume here the column value is a matrix. Start with the new column names.
q)show ncn:`$string[`b],/:string 1+til count first t`b `b1`b2`b3
Pair them with the matrix columns.
q)show d:ncn!flip t`b b1| 4 6 12 b2| 5 12 36 b3| 6 23 14
Heres the table with the nested column removed. (We use Functional qSQL rather than hard-code the column name.)
q)show twc:![t;();0b;enlist`b] / table without column a - 1 2 3 q)twc,'flip d / Join Each new tuple a b1 b2 b3 ---------- 1 4 5 6 2 6 12 23 3 12 36 14
Putting it together:
un:{[tbl;col] / un-nest col in tbl mat:flip tbl col; / un-nest column ncn:`$(,/:) . string(col;) 1+til count mat; / new column names ![tbl;();0b;enlist col],'flip ncn!mat } q)un[t;`b] a b1 b2 b3 ---------- 1 4 5 6 2 6 12 23 3 12 36 14 -
Hi – try launching instead from incognito/private browsing mode in another browser provider. This should give you a fresh instance with all original files.
Thanks,
Michaela
-
The error message JPEG looks like source code; I dont see an HDF file attached. Did I miss something?
-
This answer assumes your
rolloverandroll_rankcolumns are instrumental to the Expected Output and not required in it.
TL;DR find cumulative maxima; eliminate recurrences; upsert into an empty table; fill nulls
Well add two extra lines to your script: one to test the constraint described in your last comment; the other to check we find the current and not just the first maximum for the leading symbol. Well also sort
tmpafter the updates to ensure it reflects them. So the script ends://below steps are to generate/update sample data for testing // to match desired dataset which we want to create a new logic tmp:update volume:500.4 from tmp where sdate=2010.01.04, sym=`VXG8; // find current not first maximum tmp:update volume:600.6 from tmp where sdate=2010.01.05, sym=`VXG8; // confirm VXZ4 cannot recur tmp:update volume:700.7 from tmp where sdate=2010.01.06, sym=`VXZ4; tmp:`sdate xasc `volume xdesc tmp; / tmp:update sums roll_rank by sdate from tmp;
The first step enlarges Terry Lynchs answer on StackOverflow. We select the rows where the maximum changes and also mark where the symbol changes.
q)show q:update rollover:differ sym from select sdate,sym,name,volume from tmp where differ maxs volume sdate sym name volume rollover ----------------------------------------- 2010.01.01 VXZ4 someName4 400.4 1 2010.01.04 VXG8 someName3 500.4 1 2010.01.05 VXG8 someName3 600.6 0 2010.01.06 VXZ4 someName4 700.7 1
The second step eliminates the last row of
q, because VXZ4 may not recur.For this we use an ancient APL idiom for finding duplicates in a vector. In APL its
(??x)?x?x, which translates easily into q as(til count x)<>x?x. And we key onsdate.q)show r:1!delete from q where rollover and {(til count x)<>x?x}sym sdate | sym name volume rollover ----------| ------------------------------ 2010.01.01| VXZ4 someName4 400.4 1 2010.01.04| VXG8 someName3 500.4 1 2010.01.05| VXG8 someName3 600.6 0This is all we need in the result. The rest is just filler.
q)/ make a template table q)s:1!flip`sdate`sym`name`volume!flip tdate,:(`;`;0n) q) / and fill in the gaps q)fills s upsert delete rollover from r sdate | sym name volume ----------| --------------------- 2010.01.01| VXZ4 someName4 400.4 2010.01.02| VXZ4 someName4 400.4 2010.01.03| VXZ4 someName4 400.4 2010.01.04| VXG8 someName3 500.4 2010.01.05| VXG8 someName3 600.6 2010.01.06| VXG8 someName3 600.6 2010.01.07| VXG8 someName3 600.6 2010.01.08| VXG8 someName3 600.6 2010.01.09| VXG8 someName3 600.6 2010.01.10| VXG8 someName3 600.6 2010.01.11| VXG8 someName3 600.6 2010.01.12| VXG8 someName3 600.6 ..
-
Full answer on SO.
TL;DR: find cumulative maxima; eliminate recurrences; upsert into an empty table; fill nulls
Eliminating recurrences uses a vector idiom inherited from APL that flags duplicate items in a list. In APL its
(??x)?x?x, which translates easily into q as(til count x)<>x?x. -
Rians excellent logic can be refactored to use the Over iterator. (You might not be familiar with Over with a ternary or quaternary function.)
q)g:{"b"$0^fills@/[;(x,y+1){where x=x msum y}'1 notz;:;1 0]count[z]#0N} q)g[3;2] 00001111000111001101000b 00000011110001111111110b q)g[3;2] 110010111001111b 000000001111111bThe key concept here is that the first argument of Amend At Over
@/is the initial state:count[z]#0N. The other (right) arguments are same-length lists, or atoms. Over works through the argument lists in succession.Once again we see the Zen monks as a point-free alternative to writing
(z;not z).The refactoring here doesnt save much time, but spotting opportunities like this improves your ability to find iterator solutions, some of which will save you significant CPU.
-
Hi
The expected output is a dictionary with 3 keys `edge `qty `numTrds. Their corresponding values should be the result of the correlation between the 15 minute timeseries for `edge (i.e correlation of the edge calculated in the previous question 4.6) with each of `edge `qty `numTrds
Output should look like:
edge | 1 qty | 0.8059053 numTrds| 0.7830214
The iterator that might be useful here is each right , check out the Iterators course to see it in action
-
q)u isMatched index id ------------------------- 1100b 0 1 2 3 a b c d 1100b 1 0 2 3 a b c d 0011b 0 1 2 3 a b c d q)fw:first where@ / composition q)select index:index@'j,id:id@'j from update j:fw each isMatched from u index id -------- 0 a 1 a 2 c
Or you might prefer the same thing as vector operations:
q)u[`index`id]@':fw each u`isMatched 0 1 2 a a c q)ts:10000 select index:index@'j,id:id@'j from update j:fw each isMatched from u 65 2832 q)ts:10000 u[`index`id]@':fw each u`isMatched 21 1376
In the vector expression,
fw each u`isMatchedcorresponds tofw each isMatched from u. But instead of forming columnjof a new temporary table, it is applied direct to theindexandidcolumns ofu. Notice how Each Left:takes Index At Each@'as its argument to derive binary function Index At Each Each Left.Your sublists are short; if they were longer you might prefer faster
?'[;1b]tofw each:q)ts:10000 select index:index@’j,id:id@’j from update j:?'[;1b] isMatched from u
61 3056
q)ts:10000 u[`index`id]@’:u[`isMatched]?’
1b
16 1568
-
q)show people:([]pickSeq:neg[n]?n;person:`$"person",/:string 1+til n;allowedToPick:n?01b) pickSeq person allowedToPick ------------------------------ 1 person1 0 8 person2 1 5 person3 0 7 person4 1 0 person5 0 3 person6 1 6 person7 0 4 person8 0 2 person9 1 9 person10 0 q)show prize:100*1+til 8 100 200 300 400 500 600 700 800
The winners in order:
q)`pickSeq xasc select from people where allowedToPick pickSeq person allowedToPick ----------------------------- 2 person9 1 3 person6 1 7 person4 1 8 person2 1
And their rewards:
q){select person,reward:count[x]#desc prize from x}`pickSeq xasc people where people`allowedToPick person reward -------------- person9 800 person6 700 person4 600 person2 500Or in vector form:
q){x!count[x]#desc prize}{x iasc y}. flip people[where people`allowedToPick;`person`pickSeq] person9| 800 person6| 700 person4| 600 person2| 500Above,
where people`allowedToPickfinds the winners; and{x iasc y}.sorts the winners names by pick order. The vector form has less work to do:q)ts:1000 {select person,reward:count[x]#desc prize from x}`pickSeq xasc people where people`allowedToPick 12 3184 q)ts:1000 {x!count[x]#desc prize}{x iasc y}. flip people[where people`allowedToPick;`person`pickSeq] 3 3312