rocuinneagain
Forum Replies Created
-
1. I might suggest to think about it the other way. Dictionaries are more like special paired lists.
q)dict:`a`b!1 2 q)lists:{(key x;value x)} dict q)dict a| 1 b| 2 q)lists a b 1 2 q)dict `a 1 q)lists[0]?`a 0 q)lists[1] lists[0]?`a / The same as: dict `a 1https://code.kx.com/q/ref/find/
2. Yes a list of conforming dictionaries is promoted a table
q)(`a`b!1 2;`a`b!1 2) a b --- 1 2 1 2
Importantly in memory the way it is actually stored is ‘flipped’ so it is a dictionary of lists. (no longer a list of dictionaries)
q).Q.s1 (`a`b!1 2;`a`b!1 2) "+`a`b!(1 1;2 2)"
This way the keys/column-names only need to be stored once for the whole table and not for each row.
The columns then are vectors which is more efficient and performant.
There are more details on indexing at depth here :
https://code.kx.com/q4m3/3_Lists/#38-iterated-indexing-and-indexing-at-depth
The “querying unstructured data” section of this blog may be of interest:
https://kx.com/blog/kdb-q-insights-parsing-json-files/The code in it focuses on tables but can be adapted to lists/dictionaries as well:
q)asLists:sample cols sample q)asLists[0;;`expiry] 17682D19:58:45.000000000 ` `long$() ,"" `long$() 0N ,""
q)@[`asLists;0;{(enlist[`]!enlist (::))(,)/:x}]
`asLists
q)asLists[0;;`expiry]
17682D19:58:45.000000000
::
::
::
::
::
::
q)fill:{n:count i:where (::)~/:y;@[y;i;:;n#x]}
q)fill[0Wn]asLists[0;;`expiry]
17682D19:58:45.000000000 0W 0W 0W 0W 0W 0W -
ssr is a much slower operation than the above examples.
Using wildcards such as “*#” is also difficult as it matches more than one suffix (.e.g “#” and “^#”).
To work around this you can choose the longest matching suffix.Using ‘like’ and ‘@’ where possible rather than ‘ssr’ as below is a big speed improvement.
Searching for “*” is also difficult as it is a wildcard. Instead, I use tab “t” in it’s place.
If “t” is possibly in your data you would need to change this for another character.
symbology:.Q.id ("****";enlist ",")0:`:symbology.csv update searchNASDAQ:{"*",@[x;where x="*";:;"t"]} each NASDAQ from `symbology func:{s:string x; m:select from symbology where @[s;where s="*";:;"t"] like/:searchNASDAQ; l:max count each m`NASDAQ; c:first exec CMS from m where l=count each NASDAQ; `$$[c~();s;(neg[l]_s),c] };In a test it does seem to operate correctly.
Overall it still has risks as if bad data is sent in the function cannot truly validate what it is doing. Ideally you would have root and suffix separated by a known delimiter such as a space ” ” in the source data.
-6 sublist {([] symNASDAQ:n;symCMS:func each n:`$"AAPL",/:x)}symbology`NASDAQ symNASDAQ symCMS ------------------ AAPL# AAPLWI AAPL^# AAPLRTWI AAPL-# AAPLPRWI AAPL.A# AAPLAWI AAPL+# AAPLWSWI AAPL~ AAPLTESTThe extra complexity does come at a cost of speed.
\ts func each 10000#`$"AAPL+#" 41 553776
symbologyOld:`NASDAQ xkey .Q.id (“****”;enlist “,”)0:`:symbology.csv
\ts {s:string x;`$(4#s),symbologyOld[4 _ s]`CQSSuffix} each 10000#`$”AAPL+#”
12 554448
\ts {s:string x;r:first where not s in .Q.A;`$(r#s),symbologyOld[r _ s]`CQSSuffix} each 10000#`$”AAPL+#”
16 554464As you used in your original example .Q.fu is a great tool when performing an intensive task repeatedly.
Extreme example with only one unique input:
\ts .Q.fu[func each] 10000#`$"AAPL+#" 0 394032
One limitation of .Q.fu is that it has no memory between executions.
One example of a way to bypass this would be a memory cache.
The library https://github.com/gitrj95/q-memo could be used.
In this use-case your function is fast enough that the cache is overkill and should not actually be used but would be useful for very slow operations you may have to run many times.
l memo.k .memo.init[`.;10000h] //Create a cache with 10k limit .memo.mk[`func;`memoFunc;`cache.0] //Create a wrapped function to use the cache
ts memoFunc each 10000#`$(“AAPL~”;”AAPL+#”)
126 553936//The cache pre stores the results cache.0 f a | r -------------------| --------- :: | :: ..memoFunc ,`AAPL~ | `AAPLTEST ..memoFunc ,`AAPL+#| `AAPLWSWI
-
Without knowing the full rules and logic here are some thoughts.
//Table taken from https://www.nasdaqtrader.com/trader.aspx?id=CQSsymbolconvention q)symbology:`NASDAQIntegratedPlatformSuffix xkey .Q.id ("****";enlist ",")0:`:symbology.csv q)5 sublist symbology NASDAQIntegratedPlatformSuffix| Security CQSSuffix CMSS.. ------------------------------| ---------------------------------------------.. ,"-" | "Preferred" ,"p" "PR".. "-A" | "Preferred Class "A"*" "pA" "PRA.. "-B" | "Preferred Class "B"*" "pB" "PRB.. ".A" | "Class "A"*" ".A" ,"A".. ".B" | "Class "B"*" ".B" ,"B"..
//Assuming 4 character root q){s:string x;`$(4#s),symbology[4 _ s]`CQSSuffix} each (`$"AAPL.B*";`$"AAPL^#") `AAPL`AAPLrw//Assuming there is always a suffix and first non capital letter is beginning of suffix {s:string x;r:first where not s in .Q.A;`$(r#s),symbology[r _ s]`CQSSuffix} each (`$"AAPL.B*";`$"AAPL^#") `AAPL`AAPLrwNote: ‘.B*’ does not appear in that mapping table so ‘AAPL.B*’ does not receive a new CQS Suffix.
-
You can use .p.wrap https://code.kx.com/q/ml/embedpy/userguide/#embedpy-objects_1
Small example giving each subplot a title:
plt:.p.import[`matplotlib;`:pyplot] plts:plt[`:subplots;<;pykwargs `ncols`figsize`dpi!(2;24 8;100)] .p.wrap[plts[1;0]][`:set_title]"Plot1"; .p.wrap[plts[1;1]][`:set_title]"Plot2"; plt[`:show][];
Results in:
-
On the KX Platform you should not use .z.exit directly as it performs some inbuilt operations.
Instead use .ch.addExit which is a handler for you to add your custom behaviour on top of what is inbuilt.
-
Yes the w32 version has a limit to how much memory it can address, w64 does not have this restriction.
You could also stream the data to an on disk table:
.Q.fs[{`:trade/ upsert flip colnames!("**********";",")0:x}]`:filename trade:get `:trade/