Forum Replies Created

Page 3 of 21
  • rocuinneagain

    Member
    July 8, 2024 at 8:24 am in reply to: create random symbol with length >8

    8 is a documented limit

    https://code.kx.com/q/ref/deal/#generate

    symbols, each of n chars (n≤8)

    To create 3 random symbols of length 20:

    q){`$y cut (x*y)?16#.Q.a}[3;20]

    `hgpfbbonnjlpppnpjago`mebadmhpkcjgoheikpmh`pbkbabaofjhmacenfabc

  • rocuinneagain

    Member
    July 3, 2024 at 12:20 pm in reply to: Compression for null string column

    I expect this entry in 4.0 README for 2022.04.15 is the version from where you will see the improvement:


    2022.04.15
    NEW
    anymap write now detects consecutive deduplicated (address matching) toplevel objects, skipping them to save space
    q)a:("hi";"there";"world");`:a0 set a;`:a1 set a@where 1000 2000 3000;(hcount`$":a0#")=hcount`$":a1#"
    improved memory efficiency of writing nested data sourced from a type 77 file, commonly encountered during compression of files. e.g.
    q)`:a set 500000 100#"abc";system"ts `:b set get`:a" / was 76584400 bytes, now 8390720
  • rocuinneagain

    Member
    July 3, 2024 at 12:07 pm in reply to: Compression for null string column

    <div>Can you test against a newer version of 4.0? </div>

    My 4.1 gets much improved numbers:

    q)(.z.K;.z.k)
    4.1
    2024.04.29
    q)n:10000000;tab:([]time:n#.z.p;val:n?1000;str:n#enlist "");(`:tab/;17;2;5) set tab
    `:tab/
    q)-21!`:tab/str
    compressedLength | 136807
    uncompressedLength| 80004096
    algorithm | 2i
    logicalBlockSize | 17i
    zipLevel | 5i
    //Your compression 5.6x
    q)80004096%14074225
    5.684441
    //Compression now 584x
    80004096%136807
    584.7953
    q)-21!`$":tab/str#"
    compressedLength | 93
    uncompressedLength| 4098
    algorithm | 2i
    logicalBlockSize | 17i
    zipLevel | 5i
  • rocuinneagain

    Member
    July 2, 2024 at 10:56 am in reply to: Apply attributes on-disk in 4.1

    Yes. The website changelogs cover main/highlighted features only.

    For full changelog reference the README.txt for your version.

  • rocuinneagain

    Member
    July 1, 2024 at 12:57 pm in reply to: help with VWAP function
    Sample table for testing:
    q)data:([] timestamp:til 10;bq0:til 10;bq1:til 10;bq2:til 10;bp0:til 10;bp1:til 10;bp2:til 10;aq0:til 10;aq1:til 10;aq2:til 10;ap0:til 10;ap1:til 10;ap2:til 10) 
    q)data 
    timestamp bq0 bq1 bq2 bp0 bp1 bp2 aq0 aq1 aq2 ap0 ap1 ap2 
    --------------------------------------------------------- 
    0         0   0   0   0   0   0   0   0   0   0   0   0 
    1         1   1   1   1   1   1   1   1   1   1   1   1 
    2         2   2   2   2   2   2   2   2   2   2   2   2 
    3         3   3   3   3   3   3   3   3   3   3   3   3 
    4         4   4   4   4   4   4   4   4   4   4   4   4 
    5         5   5   5   5   5   5   5   5   5   5   5   5 
    6         6   6   6   6   6   6   6   6   6   6   6   6 
    7         7   7   7   7   7   7   7   7   7   7   7   7 
    8         8   8   8   8   8   8   8   8   8   8   8   8 
    9         9   9   9   9   9   9   9   9   9   9   9   9
    Test provided query:
    q)select timestamp, depth_vwap1: (bq0; bq1; aq0; aq1) wavg (bp0; bp1; ap0; ap1) from data 
    timestamp depth_vwap1 
    --------------------- 
    0 
    1         1 
    2         2 
    3         3 
    4         4 
    5         5 
    6         6 
    7         7 
    8         8 
    9         9
    parse query to functional form:
    q)parse"select timestamp, depth_vwap1: (bq0; bq1; aq0; aq1) wavg (bp0; bp1; ap0; ap1) from data" 
    ? 
    `data 
    () 
    0b 
    `timestamp`depth_vwap1!(`timestamp;(wavg;
    (enlist;`bq0;`bq1;`aq0;`aq1);
    (enlist;`bp0;`bp1;`ap0;`ap1)))
    Test functional query:
    q)?[data;();0b;`timestamp`depth_vwap1!(`timestamp;(wavg;
    (enlist;`bq0;`bq1;`aq0;`aq1);
    (enlist;`bp0;`bp1;`ap0;`ap1)))] 
    timestamp depth_vwap1 
    --------------------- 
    0 
    1         1 
    2         2 
    3         3 
    4         4 
    5         5 
    6         6 
    7         7 
    8         8 
    9         9
    Set up variables:
    q)maxDepth:2 
    q)quantities: `$raze(("bq"; "aq"),/:\:string til maxDepth) // Note `$ to convert strings to symbols 
    q)prices : `$raze(("bp"; "ap"),/:\:string til maxDepth) // Note `$ to convert strings to symbols
    Create needed list to pass to functional query:
    q)enlist,quantities 
    enlist 
    `bq0 
    `bq1 
    `aq0 
    `aq1 
    q)-1 .Q.s1 enlist,quantities; //.Q.s1 very useful to show full structure of object 
    (enlist;`bq0;`bq1;`aq0;`aq1)
    Test new dynamic functional select:
    q)?[data;();0b;`timestamp`depth_vwap1!(`timestamp;
    (wavg;enlist,quantities;enlist,prices))] 
    timestamp depth_vwap1 
    --------------------- 
    0 
    1         1 
    2         2 
    3         3 
    4         4 
    5         5 
    6         6 
    7         7 
    8         8 
    9         9
  • rocuinneagain

    Member
    July 1, 2024 at 10:24 am in reply to: Question about TCP and IPC handles limit

    https://code.kx.com/q/ref/dotz/#zpo-open

    https://code.kx.com/q/ref/dotz/#zpc-close

    You can use callbacks to see when handles open and close

    .z.po:{string[.z.p]," opened ",string x;x}
    .z.pc:{string[.z.p]," closed ",string x;x}

    Test this against your c.Close() calls to confirm it is closing the handles.

    If still seeing a problem raising an issue can be done at: https://github.com/KxSystems/csharpkdb/issues

  • rocuinneagain

    Member
    July 1, 2024 at 10:14 am in reply to: Question about TCP and IPC handles limit

    This limitation was removed in 4.1 release

    https://code.kx.com/q/basics/errors/#runtime-errors

    conn

    Too many connections. Max connections was 1022 prior to 4.1t 2023.09.15, otherwise the limit imposed by the operating system (operating system configurable for system/protocol).

    https://code.kx.com/q/releases/ChangesIn4.1/#unlimited-ipcwebsocket-connections

    The number of connections is now limited only by the operating system and protocol settings (system configurable).

  • rocuinneagain

    Member
    June 18, 2024 at 12:34 pm in reply to: Best way to access to kdb+ personally on windows

    Command prompt is the default, but the new Windows Terminal app is much nicer. I also use it to open sessions to WSL (Windows Subsystem for Linux).

    Visual Studio Code is a great option if building up bigger projects. An official kdb plugin is available for it.

  • rocuinneagain

    Member
    June 18, 2024 at 9:48 am in reply to: Help understanding scan adverb

    Testing speed and equivalence:

    fun_ema:{[lambda; liste];({[lambda; x; y]; (lambda*y)+ (x*1-lambda)}[lambda]\) liste}
    expma1:{[lambda;vector] {[x;y;z] (x*y)+z}\[ first vector; 1-lambda; vector * lambda]};
    q)a:10
    q)b:til 1000000
    q)\ts r1:fun_ema[a;b]
    174 32777680
    q)\ts r2:expma1[a;b]
    91 41166288
    q)r1~r2
    1b

    Your fun_ema performs *, +, *, a total of count[b] times. The operations are all on atom type data.

    expma1 performs *, + a total of count[b] times and , * once. The single use of * is on vector type data.

    q uses vector instructions for speed. These take advantage of CPU instructions to be more efficient

    q)a:til 1000000

    q)b: til 1000000

    q)\ts a+b // + operates on vectors and is fast

    1 8388800

    q)\ts a+'b // Using ' to force + to loop through each pair of atoms is much slower

    26 32777488

    q)\ts {x+y}'[a;b] //Wrapping in an unnecessary lambda is slower again

    62 32777792

    Since 4.0, kdb further sped up large vector operations by adding Multithreaded primitives which will spread the vector operations across multiple threads when available.

  • rocuinneagain

    Member
    June 17, 2024 at 7:54 am in reply to: select with ssr function

    https://code.kx.com/q/ref/each/

    Use each:

    q)select {[x]ssr[x;".C";""]} each orderID, cancelTime from t

    https://code.kx.com/q/basics/application/#projection

    Neater with projection:

    select ssr[;".C";""] each orderID, cancelTime from t

    https://code.kx.com/q/ref/drop/

    If you know .C is always at the end it’s much better to use drop _ rather than ssr as ssr is compute intensive

    select {-2_ x} each orderID, cancelTime from t

    https://code.kx.com/q/ref/maps/#each-left-and-each-right

    each-right /: is neater

    q)select -2_/:orderID, cancelTime from t
  • rocuinneagain

    Member
    June 14, 2024 at 11:55 am in reply to: when the backtick is required

    This is a qSQL statement:

    https://code.kx.com/q/ref/delete/

    q)delete ps from x

    qSQL has some differences to q.

    “delete is a qSQL query template and varies from regular q syntax.”

    In qSQL statements backticks are not needed for looking up columns in a table or keys in a dictionary.

    These are normal q statements:

    q)load `:flatT //Uses ` to declare a symbol, this case is known as a hsym, denoting a filepath as it starts with :
    q)`flatT in key `. //` used to declare a symbol `flatT and also to refer to . the default namespace
  • form of each https://code.kx.com/q/ref/maps/#each

    q){x[y]:99;x}'[(1 2 3;4 5 6;7 8 9);0 1 2]
    99 2 3
    4 99 6
    7 8 99

    Amend at @ version https://code.kx.com/q/ref/apply/#amend-amend-at

    q){@[x;y;:;99]}'[(1 2 3;4 5 6;7 8 9);0 1 2]
    99 2 3
    4 99 6
    7 8 99

    q)@[;;:;99]'[(1 2 3;4 5 6;7 8 9);0 1 2]
    99 2 3
    4 99 6
    7 8 99
  • rocuinneagain

    Member
    June 10, 2024 at 10:33 am in reply to: need help on a query
    q)tab:([] timestamp:1 2 3;timeWin:(1 2 3;1 2 4 5;3 1 1 3 5))
    q)tab
    timestamp timeWin
    -------------------
    1 1 2 3
    2 1 2 4 5
    3 3 1 1 3 5
    q)update timeWin:except'[timeWin;timestamp] from tab
    timestamp timeWin
    -----------------
    1 2 3
    2 1 4 5
    3 1 1 5
  • rocuinneagain

    Member
    June 18, 2024 at 9:15 am in reply to: when the backtick is required
  • rocuinneagain

    Member
    June 12, 2024 at 8:34 am in reply to: interrupt while xasc on disk

    https://code.kx.com/q/ref/asc/#sorting-data-on-disk

    “xasc can sort data on disk directly, without loading the entire table into memory.”

    If you interrupt an on disk xasc you could be in a state where some columns were sorted but not others.

Page 3 of 21