Forum Replies Created

Page 2 of 21
  • rocuinneagain

    Member
    August 26, 2024 at 12:37 pm in reply to: How to deserialize a Kafka topic message with kfk.q

    **Untested** but a basic comsumer for testing for you would look something like:

    \l ../kfk.q
    kfk_cfg:(!) . flip(
    (`metadata.broker.list;`localhost:9092);
    (`group.id;`0);
    (`fetch.wait.max.ms;`10);
    (`statistics.interval.ms;`10000)
    );
    client:.kfk.Consumer[kfk_cfg];

    topic1:`trades;

    generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
    castRules:`time`sym`size`side!("P"$;`$;`long$;first)
    trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
    topcb1:{[msg]
    `trades insert generalHelper[enlist .j.k "c"$msg[`data];castRules]
    }
    .kfk.Metadata[client];
    .kfk.Subscribe[client;enlist topic1;enlist .kfk.PARTITION_UA;enlist topcb1]
  • rocuinneagain

    Member
    August 26, 2024 at 12:34 pm in reply to: How to deserialize a Kafka topic message with kfk.q

    If you review this blog post you will find it useful:

    Kdb+/q Insights: Parsing JSON Files | KX

    //Create a sample msg for testing
    q)msg:`time`sym`price`size`side!("2024-08-25T10:50:10.743928";"ABC";117.4;67f;enlist "B")
    q)msg
    time | "2024-08-25T10:50:10.743928"
    sym | "ABC"
    price| 117.4
    size | 67f
    side | ,"B"
    //Enlisting a dictionary beomcesa single row table
    q)enlist msg
    q)time sym price size side
    --------------------------------------------------
    "2024-08-25T10:50:10.743928" "ABC" 117.4 67 ,"B"
    //The datatypes are not what we want
    q)meta enlist msg
    c | t f a
    -----| -----
    time | C
    sym | C
    price| f
    size | f
    side | C
    //Use the function from the linked blog
    q)generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
    //Define the casting rules - adjust to what you would like
    q)castRules:`time`sym`size`side!("P"$;`$;`long$;first)
    //Now the msg can be handled to the datatypes we require
    q)generalHelper[enlist msg;castRules]
    time sym price size side
    -------------------------------------------------
    2024.08.25D10:50:10.743928000 ABC 117.4 67 B
    q)meta generalHelper[enlist msg;castRules]
    c | t f a
    -----| -----
    time | p
    sym | s
    price| f
    size | j
    side | c
    //Create an empty table to test saving the data
    q)trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
    q)trades
    time sym price size side
    ------------------------
    //We can insert the rows as they arrive
    q)`trades insert generalHelper[enlist msg;castRules]
    ,0
    q)trades
    time sym price size side
    -------------------------------------------------
    2024.08.25D10:50:10.743928000 ABC 117.4 67 B

  • rocuinneagain

    Member
    August 19, 2024 at 11:21 am in reply to: Run Jupyterq notebook with limit CPU cores

    You could start Jupyter with a core restriction:

    taskset -c 0-15 jupyter notebook 

    Or see this answer I posted previously: (you could choose 0-15 rather than 0,1)

    https://stackoverflow.com/a/77481466/4256419

    Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

    {
     "argv": [
      "taskset",
      "-c",
      "0,1",
      "q",
      "jupyterq_kernel.q",
      "-cds",
      "{connection_file}"
     ],
     "display_name": "Q (kdb+)",
     "language": "q",
     "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"}
    }
    

    If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

    python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec
    

    And you can validate in a notebook with system"taskset -pc ",string .z.i

  • rocuinneagain

    Member
    August 14, 2024 at 12:31 pm in reply to: Connecting to API

    You can see an example of streaming data from http://www.coinapi.io over Websockets here:

    For REST APIs the best library to use is kurl:

  • rocuinneagain

    Member
    August 12, 2024 at 10:17 am in reply to: license error: k4.lic personal edition

    Install instructions: https://code.kx.com/q/learn/install/

    Licencing specifics: https://code.kx.com/q/learn/licensing/

    Your folder structure should look similar to:

    $ tree ~/q
    /home/user/q
    ├── m64
    │ └── q
    ├── kc.lic
    └── q.k

    And you’d set:

    export QHOME="/home/user/q"
  • The error is:

    ‘2024.08.12T08:04:56.974 couldn’t connect to license daemon — exiting

    This means the licence you have is an online-license which needs to contact a licencing server on initialisation to verify it is valid.

    The network is either not available or there are firewall rules preventing this verification to be made.

    More details here: https://code.kx.com/q/learn/licensing/#licensing-server-for-kdb-on-demand

  • rocuinneagain

    Member
    August 6, 2024 at 10:46 am in reply to: Process requests before continuing with script

    You can use a timer and wrap the rest of the code in a function.

    h:(); .z.po:{h,:x};
    {system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
    .z.ts:{if[2=count h;system"t 0";main[]]}
    \t 1000
    main:{[] show "Rest of code" }

    Or move the code out to a different file and only load that once the handles are open:

    h:(); .z.po:{h,:x};
    {system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
    .z.ts:{if[2=count h;system"t 0";system"l main.q"]}
    \t 1000
  • rocuinneagain

    Member
    August 1, 2024 at 9:26 am in reply to: simplifying code using multiple fby possible?
    t:([] payment_type:10#CARDCASH;vendor:10#abcd`e;fare:10?50;tip:10?50)
    t
    payment_type vendor fare tip
    ----------------------------
    CARD a 30 39
    CASH b 17 0
    CARD c 23 49
    CASH d 44 46
    CARD e 12 41
    CASH a 2 18
    CARD b 36 49
    CASH c 37 40
    CARD d 44 24
    CASH e 28 13
    select from t where ({exec (tip=max tip) and (fare>avg fare) from x};([] tip;fare)) fby vendor
    payment_type vendor fare tip
    ----------------------------
    CARD a 30 39
    CARD b 36 49

    You can pass in sub tables to aggregate on more than one column

    fby – Reference – kdb+ and q documentation – kdb+ and q documentation (kx.com)

  • rocuinneagain

    Member
    July 22, 2024 at 9:48 am in reply to: How can I apply a func to a grouped table?

    Sample table:

    q)t:([] stock:10000?`1;AskOrder:10000?100;BidOrder:10000?100)
    q)t
    stock AskOrder BidOrder
    -----------------------
    n 85 43
    p 40 15
    n 9 47
    j 10 64
    i 87 88
    i 79 87
    m 46 41
    h 92 63
    g 86 39
    ......

    Function which returns a dictionary of percentiles:

    q)f:{(`$x,/:string 1+til y)!az -1+(where deltas y xrank az:asc z),count z}

    Running the function returns a dictionary:

    q)r:exec f["Ask_";4;AskOrder],f["Bid_";4;BidOrder] by stock from t
    q)r
    | AskOrder BidOrder
    -| -------------------------------------------------------------------------
    a| `Ask_1`Ask_2`Ask_3`Ask_4!25 50 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 51 74 99
    b| `Ask_1`Ask_2`Ask_3`Ask_4!26 49 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 47 74 99
    c| `Ask_1`Ask_2`Ask_3`Ask_4!23 49 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!26 49 75 99
    d| `Ask_1`Ask_2`Ask_3`Ask_4!29 53 78 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 74 99
    e| `Ask_1`Ask_2`Ask_3`Ask_4!22 46 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!26 54 79 99
    f| `Ask_1`Ask_2`Ask_3`Ask_4!25 49 75 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 48 74 99
    g| `Ask_1`Ask_2`Ask_3`Ask_4!26 49 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 76 99
    h| `Ask_1`Ask_2`Ask_3`Ask_4!24 51 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 52 76 99
    i| `Ask_1`Ask_2`Ask_3`Ask_4!24 48 74 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 49 72 99
    j| `Ask_1`Ask_2`Ask_3`Ask_4!23 46 72 99 `Bid_1`Bid_2`Bid_3`Bid_4!31 53 75 99
    k| `Ask_1`Ask_2`Ask_3`Ask_4!21 44 72 99 `Bid_1`Bid_2`Bid_3`Bid_4!22 50 75 99
    l| `Ask_1`Ask_2`Ask_3`Ask_4!24 49 73 99 `Bid_1`Bid_2`Bid_3`Bid_4!27 50 73 99
    m| `Ask_1`Ask_2`Ask_3`Ask_4!23 45 70 99 `Bid_1`Bid_2`Bid_3`Bid_4!25 52 78 99
    n| `Ask_1`Ask_2`Ask_3`Ask_4!26 52 77 99 `Bid_1`Bid_2`Bid_3`Bid_4!25 53 76 99
    o| `Ask_1`Ask_2`Ask_3`Ask_4!22 50 75 99 `Bid_1`Bid_2`Bid_3`Bid_4!24 47 74 99
    p| `Ask_1`Ask_2`Ask_3`Ask_4!24 51 76 99 `Bid_1`Bid_2`Bid_3`Bid_4!23 48 74 99

    Transforming the dictionary in to a table:

    q){`stock xcols update stock:key x from (value x)[`AskOrder],'(value x)[`BidOrder]} r
    stock Ask_1 Ask_2 Ask_3 Ask_4 Bid_1 Bid_2 Bid_3 Bid_4
    -----------------------------------------------------
    a 25 50 74 99 23 51 74 99
    b 26 49 77 99 23 47 74 99
    c 23 49 74 99 26 49 75 99
    d 29 53 78 99 27 50 74 99
    e 22 46 73 99 26 54 79 99
    f 25 49 75 99 23 48 74 99
    g 26 49 73 99 27 50 76 99
    h 24 51 77 99 27 52 76 99
    i 24 48 74 99 23 49 72 99
    j 23 46 72 99 31 53 75 99
    k 21 44 72 99 22 50 75 99
    l 24 49 73 99 27 50 73 99
    m 23 45 70 99 25 52 78 99
    n 26 52 77 99 25 53 76 99
    o 22 50 75 99 24 47 74 99
    p 24 51 76 99 23 48 74 99
  • rocuinneagain

    Member
    July 17, 2024 at 12:53 pm in reply to: Using the partition type as a parameter

    It is advised to avoid using name of column or globals as function param names as much as possible. Here you are seeing issues as a result of this.

    In a HDB the issues caused by this scoping confusion can differ from in memory tables.

    This is due to map-reduce in the HDB, which means the query execution path is more complicated.

    1. type error:

    In the query path date would be expected to be a list but if the function local date atom is used a type error is being raised

    q){2024 select cnt:count i from Orders where date=2024.07.15}[.z.d] //Error
    q){[] select cnt:count i from Orders where date=2024.07.15}[.z.d] //Okay

    2. count = 0:

    You enlist the data param in one of the uses which means = returns a list and does not get the type error of query 1. However this list value does not make sense so an unexpected value is returned.

    Returns the count of the table in the first partition of the HDB

    q){2024 select cnt:count i from Orders where date=enlist 2024.07.15}[2024.07.15]

    Same result

    q){2024 select cnt:count i from Orders where date=2024.07.15}[enlist 2024.07.15]

    Returns the counts of the first 2 partitions

    q){2024 select cnt:{enlist count x}i from Orders where date=.z.d}[2#.z.d]

    You don’t even need to put in dates as the date param is being compared to itself you can put anything in:

    q){2024 select cnt:{enlist count x}i from Orders where date=1b}[11b]

    3. Works: Not using date as a param name – no issues

  • rocuinneagain

    Member
    July 11, 2024 at 1:48 pm in reply to: pykx is returning different output compared to embedQ

    PyKX also will convert a hsym directly to a Path type object

    q).pykx.print[.pykx.eval["lambda x:x"]`:/path/to/file]
    /path/to/file
    q).pykx.print[.pykx.eval["lambda x:type(x)"]`:/path/to/file]
    <class 'pathlib.PosixPath'>

    If you want a string of the Path you can call:

    streamer:.pylibs.security[`:smStreamer] .pykx.eval["lambda x:str(x)"] file
  • rocuinneagain

    Member
    July 11, 2024 at 1:26 pm in reply to: pykx is returning different output compared to embedQ

    See: Upgrading from embedPy – PyKX

    PyKX converts Python strings to q symbols which differs from embedPy

    q).pykx.eval["'hello'"]`
    `hello

    Pass your data back as Bytes for it to become q character vectors

    q)string_to_bytes:.pykx.eval["lambda x: bytes(x, 'utf-8')"]
    q)string_to_bytes[.pykx.eval["'hello'"]]`
    "hello"

  • rocuinneagain

    Member
    August 1, 2024 at 4:07 pm in reply to: How can I apply a func to a grouped table?

    In my answer I use 0N which is the long null. If your table column type for AskOrder is float then the resulting column will be of mixed type. Check by calling meta on the result. save cannot write mixed type columns like this.

    A good solution is to update f to ensure it uses a null of the correct datatype:

    f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#z count z}

    The added bit is z count z in place of 0N

    Indexing out of bounds on a q vector returns nulls of the dataype of the vector:

    q)(0 1)2 / Long null returned
    0N
    q)(0 1f)2 / Float null returned
    0n

  • rocuinneagain

    Member
    July 31, 2024 at 8:41 pm in reply to: How can I apply a func to a grouped table?

    Best would be to update the function to pad out nulls when needed:

    q)f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#0N}
    q)r:exec f["Ask_";6;AskOrder],f["Bid_";6;BidOrder] by stock from st
    q){`stock xcols update stock:key x from (value x)[`AskOrder],'(value x)[`BidOrder]} r
    stock Ask_1 Ask_2 Ask_3 Ask_4 Ask_5 Ask_6 Bid_1 Bid_2 Bid_3 Bid_4 Bid_5 Bid_6
    -----------------------------------------------------------------------------
    a 17 28 59 61 64 34 49 52 90 92
    b 41 46 74 92 94 97 26 30 70 82 88 93
    c 6 35 35 37 96 2 30 30 38 58
    d 4 4 7 10 66 96 12 23 38 39 45 61
    e 4 21 49 68 91 92 28 59 66 84 97 97
    f 10 18 45 65 83 91 8 39 43 64 77 78
    g 5 16 16 30 59 62 84 84 90 97
    h 48 51 56 59 71 12 45 47 82 95
    i 10 15 37 48 59 91 24 37 52 68 69 84
    j 0 57 57 66 74 36 63 63 73 83
    k 14 19 43 46 58 60 30 37 50 53 90 99
    l 25 29 33 49 59 98 6 26 59 60 68 93
    m 9 65 65 74 88 12 64 64 66 77
    n 14 14 40 40 98 44 44 73 73 93
    o 32 46 50 63 70 80 20 32 54 89 93 94
    p 4 48 48 51 92 23 40 41 74 88
  • rocuinneagain

    Member
    July 31, 2024 at 6:41 pm in reply to: How can I apply a func to a grouped table?
    select from t where 16<=(count;AskOrder) fby stock

    You’d want an fby for that filtering

    https://code.kx.com/q/ref/fby/

Page 2 of 21