[rocuinneagain]

If a line ends in ; then the output is not displayed.

$ cat test.q
a:til 10
f:{x*y}
f[a;5]; /Not displayed
f[a;200] /Displayed
f[a;.01]; /Not displayed
$ q test.q
0 200 400 600 800 1000 1200 1400 1600 1800
q)

[rocuinneagain]

Note that -1 will display items differently than letting q display them

$ cat test.q
a:til 10
f:{"the answer is "," "sv string x*y}
f[a;200] //Result will print but include "" around it
f2:{-1 "the answer is "," "sv string x*y}
f2[a;200] //Result printed to standard out does not display "". -1 is also shown as the returned value.
f2[a;200]; // Same as above but ; suppresses -1 bring shown
f3:{-1 "the answer is "," "sv string x*y;}
f3[a;200] //; moved inside the function to supress -1
$q test.q
"the answer is 0 200 400 600 800 1000 1200 1400 1600 1800" //f[a;200]
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200]
-1                                                       //f2[a;200] continued
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200];
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f3[a;200]

[rocuinneagain]

Add .reset_index() when you create the tables to unkey them, then the for loop will run as you expect. For example:

group_taxi_dates = kx.q.xgroup("pickup_date", taxi_tab).reset_index()

[rocuinneagain]

**Untested** but a basic comsumer for testing for you would look something like:

\l ../kfk.q
kfk_cfg:(!) . flip(
    (`metadata.broker.list;`localhost:9092);
    (`group.id;`0);
    (`fetch.wait.max.ms;`10);
    (`statistics.interval.ms;`10000)
    );
client:.kfk.Consumer[kfk_cfg];

topic1:`trades;

generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
castRules:`time`sym`size`side!("P"$;`$;`long$;first)
trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
topcb1:{[msg]
  `trades insert generalHelper[enlist .j.k "c"$msg[`data];castRules]
  }
.kfk.Metadata[client];
.kfk.Subscribe[client;enlist topic1;enlist .kfk.PARTITION_UA;enlist topcb1]

[rocuinneagain]

If you review this blog post you will find it useful:

Kdb+/q Insights: Parsing JSON Files | KX

//Create a sample msg for testing
q)msg:`time`sym`price`size`side!("2024-08-25T10:50:10.743928";"ABC";117.4;67f;enlist "B")
q)msg
time | "2024-08-25T10:50:10.743928"
sym  | "ABC"
price| 117.4
size | 67f
side | ,"B"
//Enlisting a dictionary beomcesa single row table
q)enlist msg
q)time                         sym   price size side
--------------------------------------------------
"2024-08-25T10:50:10.743928" "ABC" 117.4 67   ,"B"
//The datatypes are not what we want
q)meta enlist msg
c    | t f a
-----| -----
time | C
sym  | C
price| f
size | f
side | C
//Use the function from the linked blog
q)generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
//Define the casting rules - adjust to what you would like
q)castRules:`time`sym`size`side!("P"$;`$;`long$;first)
//Now the msg can be handled to the datatypes we require
q)generalHelper[enlist msg;castRules]
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B
q)meta generalHelper[enlist msg;castRules]
c    | t f a
-----| -----
time | p
sym  | s
price| f
size | j
side | c
//Create an empty table to test saving the data
q)trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
q)trades
time sym price size side
------------------------
//We can insert the rows as they arrive
q)`trades insert generalHelper[enlist msg;castRules]
,0
q)trades
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B

[rocuinneagain]

You could start Jupyter with a core restriction:

taskset -c 0-15 jupyter notebook 

Or see this answer I posted previously: (you could choose 0-15 rather than 0,1)

https://stackoverflow.com/a/77481466/4256419

Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

{
 "argv": [
  "taskset",
  "-c",
  "0,1",
  "q",
  "jupyterq_kernel.q",
  "-cds",
  "{connection_file}"
 ],
 "display_name": "Q (kdb+)",
 "language": "q",
 "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"}
}

If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec

And you can validate in a notebook with system"taskset -pc ",string .z.i

xvGA7.png 4 KB Image File - Click to view

Options

• Copy Download Link

[rocuinneagain]

You can see an example of streaming data from http://www.coinapi.io over Websockets here:

• https://github.com/rianoc/qCoinAPI

For REST APIs the best library to use is kurl:

• https://code.kx.com/insights/1.10/core/kurl/quickstart.html

[rocuinneagain]

Install instructions: https://code.kx.com/q/learn/install/

Licencing specifics: https://code.kx.com/q/learn/licensing/

Your folder structure should look similar to:

$ tree ~/q
/home/user/q
├── m64
│   └── q
├── kc.lic
└── q.k

And you’d set:

export QHOME="/home/user/q"

[rocuinneagain]

The error is:

‘2024.08.12T08:04:56.974 couldn’t connect to license daemon — exiting

This means the licence you have is an online-license which needs to contact a licencing server on initialisation to verify it is valid.

The network is either not available or there are firewall rules preventing this verification to be made.

More details here: https://code.kx.com/q/learn/licensing/#licensing-server-for-kdb-on-demand

[rocuinneagain]

You can use a timer and wrap the rest of the code in a function.

h:(); .z.po:{h,:x};
{system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
.z.ts:{if[2=count h;system"t 0";main[]]}
\t 1000
main:{[] show "Rest of code" }

Or move the code out to a different file and only load that once the handles are open:

h:(); .z.po:{h,:x};
{system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
.z.ts:{if[2=count h;system"t 0";system"l main.q"]}
\t 1000

[rocuinneagain]

t:([] payment_type:10#CARDCASH;vendor:10#abcd`e;fare:10?50;tip:10?50)
t
payment_type vendor fare tip
----------------------------
CARD         a      30   39
CASH         b      17   0
CARD         c      23   49
CASH         d      44   46
CARD         e      12   41
CASH         a      2    18
CARD         b      36   49
CASH         c      37   40
CARD         d      44   24
CASH         e      28   13
select from t where ({exec (tip=max tip) and (fare>avg fare) from x};([] tip;fare)) fby vendor
payment_type vendor fare tip
----------------------------
CARD         a      30   39
CARD         b      36   49

You can pass in sub tables to aggregate on more than one column

fby – Reference – kdb+ and q documentation – kdb+ and q documentation (kx.com)

[rocuinneagain]

You do not need pollKafkaAsync.

Otherwise things look fine.

[rocuinneagain]

In my answer I use 0N which is the long null. If your table column type for AskOrder is float then the resulting column will be of mixed type. Check by calling meta on the result. save cannot write mixed type columns like this.

A good solution is to update f to ensure it uses a null of the correct datatype:

f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#z count z}

The added bit is z count z in place of 0N

Indexing out of bounds on a q vector returns nulls of the dataype of the vector:

q)(0 1)2 / Long null returned
0N
q)(0 1f)2 / Float null returned
0n

[rocuinneagain]

Best would be to update the function to pad out nulls when needed:

q)f:{i:az -1+(where deltas y xrank az:asc z),count z;(`$x,/:string 1+til y)!i,(y-count i)#0N}
q)r:exec f["Ask_";6;AskOrder],f["Bid_";6;BidOrder] by stock from st
q){`stock xcols update stock:key x from (value x)[`AskOrder],'(value x)[`BidOrder]} r
stock Ask_1 Ask_2 Ask_3 Ask_4 Ask_5 Ask_6 Bid_1 Bid_2 Bid_3 Bid_4 Bid_5 Bid_6
-----------------------------------------------------------------------------
a     17    28    59    61    64          34    49    52    90    92
b     41    46    74    92    94    97    26    30    70    82    88    93
c     6     35    35    37    96          2     30    30    38    58
d     4     4     7     10    66    96    12    23    38    39    45    61
e     4     21    49    68    91    92    28    59    66    84    97    97
f     10    18    45    65    83    91    8     39    43    64    77    78
g     5     16    16    30    59          62    84    84    90    97
h     48    51    56    59    71          12    45    47    82    95
i     10    15    37    48    59    91    24    37    52    68    69    84
j     0     57    57    66    74          36    63    63    73    83
k     14    19    43    46    58    60    30    37    50    53    90    99
l     25    29    33    49    59    98    6     26    59    60    68    93
m     9     65    65    74    88          12    64    64    66    77
n     14    14    40    40    98          44    44    73    73    93
o     32    46    50    63    70    80    20    32    54    89    93    94
p     4     48    48    51    92          23    40    41    74    88

[rocuinneagain]

select from t where 16<=(count;AskOrder) fby stock

You’d want an fby for that filtering

https://code.kx.com/q/ref/fby/