[rocuinneagain]

This article may be of interest if you have not seen it:

https://dataintellect.com/blog/level-2-storage-formats/

[rocuinneagain]

q does not have a type for lists of lists so you use the generic untyped list ()

q)tab:([] bytesCol:())
q)`tab insert enlist (0x0021;0x0002232)
0 1
q)tab
bytesCol
----------
0x0021
0x00002232
q)meta tab
c       | t f a
--------| -----
bytesCol| X
q)type tab`bytesCol
0h

[rocuinneagain]

.Q.en uses ?. It is ? that does the locking. So both at the same time is fine.

See:

https://code.kx.com/q/ref/enum-extend/#filepath

code.kx.com

Enum Extend | Reference | kdb+ and q documentation - kdb+ and q documentation

Enum Extend is a q operator that extends an enumeration.

[rocuinneagain]

If a line ends in ; then the output is not displayed.

$ cat test.q
a:til 10
f:{x*y}
f[a;5]; /Not displayed
f[a;200] /Displayed
f[a;.01]; /Not displayed
$ q test.q
0 200 400 600 800 1000 1200 1400 1600 1800
q)

[rocuinneagain]

Note that -1 will display items differently than letting q display them

$ cat test.q
a:til 10
f:{"the answer is "," "sv string x*y}
f[a;200] //Result will print but include "" around it
f2:{-1 "the answer is "," "sv string x*y}
f2[a;200] //Result printed to standard out does not display "". -1 is also shown as the returned value.
f2[a;200]; // Same as above but ; suppresses -1 bring shown
f3:{-1 "the answer is "," "sv string x*y;}
f3[a;200] //; moved inside the function to supress -1
$q test.q
"the answer is 0 200 400 600 800 1000 1200 1400 1600 1800" //f[a;200]
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200]
-1                                                       //f2[a;200] continued
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200];
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f3[a;200]

[rocuinneagain]

Add .reset_index() when you create the tables to unkey them, then the for loop will run as you expect. For example:

group_taxi_dates = kx.q.xgroup("pickup_date", taxi_tab).reset_index()

[rocuinneagain]

**Untested** but a basic comsumer for testing for you would look something like:

\l ../kfk.q
kfk_cfg:(!) . flip(
    (`metadata.broker.list;`localhost:9092);
    (`group.id;`0);
    (`fetch.wait.max.ms;`10);
    (`statistics.interval.ms;`10000)
    );
client:.kfk.Consumer[kfk_cfg];

topic1:`trades;

generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
castRules:`time`sym`size`side!("P"$;`$;`long$;first)
trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
topcb1:{[msg]
  `trades insert generalHelper[enlist .j.k "c"$msg[`data];castRules]
  }
.kfk.Metadata[client];
.kfk.Subscribe[client;enlist topic1;enlist .kfk.PARTITION_UA;enlist topcb1]

[rocuinneagain]

If you review this blog post you will find it useful:

Kdb+/q Insights: Parsing JSON Files | KX

//Create a sample msg for testing
q)msg:`time`sym`price`size`side!("2024-08-25T10:50:10.743928";"ABC";117.4;67f;enlist "B")
q)msg
time | "2024-08-25T10:50:10.743928"
sym  | "ABC"
price| 117.4
size | 67f
side | ,"B"
//Enlisting a dictionary beomcesa single row table
q)enlist msg
q)time                         sym   price size side
--------------------------------------------------
"2024-08-25T10:50:10.743928" "ABC" 117.4 67   ,"B"
//The datatypes are not what we want
q)meta enlist msg
c    | t f a
-----| -----
time | C
sym  | C
price| f
size | f
side | C
//Use the function from the linked blog
q)generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
//Define the casting rules - adjust to what you would like
q)castRules:`time`sym`size`side!("P"$;`$;`long$;first)
//Now the msg can be handled to the datatypes we require
q)generalHelper[enlist msg;castRules]
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B
q)meta generalHelper[enlist msg;castRules]
c    | t f a
-----| -----
time | p
sym  | s
price| f
size | j
side | c
//Create an empty table to test saving the data
q)trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
q)trades
time sym price size side
------------------------
//We can insert the rows as they arrive
q)`trades insert generalHelper[enlist msg;castRules]
,0
q)trades
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B

[rocuinneagain]

You could start Jupyter with a core restriction:

taskset -c 0-15 jupyter notebook 

Or see this answer I posted previously: (you could choose 0-15 rather than 0,1)

https://stackoverflow.com/a/77481466/4256419

Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

{
 "argv": [
  "taskset",
  "-c",
  "0,1",
  "q",
  "jupyterq_kernel.q",
  "-cds",
  "{connection_file}"
 ],
 "display_name": "Q (kdb+)",
 "language": "q",
 "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"}
}

If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec

And you can validate in a notebook with system"taskset -pc ",string .z.i

xvGA7.png 4 KB Image File - Click to view

Options

• Copy Download Link

[rocuinneagain]

You can see an example of streaming data from http://www.coinapi.io over Websockets here:

• https://github.com/rianoc/qCoinAPI

For REST APIs the best library to use is kurl:

• https://code.kx.com/insights/1.10/core/kurl/quickstart.html

[rocuinneagain]

Install instructions: https://code.kx.com/q/learn/install/

Licencing specifics: https://code.kx.com/q/learn/licensing/

Your folder structure should look similar to:

$ tree ~/q
/home/user/q
├── m64
│   └── q
├── kc.lic
└── q.k

And you’d set:

export QHOME="/home/user/q"

[rocuinneagain]

The error is:

‘2024.08.12T08:04:56.974 couldn’t connect to license daemon — exiting

This means the licence you have is an online-license which needs to contact a licencing server on initialisation to verify it is valid.

The network is either not available or there are firewall rules preventing this verification to be made.

More details here: https://code.kx.com/q/learn/licensing/#licensing-server-for-kdb-on-demand

[rocuinneagain]

You can use a timer and wrap the rest of the code in a function.

h:(); .z.po:{h,:x};
{system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
.z.ts:{if[2=count h;system"t 0";main[]]}
\t 1000
main:{[] show "Rest of code" }

Or move the code out to a different file and only load that once the handles are open:

h:(); .z.po:{h,:x};
{system "q ",x," -p 0W &"} each ("feed1.q";"feed2.q");
.z.ts:{if[2=count h;system"t 0";system"l main.q"]}
\t 1000

[rocuinneagain]

q)reply:"\360\237\221\215"
q)reply
"\360\237\221\215"
q)-1 reply;
👍

[rocuinneagain]

You do not need pollKafkaAsync.

Otherwise things look fine.