Replies – Forums – rocuinneagain

For example code use backslash to leave it as a multi line comment so those wishing to use this as a library can load the script without it running the example every time.

https://code.kx.com/q/basics/syntax/#comments

See example: https://github.com/KxSystems/kdb-tick/blob/master/tick.q#L50

This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 2 days, 11 hours ago by supportelearningwp-com.

code.kx.com

Syntax | Basics | kdb+ and q documentation - kdb+ and q documentation

Syntax of the q programming language

rocuinneagain

Member

September 27, 2024 at 10:36 am in reply to: .Q.gz – Not in Gzip format in java

The question is vague on what you are trying to do and how (no example code etc.)

Below is an example which when run results in:

HelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHelloHello

Java can decompress the data which kdb+ compressed with .Q.gz without issue:

import java.io.IOException;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.zip.*;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.lang.String;
import com.kx.c;
public class Main {
    public static String decompress(final byte[] bytes) throws IOException {
        if (bytes == null || bytes.length == 0) {
            return "";
        }
        try (final GZIPInputStream ungzip = new GZIPInputStream(new ByteArrayInputStream(bytes))) {
            final ByteArrayOutputStream out = new ByteArrayOutputStream();
            final byte[] data = new byte[8192];
            int nRead;
            while ((nRead = ungzip.read(data)) != -1) {
                out.write(data, 0, nRead);
            }
            return out.toString();
        }
    }
    public static void main(String[] args) {
        c c = null;
        Object result = null;
        byte[] bytes = null;
        try {
            c = new c("localhost", 5001);
            String query="`byte$.Q.gz(9;400#"Hello")";
            result = c.k(query);
            bytes = ((byte[]) result);
            System.out.println(decompress(bytes));
        } catch (Exception e){
            System.err.println(e.toString());
        }
    }
}

This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 1 month, 3 weeks ago by rocuinneagain.
This reply was modified 2 days, 11 hours ago by supportelearningwp-com.
This reply was modified 2 days, 11 hours ago by supportelearningwp-com.

rocuinneagain

Member

September 25, 2024 at 11:11 am in reply to: Tickerplant Architecture

This article may be of interest if you have not seen it:

https://dataintellect.com/blog/level-2-storage-formats/

rocuinneagain

Member

September 11, 2024 at 4:18 pm in reply to: Store array of bytes in a table column

q does not have a type for lists of lists so you use the generic untyped list ()

q)tab:([] bytesCol:())
q)`tab insert enlist (0x0021;0x0002232)
0 1
q)tab
bytesCol
----------
0x0021
0x00002232
q)meta tab
c       | t f a
--------| -----
bytesCol| X
q)type tab`bytesCol
0h

rocuinneagain

Member

September 9, 2024 at 8:14 am in reply to: Is it save to write to the same sym file by 2 or more q processes currently

.Q.en uses ?. It is ? that does the locking. So both at the same time is fine.

See:

https://code.kx.com/q/ref/enum-extend/#filepath

code.kx.com

Enum Extend | Reference | kdb+ and q documentation - kdb+ and q documentation

Enum Extend is a q operator that extends an enumeration.

rocuinneagain

Member

August 27, 2024 at 2:42 pm in reply to: Scripting and logging question

If a line ends in ; then the output is not displayed.

$ cat test.q
a:til 10
f:{x*y}
f[a;5]; /Not displayed
f[a;200] /Displayed
f[a;.01]; /Not displayed
$ q test.q
0 200 400 600 800 1000 1200 1400 1600 1800
q)

rocuinneagain

Member

August 27, 2024 at 2:49 pm in reply to: Scripting and logging question

Note that -1 will display items differently than letting q display them

$ cat test.q
a:til 10
f:{"the answer is "," "sv string x*y}
f[a;200] //Result will print but include "" around it
f2:{-1 "the answer is "," "sv string x*y}
f2[a;200] //Result printed to standard out does not display "". -1 is also shown as the returned value.
f2[a;200]; // Same as above but ; suppresses -1 bring shown
f3:{-1 "the answer is "," "sv string x*y;}
f3[a;200] //; moved inside the function to supress -1
$q test.q
"the answer is 0 200 400 600 800 1000 1200 1400 1600 1800" //f[a;200]
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200]
-1                                                       //f2[a;200] continued
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f2[a;200];
the answer is 0 200 400 600 800 1000 1200 1400 1600 1800 //f3[a;200]

rocuinneagain

Member

August 27, 2024 at 10:50 am in reply to: xgroup error in taxi and weather analysis

Add .reset_index() when you create the tables to unkey them, then the for loop will run as you expect. For example:

group_taxi_dates = kx.q.xgroup("pickup_date", taxi_tab).reset_index()

rocuinneagain

Member

August 26, 2024 at 12:37 pm in reply to: How to deserialize a Kafka topic message with kfk.q

**Untested** but a basic comsumer for testing for you would look something like:

\l ../kfk.q
kfk_cfg:(!) . flip(
    (`metadata.broker.list;`localhost:9092);
    (`group.id;`0);
    (`fetch.wait.max.ms;`10);
    (`statistics.interval.ms;`10000)
    );
client:.kfk.Consumer[kfk_cfg];

topic1:`trades;

generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
castRules:`time`sym`size`side!("P"$;`$;`long$;first)
trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
topcb1:{[msg]
  `trades insert generalHelper[enlist .j.k "c"$msg[`data];castRules]
  }
.kfk.Metadata[client];
.kfk.Subscribe[client;enlist topic1;enlist .kfk.PARTITION_UA;enlist topcb1]

rocuinneagain

Member

August 26, 2024 at 12:34 pm in reply to: How to deserialize a Kafka topic message with kfk.q

If you review this blog post you will find it useful:

Kdb+/q Insights: Parsing JSON Files | KX

//Create a sample msg for testing
q)msg:`time`sym`price`size`side!("2024-08-25T10:50:10.743928";"ABC";117.4;67f;enlist "B")
q)msg
time | "2024-08-25T10:50:10.743928"
sym  | "ABC"
price| 117.4
size | 67f
side | ,"B"
//Enlisting a dictionary beomcesa single row table
q)enlist msg
q)time                         sym   price size side
--------------------------------------------------
"2024-08-25T10:50:10.743928" "ABC" 117.4 67   ,"B"
//The datatypes are not what we want
q)meta enlist msg
c    | t f a
-----| -----
time | C
sym  | C
price| f
size | f
side | C
//Use the function from the linked blog
q)generalHelper:{[t;d]![t;();0b;key[d]!{(x;y)}'[valued;keyd]]}
//Define the casting rules - adjust to what you would like
q)castRules:`time`sym`size`side!("P"$;`$;`long$;first)
//Now the msg can be handled to the datatypes we require
q)generalHelper[enlist msg;castRules]
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B
q)meta generalHelper[enlist msg;castRules]
c    | t f a
-----| -----
time | p
sym  | s
price| f
size | j
side | c
//Create an empty table to test saving the data
q)trades:([] time:`timestamp$();sym:`$();price:`float$();size:`long$();side:`char$())
q)trades
time sym price size side
------------------------
//We can insert the rows as they arrive
q)`trades insert generalHelper[enlist msg;castRules]
,0
q)trades
time                          sym price size side
-------------------------------------------------
2024.08.25D10:50:10.743928000 ABC 117.4 67   B

rocuinneagain

Member

August 19, 2024 at 11:21 am in reply to: Run Jupyterq notebook with limit CPU cores

You could start Jupyter with a core restriction:

taskset -c 0-15 jupyter notebook

Or see this answer I posted previously: (you could choose 0-15 rather than 0,1)

https://stackoverflow.com/a/77481466/4256419

Edit the file https://github.com/KxSystems/jupyterq/blob/master/kernelspec/kernel.json

{
 "argv": [
  "taskset",
  "-c",
  "0,1",
  "q",
  "jupyterq_kernel.q",
  "-cds",
  "{connection_file}"
 ],
 "display_name": "Q (kdb+)",
 "language": "q",
 "env": {"JUPYTERQ_SERVERARGS":"","MPLBACKEND":"Agg"}
}

If you have already installed you can ensure Jupyter picks up the changes by reinstalling the kernel:

python -m jupyter kernelspec install --user --name=qpk /path/to/kernelspec

And you can validate in a notebook with system"taskset -pc ",string .z.i

Options

Copy Download Link

rocuinneagain

Member

August 27, 2024 at 2:54 pm in reply to: Scripting and logging question

q)reply:"\360\237\221\215"
q)reply
"\360\237\221\215"
q)-1 reply;
👍

rocuinneagain

Member

August 27, 2024 at 10:46 am in reply to: How to deserialize a Kafka topic message with kfk.q

You do not need pollKafkaAsync.

Otherwise things look fine.

rocuinneagain

Forum Replies Created