[eohara_kdb]

@megan_mcp no problem

Screenshot-2024-07-17-125652.png 23 KB Image File - Click to view

Options

• Copy Download Link

[eohara_kdb]

Thanks @rocuinneagain , will test.

FYI we’d also tested changing the type from string to symbol, the symbol column compresses at the same ratio as in your example

q)show c:count get`:eohara_dev/strCol
18809996
q)vals:sym?c#
q)sym

symbol$()
q)(

:eohara_dev/test;17;2;5)set vals

:eohara_dev/test
q)-21!

:eohara_dev/test
compressedLength | 257281
uncompressedLength| 150484064
algorithm | 2i
logicalBlockSize | 17i
zipLevel | 5i

q)150484064%257281
584.9016

[eohara_kdb]

Thanks fellas! Didn’t have the README to hand

[eohara_kdb]

Hi,

 

There is a version of the C# interface here https://github.com/KxSystems/csharpkdb

that has an optional `useTLS` parameter: csharpkdb/kx/c.cs at main KxSystems/csharpkdb (github.com)

 

Thanks,

Eoghan

[eohara_kdb]

Here’s one option:

 

 

q)b:(1 5 6;9 10 76; 43 12 11)

q)n:(0;2;1)

q){x y}'[b;n]

1 76 12

 

 

 

[eohara_kdb]

As far as I know it’s not possible without some kind of each, so here’s the same approach for a table:

q)tab:([time:10:03:54.347 10:04:05.827]price:(20.83 21.44 26.83 29.83e;88.88 88.75 83.27 823.77f);n:0 1) 
q)update m:{x y}'[price;n] from tab 
time        | price n m 
------------| --------------------------------- 
10:03:54.347| 20.83 21.44 26.83 29.83 0 20.83e 
10:04:05.827| 88.88 88.75 83.27 823.77 1 88.75

 

[eohara_kdb]

Assuming your initial table can grow to a large size, probably best to only query it once to get the dates and symbols you’re interested in:

 

q)l:((2023.05.20;`a`b);(2023.05.19;enlist `b);(2023.05.18;`c`d`a);(2023.05.17;`d`a)); 
q)tab:flip `date`data!(2023.05.20 2023.05.20 2023.05.20 2023.05.19 2023.05.19 2023.05.19;`a`c`b`b`b`c); 
q)subset:select from tab where date in l[;0],data in raze l[;1]; 
q)raze{select from y where date=x[0],data in x[1]}[;subset]each l 
date      data 
--------------- 
2023.05.20 a 
2023.05.20 b 
2023.05.19 b 
2023.05.19 b

 

There’s probably a more efficient way but this should do it

[eohara_kdb]

Handles to files and processes | Basics | q and kdb+ documentation – Kdb+ and q documentation (kx.com)

 

1 means send to stdout

-1 means send to stdout, then append a newline character

[eohara_kdb]

The TP logs from those dates have been deleted in housekeeping jobs. We’ll investigate the process logs related to these write downs if they still exist, thanks!

[eohara_kdb]

1. I don’t, most likely the same as we are using now
2. No
3. No
4. No
5. No, the four columns that cause the mmap to increase all have count of 22210, whereas the rest have a count of 33199
6. When I re-write a bad partition and load it, all columns have a length of 22210 and I don’t see any mmap increase when I query the table

[eohara_kdb]

Yeah sounds like a good approach. We haven’t gotten to upgrade yet, in the meantime I see errors like this in my HDB when I’ve queried the same table a number of times

./2021.07.27/orders/aSymbolColumn. OS reports: Cannot allocate memory

When this happens .Q.w[] looks like the below example, where `mmap` has ballooned but `used` is ok:

used 204451472 
heap 805306368 
peak 3422552064 
wmax 0 
mmap 5866853440032 
mphy 809955348480 
syms 453383 
symw 44899939

 

[eohara_kdb]

Thanks for the info  . We were thrown by the fact that we had ~200 date partitions for this table, and only saw this behaviour in 3 of them.

The data doesn’t have any clear differences that we can see

[eohara_kdb]

Thanks for the reply,

Yeah looks like there’s no way around it. Thankfully sym links work as expected in par.txt

[eohara_kdb]

Hi @unterrainer_ale , it seems that the error only occurs when I’m querying a HDB table and when the parameter used in the where clause to filter on my partition column has the same name as the virtual column. Some different variations of the query in attached screenshot

Screenshot-2024-07-17-132324.png 59 KB Image File - Click to view

Options

• Copy Download Link