[jlucid]

I’ve since found a solution to this, specifically, a way to specify which network interface Q should use when establishing a socket connection.

1. Download and rename the attached file to interpose.c.
This file contains code that intercepts the creation of socket connections to bind them to a specific network interface

2. Compile it into a shared library gcc -shared -fPIC -o libinterpose.so interpose.c -ldl

3. Then start q using LD_PRELOAD=/path/to/libinterpose.so q
You need to set this LD_PRELOAD env variable to ensure that the socket creation function in the library is used instead of the default one
4. Within Q, specify the socket interface you want to use by setting the

environmental variable setenv[SOCKET_INTERFACE;”eth1″]

5. Open your socket connection as normal. Before opening additional socket connections on other interfaces, just change the SOCKET_INTERFACE value again

6. Verify its working. Download verify.txt and rename to verify.c. Compile: gcc -DKXVER=3 -shared -fPIC -o verify.so verify.c , You need k.h in the same directory. Then Load the shared library into Q and pass the handle value of the connection to the function get_socket_interface

libPath:/path/to/verify;

get_socket_interface:libPath 2: (get_socket_interface; 1);

q)get_socket_interface[5i] // 5i is just your handle value
“eth1”

User beware, while this works for me currently, we would need a core Kx person to say if its ok to do this or not, because I don’t know whats going on under the hood. There might be some negative side effect I am currently not seeing. It would be better if this could be done natively in Q, possibly an idea for a future release.

interpose.txt 1 KB Text File - Click to view

Options

• Copy Download Link

#define _GNU_SOURCE #include #include #include #include #include #include #include #include #include #include #include static int (*real_socket)(int domain, int type, int protocol) = NULL; __attribute__((constructor)) void init(void) { real_socket = dlsym(RTLD_NEXT, "socket"); if (!real_socket) { fprintf(stderr, "Error loading original socket function\n"); exit(1); } } int socket(int domain, int type, int protocol) { int sockfd = real_socket(domain, type, protocol); if (sockfd == -1) { return -1; } char *interface = getenv("SOCKET_INTERFACE"); if (interface) { FILE *log = fopen("/tmp/socket_bind.log", "a"); if (log) { fprintf(log, "Binding socket %d to interface %s\n", sockfd, interface); fflush(log); // Ensure the log is written immediately fclose(log); } if (setsockopt(sockfd, SOL_SOCKET, SO_BINDTODEVICE, interface, strlen(interface)) < 0) { FILE *log = fopen("/tmp/socket_bind.log", "a"); if (log) { fprintf(log, "Failed to bind socket to interface %s: %s\n", interface, strerror(errno)); fflush(log); // Ensure the log is written immediately fclose(log); } close(sockfd); return -1; } } return sockfd; }

Expand

verify.txt 728 B Text File - Click to view

Options

• Copy Download Link

#define _GNU_SOURCE #include #include #include #include #include #include #include #include #include #include "k.h" K get_socket_interface(K handle) { // Check that the handle is of type int if (handle->t != -KI) { return krr("handle type"); } int sock = handle->i; char if_name[IFNAMSIZ] = {0}; socklen_t optlen = sizeof(if_name); // Get the network interface associated with the socket if (getsockopt(sock, SOL_SOCKET, SO_BINDTODEVICE, if_name, &optlen) == -1) { return krr(strerror(errno)); } // Return the network interface name return kp(if_name); }

Expand

[jlucid]

Ok great, I didnt know about option (1), so the expiry date on the kdb+ license can be used to extend the dashboard expiry date.

(2) Seems like another fast option. We ended up just copying the existing dashboards and connections into the latest package and redeploying, but copying go.q_ is good to know.

Thanks Kieran

[jlucid]

I think you need the Edit Mode to be set to instant for those click boxes to appear. See edit mode section here.

 

https://code.kx.com/dashboards/datagrid/

[jlucid]

Thanks for the ideas Rian, yes the single anymap file would be too large, but I could try distributing the keys across a set of int partitions, so grouping them in some way, perhaps using a hash. That would reduce the search space. Then I could split the partitions again if they get too big.

 

Another idea was having a Bloom or Cuckoo filter associated with each date partition, using that to determine if a string is definitely not present in a partition to avoid searching, but it’s not a native feature and I can’t find any examples of people using that.

 

For the levelDB option, I see that I can compile the C++ library into a shared library and then load that into my q process. At least with that approach I am just writing a wrapper library for the main “Get” and “Put” methods. So that should be relatively quick to test against and use as a benchmark

 

[jlucid]

Ok no problem Kieran, thanks for looking into it

[jlucid]

My question is about calling the k function, in the C API, from different threads within a non q process. A C or C++ process in this case.  The q processes which are receiving the data are all independent and single threaded

[jlucid]

In version, 2.1.0 I see the same issue in modules/Datagrid_b71a7f6/main.js

replacing e[0] with e does the job to show the full error message.

Please let me know if this one is being tracked or was already fixed in a patch, thanks

[jlucid]

Ok thanks for confirming the symw query and for the work around suggestion.

I will give the mkdir+cd combo a go

[jlucid]

Perfect, that was the exact info I was looking for, thanks

[jlucid]

After a bit more digging I realised I can do this easily using the Mapping option, https://code.kx.com/dashboards/datasources/#mapping

I just changed the table structure to  have a sym column with values  `a `b and `c  and keyed it

 

 

[jlucid]

Thanks for confirming Megan. I was also looking at the data form but I didn’t see an increments option there. The best I could do there is provide the user with a drop down of values to select from.

[jlucid]

Thanks for that info and link, Kieran, just what I need. Looking forward to v2.1.0

 

[jlucid]

Thanks Kieran, that did the trick, updating as expected now

[jlucid]

Thanks for providing clarification on the modes Rian. So the replacement for the embedPy part is a licensed model. Noting that Pykx has more advanced functionality than embedPy.

Still, I hope they continue to maintain embedPy for users who only require what it offers, while keeping costs limited to the kdb license.

[jlucid]

PyKX under q requires the pykx license feature flag right, so there is a license fee to use it. This is the part where you are using python within the q interpreter (loading pykx.q). You can use PyKx free when sending data from python to kdb, when you set export PYKX_UNLICENSED=”true”. Whereas embedPy is free to use (apache 2.0 license). This is my understanding, please correct me if I am wrong