gyorokpeter-kx
Forum Replies Created
-
Previously you mentioned using KDB+ 4.0 2023.08.11 installed on Ubuntu 20.04. But in your latest reply the release date is 2021.03.12. So you were probably using two different versions for two different tests. Can you try using the latest version of Cloud Edition as it should have some bug fixes related to object storage?
-
Can you try again after setting this environment variable?
export KX_TRACE_OBJSTR=1
Does it print any additional info?
-
Are you using the Insights Core version of kdb+ instead of the vanilla one (which lacks the cloud-related features such as object storage)?
-
Have you tried stracing the q process to see what is happening just before the error is printed?
l calls the .Q.l function to load the database, so you could try stepping through that if you are comfortable with reading k code.
-
Has your database grown in size such that it now goes over some threshold?
Alternatively some files in your HDB might be corrupted. A file with a corrupted size field might result in an attempt to allocate a very large amount of memory.
-
What is your kdb+ version? Older versions have a limit on the number of nested columns, although in that case the error would usually be “too many open files”.
Does the load always complain about the same file or a different one every time? If it’s the same, can you try making a backup and overwriting it with a column full of nulls with the correct size to see if the load progresses past it next time?
-
Can you check resource usage in your cluster? If you ingest too much data to fit into the memory of the EOI process, it will exit with a memory limit error (-w abort) and be unable to persist your data.
-
It doesn’t necessarily mean that the port is not available, but that there is currently no process listening on it.
If you are expecting a server to be running on that port, you might want to check what’s up with it – it might be crashing or hanging in an error trap, not reaching the point where it opens the port.
If you are just experimenting, you first need to start a second q process with the command line parameters -p 5001 which will make it listen on that port.
-
You have a keyed table in .perm.users. Dot syntax (.perm.users.user) doesn’t work for indexing in this case.
-
From the error message, it seems to be complaining that .perm.users.user does not exist. Can you check if this is the case? The variable needs to be defined before the first invocation of .z.pg happens.
-
There is some smart logic behind querying partitioned tables. If you filter on the partition column only, the data is actually not loaded into memory until you actually do some operations on it. So the first query doesn’t have to load all the data, just references to that specific partition. Then the second query does need to read in the data because you are querying an actual column in the table, so now q has to read the filtering column to find which rows match, and since you didn’t specify which columns to keep, it must also load the rest of the columns and filter them.
-
In reality there is no such thing as a “zero-argument function” in q. Even if you don’t have an argument list and you don’t use any of x, y and z in your function, or if you specify the argument list as [], the function still has one argument. If you call a function with “no arguments” [], that actually means calling it with :: as an argument.
Projection for the purpose of a deferred function call is achieved by adding a dummy argument, e.g. instead of
fn: {[x] x+1};you would have
fn: {[x;u] x+1};then fn[1] creates a projection that will execute the calculation with any argument passed.
For the second question, one possible way is by composing a projection of enlist with the dot-apply of your function:
q)proj:(‘)[fn .;(1; ; 3)]; q)proj .[{[x;y;z] x+y-z}]enlist[1;;3] q)proj 2 0 q)proj 10 8
-
You can perform the “cd” command over IPC, but as this will change the current directory in the process, you should be sure that this won’t cause problems, as any code that depends on the current value of “cd” might break.
Also if you are sending the command as a string to the handle, you have to escape any embedded quotes:
h"system "cd /my/directory/f1""
Or you could use a lambda with a dummy parameter instead:
h({system "cd /my/directory/f1"};::) -
Do you see the files for the new columns on disk?
Did you make sure to reload the database with l after the .Q.chk is done?
-
How is the thread started? It is only safe to call k(…) from a thread started by q itself. So it’s OK to call from C functions called by .z.pg regardless of multithreaded input mode, but it should not be called from a std::thread.