-
Sym File Locking
Does KDB automatically lock the sym file when writing to it? I have noticed what seems to be corruption when multiple processes are writing to the sym file at the same time, and was thinking of implementing write locking. However, I am not sure if this is already baked into .Q.en[] itself.
-
4 Replies
-
.Q.enuses?which does locking- https://code.kx.com/q/ref/dotq/#en-enumerate-varchar-cols
- https://code.kx.com/q/ref/enum-extend/#filepath
“The file is locked at a process level for writing during
.Q.enonly. Avoid reading from any file which may be being written to.”“The system call used is https://linux.die.net/man/3/lockf.”
-
Adding to ‘s information surrounding the underlying issue of .Q.en for concurrent writes, you might find This White Paper useful. It covers specifically how dataloaders handle mass writedowns which explicitly deals with the integrity of the sym file using multiple processes. Leveraging this will mean you can still benefit from the performance of multiple processes while avoiding any conflicts with the sym file.
This may be an overengineered solution for your use case and a more straightforward solution would just be IPC calls between writing processes communicating when a sym file is being written to, ensuring only one process writing at any given time but this should give you another option if concurrent writedowns is a requirement.
-
What precisely does at a process level mean? Does that mean only if a single port is multithreading (i.e. peaching and writing to the sym file the lock will be used)? Or I have two distinct ports both writing to the sym file, this built in locking won’t work?
My hypothesis is in the latter (multiple ports writing to the sym file at once), there is no locking of the sym file and we are seeing some concurrent writes. Solution for us to write our own lock that can handle these cases. -
The locking is done by the system call lockf (see link in previous message), this locks the file at the system level. No other process can get a lock on the file until the first process releases the lock.
Log in to reply.