[clivelo]

Replying my solution in case someone else is experiencing this.

I created a tiny subset of the data and tested in a barebone environment. Although the error didn’t occur, the mmap value was still non-zero (all HDBs I looked at had 0 mmap). As such, I suspected it was related to our setup of HDB that consumes mmap, and as the data grows, it will throw that error.

Turns out we are using a segmented database and the segments are listed using a par.txt file. Importantly, the par.txt file must be in a standalone directory, but we put it in the root of the segmented database. Basically, when we loaded the HDB, it loaded the entire segmented database directly.

https://code.kx.com/q4m3/14_Introduction_to_Kdb%2B/#144-segmented-tables

The above should provide a good example of how the par.txt should be in a standalone folder, and how a segmented database should be structured.

[clivelo]

I ran .Q.w[] on the HDB and noticed that the mmap is 98GB. AFAIK, shouldn’t mmap generally be at 0 since .Q.l loads the database in deferred mapping? Wondering if our setup is causing an issue.

I don’t know k code at all, but will try to walk through it.

[clivelo]

Thanks for the reply.

Unless the reload command l . loads the database differently than just restarting the Q instance, I don’t think it would be due to size. The HDB is only using around 350 MB of RAM.

With corrupted file, is there a way to check whether the file is corrupted? When the error pops up, it also associates it with a file in the database that it’s unable to allocate memory to. But if I run a query to view the data in the slice of the database, I can view the data without issues.

[clivelo]

I am running version 3.6.

The error is not always the same file. Strangely if I save more data into the database, the data that it complains about seems to shift back; otherwise with the amount of data unchanged, it complains about the same file.

I have tried overwriting the problem data with nulls of the exact same size and it’s still throwing the same error on the same file.

Is there an alternative command to running l . that can reload indexing the HDB?