KX Community

Find answers, ask questions, and connect with our KX Community around the world.
KX Community Guidelines

Home Forums kdb+ How to notify RDB that there’s a new table in the tp that it can subscribe to?

  • How to notify RDB that there’s a new table in the tp that it can subscribe to?

    Posted by user931206 on February 1, 2024 at 12:00 am

    One of our data plants it’s a self-updating data plant that handles column additions/deletions (type changes still manual)

    The problem I am facing now is that .u.sub subscribes the first time the rdb connects to the tp and then it doesn’t check if new tables end up in the tp so if we are being sent a new table, the schema is set in the tp, but the rdb doesn’t know about it.

    How can I notify the rdb that it needs to subscribe to this new table ?

    Should I just send an async call with the name of the table and make the rdb subscribe to it?

    Could that lead to loss of data?

    user931206 replied 3 months, 1 week ago 3 Members · 2 Replies
  • 2 Replies
  • megan_mcp

    Administrator
    February 6, 2024 at 12:00 am

    Hi ,

    Assuming vanilla kdb-tick, without any modifications to pub/sub logic (and this probably advised against), the TP would have to send some sort of signal to the RDB. The RDB would also have to be configured with some logic/callback to then subscribe to new table and instantiate it’s schema. As for data loss, the RDB may also need a customized function to read from the tick log. However, it doesn’t want to replay every entry in the tick log, jut the ones pertaining to the new table (this can be quite expensive depending on size of tick log and can potentially block RDB for significant period of time)There are possible structures that are more advantageous – like using a segmented tick plant where each table has it’s own tick log. In any event, while not supported by vanilla tick setup, it is possible with some custom configuration.

    Hope this helps,

    Megan

  • sujoy

    Member
    February 8, 2024 at 12:00 am

    Based on your problem statement,

    i would first ensure tp is sending the data as a table to subscriber’s and same is stored in tplog.

    Next whenever TP has a new Table, tp should append the same in .u.w for each (or required) subscriber (rdb)

    rdb on upd will simply insert. So a new table should appear without any data loss.

    Thanks

    Sujoy

Log in to reply.