• kdb+

  Power BI Scalable Solution

  Posted by dunny on January 3, 2024 at 12:00 am

  Hi All,

  Happy New Year. Im just reaching out today to see if anyone has used Power BI as a front end for kdb+ and managed to return results in a scalable manner.
  We have managed to get Power BI to run live queries to kdb+ and get results back in real time using M (The Power BI Formula Language) and passing in parameters from the front end. We used Direct Query (with a SQL Server as a jump box) and ODBC connectors from Power BI as two potential options.
  Dynamic M query parameters in Power BI Desktop – Power BI | Microsoft Learn

  However this is all done synchronously and it doesn’t scale well. I.e. if we want to have multiple users run queries at the same time.
  We were hoping to point Power BI to our kdb async gateway to service front end users but outside of building a custom connector for Power BI, there doesn’t seem to be many other options to achieve this goal.
  I wanted to reach out to see if anyone had managed to do something similar? Or how have people solved having similar problems?

   

  Note,
  Exporting the data to a more Power BI Friendly technology was considered but with 1000s of possible different user inputs that could be used to calculate the data needed we are hoping to avoid that
  Thanks!

  dunny replied 8 months, 2 weeks ago 2 Members · 2 Replies
• 2 Replies

• 

  dunny

  Member

  January 5, 2024 at 12:00 am

  Hi All,
  Thanks Laura
  Just a note for anyone else who might be following. We are going to go down the route of building a new gateway system that accepts sync messages from the client and using Deferred sync messages to handle the messages on the gateway to send to load balanced nodes

• 

  Laura

  Administrator

  January 5, 2024 at 12:00 am

  Hi  ,

  I’ve been following the cross-post of this topic on our K4 Listbox, so I’m going to paste some of the suggested solutions below just for future reference!

  1. How about pointing at a load balanced set of q replicas running the same code?
  2. You can define my end points, neg ports (Multithreaded input queue mode). Depending upon your user queries, replication is the next option.
  3. You can also use deferred response to achieve load balancing without clients being aware of any implementation changes.

  I understand from the thread that you were successful using deferred response?

  Feel free to share more on this channel about your solution

  Thanks,

  Laura

Log In to Reply