publish data to kdb uisng qpython

[darrenwsun]

1. Doing this via qpython is no different from those via other interfaces (including q itself): you open a connection (e.g. to tickerplant, assuming a tp+rdb+hdb setup) and call a function that will get your data persisted (e.g. .u.upd[`trade; data]).

 

2. Generally speaking, backpopulating historical data in a right way isn’t straightforward: it could be done as simple as splaying data using set, one for each partition (date), if you have complete data to write; if you’ve got partial data to write (e.g. trades for a new instrument type), you may have to re-sort the combined data and re-apply attributes wherever needed. The q language itself is short of higher-level APIs that handles the nuances, but I find TorQ has a good collections of those, which makes the task easier.

[rocuinneagain]

1.

An example data publisher is included in documentation https://qpython.readthedocs.io/en/latest/usage-examples.html#data-publisher

 

2.

For going straight to HDB you can write a custom .u.upd yourself to write direct

• https://code.kx.com/q/ref/upsert/#splayed-table
• https://code.kx.com/q/ref/asc/#sorting-data-on-disk
• https://code.kx.com/q/ref/set-attribute/#grouped-and-parted

Or use a file based batch load approach to backfill.

Articles:

• https://code.kx.com/q/wp/data-loaders/
• https://kx.com/blog/kx-product-insights-parsing-data-in-kdb/
• https://kx.com/blog/kdb-q-insights-parsing-json-files/

 

Documentation:

• https://code.kx.com/q/ref/dotj/
• https://code.kx.com/q/ref/file-text/#load-csv
• https://code.kx.com/q/interfaces/arrow/
• https://code.kx.com/q/interfaces/hdf5/