-
publish data to kdb uisng qpython
1. how to push data to KDB using qpython?
2. is there any script to push data to HDB? assume I have lots of dated trade data, how can I push data to kdb by setting trade date as the data of HDB.
thanks
-
2 Replies
-
1. Doing this via qpython is no different from those via other interfaces (including q itself): you open a connection (e.g. to tickerplant, assuming a tp+rdb+hdb setup) and call a function that will get your data persisted (e.g. .u.upd[`trade; data]).
2. Generally speaking, backpopulating historical data in a right way isn’t straightforward: it could be done as simple as splaying data using set, one for each partition (date), if you have complete data to write; if you’ve got partial data to write (e.g. trades for a new instrument type), you may have to re-sort the combined data and re-apply attributes wherever needed. The q language itself is short of higher-level APIs that handles the nuances, but I find TorQ has a good collections of those, which makes the task easier.
-
1.
An example data publisher is included in documentation https://qpython.readthedocs.io/en/latest/usage-examples.html#data-publisher
2.
For going straight to HDB you can write a custom
.u.updyourself to write direct- https://code.kx.com/q/ref/upsert/#splayed-table
- https://code.kx.com/q/ref/asc/#sorting-data-on-disk
- https://code.kx.com/q/ref/set-attribute/#grouped-and-parted
Or use a file based batch load approach to backfill.
Articles:
- https://code.kx.com/q/wp/data-loaders/
- https://kx.com/blog/kx-product-insights-parsing-data-in-kdb/
- https://kx.com/blog/kdb-q-insights-parsing-json-files/
Documentation:
- https://code.kx.com/q/ref/dotj/
- https://code.kx.com/q/ref/file-text/#load-csv
- https://code.kx.com/q/interfaces/arrow/
- https://code.kx.com/q/interfaces/hdf5/
Log in to reply.