KX Community

Find answers, ask questions, and connect with our KX Community around the world.
KX Community Guidelines

Home Forums kdb+ Loading a database from GCS storage

  • Loading a database from GCS storage

    Posted by liordra on November 20, 2023 at 12:00 am

    Hi

    I am trying to load a database using GCS.

    • I have read and followed  this documentation
    • I have created a database (by the example):
    d:2021.09.01+til 20
    {[d;n]sv[`;.Q.par[`:test/db/;d;`trade],`]set .Q.en[`:test/;([]sym:`$'n?.Q.A;time:.z.P+til n;price:n?100f;size:n?50)];}[;10000]each d
    • I have uploaded it to GCS.

    This is how my bucket looks:

    lior_kdb/db/
    • I have set environment variable GCLOUD_PROJECT_ID to the correct project.
    • The contents of par.txt
    $ cat par.txt 
    gs://lior_kdb/db
    • When I try to load this way:
    q)l db

    Thee is no error.

    • When I try to access the data, I get an error
    q)t:get`$":gs://lior_kdb/db/2021.09.01/trade/"
    'gs://lior_kdb/db/2021.09.01/trade/. OS reports: No such file or directory
    [0] t:get`$":gs://lior_kdb/db/2021.09.01/trade/"

    All other attempts to access the database also fail.

    I am using KDB+ 4.0 2023.08.11 installed on Ubuntu 20.04

    Could you please assist,

    Thank you,

    Lior

     

     

     

     

     

     

     

    liordra replied 7 months ago 2 Members · 6 Replies
  • 6 Replies
  • gyorokpeter-kx

    Member
    November 20, 2023 at 12:00 am

    Can you try again after setting this environment variable?

    export KX_TRACE_OBJSTR=1

    Does it print any additional info?

  • liordra

    Member
    November 20, 2023 at 12:00 am

    Hi,

    Thanks for your reply.

    I am getting the same results:

    q)l db
    
    q)t:get`$":gs://lior_kdb/db/2021.09.01/trade/" 'gs://lior_kdb/db/2021.09.01/trade/. OS reports: No such file or directory [0] t:get`$":gs://lior_kdb/db/2021.09.01/trade/"
    q)select from `$":gs://lior_kdb/db/2021.09.01/trade/" 'gs://lior_kdb/db/2021.09.01/trade/. OS reports: No such file or directory [0] select from `$":gs://lior_kdb/db/2021.09.01/trade/"

    Just to confirm that the data is there I double checked

    $gsutil ls gs://lior_kdb/db/2021.09.01/trade/
    gs://lior_kdb/db/2021.09.01/trade/.d
    gs://lior_kdb/db/2021.09.01/trade/price
    gs://lior_kdb/db/2021.09.01/trade/size
    gs://lior_kdb/db/2021.09.01/trade/sym
    gs://lior_kdb/db/2021.09.01/trade/tim
  • gyorokpeter-kx

    Member
    November 20, 2023 at 12:00 am

    Are you using the Insights Core version of kdb+ instead of the vanilla one (which lacks the cloud-related features such as object storage)?

  • liordra

    Member
    November 20, 2023 at 12:00 am

    Hi I am using the version available at GCP

    KDB+ 4.0 Cloud Edition 2021.03.12 Copyright (C) 1993-2021 Kx Systems
    l64/ 8()core 64297MB   kx-insights-1-vm 10.128.0.4 EXPIRE 2024.06.02 654962966702 GCP KXCE #5014615

     

  • gyorokpeter-kx

    Member
    November 21, 2023 at 12:00 am

    Previously you mentioned using KDB+ 4.0 2023.08.11 installed on Ubuntu 20.04. But in your latest reply the release date is 2021.03.12. So you were probably using two different versions for two different tests. Can you try using the latest version of Cloud Edition as it should have some bug fixes related to object storage?

  • liordra

    Member
    November 21, 2023 at 12:00 am

    I am using cloud edition for production and integration.

    I have another installation for development only.

    The results are the same on both installations.

    I will create another cloud instance with the latest edition and report.

    Edit: I can can confirm that the current version available in the market:

    KDB+ 4.0 Cloud Edition 2023.01.20 Copyright (C) 1993-2023 Kx Systems

    solves the issue.

     

    Thanks for your help!

    L.

     

Log in to reply.