KX Community

Find answers, ask questions, and connect with our KX Community around the world.
KX Community Guidelines

Home Forums kdb+ ArrowKDB

  • ArrowKDB

    Posted by kxnoob on August 8, 2023 at 12:00 am

    I have tried installing ArrowKDB from source.

     

    I followed the installation instructions and the instructions to build worked properly.

     

    git clone https://github.com/KxSystems/arrowkdb.git cd arrowkdb
    mkdir build cd build cmake --build . --config Release
    cmake --build . --config Release --target install

     I had to change somethings in the config and some environment variables to make it works but it builds successfully and it builds successfully.

    $ cmake –build . –config Release –target install
    [100%] Built target arrowkdb
    Install the project…
    — Install configuration: “Release”
    — Installing: <>/q/l64/arrowkdb.so
    — Set runtime path of ” <>/q/l64/arrowkdb.so” to “”
    — Up-to-date: <>/q
    — Installing:<>/q/arrowkdb.q

    However, when I try to load arrowkdb I get the following:

    q arrowkdb.q
    KDB+ 4.0 Cloud Edition 2022.09.30 Copyright (C) 1993-2022 Kx Systems
    l64/ 32()core 253885MB <> EXPIRE 2024.03.14 <> KXCO KXCE <>

    ‘<>/q/l64/arrowkdb.so: undefined symbol: _ZNK5arrow9ArraySpan24UnionMayHaveLogicalNullsEv
    [2] <>/q/arrowkdb.q:6: dt.boolean:`arrowkdb 2:(`boolean;1);

    Can someone point me to what might have gone wrong in the build process? I am building against arrow 9. Using the release version did not work.

    kxnoob replied 9 months ago 2 Members · 4 Replies
  • 4 Replies
  • nmcdonnell

    Member
    August 8, 2023 at 12:00 am

    Arrowkdb should work with earlier libarrow versions, say 5.0 (or maybe even 3.0) since your are building from source.

    What is your distro?

    If you’re able to install libarrow.so.800 you should also be able to install libparquet.so.800.  It’ll be two separate yum commands similar to https://github.com/KxSystems/arrowkdb#linux (although that’s apt on ubuntu)

    See https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos:

    Something like this from centos 7:

    sudo yum install -y epel-release || sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1).noarch.rpm sudo yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm sudo yum install -y --enablerepo=epel arrow-devel # For C++ sudo yum install -y --enablerepo=epel parquet-devel # For Apache Parquet C++

     

  • kxnoob

    Member
    August 8, 2023 at 12:00 am

    > I had to change somethings in the config and some environment variables to make it works but it builds successfully

    Yeah – I had to change the CMakeLists to point to that libarrow12 (ARROW_LIBRARY and PARQUET_LIBRARY). I tried pointing to a libarrow9 version in a different miniconda environment. I also had to force it to use C++ 17.

     

    Unfortunately, I am not able to install libarrow 9 on my system outside of conda. The most my system is able to yum install is libarrow8. It installs  libarrow.so.800 to /usr/lib64 but not a libparquet.so.800.

    Do you have any suggestions for any workarounds to this? I’m not an expert on CMake or building C++ packages and whatnot.

  • nmcdonnell

    Member
    August 8, 2023 at 12:00 am

    Hi,

    I suspect you’re missing the libarrow dependencies.
    Could run `ldd $QHOME/l64/arrowkdb.so`?
    It should return something like:

    ldd arrowkdb.so
    linux-vdso.so.1 (0x00007ffed21f9000)
    libarrow.so.900 => /lib/x86_64-linux-gnu/libarrow.so.900 (0x00007f92d5ded000)
    libparquet.so.900 => /lib/x86_64-linux-gnu/libparquet.so.900 (0x00007f92d5ac2000)
    libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f92d58e0000)
    libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f92d58c5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d56d3000)
    libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1 (0x00007f92d564a000)
    libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f92d563a000)
    libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2 (0x00007f92d55ed000)
    libre2.so.5 => /lib/x86_64-linux-gnu/libre2.so.5 (0x00007f92d557c000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d5576000)
    libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f92d529f000)
    libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f92d528c000)
    liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f92d5269000)
    libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f92d525e000)
    libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f92d5242000)
    libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f92d5199000)
    libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f92d5107000)
    libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d4fb8000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d4f93000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f92d7c87000)
    libthrift-0.13.0.so => /lib/x86_64-linux-gnu/libthrift-0.13.0.so (0x00007f92d4edc000)
    libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f92d4eb9000)
    libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f92d4e90000)
    libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f92d4e6f000)
    librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f92d4e4d000)
    libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f92d4ddf000)
    libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f92d4dcc000)
    libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f92d4d39000)
    libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f92d4cec000)
    libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f92d4c96000)
    liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f92d4c83000)
    libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f92d4b01000)
    libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f92d492b000)
    libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f92d48f4000)
    libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f92d48ba000)
    libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f92d4834000)
    libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f92d4757000)
    libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f92d4726000)
    libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f92d471f000)
    libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f92d4710000)
    libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f92d46f2000)
    libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f92d46d5000)
    libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f92d4690000)
    libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f92d455a000)
    libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f92d4544000)
    libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f92d453b000)
    libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f92d452f000)
    libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f92d449c000)
    libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f92d43f5000)
    libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f92d43bd000)
    libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f92d43a2000)
    libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f92d4396000)
    libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f92d436c000)
    libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f92d435a000)
    libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f92d430c000)
    libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f92d41e3000)
    libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f92d41a6000)

     

  • kxnoob

    Member
    August 8, 2023 at 12:00 am

    Hello, thank you for your response!

    I got the following output.

     

    $ ldd $QHOME/l64/arrowkdb.so linux-vdso.so.1 (0x00007ffd3e3fc000) libarrow.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.1200 (0x00007fbaf4ee9000) libparquet.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libparquet.so.1200 (0x00007fbaf4760000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbaf43cb000) libm.so.6 => /lib64/libm.so.6 (0x00007fbaf4049000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbaf3e31000) libc.so.6 => /lib64/libc.so.6 (0x00007fbaf3a6c000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fbaf3868000) librt.so.1 => /lib64/librt.so.1 (0x00007fbaf3660000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbaf3440000) /lib64/ld-linux-x86-64.so.2 (0x00007fbaf7c8a000)

Log in to reply.