ArrowKDB

kdb+

ArrowKDB

Posted by kxnoob on August 8, 2023 at 12:00 am
I have tried installing ArrowKDB from source.

I followed the installation instructions and the instructions to build worked properly.
```
git clone https://github.com/KxSystems/arrowkdb.git cd arrowkdb
mkdir build cd build cmake --build . --config Release
cmake --build . --config Release --target install
```
I had to change somethings in the config and some environment variables to make it works but it builds successfully and it builds successfully.

$ cmake –build . –config Release –target install
[100%] Built target arrowkdb
Install the project…
— Install configuration: “Release”
— Installing: <>/q/l64/arrowkdb.so
— Set runtime path of ” <>/q/l64/arrowkdb.so” to “”
— Up-to-date: <>/q
— Installing:<>/q/arrowkdb.q
However, when I try to load arrowkdb I get the following:

q arrowkdb.q KDB+ 4.0 Cloud Edition 2022.09.30 Copyright (C) 1993-2022 Kx Systems l64/ 32()core 253885MB <> EXPIRE 2024.03.14 <> KXCO KXCE <>

‘<>/q/l64/arrowkdb.so: undefined symbol: _ZNK5arrow9ArraySpan24UnionMayHaveLogicalNullsEv
[2] <>/q/arrowkdb.q:6: dt.boolean:`arrowkdb 2:(`boolean;1);

Can someone point me to what might have gone wrong in the build process? I am building against arrow 9. Using the release version did not work.
kxnoob replied 8 months, 2 weeks ago 2 Members · 4 Replies
4 Replies

nmcdonnell

Member
August 8, 2023 at 12:00 am
Arrowkdb should work with earlier libarrow versions, say 5.0 (or maybe even 3.0) since your are building from source.

What is your distro?

If you’re able to install libarrow.so.800 you should also be able to install libparquet.so.800. It’ll be two separate yum commands similar to https://github.com/KxSystems/arrowkdb#linux (although that’s apt on ubuntu)

See https://arrow.apache.org/install/#c-and-glib-c-packages-for-debian-gnulinux-ubuntu-and-centos:

Something like this from centos 7:
```
sudo yum install -y epel-release || sudo yum install -y https://dl.fedoraproject.org/pub/epel/epel-release-latest-$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1).noarch.rpm sudo yum install -y https://apache.jfrog.io/artifactory/arrow/centos/$(cut -d: -f5 /etc/system-release-cpe | cut -d. -f1)/apache-arrow-release-latest.rpm sudo yum install -y --enablerepo=epel arrow-devel # For C++ sudo yum install -y --enablerepo=epel parquet-devel # For Apache Parquet C++
```
kxnoob

Member
August 8, 2023 at 12:00 am

> I had to change somethings in the config and some environment variables to make it works but it builds successfully

Yeah – I had to change the CMakeLists to point to that libarrow12 (ARROW_LIBRARY and PARQUET_LIBRARY). I tried pointing to a libarrow9 version in a different miniconda environment. I also had to force it to use C++ 17.

Unfortunately, I am not able to install libarrow 9 on my system outside of conda. The most my system is able to yum install is libarrow8. It installs libarrow.so.800 to /usr/lib64 but not a libparquet.so.800.

Do you have any suggestions for any workarounds to this? I’m not an expert on CMake or building C++ packages and whatnot.

nmcdonnell

Member

August 8, 2023 at 12:00 am

Hi,

I suspect you’re missing the libarrow dependencies.
Could run `ldd $QHOME/l64/arrowkdb.so`?
It should return something like:

ldd arrowkdb.so
linux-vdso.so.1 (0x00007ffed21f9000)
libarrow.so.900 => /lib/x86_64-linux-gnu/libarrow.so.900 (0x00007f92d5ded000)
libparquet.so.900 => /lib/x86_64-linux-gnu/libparquet.so.900 (0x00007f92d5ac2000)
libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f92d58e0000)
libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f92d58c5000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f92d56d3000)
libbrotlienc.so.1 => /lib/x86_64-linux-gnu/libbrotlienc.so.1 (0x00007f92d564a000)
libbrotlidec.so.1 => /lib/x86_64-linux-gnu/libbrotlidec.so.1 (0x00007f92d563a000)
libutf8proc.so.2 => /lib/x86_64-linux-gnu/libutf8proc.so.2 (0x00007f92d55ed000)
libre2.so.5 => /lib/x86_64-linux-gnu/libre2.so.5 (0x00007f92d557c000)
libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f92d5576000)
libcrypto.so.1.1 => /lib/x86_64-linux-gnu/libcrypto.so.1.1 (0x00007f92d529f000)
libbz2.so.1.0 => /lib/x86_64-linux-gnu/libbz2.so.1.0 (0x00007f92d528c000)
liblz4.so.1 => /lib/x86_64-linux-gnu/liblz4.so.1 (0x00007f92d5269000)
libsnappy.so.1 => /lib/x86_64-linux-gnu/libsnappy.so.1 (0x00007f92d525e000)
libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f92d5242000)
libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f92d5199000)
libcurl.so.4 => /lib/x86_64-linux-gnu/libcurl.so.4 (0x00007f92d5107000)
libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f92d4fb8000)
libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f92d4f93000)
/lib64/ld-linux-x86-64.so.2 (0x00007f92d7c87000)
libthrift-0.13.0.so => /lib/x86_64-linux-gnu/libthrift-0.13.0.so (0x00007f92d4edc000)
libbrotlicommon.so.1 => /lib/x86_64-linux-gnu/libbrotlicommon.so.1 (0x00007f92d4eb9000)
libnghttp2.so.14 => /lib/x86_64-linux-gnu/libnghttp2.so.14 (0x00007f92d4e90000)
libidn2.so.0 => /lib/x86_64-linux-gnu/libidn2.so.0 (0x00007f92d4e6f000)
librtmp.so.1 => /lib/x86_64-linux-gnu/librtmp.so.1 (0x00007f92d4e4d000)
libssh.so.4 => /lib/x86_64-linux-gnu/libssh.so.4 (0x00007f92d4ddf000)
libpsl.so.5 => /lib/x86_64-linux-gnu/libpsl.so.5 (0x00007f92d4dcc000)
libssl.so.1.1 => /lib/x86_64-linux-gnu/libssl.so.1.1 (0x00007f92d4d39000)
libgssapi_krb5.so.2 => /lib/x86_64-linux-gnu/libgssapi_krb5.so.2 (0x00007f92d4cec000)
libldap_r-2.4.so.2 => /lib/x86_64-linux-gnu/libldap_r-2.4.so.2 (0x00007f92d4c96000)
liblber-2.4.so.2 => /lib/x86_64-linux-gnu/liblber-2.4.so.2 (0x00007f92d4c83000)
libunistring.so.2 => /lib/x86_64-linux-gnu/libunistring.so.2 (0x00007f92d4b01000)
libgnutls.so.30 => /lib/x86_64-linux-gnu/libgnutls.so.30 (0x00007f92d492b000)
libhogweed.so.5 => /lib/x86_64-linux-gnu/libhogweed.so.5 (0x00007f92d48f4000)
libnettle.so.7 => /lib/x86_64-linux-gnu/libnettle.so.7 (0x00007f92d48ba000)
libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007f92d4834000)
libkrb5.so.3 => /lib/x86_64-linux-gnu/libkrb5.so.3 (0x00007f92d4757000)
libk5crypto.so.3 => /lib/x86_64-linux-gnu/libk5crypto.so.3 (0x00007f92d4726000)
libcom_err.so.2 => /lib/x86_64-linux-gnu/libcom_err.so.2 (0x00007f92d471f000)
libkrb5support.so.0 => /lib/x86_64-linux-gnu/libkrb5support.so.0 (0x00007f92d4710000)
libresolv.so.2 => /lib/x86_64-linux-gnu/libresolv.so.2 (0x00007f92d46f2000)
libsasl2.so.2 => /lib/x86_64-linux-gnu/libsasl2.so.2 (0x00007f92d46d5000)
libgssapi.so.3 => /lib/x86_64-linux-gnu/libgssapi.so.3 (0x00007f92d4690000)
libp11-kit.so.0 => /lib/x86_64-linux-gnu/libp11-kit.so.0 (0x00007f92d455a000)
libtasn1.so.6 => /lib/x86_64-linux-gnu/libtasn1.so.6 (0x00007f92d4544000)
libkeyutils.so.1 => /lib/x86_64-linux-gnu/libkeyutils.so.1 (0x00007f92d453b000)
libheimntlm.so.0 => /lib/x86_64-linux-gnu/libheimntlm.so.0 (0x00007f92d452f000)
libkrb5.so.26 => /lib/x86_64-linux-gnu/libkrb5.so.26 (0x00007f92d449c000)
libasn1.so.8 => /lib/x86_64-linux-gnu/libasn1.so.8 (0x00007f92d43f5000)
libhcrypto.so.4 => /lib/x86_64-linux-gnu/libhcrypto.so.4 (0x00007f92d43bd000)
libroken.so.18 => /lib/x86_64-linux-gnu/libroken.so.18 (0x00007f92d43a2000)
libffi.so.7 => /lib/x86_64-linux-gnu/libffi.so.7 (0x00007f92d4396000)
libwind.so.0 => /lib/x86_64-linux-gnu/libwind.so.0 (0x00007f92d436c000)
libheimbase.so.1 => /lib/x86_64-linux-gnu/libheimbase.so.1 (0x00007f92d435a000)
libhx509.so.5 => /lib/x86_64-linux-gnu/libhx509.so.5 (0x00007f92d430c000)
libsqlite3.so.0 => /lib/x86_64-linux-gnu/libsqlite3.so.0 (0x00007f92d41e3000)
libcrypt.so.1 => /lib/x86_64-linux-gnu/libcrypt.so.1 (0x00007f92d41a6000)

kxnoob

Member
August 8, 2023 at 12:00 am

Hello, thank you for your response!

I got the following output.

$ ldd $QHOME/l64/arrowkdb.so linux-vdso.so.1 (0x00007ffd3e3fc000) libarrow.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libarrow.so.1200 (0x00007fbaf4ee9000) libparquet.so.1200 => <>/miniconda3/lib/python3.9/site-packages/pyarrow/libparquet.so.1200 (0x00007fbaf4760000) libstdc++.so.6 => /lib64/libstdc++.so.6 (0x00007fbaf43cb000) libm.so.6 => /lib64/libm.so.6 (0x00007fbaf4049000) libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x00007fbaf3e31000) libc.so.6 => /lib64/libc.so.6 (0x00007fbaf3a6c000) libdl.so.2 => /lib64/libdl.so.2 (0x00007fbaf3868000) librt.so.1 => /lib64/librt.so.1 (0x00007fbaf3660000) libpthread.so.0 => /lib64/libpthread.so.0 (0x00007fbaf3440000) /lib64/ld-linux-x86-64.so.2 (0x00007fbaf7c8a000)

KX Community

nmcdonnell

kxnoob

nmcdonnell

kxnoob