[simon_watson_sj]

Thanks – I have actually downloaded the linux and docker version of this but just need to get a quiet afternoon to set it up. (and I do have the additional license). There are a couple of things of interest to me with the ‘insights approach’ that are hazy – in my current setup I use a collection of 3 functions with a load balancer to allow me to throw a query to be processed into elements that can be farmed off and processed by other q processes and I then process the inbound results via .z.po. To set up my load-balancer and q processes I have a bash script that takes flags to determine the precise config. Updating that, I imagine I just swap out each q staring expression for one docker insights startup and ensure they are all mapped to the same file hierarchy?That means if I had say 10 q processes previously, now I have 10 insight docker containers running (not as bad as it sounds – maybe perfect use case for docker in fact). Also, to run a Developer instance, would I just treat the q prompt generated from a docker insights process as a standard q prompt and just use the standard developer expression in as usual? so: alias developer=’source /home/userName/developer/config/config.profile; q /home/userName/developer/launcher.q_ ‘ Finally, although I am drawn to the contained nature of the dockerised solution over the Linux install (I’m on Ubuntu) but my concern is the loss of processing capability. As I understand it the usual ‘docker penalty’ is about 10% of processing speed over a ‘bare metal’ deployment. Once I get that afternoon, I will let you know how I go. Simon

[simon_watson_sj]

Thanks – I have actually downloaded the linux and docker version of this but just need to get a quiet afternoon to set it up. (and I do have the additional license).

There are a couple of things of interest to me with the ‘insights approach’ that are hazy – in my current setup I use a collection of 3 functions with a load balancer to allow me to throw a query to be processed into elements that can be farmed off and processed by other q processes and I then process the inbound results via .z.po. To set up my load-balancer and q processes I have a bash script that takes flags to determine the precise config. Updating that, I imagine I just swap out each q staring expression for one docker insights startup and ensure they are all mapped to the same file hierarchy?That means if I had say 10 q processes previously, now I have 10 insight docker containers running (not as bad as it sounds – maybe perfect use case for docker in fact).

Also, to run a Developer instance, would I just treat the q prompt generated from a docker insights process as a standard q prompt and just use the standard developer expression in as usual?
so:

alias developer='source /home/userName/developer/config/config.profile; q /home/userName/developer/launcher.q_ '

Finally, although I am drawn to the contained nature of the dockerised solution over the Linux install (I’m on Ubuntu) but my concern is the loss of processing capability. As I understand it the usual ‘docker penalty’ is about 10% of processing speed over a ‘bare metal’ deployment.

Once I get that afternoon, I will let you know how I go.

Simon

 

[simon_watson_sj]

Thanks so much for this – sorry for the delay in response. This does look an exciting new development and it’s great for KDB/Q – As well as enabling the integration of a vast library of useful stuff into Q (with, as I understand it some work under the hood to help with speed), let’s hope pyKX becomes a gateway drug to get Python users hooked in Vector programming and KDB/Q! Simon

[simon_watson_sj]

Thanks  – I’ll give this a go. I was saying to somewhere else I think an issue is that whilst I can correctly initialise my nested dictionaries (starting them with (enlist `)!(enlist ::), any data objects which I’ve de-serialised from JSON come as they are. I had discussed the idea of extending the ‘apply’ form to it could have functionality to either

• impose a given initialisation per nesting layer rather than taking it from the data object
• ignore any key errors

I have a feeling that my second idea might be redundant given your suggestion for the use of :: does pretty much the same thing. Thinking about my first idea – I think this is actually about addressing problems from sets of dictionary key with mixed type. I imagine anything that could handle that would end up being horribly slow. Also I’m not sure on the principle of building native functionality which ultimately only addresses poor data structuring in the first place (unless you stuff it in some sort of separate ‘cleanup’ namespace).

Thanks again for getting back to me on this I’ll give it a go and report back on results.

Simon

[simon_watson_sj]

Hi unterrainer_ale – I’m afraid it’s work related and I get enough heat just moving it around internally. I have a todo to get to grips with navigating nested data objects so when I get chance, I’ll write up some cases.

(Hopefully with solutions!)

Thanks for having a look and giving it a go. I’ll investigate ‘cross-section’.

Simon

[simon_watson_sj]

Thanks rocuinneagain – that’s good insight.

Given the above constraint, I have the following issue:

// define a table with the intention of loading a string to a columnn.

// as discussed, it is empty so ‘uptar’ is defined as having type “c”.

test: ([]test: `$(); uptar:"C"$());

// With this done, I can’t see a valid upsert to get a string into the uptar column.

test upsert ([]test: `rta; uptar: enlist "ra"); 
test upsert ([]test: enlist `rta; uptar: "ra");
test upsert ([]test: enlist `rta; uptar:enlist enlist "ra");

// finally, this one projects – so works but still not correctly.

test upsert ([]test: `rta; uptar: "ra");

 

Is it possible to insert a nested list into an empty table or must we use a ‘set’ workaround?

Thanks and regards,

Simon

[simon_watson_sj]

Hey David – updates!

I think the Query Router white paper might have a couple of quirks that are me causing issues. (Although at the moment I feel like the guy in the 2000’s who blames Bill Gates when he stuffs up is VB code)

I think the main issue is this:

returnRes:{[res] uh:first exec uh from queryTable where sq=(res 0); // (res 0) is the sequence number if[not null uh;(neg uh)(res 1)]; // (res 1) is the result queryTable[(res 0);`ret]:.z.p }

In particular:

(neg uh)(res 1)

I think res causes a `Type failure because it needs to be a string. I’ve tried:

(neg uh) .Q.s1 (res 1)

but that doesn’t return anything that shows up anywhere. Interestingly, if I do something like (neg uh)”foo:123″;(neg uh)””;

from the gateway, then I do get a variable called foo with that value on the client.

My next attempt will be to either adjust the query process so it assigns return data to a variable or go through the async paper again and see what the standard method is.

I will let you know how I go.

Simon

[simon_watson_sj]

Hey David – I wanted to follow up on this. I have recently built myself a fancypants new PC. I’m keen to squeeze as much as possible out of it so I went for a bare metal environment rather than in containers. I have just tried loading up the load balancer example using mserve.q and it seems to be working fine.

I’ll update when I try the full query router example and let you know if there are still issues. I would like to get it going in containers – I wonder if the docker approach would mean using one process per container. I will advise as I discover!!!

Cheers,

Simon

[simon_watson_sj]

Thanks Dave,

I still get no luck – I should say that if I open a port to 5003 directly using

`h_gw set hopen 5003;

I can get it to do say “5+6” (returning 11) or “til 5” (returning 1 2 3 4).

As you advised, I restructured the gateway function:

gw:{(neg `h_gw[])(`userQuery;x);`h_gw[]""}

I now get an error back from

gw(`EQUITY_MARKET_RDB;"select from trade where date=max date")

where it doesn’t seem to recognize the table. It does return properly now though. I will add log functions to each component on the gw and see what gets done when. Also I should say, I took all the code directly from the query routing white paper. Perhaps I’m being too flippant with my cut and paste – either way, this process will get the skills built. I will report back with news from the logs.

Simon

[simon_watson_sj]

Hi David,

Thank you for your speedy response! I have tried the load balancer on basic q processes.

I did the following:

copy the following files to QHOME:

• loadBalancer.q
• service.q
• gateway.q

Next, I defined function qmserve in .bashrc  as follows:

qmserve(){ q loadBalancer.q -p 5001 & q service.q -p 5002 & q gateway.q -p 5003 & q -p 5004 & q -p 5005 }

and ran

source .bashrc

from the command line in the home directory to load up the function.

I then executed the function qmserve and noted 5 processes starting.

at q prompt did this:

gw:{h:hopen x;{(neg x)(`userQuery;y);x[]}[h]}[`:localhost:5003]

followed by this:

gw(`EQUITY_MARKET_RDB;"select from trade where date=max date")

The result was a hanging cursor.

Looking in the service.q code, what does LB do when it is used in manageConn?

My thoughts for troubleshooting was that I should load up the processes separately on multiple terminals as using ‘&’ suppresses error returns as I’m sure you will know. An alternative view though is that my final process doesn’t have a ‘&’ so should return errors normally.

Keen for any thoughts you might have even if it’s on the next steps for my troubleshooting. I wondered if it was worth loading up the scripts in a q startup process wrapped in my qmserve function but then start up my client q process directly on the command line. I don’t think the function wrapper in the bash script should change things but it never hurts to try.

Simon

[simon_watson_sj]

Hey Kevin,

Thanks so much for this – exactly what I was looking for. With this capability, GoG is a great way of generating visualisations if a ‘server side’ approach works for you.

Simon

[simon_watson_sj]

Hey Rian/Sam,

I finally got around to investigating this more fully.

The issue I have comes down to the below example of a nested data structure called dsEg here.

dsEg: (`doctype`html)!(enlist “html”;`text`body!(enlist”test”;enlist ([]a: `d`f`g;b: 23 43 777)));

My problem is that I don’t know how to use apply (@;.) to get to the columns on that nested table ([]a: `d`f`g;b: 23 43 777).

I feel like it should be

cols .[dsEg;(`html;`body;0)]

since the table is enlisted which means it’s in a single element list.

However, I can’t get it, or any other approach to work using Apply alone so I can’t use Apply as a method to traverse any general data structure  in cases like the above where the structure descends within a nested table. The best I can do is get to the layer above and apply raze. I know that sounds like a small thing but the problem comes when nesting then continues down into the table – there is no way to use apply with a list of keys to get past that application of raze at the table level.

Building a function which will allow that to work was ultimately the motivation for this whole tangent. I think I now appreciate that Apply is intended to be fully generic so I’m basically reinventing the wheel. However, to help me abandon my method, can you advise how Apply might be used with a list of keys to get to the column names or any other nested elements within that table?

Regards,

Simon

[simon_watson_sj]

Hi All,

I got a response from Conor at KX about this. It’s due to the python library having output in a nonstandard python data type.

To get around this, you convert the output to standard Python data types when in python prior to bringing it over to Q. The key to his solution was the python function he defines to do this where you can see he changes the output to a string python type:

q)p)def func(x):return str(x)

Also note that the [<] symbol in  )qfunc[<] below is just a better way of telling embedPy to interpret the object in the memory space as a Q object rather than a Python one (which would look like ‘foreign’ when represented by Q if you recall from the docs).

His full solution is as follows (based on the example_html doc I defined in my initial post):

q)bs4[`:BeautifulSoup][example_html;”html.parser”]

{[f;x]embedPy[f;x]}[foreign]enlist

q)bs4[`:BeautifulSoup][example_html;”html.parser”]`

foreign

q)bs4[`:BeautifulSoup][example_html;”html.parser”]

{[f;x]embedPy[f;x]}[foreign]enlist

q)bs:bs4[`:BeautifulSoup][example_html;”html.parser”]

q)result:bs[`:find_all][“a”;`href pykw 1b]

q)p)def func(x):return str(x)

q)qfunc:.p.get`func

q)qfunc[<]each result`

“<a href=”http://somegreatsite.com“>Link Name</a>”

“<a href=”mailto:support@yourcompany.com“>support@yourcompany.com</a>”

 

 

[simon_watson_sj]

Actually, subsequent to this, using:

q)p)def func(x):return x.attrs

in place of

q)p)def func(x):return str(x)

will return a python dictionary of attributes. Since this is a native python type, I found that it imports to Q as a dictionary and keeps the nested structure.