-
Timing/Memory surprise on 3 versions of <Conditional function applied to lists>
Hi,
A bit puzzled here… Say I have a table comprised of 2 columns, with the first one made of variable length lists, and the second showing a binary:
t:([] a:{x?50} each 2+100000?100; b:100000?0b)I want to apply a specific function to each list found in t`a, depending on the binary t`b. In this example, let’s simply use <first> and <last> as dependant functions.
Below are 3 possible syntax versions and their time/memory consumption. I would have expected the <scan> version to beat the 2 others by a substantial factor, at least on timing (with +/- impact on memory). The reality seems to be the opposite:
Cond$ with Each
q)ts {$[x`b; last x`a; first x`a]} each t 40 3697968
Cond? with Each within function
q)ts {?[x`b; last each x`a; first each x`a]}t 16 4746672
Scan
q)ts ({?[y`b; last; first] y`a})[::;t] 62 12746416
Is there an issue with my wording of the scan version (4x slower, 2.5x memory hungry), or is the second one the actual optimal solution? As usual, your insight is highly appreciated. Thx.
-
1 Reply
-
This is not really a suitable application of scan. Scan is an accumulator where it is useful when the calculation of a subsequent calculation depends of the result of the previous calculation. This has an overhead as the calculation is computed item by item and each result must be passed back in to the next calculation. As you never use the variable ‘x’ inside the scan it is an indication it is not the best use-case. This blog has some visualisations which aim to show how scan functions internally.
Your second version is fastest as it operates on ‘x`b’ as a vector rather than inside ‘each’.
One other possible variation is shown below:
q)\ts {?[x`b; last each x`a; first each x`a]}t 16 4746672 q)\ts {((first;last) x`b)@'x`a}t 7 4746640It’s goal is to avoid calculating ‘last each’ and ‘first each’ for every row.
Instead it uses each both (‘) to apply first or last after it is known which function is needed.
Log in to reply.