Re: [lmi] [lmi-commits] odd/test_all_products 53f77a1 2/2: Test every product in every jurisdiction

[Greg Chicares]

On 11/16/20 12:55 AM, Greg Chicares wrote:
> On 11/15/20 11:48 PM, Vadim Zeitlin wrote:
>> On Sun, 15 Nov 2020 16:39:13 -0500 (EST) Greg Chicares 
>> <gchicares@sbcglobal.net> wrote:
>> 
>> GC> branch: odd/test_all_products
> [...]
>> GC>     This implementation is casual, but it's needed now for acceptance
>> GC>     testing of changes in proprietary product files. A careful
>> GC>     reimplementation may be added to 'master' later. Beware: this takes
>> GC>     about ten minutes with about 130 products, on an E5-2630 v3 machine.
>> GC>     Threading would be most beneficial.
>> 
>>  Is this last sentence a call to action or just a general observation?
> 
> At the time I wrote it, I hadn't decided. But now I've looked into it a
> little, and concluded that in the next couple of months I can't spend
> the time it would take to figure it out myself. So, if it's pretty easy
> for you, then please show me the way
Although on that branch it's an appendage to the CLI self-test, a
refined version could be either a separate function, triggered by
a different command-line argument--or even a standalone program
(which might be preferable if it takes a while to get it right,
or if exotic compiler and linker options are needed). But this
can wait, because I've found a way to double or triple the speed:
  LMI_COMPILER=gcc ; LMI_TRIPLET=x86_64-pc-linux-gnu ; . 
/opt/lmi/src/lmi/set_toolchain.sh
IOW, I was running i686-w64-mingw32 yesterday, and the clock time
reported by 'time' was 10:24.26; but for x86_64-pc-linux-gnu today,
it's only 3:45.24 .

Here are copy-pastable instructions (if you're using a proprietary
product repository, make sure beforehand that it's up to date; and
of course it wouldn't make sense to build for pc-linux-gnu on cygwin):

LMI_COMPILER=gcc ; LMI_TRIPLET=x86_64-pc-linux-gnu ; . 
/opt/lmi/src/lmi/set_toolchain.sh
cd /opt/lmi/src/lmi
git pull
git switch odd/test_all_products
make $coefficiency install check_physical_closure 2>&1 | less -S
time make $coefficiency cli_tests >product_errors 2>&1

Along with the normal self-test results, the output should contain
the name of each product tested, on a line by itself. For each error
detected, there'll be an extra line with the name of the product and
the state where it fails, along with the lmi error message. For
example, here's a mistake I made yesterday (proprietary names
replaced with precious metals):

PreciousMetals-2009-tier-3
PreciousMetals-2012-tier-1
PreciousMetals-2012-tier-2
PreciousMetals-2012-tier-3
PreciousMetals-2012-tier-4
PreciousMetals-2013-tier-1
PreciousMetals-2013-tier-1, LA:
Platinum-2009 b->lingo_->lookup(policy_form)
Osmium-2013 PolicyForm

[ledger_invariant_init.cpp : 336]

PreciousMetals-2013-tier-1, MO:
Platinum-2009 b->lingo_->lookup(policy_form)
Osmium-2013 PolicyForm

[ledger_invariant_init.cpp : 336]

PreciousMetals-2013-tier-1, MT:
...

because certain states required Os instead of Pt.

I've built that for three architectures, all using gcc-10 with the
same source code in the same chroot. For this task, pc-linux-gnu
performs far better than the others:

i686-w64-mingw32    758.00s user 59.02s system 90% cpu 15:00.19 total
x86_64-w64-mingw32  556.97s user 59.08s system 88% cpu 11:39.00 total
x86_64-pc-linux-gnu 218.89s user 6.41s system 100% cpu  3:45.24 total

The 10:24.26 clock time reported for yesterday's runs was measured
in a different chroot, with gcc-8, i686 mingw-w64, and a different
version of 'wine'; recent experience suggests that the difference
(10:24.26 vs. 15:00.19) is due primarily to 'wine' versions.

Anyway, lmi today has only three other sets of drop-in text variants
that depend on state, and spending four minutes to test each one
exhaustively isn't a very great cost.