Re: [lmi] Group premium quotes

[Greg Chicares]

On 2015-06-21 20:48, Vadim Zeitlin wrote:
> On Sun, 21 Jun 2015 17:05:51 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> On 2015-06-19 23:30, Greg Chicares wrote:
> GC> > On 2015-06-19 14:20, Vadim Zeitlin wrote:
> GC> >> On Wed, 17 Jun 2015 15:24:52 +0000 Greg Chicares <address@hidden> 
> wrote:
> GC> 
> GC> Should we use the existing class Ledger, with its somewhat "stringly"
> GC> interface, or create something new?
> 
>  I agree that your arguments for reusing the existing "Ledger" class sound
> convincing.
[...]
>  However I still have some questions/uncertainties about using it, please
> see below.

It might help you to think in terms of the (simply flattened) xml output,
even if you actually use the (structured) class Ledger. You can create a
specimen like this:

  ./lmi_wx_shared --ash_nazg --data_path=/opt/lmi/data --pyx=xml
                                                       ^^^^^^^^^
with
  File | New | Illustration
  OK
  File | Print preview

giving an 'unnamed1.xml' like this:

<?xml version="1.0"?>
<illustration>
  <scalar/>              <!-- scalars -->
  <data/>                <!-- vectors -->
  <supplementalreport/>  <!-- ignore this -->
</illustration>

Think first of the xml, then grep to see where the data come from in the
ledger classes. Take "SpecAmt" for example:
  /lmi/src/lmi[0]$grep -w SpecAmt ledger*.?pp |less -S
ledger_invariant.?pp
  datatype: vector<double>
  [also source of data within lmi, which you don't need]
ledger_xml_io.cpp
  [title used in illustrations--but premium quotes have their own]
  [format used in illustrations--here, 123456.43 --> "123,456"]
ledger_text_formats.cpp
  a concrete example you can use:

void PrintRosterTabDelimited
    (Ledger const& ledger_values
...
// 'SpecAmt' is in the "invariant" ledger object:
    LedgerInvariant const& Invar = ledger_values.GetLedgerInvariant();
...
// Here you'll want "0" instead of "d", to get the first element:
        << Invar.value_str("SpecAmt"              ,d) << '\t'

You don't need to understand why 'SpecAmt' is "invariant" (or what it would
mean to be "variant" instead), or why it's an "EndYearVector" (or what
beginning-of-year vectors are, or "other" vectors).

> GC> >>  As for the "two pieces" part, for me this is more of an advantage than
> GC> >> anything else as it would make each of them simpler and, for the PDF 
> one,
> GC> >> easier to test.
> GC> > 
> GC> > Very good point. The easiest way to regression-test the data that a
> GC> > PDF contains is to dump the data as flat text, and test that.
> GC> 
> GC> We already have regression tests for class Ledger.
> 
>  But what about the PDF testing?

We're creating a report with rows consisting of
  name, age, DOB, salary, "SpecAmt", and a few premiums
along with some headers. We'll probably want to dump the data as flat text
just for regression testing (because most of these premiums are used nowhere
else). I suppose we'll judge whether the PDF corresponds to the flat text by
code review and by visual inspection of some samples--there's little chance
any problem will sneak past us, and PDFs are hard to test automatically.

> There doesn't seem to be any way to create
> a (dummy) Ledger for testing purposes, or am I missing some way to do it?

I don't quite understand. I'm not really sure whether class Ledger is
default constructible, but it's easy to construct a concrete instance
with the "File | New | Illustration ..." commands above.

> GC> Members of class Ledger and its HAS-A subordinates aren't quite 
> "stringly".
> GC> However, Ledger::write(xml::element&) creates an inherently "stringly"
> GC> xml object.
> 
>  This is my second question: will the PDF code be supposed to work with XML?
> I'd really like to avoid this if possible and it seems like it ought to be,
> with value_str() method, but you keep mentioning XML in your email which is
> making me nervous, so I'd appreciate if you could please confirm that XML
> names are used for illustrative purposes only and you don't propose to
> create the XML document from the ledger and use it to produce the PDF.

Okay, confirmed. Discussing it in terms of XML makes it easy to understand,
and makes implementation simple but appallingly ugly and inefficient.
Implementing it in terms of the ledger classes is the right way, though
harder to understand at first. XML is just a bridge to understanding, which
we'll discard early.

>  The third potential problem is more nebulous, but it's the one that
> worries me the most: Ledger classes are relatively complex, I've never
> worked with them before and looking at them right now I have to admit that
> I'm a bit lost. I hope I can find my way in them quickly, but there is
> still a risk of misunderstanding something and making some stupid mistake.

I'll be your safari guide, so no hippopotamus will become your sarcophagus.

> These classes also seem to be pretty central to lmi, so modifying them is
> probably out of question and if I run into any problems (e.g. performance,
> although I still think it's unlikely) I wouldn't have much latitude in
> solving them, compared to using an ad hoc intermediate struct discussed
> before which would have been simple and easily modifiable/optimizable if
> needed.

Modifying the ledger classes should be pretty safe because they're so
carefully regression tested. If we need to add data to them, I'll add it.
I don't think we'll need any other change.

Performance really shouldn't matter. A typical "quote" might have a few
dozen lives, and we don't care if that takes one second or one hundredth.

If you like, you can use an ad hoc intermediate struct internal to the
"premium quotes" code. For example: as the Ledger objects drift by
(driven by a chain of events you can trace from CensusView::DoAllCells()
into as much depth as you please), grab the tiny subset you need, and
store it in your local data structure, then use that subset to create
the PDF. That would be more economical of RAM than holding all the
Ledger objects for potentially thousands of rows.

> GC> I'm starting to see this as a generalized "roster" facility. The existing
> GC> group roster is one flavor; premium quotes are another; and the inforce
> GC> bill that I see on the horizon is a third.
> 
>  I have no idea what the inforce bill is,

Nobody knows exactly what an inforce bill will look like, but it will
be much the same as a "premium quote" except with different columns,
headers, and footnotes.

All these things look like your monthly bank statement:
  Bank name and address
  Your name and account number
  for each debit or credit transaction:
    a bunch of rows, e.g. beginning balance, transaction amount, ending balance
  Some legal statement at the bottom

> but even I can see that the
> existing roster.tsv output is indeed pretty similar to the PDF summary we
> want to add.

That's all there is to it.

> GC> - GUI: The existing "Print roster to spreadsheet" command could become
> GC>   "Print roster..." with a submenu, with its toolbar button perhaps
> GC>   becoming a combobox (which I suppose wx can put in a toolbar).
> 
>  (it can put a button with a drop down menu, which is the more standard UI
> for this kind of things)

Good. Then it'll be like this:
  http://virt-tools.org/learning/start-vm-with-virt-manager/force-off.png

> GC> - Calling code: we could either expand enum mcenum_emission (which has
> GC>   thirteen enumerators now, and I guess we can portably have thirty-two
> GC>   total, being log2(1.0+ULONG_MAX), which should suffice for another
> GC>   decade or so; or add an argument:
[...]
> GC> +   ,mcenum_roster_t roster_type
[...]
>  I really don't know if it's more logical to think of the new feature as
> another roster type or as a new emission. For me it seems to rather be the
> latter (i.e. I'd prefer to extend mcenum_emission) but, as I keep repeating
> since the beginning of this discussion, I am absolutely not sure about it.

I'm not sure, either. Let me say this, though: we might want to "emit"
more than one "emission" in a single pass through the code, though I'm
not sure we've ever done that; but we really won't need to emit more
than one "roster type": IOW, the toolbar-button-with-dropdown-menu
has a single-choice menu only. That seems to argue for a separate
roster-type argument.

Let me suggest:

    if(emission & mce_emit_group_roster)
        {
        LMI_ASSERT(!tsv_filepath.empty());
        PrintRosterTabDelimited
            (ledger
            ,   tsv_filepath.string()
            +   ".roster"
            +   configurable_settings::instance().spreadsheet_file_extension()
            );
        if(contains(ledger.GetLedgerInvariant()..value_str("Comments"), 
"TeStInG"))
            WritePremiumQuotePdfFile();
        }

That trick makes it easy to postpone the decision. (Every PDF contains that
WeIrD sTrInG during development only, but that's all right.)

> GC> >>  What do you think?
> GC> 
> GC> Right back atcha!
> 
>  Well, it seems clear that we will be using Ledger for this. I'll have to
> spend some time to understand how to create Ledger with the test data first

We'll use CensusView::DoAllCells() in production. For isolated, lightweight
testing during development, just clone that small function, not worrying
about the effect of user cancellation. The svn repository contains a
'sample.cns' that is probably good enough to start testing.

> and then how to get the data back from it. If you have any words of wisdom
> on either or both of these topics, they would be gratefully received, but
> if not I can at least try to learn this on my own during the next couple of
> days.

I hope the blazes and cairns I've laid above show the trail well enough,
but let me know if you want more detail.