Re: [lmi] actuarial tables format (was Re: Terse list of valuable projects)

[Václav Slavík]

Hi,

On 21 Mar 2012, at 21:57, Greg Chicares wrote:
>> <table>
>>    <select period="3">
>>        <age min="31" max="33">
>>            <value>0.198</value>
>>            <value>0.194</value>
>>            <value>0.190</value>
>>        </age>
>>        <age min="31" max="33">
>>            <value>0.198</value>
>>            <value>0.194</value>
>>            <value>0.190</value>
>>        </age>
>>        <age min="31" max="33">
>>            <value>0.198</value>
>>            <value>0.194</value>
>>            <value>0.190</value>
>>        </age>
>>    </select>
>> </table>
>> 
>> Notice that I make the age/duration the inner-most axis, I think that's
>> consistent with sample.database and is generally the most reasonable;
>> please let me know if I got this wrong.
> 
> I'm not quite sure, because the <value> triplets are all the same.
> Let me answer in terms of table_256() in 'actuarial_table_test.cpp':
> 
>        //     1        2        3
>        {0.00106 ,0.00140 ,0.00165 // 10
>        ,0.00113 ,0.00148 ,0.00175 // 11
>        ,0.00120 ,0.00157 ,0.00184 // 12
> ...
>        ,0.06520 ,0.10486 ,0.13557 // 80

I'm sorry, my laziness got the better of me. I assumed it would be clear the 
numbers are just placeholders, but it did hurt understandability. Here's how I 
would represent the above table:

<table>
   <select period="3">
       <age min="10" max="80">
           <value>0.00106</value>
           <value>0.00113</value>
           <value>0.00120</value>
           ...
           <value>0.06520</value>
       </age>
       <age min="10" max="80">
           <value>0.00140</value>
           <value>0.00148</value>
           <value>0.00157</value>
           ...
           <value>0.10486</value>
       </age>
       <age min="10" max="80">
           <value>0.00165</value>
           <value>0.00175</value>
           <value>0.00184</value>
           ...
           <value>0.13557</value>
       </age>
   </select>
</table>

> There, the rows are primary, and columns are secondary: that is,
> an actuary who looks up values in this table would select a single
> row (ignoring all other rows) and read values across the columns.
> If a 12-year-old person is to be insured, we want this row vector:
>        ,0.00120 ,0.00157 ,0.00184 // 12
> OTOH, a column vector such as this:
>              2
>        0.00140
>        0.00148
>        0.00157
> is never interesting. So I suppose this example might be written thus
> (with some xml comments referring to class actuarial_table):

This is a fatal problem with my proposal, of course. For the record, my I 
picked this organization, because it has some nice properties:

(1) A 1D table representation can be used as component of a 2D table: it 
corresponds to a column of a 2D table, so 2D table is represented by some 
markup to specify the columns and then the columns are simply included as 1D 
tables as child elements.

(2) The axis with the (typically) largest domain is inner-most. There are tens 
of rows in a typical table and only a few columns. Putting as many values as 
possible together, and representing them with concise markup, is more readable 
than having the flow of the document broken into a large number of small 
chunks. Compare this (the numbers are nonsense, I repeat the same 4 rows three 
times to accentuate the effect):

   <select period="3">
       <age min="10" max="21">
           <value>0.00106</value>
           <value>0.00113</value>
           <value>0.00120</value>
           <value>0.06520</value>
           <value>0.00106</value>
           <value>0.00113</value>
           <value>0.00120</value>
           <value>0.06520</value>
           <value>0.00106</value>
           <value>0.00113</value>
           <value>0.00120</value>
           <value>0.06520</value>
       </age>
       <age min="10" max="21">
           <value>0.00140</value>
           <value>0.00148</value>
           <value>0.00157</value>
           <value>0.10486</value>
           <value>0.00140</value>
           <value>0.00148</value>
           <value>0.00157</value>
           <value>0.10486</value>
           <value>0.00140</value>
           <value>0.00148</value>
           <value>0.00157</value>
           <value>0.10486</value>
       </age>
       <age min="10" max="21">
           <value>0.00165</value>
           <value>0.00175</value>
           <value>0.00184</value>
           <value>0.13557</value>
           <value>0.00165</value>
           <value>0.00175</value>
           <value>0.00184</value>
           <value>0.13557</value>
           <value>0.00165</value>
           <value>0.00175</value>
           <value>0.00184</value>
           <value>0.13557</value>
       </age>
   </select>

to this:

   <row age="10">
       <value>0.00106</value>
       <value>0.00140</value>
       <value>0.00165</value>
   </row>
   <row age="11">
       <value>0.00113</value>
       <value>0.00148</value>
       <value>0.00175</value>
   </row>
   <row age="12">
       <value>0.00120</value>
       <value>0.00157</value>
       <value>0.00184</value>
   </row>
   <row age="13">
       <value>0.06520</value>
       <value>0.10486</value>
       <value>0.13557</value>
   </row>
   <row age="14">
       <value>0.00106</value>
       <value>0.00140</value>
       <value>0.00165</value>
   </row>
   <row age="15">
       <value>0.00113</value>
       <value>0.00148</value>
       <value>0.00175</value>
   </row>
   <row age="16">
       <value>0.00120</value>
       <value>0.00157</value>
       <value>0.00184</value>
   </row>
   <row age="17">
       <value>0.06520</value>
       <value>0.10486</value>
       <value>0.13557</value>
   </row>
   <row age="18">
       <value>0.00106</value>
       <value>0.00140</value>
       <value>0.00165</value>
   </row>
   <row age="19">
       <value>0.00113</value>
       <value>0.00148</value>
       <value>0.00175</value>
   </row>
   <row age="20">
       <value>0.00120</value>
       <value>0.00157</value>
       <value>0.00184</value>
   </row>
   <row age="21">
       <value>0.06520</value>
       <value>0.10486</value>
       <value>0.13557</value>
   </row>


But you pointed out that its another property is that

(3) It's hard to read by humans.

And that beats (1)+(2), of course — what would be the point of using XML if it 
wouldn't be readable and editable by humans easily enough? 

> <table>
>    <select period="3"/>           <!-- NOTE 1 -->
>    <age min="10" max="80">        <!-- min_age(), max_select_age() -->
>        <age_index="10">           <!-- NOTE 2 -->
>            <value>0.00106</value>
>            <value>0.00140</value>
>            <value>0.00165</value>
>        </age_index>
>        <age_index="11">
>            <value>0.00113</value>
>            <value>0.00148</value>
>            <value>0.00175</value>
>        </age_index>
>    ...
> 
> NOTE 1: instead of
>  <table>
>      <select period="3"/>
> would
>  <table type="select" select_period="3">
> be preferable? In the public Society of Actuaries interface, it's this:
>  ///   3    [unsigned] char: Table type: {A, D, S} --> {age, duration, select}
> which is a property of the whole table.

I don't want to put things that affect the validation of child elements into 
attributes, for compatibility with XML Schema and for general feeling of it not 
"being right". I also want to avoid restricting whatever a "table" is to 1D or 
2D ones — you said you perceived that as a flaw in SOA's designs.

> Perhaps a textbook reference will be clearer, such as page 24 here:
>  http://www.math.purdue.edu/~rcp/STAT472/Notes/select.pdf

I did some reading of my own in the past week and understand this particular 
point, but thanks for the reference, this did come handy.

> The word "select" maps to multiple concepts in the solution domain.
> A table can be "select" in the sense mentioned above:
>  ///   3    [unsigned] char: Table type: {A, D, S} --> {age, duration, select}
> and a row of that table such as
>        //     1        2        3
>        ,0.00120 ,0.00157 ,0.00184 // 12  <-- THIS ROW
> gives rates for "select" age twelve (I called it "age_index" above,
> but "select_age" would have been more descriptive; and now I guess
> your <select> meant exactly that).

Yes.

> I think your answer
> to that question is that it's better to distinguish the names;
> otherwise, we have nothing more than Axis1, Axis2, ... and
> xml attributes to say what the Axes mean, and that's weak xml.
> I certainly agree (if I've understood your answer correctly).

Exactly. 

>> This has the side-effect of making it possible to omit some values
>> (e.g. have a table for only male & female data). Is that desirable?
> 
> Yes, that would often be useful.
> 
>> What do you think? Did I miss something important?
> 
> Yes: select-and-ultimate tables. Maybe I should ask you to look at
> the textbook reference above first;

Thanks, that helped. I noticed before, in some of the published SOA tables that 
I randomly chose to look at, that the ultimate column may have different min & 
max age boundaries than the select columns. So I think the best markup 
representation is as a composite of two child elements: one for ordinary 2D 
select table and one for a separate 1D ultimate table. (BTW, that's something 
that the in-memory representation would need to use too, in order to return 
[x]+n values for any sane n > select period transparently.)

I'll write updated format proposal in another mail later.

Thanks for the feedback!
Vaclav