Re: [lmi] Limits on pastable census size

[Greg Chicares]

On 2018-02-09 15:19, Vadim Zeitlin wrote:
> On Fri, 9 Feb 2018 14:06:14 +0000 Greg Chicares <address@hidden> wrote:
> 
> GC> Yesterday we had to work with a census with over ten thousand cells.
> GC> (We've seen one that large only once before.) It seems to work now,
> GC> but it was difficult to create with "Census | Paste census", which
> GC> ran out of memory (it threw bad_alloc).
> 
>  This is surprising. In the current 32 bit build, sizeof(Input) is "just"
> 8920 (BTW, with MSVC it's 8248, so it looks like some space could be saved
> just by using tighter packing), so allocating enough space for 10000
> objects of this type requires about 85MiB of RAM, which shouldn't create
> any trouble for a system capable of running any contemporary software
> (looking at Firefox and 2.5GiB in its working set column on my system right
> now...). So either the machine this was tested on was really underpowered
> or its RAM was heavily fragmented for some reason. BTW, what version of MSW
> and of which bitness did this happen under?

It failed on these three machines:

Kim's machine:  64-bit msw-seven, eight GB RAM (7.88 GB "usable")
user's machine: 64-bit msw-ten (that's all we know)
my machine:     64-bit debian chroot, 64GB RAM, 64- and 32-bit wine:
  [`ls -di` indicates running in chroot, but you probably knew that]

/tmp[0]$ls -di /
19005443 /
/tmp[0]$wine --version
wine-1.8.7 (Debian 1.8.7-2)
/tmp[0]$cat /etc/debian_version 
9.3

>  If you'd like me to debug this further, please send me the data you
> attempted to paste so that I could at least see if I can reproduce this
> here.

I can't share that exact file because it contains personal client data,
but it's easy to fabricate an equivalent testcase. First of all, does
the example here:
  http://www.nongnu.org/lmi/pasting_to_a_census.html
work for you? (It is my impression that the blank line below the headers
must be removed.) Using your tool of choice, copy the last line of that
until you have ten thousand nonblank lines, then copy them all to the
clipboard and paste into lmi, thus:
  File | New | Census
  Census | Paste
I see:
  [statusbar] Added cell number 9999.
  [messagebox] Error   std::bad_alloc