Re: performance bug of `wc -m`

[Pádraig Brady]

On 16/05/18 17:31, Eric Fischer wrote:
> I should also add that the core reason that wc is slow and Python is fast
> is not that UTF-8 decoding in wc is slow, it is that the Python code is
> just counting characters, while wc is also maintaining a line width
> for --max-line-length. It doesn't really need to do this, and probably
> shouldn't do this, unless --max-line-length is specified.

I had a look at adding some short-circuiting like you suggested.
The attached gives these results on Fedora 25

$ yes áááááááááááááááááááá | head -n100000 > mbc.txt
$ yes 12345678901234567890 | head -n100000 > num.txt

===== Before ====

$ time src/wc -Lm < mbc.txt
2100000      20
real    0m0.186s

$ time src/wc -m < mbc.txt
2100000
real    0m0.186s

$ time src/wc -Lm < num.txt
2100000      20
real    0m0.055s

$ time src/wc -m < num.txt
2100000
real    0m0.056s

==== After ====

$ time src/wc -Lm < mbc.txt
2100000      20
real    0m0.196s

$ time src/wc -m < mbc.txt
2100000
real    0m0.173s

$ time src/wc -Lm < num.txt
2100000      20
real    0m0.031s

$ time src/wc -m < num.txt
2100000
real    0m0.028s

================

>From this we can deduce that short-circuiting iswprint()
is a small win, but avoiding wcwidth() can be a small loss
on Linux due to the low overhead which is negated by
the additional conditional.

Note this also shows that I'm not reproducing the very slow wc
operation that Peng Yu noticed (the python test program operates
at much the same speed, but wc is _much_ quicker here).
I suspect that this is due to iswprint()/wcwidth() being mega slow on OSX?
If that's the case then the wcwidth() avoidance is worth it
even though it slows us down a little in one case on Linux.

Now I see we may be replacing wcwidth() on OSX as there are issues
with OSX handling of combining characters in UTF-8.
So maybe the slow down is with the gnulib wcwidth!?
To test that I did:

  $ gl_cv_func_wcwidth_works=no ./configure --quiet
  $ time src/wc -Lm < mbc.txt
  2100000      20
  real  0m0.225s

So only slightly slower.  That suggests we're not replacing
wcwidth on this OSX system, and that the system implementation
is just very slow, which the attached patch should avoid if possible.

cheers,
Pádraig

p.s. I'm slightly worried that the _existing_ fast path processing
for the is_basic() set, may be too big a set for mbrtowc() avoidance
for GB18030 for example?

wc-osx-speedup.patch
Description: Text Data