Re: igc: Some benchmarks

[Eli Zaretskii]

> Date: Mon, 10 Feb 2025 14:56:14 +0000
> From: Pip Cet <pipcet@protonmail.com>
> Cc: oscarfv@eclipso.eu, emacs-devel@gnu.org
> 
> "Eli Zaretskii" <eliz@gnu.org> writes:
> 
> > The SIGIO handler records the time when the signal was delivered to
> > Emacs, so it should be possible to measure the response time including
> 
> But it omits whatever time the signal was blocked for before it was
> delivered.

Why would SIGIO be blocked for any significant time in the scenario of
this benchmark?

> > In this case, average is not a good statistic (because there are
> > outliers); median is better.  The median values are 38.7 msec for
> > master and 24.4 msec for igc, a 37% speedup, which IMO is impressive
> > for such a simple operation.
> 
> While it's hard to do worse than the mean/average, the median isn't a
> much better choice: We want to know which percentage of key presses
> resulted in unacceptably long delays and weight those appropriately by
> how long the delay was.

That's not what a median is about: it's an estimator of the population
average, that's all.  I suggest using it instead of the mean in every
case where the distribution is unknown, especially if there's a
significant fraction of very low and very high samples.

If we want to compute higher moments of the distribution, or specific
percentiles, there are good robust estimators for that as well, but
they are not the median.

Btw, the main problem I see with the samples we saw until now is that
they are very small.  If we want representative figures, let alone
high percentiles, we need much larger samples.