Re: Any idea about what makes Emacs slow reading on pipes?

[Stefan Monnier]

> Actually, this isn't it.  Pipes behave just the same, and I now know
> why.  You won't see this problem on an SMP machine.  The writing

Indeed my old 266Mhz PII is bi-processor and I don't see it.

> process makes a _small_ write, and Emacs, waiting on the receiving end
> of it with select, gets woken up immediately and gets to process the
> small write.  But while Emacs is processing the small write, the
> writing process does not get any processing time at all, so it won't
> produce anything new until Emacs either goes back to sleep, or is
> preempted because of taking too long (for example garbage collecting).

Sounds pretty convincing.

> So what can we do about this?
> 
> a) use only SMP systems
> b) pester Linux developers to be less eager with context switches on
>    select.  I am trying this course now, but it will of course take
>    time to register and other OS might have similar problems
> c) make a super-efficient path for process output arriving in tiny
>    chunks, since this is what we will be force-fed most of the time.

The problem with (c) is that we can't cut many corners for
this special case, since we have no idea whether some other
output is about to come or not, so we have to do the full
processing, just in case.

I think the problem is really a kernel issue, and it even sounds
like something very vaguely familiar (as in "some kind of textbook
example about why you don't want to systematically switch when
an event wakes up another process").  I haven't hung out in kernel
world for a long time, but I suspect that the problem you describe
depends on the kernel version.  I would also consider it as a
kernel-scheduler bug, since the `dd' process ends up starved.

I'd appreciate if you could CC me on any email you might send
to the kernel crowd.


        Stefan