Re: [Fastcgipp-users] SIGPIPE causes SEGFAULT

[Eddie Carle]

On Wed, 2010-11-10 at 00:51 +0100, Volker Schreiner wrote:
> The system i am working with is a Core2Duo with 2 Gbytes of ram
> running a Ubuntu 10.04 LTS - Lucid Lynx with the installed
> prerequisites of boost 1.40 and libmysqlclient16. I am using nginx
> version 0.7.65 that connects the fastcgi application through a TCPIP
> socket on the
> IP address 127.0.0.1 (localhost). Furthermore nginx starts 10
> worker_processes with up to 1024 concurrent worker_connections per
> process (10240 concurrent connections). The test i
> was running on localhost with Apache Benchmark counted 10000 Requests
> with 1000 concurrent requests. I think the 1000 concurrent requests in
> combination with the busy system caused the nginx to create a large
> number of SIGPIPE signals that needs to be handled
> by the busy webapplication and causes a confusion in the transceiver
> that leads to the segmentation fault.

Alrighty, I've finally gotten around to messing around with this and
I've discovered some interesting issues. I can't reproduce this so after
perusing through the code and your data I've come up with a hypothesis.

First off I'll mention that I'm pretty sure this ironically has nothing
to do with signals. The following is a description of how fastcgi++
handles opening/closing file descriptors.

     1. The FastCGI protocol specifies that the server should tell the
        FastCGI application whether or not to handle closing the pipe/fd
        upon request completion when the request is first set up (see
        Protocol::BeginRequest in the documentation). Most servers do
        one Request per fd (which really annoys me because it is so
        stupidly inefficient) and therebye leave management of the pipes
        to the applications. It looks like when Nginx gets a lot on it's
        plate it starts closing pipes despite this.
     2. fastcgi++ takes this value and sets it into the Request objects
        at manager.hpp:264. 
     3. When the Fastcgipp::Request has data to send back to the server,
        it writes it into a ring buffer and when the request is complete
        it writes a "request complete" record into the ring buffer at
        request.cpp:160. With this line we also embed a boolean value
        (the value from step 1) into the chunk of data in the ring
        buffer telling Transceiver to close the pipe/fd when said chunk
        is transmitted. 
     4. Once the data chunk is transmitted,
        Transceiver::Buffer::freeRead() is called to free the data in
        the ring buffer. Notice at transceiver.cpp:175 it tests whether
        or not the pipe should be closed or not. 
     5. transceiver.cpp:177 searches the Transceiver::pollFds container
        for the appropriate fd and immediately calls erase on said fd. 
     6. This is all fine and dandy except that transceiver.cpp:36 has
        already called an erase on that fd should we run into an EPIPE
        error while trying to write to said fd. 
     7. What is segfaulting? Well, my guess is that we are calling
        pollFds.erase() on pollFds.end() and that is doing it.

So to fix this I guess a check to see if the iterator is pollFds.end()
before calling erase on it is the best solution. I've pushed an
attempted fix to the git repo. Please test it out as apache doesn't have
the same issue.
-- 
        Eddie Carle