Command mode deadlock when stdout is fully buffered

[Jeffrey Bosboom]

To: bug-ddrescue@gnu.org
Subject: Command mode deadlock when stdout is fully buffered

The following short Python program forks a ddrescue command mode
process, sends it a command, and tries to read the result. However, it
ends up waiting forever. ddrescue never sends the response, because
stdout is fully buffered and ddrescue doesn't flush it before trying to
read the next command. (This uses the files in the testsuite directory
of a ddrescue source tarball.)

#!/usr/bin/env python3
import subprocess
with subprocess.Popen(['ddrescue', '--command-mode', 'test1.txt',
        '/tmp/dest.txt', 'mapfile1'], stdin=subprocess.PIPE,
        stdout=subprocess.PIPE, stderr=subprocess.DEVNULL,
        encoding='utf-8') as p:
    p.stdin.write('s 0 512\n')
    p.stdin.flush()
    response = p.stdout.readline()
    print(response)

To confirm buffering is the problem, run this script under strace,
following forks:

  strace -ff --decode-fds=all -o /tmp/trace.txt ./script.py

When the deadlock occurs, the tail of the trace of the ddrescue process
will look like:

newfstatat(0<pipe:[475022]>, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, 
AT_EMPTY_PATH) = 0
read(0<pipe:[475022]>, "s 0 512\n", 4096) = 8
newfstatat(1<pipe:[475023]>, "", {st_mode=S_IFIFO|0600, st_size=0, ...}, 
AT_EMPTY_PATH) = 0
read(0<pipe:[475022]>, 

ddrescue (actually the C library on ddrescue's behalf) stat()s stdout,
but then reads stdin again without actually writing to stdout.

If you modify the script to ...Popen(['stdbuf', '-o', 'L', 'ddrescue'...
the deadlock does not occur. stdbuf sets stdout to line-buffered even
though it is non-interactive (a pipe). To do this from within ddrescue,
I think you would call setvbuf(stdout, NULL, _IOLBF, BUFSIZ). (But
cppreference[1] warns that line buffering on non-terminals is not
portable.) Of course, inserting calls to std::fflush(stdout) at
appropriate places will also work. (I think in command_mode.cc
Rescuebook::do_commands?)

Note that programs that send ddrescue commands and then close ddrescue's
stdin currently work properly even when ddrescue's stdout is fully
buffered. ddrescue will read until it gets EOF on stdin, then exit, and
the C library will flush stdout before the process terminates. (Try it
by adding "p.stdin.close()" to the Python script above.  It doesn't
deadlock, and strace shows the write to stdout happening just before the
exit_group syscall.)

Thank you for your time and effort on ddrescue.

--Jeffrey

[1]: https://en.cppreference.com/w/c/io/setvbuf