bug-gnulib
[Top][All Lists]
Advanced

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: gnulib/gnulib-tool: line 1512: echo: write error: Broken pipe


From: Bruno Haible
Subject: Re: gnulib/gnulib-tool: line 1512: echo: write error: Broken pipe
Date: Sun, 31 Aug 2008 01:39:41 +0200
User-agent: KMail/1.5.4

Eric Blake explained on 2008-06-30:

> echo "$handledmodules" | LC_ALL=C join -v 2 - "$tmp"/queued-modules
> 
> > 1) The 'join' command (or 'join --nocheck-order' in coreutils 6.11 or newer)
> >    apparently closes its standard input and exits when the contents of the
> >    second file (small enough or even empty) allows to determine the complete
> >    'join' output - without having read all of its standard input.
> 
> Closing stdin on exit is expected by POSIX.  What's more, if stdin is
> seekable, it must be reset back to the next position that was not
> consumed by join.  But in this case, stdin is a pipe, so the state of
> the pipe is explicitly called out as unspecified by POSIX, so it is
> neither required to be fully consumed, nor to avoid losing data that
> was buffered but not consumed by join.
> 
> > 
> > 2) The kernel then transforms the pipe into a pipe with no readers.
> 
> Correct - if join finishes prior to echo, this is expected.
> 
> > 
> > 3) The 'echo' command, when writing to a pipe with no readers, gets a 
> > SIGPIPE.
> 
> Yes.  But it depends on whether the shell is trapping/ignoring SIGPIPE
> before this is detectable as a write error instead of an abrupt exit.
> 
> > 
> > 4) The 'echo' command or its parent shell prints an error message on stderr.
> 
> Yes, echo (and all POSIX utilities) are required to print an error message
> to stderr if exiting normally with non-zero status.  And if SIGPIPE is
> being ignored, then echo must exit normally with non-zero status
> (if SIGPIPE is not being ignored, then the exit is due to a signal, so
> nothing need be printed).
> 
> > 
> > Are these 4 behaviours all correct (POSIX compliant and to be expected)?
> > If not, which one is incorrect/buggy?
> 
> It seems like they are all expected, so you should either change
> gnulib-tool to not ignore SIGPIPE before using echo, or you should
> change it to silence stderr when it is known that a piped stdout
> might be closed early.

Thank you for explaining all these. If 'join' was to be blamed for closing
stdin too early, I would have applied the attached patch "blaming-join".

But more generally: How should programs react on SIGPIPE? I think the
answer is: If the program produces only output to stdout, then terminating
without an error message is the right behaviour. But if the program has
side effects other than writing to stdout, then the program should show an
error message.

The POSIX default behaviour is to terminate without an error message; this
is the right thing for the classical Unix filters which read from stdin and
write to stdout. For gnulib-tool as a whole, it is not right; that's the reason
why we have the 'trap' command in gnulib-tool line 1100.

In a pipe like
  { while true; do echo foo; done; } | head -10
strace shows me that "head -10" is closing stdin early.

With bash-3.2.33:
  { while true; do echo foo; done; } | head -10      => no error message, stops
  { while true; do /bin/echo foo; done; } | head -10 => no error message, hangs
  trap '' SIGPIPE
  { while true; do echo foo; done; } | head -10      => error message, loops
  { while true; do /bin/echo foo; done; } | head -10 => error message, hangs

So it seems that something in Sam's environment is setting the SIGPIPE default
behaviour to the empty command. The output of Sam's test commands in
<http://lists.gnu.org/archive/html/bug-gnulib/2008-06/msg00296.html>
(where he had no error message) indicates that this SIGPIPE default comes in
while executing gnulib-tool somewhere.

I'm applying this. Sam, please speak up if the problem persists.


2008-08-30  Bruno Haible  <address@hidden>

        * gnulib-tool (func_reset_sigpipe): New function.
        (func_get_automake_snippet, func_modules_transitive_closure,
        func_import): Invoke it before a join command that reads from stdin,
        to avoid "echo: write error: Broken pipe" error messages on stderr.
        Reported by Sam Steingold <address@hidden>.

--- gnulib-tool.orig    2008-08-31 01:36:52.000000000 +0200
+++ gnulib-tool 2008-08-31 01:36:46.000000000 +0200
@@ -548,6 +548,31 @@
   fi
 }
 
+# func_reset_sigpipe
+# Resets SIGPIPE to its default behaviour. SIGPIPE is signalled when a process
+# writes into a pipe with no readers, i.e. a pipe where all readers have
+# already closed their file descriptor that read from it or exited entirely.
+# The default behaviour is to terminate the current process without an error
+# message.
+# When "trap '' SIGPIPE" is in effect, the behaviour (at least with bash) is to
+# terminate the current process with an error message.
+# This function should be called at the beginning of a command that only
+# produces output to stdout (i.e. no side effects!), when the command that
+# will read from this pipe might prematurely exit or close its standard input
+# descriptor.
+if test -n "$BASH_VERSION"; then
+  # The problem has only been reported with bash.
+  # Note that Solaris sh does not understand "trap - SIGPIPE".
+  func_reset_sigpipe ()
+  {
+    trap - SIGPIPE
+  }
+else
+  func_reset_sigpipe ()
+  {
+  }
+fi
+
 # Ensure an 'echo' command that does not interpret backslashes.
 # Test cases:
 #   echo '\n' | wc -l                 prints 1 when OK, 2 when KO
@@ -1342,7 +1367,8 @@
                  done | sed -e 's,^lib/,,'`
       # Remove $already_mentioned_files from $lib_files.
       echo "$lib_files" | LC_ALL=C sort -u > "$tmp"/lib-files
-      extra_files=`for f in $already_mentioned_files; do echo $f; done \
+      extra_files=`func_reset_sigpipe; \
+                   for f in $already_mentioned_files; do echo $f; done \
                    | LC_ALL=C sort -u | LC_ALL=C join -v 2 - "$tmp"/lib-files`
       if test -n "$extra_files"; then
         echo "EXTRA_DIST +=" $extra_files
@@ -1509,7 +1535,7 @@
     handledmodules=`for m in $handledmodules $inmodules_this_round; do echo 
$m; done | LC_ALL=C sort -u`
     # Remove $handledmodules from $inmodules.
     for m in $inmodules; do echo $m; done | LC_ALL=C sort -u > 
"$tmp"/queued-modules
-    inmodules=`echo "$handledmodules" | LC_ALL=C join -v 2 - 
"$tmp"/queued-modules`
+    inmodules=`func_reset_sigpipe; echo "$handledmodules" | LC_ALL=C join -v 2 
- "$tmp"/queued-modules`
   done
   modules=`for m in $outmodules; do echo $m; done | LC_ALL=C sort -u`
   rm -f "$tmp"/queued-modules
@@ -2406,7 +2432,7 @@
   fi
   # Determine tests-related module list.
   echo "$final_modules" | LC_ALL=C sort -u > "$tmp"/final-modules
-  testsrelated_modules=`echo "$main_modules" | LC_ALL=C sort -u | LC_ALL=C 
join -v 2 - "$tmp"/final-modules`
+  testsrelated_modules=`func_reset_sigpipe; echo "$main_modules" | LC_ALL=C 
sort -u | LC_ALL=C join -v 2 - "$tmp"/final-modules`
   if test $verbose -ge 1; then
     echo "Tests-related module list:"
     echo "$testsrelated_modules" | sed -e 's/^/  /'
@@ -3272,10 +3298,12 @@
         if test -f "$destdir/$dir$ignore"; then
           if test -n "$dir_added" || test -n "$dir_removed"; then
             sed -e "s|^$anchor||" < "$destdir/$dir$ignore" | LC_ALL=C sort > 
"$tmp"/ignore
-            echo "$dir_added" | sed -e '/^$/d' | LC_ALL=C sort -u \
-              | LC_ALL=C join -v 2 "$tmp"/ignore - > "$tmp"/ignore-added
-            echo "$dir_removed" | sed -e '/^$/d' | LC_ALL=C sort -u \
-              | LC_ALL=C join -v 2 "$tmp"/ignore - > "$tmp"/ignore-removed
+            (func_reset_sigpipe
+             echo "$dir_added" | sed -e '/^$/d' | LC_ALL=C sort -u \
+               | LC_ALL=C join -v 2 "$tmp"/ignore - > "$tmp"/ignore-added
+             echo "$dir_removed" | sed -e '/^$/d' | LC_ALL=C sort -u \
+               | LC_ALL=C join -v 2 "$tmp"/ignore - > "$tmp"/ignore-removed
+            )
             if test -s "$tmp"/ignore-added || test -s "$tmp"/ignore-removed; 
then
               if $doit; then
                 echo "Updating $destdir/$dir$ignore (backup in 
$destdir/$dir${ignore}~)"


Attachment: blaming-join
Description: Text Data


reply via email to

[Prev in Thread] Current Thread [Next in Thread]