bug#23837: grep 2.7 EXIT STATUS inconsistency

[O'Brien, Michael]

GNU grep team,

I've found some inconsistency with the EXIT STATUS processing of grep 2.7.   
All data used in the test is shown below, along with the command line arguments 
to grep.   Everything needed to repeat my test cases is described in this 
email.  Below are the details.

Version Information:

> grep --version
grep (GNU grep) 2.7
Copyright (C) 2010 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and others, see 
<http://git.sv.gnu.org/cgit/grep.git/tree/AUTHORS>.
>

Details:

>From the Manual page of grep, the text of the EXIT STATUS follows:

EXIT STATUS
       Normally, the exit status is 0 if selected lines are found and 1 
otherwise.  But the exit status is 2 if an  error  occurred,  unless
       the  -q  or  --quiet  or --silent option is used and a selected line is 
found.  Note, however, that POSIX only mandates, for programs
       such as grep, cmp, and diff, that the exit status in case of error be 
greater than 1; it is therefore  advisable,  for  the  sake  of
       portability, to use logic that tests for this general condition instead 
of strict equality with 2.

Below are two test cases that show the EXIT STATUS from grep is not necessarily 
described as "Normally, the exit status is 0 if selected lines are found and 1 
otherwise".

*         Note1: $GREP_EXCLUDE used below is simply a shell variable pointing 
to a file containing the lines to exclude from the output, and set with a 
command similar too: GREP_EXCLUDE=./testing.grep_exclude

*         Note2: During the testing below, the *only change* was to the 
contents of the $GREP_EXCLUDE file.

*         Note3: The input data remained the same between test cases.

*         Note4: The input data is ascii text, with each line terminated with a 
newline.   Each line in the input file begins with the pattern: "n,n," and 
continues to the EOL, or newline character.  There are 7 lines in the input 
file.

Test case one will show the return code from grep when the -ivEF "exclude" 
removes *all* data records from the input.   Notice, the input file is fed to 
grep, and there's no output -- in other words, all records were (or "selected 
to be") excluded.   The return code (or EXIT STATUS) is 1.

Thu Jun 23 22:12:46 UTC 2016
Test Case 1: grep exclude file with filters that will exclude all data from 
input
hpehrshdev:~/tmp> cat mjodata.txt | grep -ivEf $GREP_EXCLUDE
hpehrshdev:~/tmp> echo $?
1
hpehrshdev:~/tmp> cat mjodata.txt
0,5,"2016-06-17 16:30:54.404000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 16:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=1, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
16:30:53.0000000."
0,5,"2016-06-17 15:30:54.158000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 15:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
15:30:53.0000000."
0,5,"2016-06-17 14:30:53.780000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 14:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
14:30:53.0000000."
0,5,"2016-06-17 13:30:54.906000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 13:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
13:30:53.0000000."
0,5,"2016-06-17 12:30:54.665000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 12:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
12:30:53.0000000."
77,4,"2016-06-17 12:30:56.582000000","The disk usage of this database has 
exceeded 7498 GB. See system view M_DISK_USAGE for more details."
69,4,"2016-06-17 12:33:20.228000000","Automatic log backup is disabled."
hpehrshdev:~/tmp>
hpehrshdev:~/tmp>
hpehrshdev:~/tmp> cat $GREP_EXCLUDE
^69,4,.*Automatic log backup is disabled.*$
^77,4,.*The disk usage of this database has exceeded.*See system view 
M_DISK_USAGE for more details.*$
^0,5,.*301 unique constraint violated: TrexUpdate failed on table
hpehrshdev:~/tmp>

Test case two will show the return code from grep when the -ivEF "exclude" 
removes *only some* of the data records from the input.   Notice, the input 
file is fed to grep, and output is returned -- in other words, some records 
were (selected to be) excluded, but some of the records did not get excluded 
(because there was no filter for them).   The return code (or EXIT STATUS) is 0.

Thu Jun 23 22:15:35 UTC 2016
Test Case 2: grep exclude file with filters that will exclude some of the data 
from input
hpehrshdev:~/tmp> cat mjodata.txt | grep -ivEf $GREP_EXCLUDE
0,5,"2016-06-17 16:30:54.404000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 16:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=1, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
16:30:53.0000000."
0,5,"2016-06-17 15:30:54.158000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 15:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
15:30:53.0000000."
0,5,"2016-06-17 14:30:53.780000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 14:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
14:30:53.0000000."
0,5,"2016-06-17 13:30:54.906000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 13:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
13:30:53.0000000."
0,5,"2016-06-17 12:30:54.665000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 12:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
12:30:53.0000000."
hpehrshdev:~/tmp> echo $?
0
hpehrshdev:~/tmp> cat mjodata.txt
0,5,"2016-06-17 16:30:54.404000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 16:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=1, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
16:30:53.0000000."
0,5,"2016-06-17 15:30:54.158000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 15:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
15:30:53.0000000."
0,5,"2016-06-17 14:30:53.780000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 14:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=3, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
14:30:53.0000000."
0,5,"2016-06-17 13:30:54.906000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 13:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
13:30:53.0000000."
0,5,"2016-06-17 12:30:54.665000000","[29] 301 unique constraint violated: 
TrexUpdate failed on table '_SYS_STATISTICS:STATISTICS_ALERTS_BASE' with error: 
unique constraint violation in self check for table 
_SYS_STATISTICS:STATISTICS_ALERTS_BASEen, constraint='$trexexternalkey$', 
udiv='2016-06-17 12:30:53;29;27,PBM_ECC_DB.ZFE_FDSTAGDETAIL', pos=4, 
indexname=_SYS_TREE_CS_#145806_#0_#P0, rc=55 SNAPSHOT_ID: 2016-06-17 
12:30:53.0000000."
77,4,"2016-06-17 12:30:56.582000000","The disk usage of this database has 
exceeded 7498 GB. See system view M_DISK_USAGE for more details."
69,4,"2016-06-17 12:33:20.228000000","Automatic log backup is disabled."
hpehrshdev:~/tmp> cat $GREP_EXCLUDE
^69,4,.*Automatic log backup is disabled.*$
^77,4,.*The disk usage of this database has exceeded.*See system view 
M_DISK_USAGE for more details.*$
hpehrshdev:~/tmp>

Issue:

In scripting, it is normal to process the EXIT STATUS of grep in such a way 
that an EXIT STATUS of 0 (zero) indicates that matches to the grep pattern were 
found.   In *both of* the above test cases matches *were* found, yet the EXIT 
STATUS was not consistently 0.  In test case 1, *all* records were matched, yet 
the EXIT STATUS was 1.  With this behaviour of grep, it is * not possible to 
correctly* perform the following simple logic (bash syntax):

typeset -i RC=0

cat mjodata.txt | grep -ivEf $GREP_EXCLUDE
RC=$?

if [ $RC -eq 0 ]; then
     #Grep found matches
     #Do something...
else
     #No match found, or possibly some error -- needs more checking
fi


Michael J. O'Brien
Consultant Architect, AMS ABS Global Practice
address@hidden<mailto:address@hidden>

+1 650 258 2628  Office
+1 916 335 3010  Mobile

US/Pacific time zone
hpe.com


[HPE logo]<http://www.hpe.com/>