Re: Consolidate recommended options?

[Greg Wooledge]

On Tue, Dec 31, 2024 at 12:15:27 -0500, Chet Ramey wrote:
> On 12/29/24 4:34 PM, Lawrence Velázquez wrote:
> > I'm aware of the pitfalls described therein, and more.  I don't
> > think anyone thinks multiple evaluation in indexes is good per se,
> > but many find the side effects of the attempted cures worse than
> > the disease (e.g., https://mywiki.wooledge.org/BashPitfalls#pf62).
> 
> In its zeal to demonstrate how bad bash is, that little code snippet
> ignores the obvious solution:
> 
> declare -A hash
> key=\'\]
> hash[$key]=17
> (( hash[key]++ ))
> declare -p hash
> 
> Since arithmetic evaluation performs its own expansion of identifiers.
> assoc_expand_once also works. But that wasn't really the point, was it?

What's obvious to you is not obvious to the rest of us.  Also, the main
point of the BashPitfalls page isn't just to show how "bad" bash is; it's
to tell users how to safely and correctly do the thing they want to do.

So, let's see if I can at least lay out what the issues are, and try to
find *something* that will work.

There are 8 versions of bash that have associative arrays (4.0 through 5.2),
and 3 of those versions have an "assoc_expand_once" shopt that may
change how that version works.  So, something like 11 version of bash that
we have to deal with, if you count them that way.

First issue (pitfall 61): [[ -v hash[$key] ]] fails for some keys.
This one has two failure modes: either it just gives the wrong answer,
or it allows code injections.

Here's a "script" (minus shebang and shopt) that demonstrates the first
failure mode:

hobbit:~$ cat x61
declare -A hash
key=\'\]
hash[$key]=foo
if [[ -v hash[$key] ]]; then echo yes; else echo no; fi

It should say "yes".

Here's a wrapper script to set the shopt:

hobbit:~$ cat y61
shopt -s assoc_expand_once
source ./x61

And here's what we get:

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x61; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y61; done
4.0: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `hash[$key]'
x61: line 4: `if [[ -v hash[$key] ]]; then echo yes; else echo no; fi'
4.1: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `hash[$key]'
x61: line 4: `if [[ -v hash[$key] ]]; then echo yes; else echo no; fi'
4.2: no
4.3: no
4.4: no
5.0: no
5.1: no
5.2: yes
5.0 with shopt: no
5.1 with shopt: no
5.2 with shopt: yes

So, this one fails completely in 4.0 and 4.1, gives the wrong answer in
4.2 through 5.1 (regardless of shopt), and gives the right answer only
in 5.2 (regardless of shopt).

Pitfall 61 claims that [[ -v 'hash[$key]' ]] will work, so let's try that:

hobbit:~$ cat x61
declare -A hash
key=\'\]
hash[$key]=foo
if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x61; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y61; done
4.0: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `'hash[$key]''
x61: line 4: `if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi'
4.1: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `'hash[$key]''
x61: line 4: `if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi'
4.2: no
4.3: yes
4.4: yes
5.0: yes
5.1: yes
5.2: yes
5.0 with shopt: yes
5.1 with shopt: no
5.2 with shopt: yes

This one fails completely in 4.0 and 4.1, gives the wrong answer in 4.2,
also gives the wrong answer in 5.1 with shopt, and gives the right answer
in the other test cases.

So, I've already got some changes to make on the wiki.

Now, the second failure mode: if key contains a command substitution,
the evaluation may execute it.

hobbit:~$ cat x61
declare -A hash
key='x$(date >&2)'
hash[$key]=foo
if [[ -v hash[$key] ]]; then echo yes; else echo no; fi

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x61; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y61; done
4.0: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `hash[$key]'
x61: line 4: `if [[ -v hash[$key] ]]; then echo yes; else echo no; fi'
4.1: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `hash[$key]'
x61: line 4: `if [[ -v hash[$key] ]]; then echo yes; else echo no; fi'
4.2: no
4.3: Tue Dec 31 18:05:55 EST 2024
no
4.4: Tue Dec 31 18:05:55 EST 2024
no
5.0: Tue Dec 31 18:05:55 EST 2024
no
5.1: Tue Dec 31 18:05:55 EST 2024
no
5.2: yes
5.0 with shopt: Tue Dec 31 18:05:55 EST 2024
no
5.1 with shopt: yes
5.2 with shopt: yes

This one fails completely in 4.0 and 4.1, suffers from code injection
in 4.3 through 5.1 without shopt, suffers code injection in 5.0 with shopt,
and *doesn't* suffer code injection in 4.2 or 5.2 (without shopt), and
5.1 or 5.2 (with shopt).

With the single quotes:

hobbit:~$ cat x61
declare -A hash
key='x$(date >&2)'
hash[$key]=foo
if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x61; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y61; done
4.0: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `'hash[$key]''
x61: line 4: `if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi'
4.1: x61: line 4: conditional binary operator expected
x61: line 4: syntax error near `'hash[$key]''
x61: line 4: `if [[ -v 'hash[$key]' ]]; then echo yes; else echo no; fi'
4.2: no
4.3: yes
4.4: yes
5.0: yes
5.1: yes
5.2: yes
5.0 with shopt: yes
5.1 with shopt: no
5.2 with shopt: yes

The single quotes successfully stop the code injection in all versions,
which is good.

The conclusion here is that this feature simply doesn't exist before
version 4.2, has no known way to make it work correctly in 4.2, can be
made to work correctly with a single-quote workaround in versions 4.3
to 5.1 (as long as assoc_expand_once is NOT set), and works out of the
box only in 5.2.

Next, we have pitfall 62: (( hash[$key]++ ))

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
(( hash[$key]++ ))
printf '%s\n' "${hash[$key]}"

This one should print "4".  Here's what we get with no workarounds:

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
4.1: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
4.2: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
4.3: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
4.4: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
5.0: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
5.1: x62: line 4: ((: hash[']]++ : bad array subscript (error token is 
"hash[']]++ ")
3
5.2: 4
5.0 with shopt: ./x62: line 4: ((: hash[']]++ : syntax error: invalid 
arithmetic operator (error token is "]++ ")
3
5.1 with shopt: ./x62: line 4: ((: hash[']]++ : syntax error: invalid 
arithmetic operator (error token is "]++ ")
3
5.2 with shopt: 4

This one gives the right answer only in 5.2, with or without the shopt.

The single-quote workaround from pitfall 61 only works in some versions:

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
(( 'hash[$key]++' ))
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 4
4.1: 4
4.2: 4
4.3: 4
4.4: 4
5.0: 4
5.1: x62: line 4: ((: 'hash[']]++' : syntax error: operand expected (error 
token is "'hash[']]++' ")
3
5.2: x62: line 4: ((: 'hash[\'\]]++' : syntax error: operand expected (error 
token is "'hash[\'\]]++' ")
3
5.0 with shopt: 3
5.1 with shopt: ./x62: line 4: ((: 'hash[']]++' : syntax error: operand 
expected (error token is "'hash[']]++' ")
3
5.2 with shopt: ./x62: line 4: ((: 'hash[\'\]]++' : syntax error: operand 
expected (error token is "'hash[\'\]]++' ")
3

This gives the right answer in 4.0 through 5.0, gives the wrong answer with
NO error message in 5.0 with shopt, and gives the wrong answer with an
error message in 5.1 and 5.2 (regardless of shopt).

The second known workaround is a backslash:

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
(( hash[\$key]++ ))
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 4
4.1: 4
4.2: 4
4.3: 4
4.4: 4
5.0: 4
5.1: 4
5.2: 3
5.0 with shopt: 3
5.1 with shopt: 3
5.2 with shopt: 3

This one gives the right answer in 4.0 through 5.1 (without shopt),
gives the wrong answer in 5.0 and 5.1 with shopt, and gives the wrong
answer in 5.2 regardless of the shopt.

Pitfall 62 gives a few workarounds which are purported to work across
all versions.  The first is using a temporary string variable:

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
tmp=${hash[$key]}
hash[$key]=$((tmp+1))
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 4
4.1: 4
4.2: 4
4.3: 4
4.4: 4
5.0: 4
5.1: 4
5.2: 4
5.0 with shopt: 4
5.1 with shopt: 4
5.2 with shopt: 4

The second is to convert the hash[$key] into ${hash[$key]}:

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
hash[$key]=$(( ${hash[$key]} + 1 ))
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 4
4.1: 4
4.2: 4
4.3: 4
4.4: 4
5.0: 4
5.1: 4
5.2: 4
5.0 with shopt: 4
5.1 with shopt: 4
5.2 with shopt: 4

The third is to use "let" with a single-quoted expression instead of "((":

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
let 'hash[$key]++'
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 4
4.1: 4
4.2: 4
4.3: 4
4.4: 4
5.0: 4
5.1: 4
5.2: 4
5.0 with shopt: 3
5.1 with shopt: 3
5.2 with shopt: 3

(Note: gotta mention that the shopt breaks this one!)

I won't even bother with the fourth one, because it's some ormaaj voodoo.

Now, Chet has proposed a fifth one:

hobbit:~$ cat x62
declare -A hash
key=\'\]
hash[$key]=3
(( hash[key]++ ))
printf '%s\n' "${hash[$key]}"

hobbit:~$ for v in 4.0 4.1 4.2 4.3 4.4 5.0 5.1 5.2; do printf '%s: ' "$v"; 
bash-"$v" x62; done; for v in 5.0 5.1 5.2; do printf '%s with shopt: ' "$v"; 
bash-"$v" y62; done
4.0: 3
4.1: 3
4.2: 3
4.3: 3
4.4: 3
5.0: 3
5.1: 3
5.2: 3
5.0 with shopt: 3
5.1 with shopt: 3
5.2 with shopt: 3

This one doesn't work in any version of bash being tested.

OK, now it's editing time.