bash: global var scoping: values lost, why?

Hi there,

I have a bash script gzipping some files and reporting file shrinkage,
but with the first method of find ... piped to a while loop the accumulated
totals are lost when script leaves the do/done loop.

But if I use a temp file between find and do/done loop the thing works as
expected, why is this?

bash script:

totsizefile=0
totsizegzip=0
shrink() # filename
{
[ -r $1 ] || return
gzip -c $1 > $1.gz
filesize=$(stat -L -c %s $1)
gzipsize=$(stat -c %s $1.gz)
totsizefile=$(( totsizefile + filesize ))
totsizegzip=$(( totsizegzip + gzipsize ))
percent=$(( 1000 * (filesize - gzipsize) / filesize ))
printf "%-66s % 3d.%01d%%\n" $name $((percent / 10 )) $((percent % 10 ))
}

find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
| grep -v robots.txt | while read name
do
shrink $name
done

# get overall shrinkage, why did these go back to zero?
echo "out1: totsizefile: $totsizefile, totsizegzip: $totsizegzip"

tmp=$(mktemp)
find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
| grep -v robots.txt > $tmp

while read name
do
shrink $name
done < $tmp
rm $tmp

# get overall shrinkage, this one works as expected
echo "out2: totsizefile: $totsizefile, totsizegzip: $totsizegzip"

- - -
script run:

~$ txt2gzip
....
../pakweb/txt2gzip.txt 52.2%
../pakweb/pak-web-site.txt 65.3%
../pakweb/t2g-run.txt 82.2%
out1: totsizefile: 0, totsizegzip: 0
....
../pakweb/txt2gzip.txt 52.2%
../pakweb/pak-web-site.txt 65.3%
../pakweb/t2g-run.txt 82.2%
out2: totsizefile: 390534, totsizegzip: 140171

Thanks,
Grant.
--
http://bugsplatter.id.au/

On August 27, 2008 19:22, in comp.unix.shell, Grant (g_r_a_n_t_-at-dodo.com.au)
wrote:

> Hi there,
>
> I have a bash script gzipping some files and reporting file shrinkage,
> but with the first method of find ... piped to a while loop the
> accumulated totals are lost when script leaves the do/done loop.
>
> But if I use a temp file between find and do/done loop the thing works as
> expected, why is this?

Because, the commands that are run in the "do/done" loop are run in a
separate process from the commands run outside the loop. Since the
accumulation logic executes in an isolated process, it cannot modify
variables in any other process, including variables in the parent process.
Thus, the parent process only sees the values it set (or did not set)
outside of the loop - the values set in the loop are set in another
process, and discarded when the loop terminates.


--
Lew Pitcher

Master Codewright & JOAT-in-training | Registered Linux User #112576
http://pitcher.digitalfreehold.ca/ | GPG public key available by request
---------- Slackware - Because I know what I'm doing. ------

On 2008-08-27, Grant wrote:
> Hi there,
>
> I have a bash script gzipping some files and reporting file shrinkage,
> but with the first method of find ... piped to a while loop the accumulated
> totals are lost when script leaves the do/done loop.
>
> But if I use a temp file between find and do/done loop the thing works as
> expected, why is this?

All components of a pipeline are run in subshells, and they cannot
affect values in the calling shell.

You can force the rest of the script into the same subshell by
grouping the commands:

find ... | grep ... | { while read var; do
....
done

# rest of script here
}


> bash script:
>
> totsizefile=0
> totsizegzip=0
> shrink() # filename
> {
> [ -r $1 ] || return
> gzip -c $1 > $1.gz
> filesize=$(stat -L -c %s $1)
> gzipsize=$(stat -c %s $1.gz)
> totsizefile=$(( totsizefile + filesize ))
> totsizegzip=$(( totsizegzip + gzipsize ))
> percent=$(( 1000 * (filesize - gzipsize) / filesize ))
> printf "%-66s % 3d.%01d%%\n" $name $((percent / 10 )) $((percent % 10 ))
> }
>
> find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
> | grep -v robots.txt | while read name
> do
> shrink $name
> done
>
> # get overall shrinkage, why did these go back to zero?
> echo "out1: totsizefile: $totsizefile, totsizegzip: $totsizegzip"
>
> tmp=$(mktemp)
> find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
> | grep -v robots.txt > $tmp
>
> while read name
> do
> shrink $name
> done < $tmp
> rm $tmp
>
> # get overall shrinkage, this one works as expected
> echo "out2: totsizefile: $totsizefile, totsizegzip: $totsizegzip"
>
> - - -
> script run:
>
> ~$ txt2gzip
> ...
> ./pakweb/txt2gzip.txt 52.2%
> ./pakweb/pak-web-site.txt 65.3%
> ./pakweb/t2g-run.txt 82.2%
> out1: totsizefile: 0, totsizegzip: 0
> ...
> ./pakweb/txt2gzip.txt 52.2%
> ./pakweb/pak-web-site.txt 65.3%
> ./pakweb/t2g-run.txt 82.2%
> out2: totsizefile: 390534, totsizegzip: 140171
>
> Thanks,
> Grant.


--
Chris F.A. Johnson, author
Shell Scripting Recipes: A Problem-Solution Approach (2005, Apress)
===== My code in this post, if any, assumes the POSIX locale
===== and is released under the GNU General Public Licence

On Wed, 27 Aug 2008 20:22:21 -0300, Grant wrote:


>
> find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
> | grep -v robots.txt | while read name
> do
> shrink $name
> done
>
> # get overall shrinkage, why did these go back to zero?
> echo "out1: totsizefile: $totsizefile, totsizegzip: $totsizegzip"
>


Make the while in the parent:

while read name
do
shrink $name
done < <(find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
| grep -v robots.txt )

On Thu, 28 Aug 2008 00:05:00 +0000, "Chris F.A. Johnson" wrote:

>On 2008-08-27, Grant wrote:
>> Hi there,
>>
>> I have a bash script gzipping some files and reporting file shrinkage,
>> but with the first method of find ... piped to a while loop the accumulated
>> totals are lost when script leaves the do/done loop.
>>
>> But if I use a temp file between find and do/done loop the thing works as
>> expected, why is this?
>
> All components of a pipeline are run in subshells, and they cannot
> affect values in the calling shell.
>
> You can force the rest of the script into the same subshell by
> grouping the commands:
>
>find ... | grep ... | { while read var; do
>...
>done
>
># rest of script here
>}

Thanks Lew and Chris for the explanation. First time I hit this issue

Grant.
--
http://bugsplatter.id.au/

On Wed, 27 Aug 2008 22:16:33 -0300, mop2 wrote:

>On Wed, 27 Aug 2008 20:22:21 -0300, Grant wrote:
>
>
>>
>> find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
>> | grep -v robots.txt | while read name
>> do
>> shrink $name
>> done
>>
>> # get overall shrinkage, why did these go back to zero?
>> echo "out1: totsizefile: $totsizefile, totsizegzip: $totsizegzip"
>>
>
>
>Make the while in the parent:
>
>while read name
>do
> shrink $name
>done < <(find . -name "*.txt" | grep -v networkmonitor | grep -v cc2ip/public \
> | grep -v robots.txt )

Very good, neater than jamming the rest of the script inside { }.

Grant.
--
http://bugsplatter.id.au/