I am stuck with some problems with my scripts in ksh. FWIW the problem which I am unable to overcome is that when I use a structure such as this

command | while read VAR1 
do
   many.commands using $VAR1
done

I often get that my scripts do not perform the loop for every line piped to the while. To test this I then change the structure to

command > /tmp/tempfile
cat -n /tmp/tempfile >&2
cat /tmp/tempfile | while read VAR1
etc

This proves that there is many lines in the output.

Further I then add an extra line immediately after the do, like

echo DEBUGGING: $VAR1  >&2

Which proves that the loop runs only once. I am really stumped.

One workaround which is not always viable, is to do

for X in $(cat /tmp/tempfile )
do
...
done

This then works correctly but besides the fact that I hate this for structure, It means you expand the entire input data on the command-line (which has hard limits)

It appears that bash is better than ksh at handling this kind of thing. In particular it seems that this may be related to read calls failing but not retrying if the loop takes a long time to run.

However it seems that bash does not have a built-in “read”, which means much of my scripts will need to be re-written. I OFTEN use large structures like

command1 | command2 | while read SOMEVAR; do awk -F: "... long awk program" | sed "long sed program" ; done | sort -u | tail -1 | read FINAL_ANSWER

The problem is that bash uses /usr/bin/read which as expected throws away the result of FINAL_ANSWER as fast as it gets it. The obvious workaround is to replace

| read FINAL_ANSWER

with

> /tmp/final_answer && FINAL_ANSWER="$(cat /tmp/final_answer)"

So…. Any scripting gurus on here able to shed some more light on this? I deliberately did not post my real scripts here both because they are part of a sensitive solution developed for a customer, and because I don’t want the actual detail of the scripts to confuse the issue.

I use the “while read” format OFTEN. It usually works. I’ve in fact never had a problem with it in 25 years of shell scripting. Now I’m having problems. Very frustrating. Perplexing.

Initially I thought the while read is only receiving, or passing along, the first line of input. But then I discovered a situation where, when I run the script over and over, it runs further and further into the input. Specifically I have something line this

command | while read NEXT_ONE DONEFLAG
do
   if [ $DONEFLAG = "yes" ]
   then
       echo Already completed work for $NEXT_ONE
   else
       dowork $NEXT_ONE && set_flag $NEXT_ONE
   fi
done

It turns out that on each run of the script, it performs dowork once. It doesn’t really matter what dowork is, so long as it takes more than a few seconds. Some kind of shell pipe timeout occurs and the rest of the input then disappears. Google tells me that dtksh may solve this issue (Apparently it will retry the read/write or something, I didn’t read enough)

I see that dtksh exists in /usr/st/bin/dtksh

Who is this? I don’t like using shells that I don’t know, but it might be worth while to split small portions of scripts into sub-scripts with /usr/dt/bin/dtksh as interpreter.

Any advice?

EDIT: Providing an example of why I can not use bash as a drop-in replacement for ksh as interpreter:

sol10-primary> # cat test.sh
#!/bin/ksh
echo hello| read VAR1
echo $VAR1
sol10-primary> # ./test.sh
hello
sol10-primary> # sed 's/ksh/bash/' <test.sh >test2.sh
sol10-primary> # chmod +x test2.sh
sol10-primary> # ./test2.sh

sol10-primary> #