Using << and an output pipe together in shell (bash)

[Kenny McCormack]

Consider this code in a bash script:

(Note: No actual indentation is intended - only shown like this for posting)

someCommand ... << EOF | someOtherCommand
some data
for someCommand
EOF

This should work, right?

In simple cases, it does (seem to) work OK.

However, in the actual real world case, it is more like:

someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF |
/usr/lib/ANotherCommand -x "with option" and "more options"
some data
for someCommand
EOF

This time, two things happen:

When edited with GVIM, everything after the line that ends with | is
highlighted as if it was an unterminated string (that is, in a purple/pink
color) and when the above file is dotted, bash complains about "Syntax
error: Unexpected end of file" - as if it never seems the EOF tag.

(ANotherCommand ...) << EOF

and all is well. But why? Should this be necessary?
Yes, I know it is hard to debug this w/o specifics, but I really don't
think the specifics matter; they would be hard to reproduce here.

I just want to know if the basic syntax is valid and/or if there is a
better workaround (better than switching to: > >(...))

[Janis Papanagnou]

Post by Kenny McCormack
(Note: No actual indentation is intended - only shown like this for posting)
someCommand ... << EOF | someOtherCommand
some data
for someCommand
EOF
This should work, right?
In simple cases, it does (seem to) work OK.
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF |
/usr/lib/ANotherCommand -x "with option" and "more options"
some data
for someCommand
EOF

This complexity is not helping much to see what issue you have also
when mentioning syntax highlighting problems.

All I can say at the moment - and you decide whether that's relevant
for your case - is that, say, instead of syntax

cat \
-n << EOF |
wc -l
some data
for someCommand
EOF

you might rather want

cat \
-n << EOF |
some data
for someCommand
EOF
wc -l


HTH.

Janis

Post by Kenny McCormack
When edited with GVIM, everything after the line that ends with | is
highlighted as if it was an unterminated string (that is, in a purple/pink
color) and when the above file is dotted, bash complains about "Syntax
error: Unexpected end of file" - as if it never seems the EOF tag.

(ANotherCommand ...) << EOF

and all is well. But why? Should this be necessary?
Yes, I know it is hard to debug this w/o specifics, but I really don't
think the specifics matter; they would be hard to reproduce here.
I just want to know if the basic syntax is valid and/or if there is a
better workaround (better than switching to: > >(...))

[Ben Bacarisse]

Post by Kenny McCormack
(Note: No actual indentation is intended - only shown like this for posting)
someCommand ... << EOF | someOtherCommand
some data
for someCommand
EOF
This should work, right?
In simple cases, it does (seem to) work OK.
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF |

I think you need "| \" at the end of this line. At least that's what I
usually do and it seems to work.

Post by Kenny McCormack
/usr/lib/ANotherCommand -x "with option" and "more options"
some data
for someCommand
EOF

[Kenny McCormack]

In article <***@bsb.me.uk>, Ben Bacarisse <***@bsb.me.uk> wrote:
...

Post by Ben Bacarisse
I think you need "| \" at the end of this line. At least that's what I
usually do and it seems to work.

It is not generally necessary to put a backslash at the end of a line that
ends with | (in shell script).

But it *is* necessary in this special case! So, you get the prize.
Note that this solves it as far as getting bash to be happy with it is
concerned. When I get a chance, I need to see about how VIM feels about it.

Here's test case:

$ nl << EOF | nl
test
this
EOF
1 1 test
2 2 this
$ nl << EOF |
nl
test
this
EOF

-bash5: syntax error: unexpected end of file
Status: 2
$ nl << EOF | \
nl
test
this
EOF
1 1 test
2 2 this
$

[Janis Papanagnou]

Post by Kenny McCormack
...

Post by Ben Bacarisse
I think you need "| \" at the end of this line. At least that's what I
usually do and it seems to work.

It is not generally necessary to put a backslash at the end of a line that
ends with | (in shell script).
But it *is* necessary in this special case!

Unless you put the 'nl' at the "right place"; you can write your
example below as

$ nl << EOF |
test
this
EOF
nl

so the backslash is not "necessary". - As you say, the '|' needs no
[spurious] continuation escape character if you have it at the end
of a command. - After the lines that define the here-doc (for 'nl's
redirection) the pipe command gets continued on the subsequent line,
which is the line after the "EOF".

Post by Kenny McCormack
So, you get the prize.
Note that this solves it as far as getting bash to be happy with it is
concerned. When I get a chance, I need to see about how VIM feels about it.
$ nl << EOF | nl
test
this
EOF
1 1 test
2 2 this
$ nl << EOF |
nl
test
this
EOF

Here, 'nl' is part of the here-doc data. so the pipe has no process
to feed its data in. You can see that if you add, say, another 'nl'
behind the "EOF" and you'll get
1 1 nl
2 2 test
3 3 this

Post by Kenny McCormack
-bash5: syntax error: unexpected end of file
Status: 2
$ nl << EOF | \
nl

(That's a spurious [unnecessary] escape I'd consider a hack. - But
okay, YMMV.)

Janis

Post by Kenny McCormack
test
this
EOF
1 1 test
2 2 this
$

[Kenny McCormack]

In article <vg18jc$2tv5s$***@dont-email.me>,
Janis Papanagnou <janis_papanagnou+***@hotmail.com> wrote:
...

Post by Janis Papanagnou
Unless you put the 'nl' at the "right place"; you can write your
example below as
$ nl << EOF |
test
this
EOF
nl
so the backslash is not "necessary". - As you say, the '|' needs no
[spurious] continuation escape character if you have it at the end
of a command. - After the lines that define the here-doc (for 'nl's
redirection) the pipe command gets continued on the subsequent line,
which is the line after the "EOF".

Yes, that works, too. Thanks.

Somewhat surprisingly (to me, anyway), I think I actually prefer to stick
with; > >(cmd ...)

That has a certain beauty to it.

[Lem Novantotto]

I think I actually prefer to stick with; > >(cmd ...)

Note that

cat << EOF | grep . ; echo end

and also

(cat | grep .) << EOF ; echo end

produce the same output:
---------------------------------------------------------
***@biggy:~$ cat << EOF | grep . ; echo end
start
EOF
start
end
***@biggy:~$
---------------------------------------------------------

start
EOF

start
end
***@biggy:~$
---------------------------------------------------------

Instead with "cat << EOF > >(grep .) ; echo end" the output is
unpredictable. You can very well get, as usual:
---------------------------------------------------------
***@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
start
end
***@biggy:~$
---------------------------------------------------------
OR maybe:
---------------------------------------------------------
***@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
end
start
***@biggy:~$
---------------------------------------------------------
OR even, most of the time:
---------------------------------------------------------
***@biggy:~$ cat << EOF > >(grep .) ; echo end
start
EOF
end
***@biggy:~$ start [you press ENTER here and get back an empty prompt]

***@biggy:~$
---------------------------------------------------------

So sometimes it's better to be careful. :)

[Kaz Kylheku]

Post by Kenny McCormack
...

Post by Ben Bacarisse
I think you need "| \" at the end of this line. At least that's what I
usually do and it seems to work.

It is not generally necessary to put a backslash at the end of a line that
ends with | (in shell script).

The << feature works outside of the ordinary shell grammar, fetching
data in a line-oriented way until the delimiter.

If you have multiple occurrences, both their data parts go after
the command:

cmd1 << EOF | cmd2 << EOF | cmd3 << EOF
data1
EOF
data2
EOF
data3
EOF

Actual run with && rather than |:

$ cat << EOF && cat << EOF && cat << EOF

Post by Kenny McCormack
foo
EOF
bar
EOF
xyzzy
EOF

foo
bar
xyzzy

It looks like the implementation of << pulls logical lines (after
backslash folding) during parsing.

If this entire command pipe were parsed first:

cmd1 << EOF | cmd2 << EOF | cmd3 << EOF

and then a second pass took place to fetch the EOF-delimited pieces of
syntax that it requires, then it could contain artbitrary backslahes.

[Helmut Waitzmann]

Post by Kenny McCormack
(Note: No actual indentation is intended - only shown like this for posting)
someCommand ... << EOF | someOtherCommand
some data
for someCommand
EOF
This should work, right?
In simple cases, it does (seem to) work OK.
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF |
/usr/lib/ANotherCommand -x "with option" and "more options"
some data
for someCommand
EOF
When edited with GVIM, everything after the line that ends with
| is highlighted as if it was an unterminated string (that is,
in a purple/pink color) and when the above file is dotted, bash
complains about "Syntax error: Unexpected end of file" - as if
it never seems the EOF tag.

I don't think so.  It doesn't see any command after the pipe
symbol (because the next and following lines are the here
document).  That's why GVIM shows the color used to indicate
strings and here documents.

Post by Kenny McCormack
In the end, I ended up replacing the " << EOF | ANotherCommand

(ANotherCommand ...) << EOF

and all is well. But why? Should this be necessary?

It isn't necessary, because there are at least three
alternatives.

Post by Kenny McCormack
Yes, I know it is hard to debug this w/o specifics, but I really
don't think the specifics matter; they would be hard to
reproduce here.
I just want to know if the basic syntax is valid and/or if there is a
better workaround (better than switching to: > >(...))

Either use a backslash for unfolding a folded line (as Ben
showed) or put the part of the command line following the pipe
symbol ("|") after the EOF of the here document (as Janis showed)
or embrace the simple command and its here document by "{ ... ;
}":


{
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF
some data
for someCommand
EOF
} |
/usr/lib/ANotherCommand -x "with option" and "more options"

[Janis Papanagnou]

Post by Helmut Waitzmann

Post by Kenny McCormack
I just want to know if the basic syntax is valid and/or if there is a
better workaround (better than switching to: > >(...))

Either use a backslash for unfolding a folded line (as Ben
showed) or put the part of the command line following the pipe
symbol ("|") after the EOF of the here document (as Janis showed)
or embrace the simple command and its here document by "{ ... ;
{
someCommand -x -y "sjdfhk" and more options \
-g and still more options -Q "another option" << EOF
some data
for someCommand
EOF
} |
/usr/lib/ANotherCommand -x "with option" and "more options"

In this vein (i.e. structuring the code) another option is to put
all these long commands with all their options into functions, so
that the complex command calls fit (with the pipes, and without
escapes) on a single line

f_someCommand <<EOF | f_anotherCommand
...
EOF

(This could make the code also better readable and maintainable.)

Janis