Discussion:
BASH Script to read a file, but ignore #comments
(too old to reply)
Luke Robertson
2004-11-13 21:29:56 UTC
Permalink
hi,

I am trying to get a script to read a list of banned sites:

while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites

At the moment, the bannedsites file just has a list of IP addresses,
one per line.

I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.

Any ideas?
mjt
2004-11-13 21:30:45 UTC
Permalink
Post by Luke Robertson
while read BANNED; do
  iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
... why not put those sites in the /etc/hosts
file and point them to 127.0.0.1 ?
--
<< http://michaeljtobler.homelinux.com/ >>
You should never bet against anything in science at
odds of more than about 10^12 to 1. - Ernest Rutherford
Peteris Krumins
2004-11-14 23:00:29 UTC
Permalink
Post by mjt
Post by Luke Robertson
while read BANNED; do
  iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
... why not put those sites in the /etc/hosts
file and point them to 127.0.0.1 ?
Because that's useless.


P.Krumins
mjt
2004-11-14 23:02:46 UTC
Permalink
Post by Peteris Krumins
Post by mjt
... why not put those sites in the /etc/hosts
file and point them to 127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all
my other boxes for YEARS.

even tools, such as "search and destroy" utilize
this simple trick to block out ad sites, etc
.
--
<< http://michaeljtobler.homelinux.com/ >>
The average woman would rather have beauty than brains, because
the average man can see better than he can think.
j***@invalid.address
2004-11-15 04:40:40 UTC
Permalink
Post by mjt
Post by Peteris Krumins
Post by mjt
... why not put those sites in the /etc/hosts
file and point them to 127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all
my other boxes for YEARS.
even tools, such as "search and destroy" utilize
this simple trick to block out ad sites, etc
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.

Joe
--
Nothing cures like time and love
- Laura Nyro
noi
2004-11-15 09:12:21 UTC
Permalink
Post by j***@invalid.address
Post by Peteris Krumins
... why not put those sites in the /etc/hosts file and point them to
127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all my other boxes for YEARS.
even tools, such as "search and destroy" utilize this simple trick to
block out ad sites, etc
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.
Joe
INE but doesn't the original poster need to generate INPUT and FORWARD
table statements to block incoming ?

Isn't it easier to change the FORWARD policy to DROP instead of ACCEPT and
allow only valid ip addresses or masked ip addresses and also match
states for TCP?
iptable -A FORWARD -s 192.168.0.0/255.255.0.0 -match state -state
established,related -j ACCEPT


Combined with the hosts file solution should be enough to kill outbound
and inbound packets from the banned sites except for spoofed addresses.
j***@invalid.address
2004-11-15 15:28:09 UTC
Permalink
Post by noi
Post by j***@invalid.address
Post by Peteris Krumins
... why not put those sites in the /etc/hosts file and point
them to 127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all my other boxes for YEARS.
even tools, such as "search and destroy" utilize this simple
trick to block out ad sites, etc
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.
Joe
INE but doesn't the original poster need to generate INPUT and
FORWARD table statements to block incoming ?
I was under the impression that that's what the -j�DROP�-s�$BANNED
options were for. The man page on my system says that -j DROP will
drop the packet specified by -s $BANNED on the floor, keeping an
incoming connection from being made.
Post by noi
Isn't it easier to change the FORWARD policy to DROP instead of
ACCEPT and allow only valid ip addresses or masked ip addresses and
also match states for TCP? iptable -A FORWARD -s
192.168.0.0/255.255.0.0 -match state -state established,related -j
ACCEPT
Combined with the hosts file solution should be enough to kill
outbound and inbound packets from the banned sites except for
spoofed addresses.
Isn't this equivalent to saying that the hosts file solution isn't
enough?

Joe
--
Nothing cures like time and love
- Laura Nyro
noi
2004-11-15 16:31:37 UTC
Permalink
Post by noi
Post by j***@invalid.address
Post by Peteris Krumins
... why not put those sites in the /etc/hosts file and point them
to 127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all my other boxes for YEARS.
even tools, such as "search and destroy" utilize this simple trick to
block out ad sites, etc
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.
Joe
INE but doesn't the original poster need to generate INPUT and FORWARD
table statements to block incoming ?
I was under the impression that that's what the -j DROP -s $BANNED options
were for. The man page on my system says that -j DROP will drop the packet
specified by -s $BANNED on the floor, keeping an incoming connection from
being made.
Yes. I should have said that the INPUT, FORWARD and OUTPUT chains are not
coordinated so FORWARD applies to packets routing through IPTABLES not
on INPUT or OUTPUT traffic. So, wouldn't someone be able to initiate
and receive traffic from the banned sites?
Post by noi
Isn't it easier to change the FORWARD policy to DROP instead of ACCEPT
and allow only valid ip addresses or masked ip addresses and also match
states for TCP? iptable -A FORWARD -s 192.168.0.0/255.255.0.0 -match
state -state established,related -j ACCEPT
Combined with the hosts file solution should be enough to kill outbound
and inbound packets from the banned sites except for spoofed addresses.
Isn't this equivalent to saying that the hosts file solution isn't enough?
Joe
noi
2004-11-15 16:40:28 UTC
Permalink
Post by noi
Post by j***@invalid.address
Post by Peteris Krumins
... why not put those sites in the /etc/hosts file and point them
to 127.0.0.1 ?
Because that's useless.
... really?!?! been working here and all my other boxes for YEARS.
even tools, such as "search and destroy" utilize this simple trick to
block out ad sites, etc
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.
Joe
INE but doesn't the original poster need to generate INPUT and FORWARD
table statements to block incoming ?
I was under the impression that that's what the -j DROP -s $BANNED options
were for. The man page on my system says that -j DROP will drop the packet
specified by -s $BANNED on the floor, keeping an incoming connection from
being made.
Post by noi
Isn't it easier to change the FORWARD policy to DROP instead of ACCEPT
and allow only valid ip addresses or masked ip addresses and also match
states for TCP? iptable -A FORWARD -s 192.168.0.0/255.255.0.0 -match
state -state established,related -j ACCEPT
Combined with the hosts file solution should be enough to kill outbound
and inbound packets from the banned sites except for spoofed addresses.
Isn't this equivalent to saying that the hosts file solution isn't enough?
Joe
I meant to add that if the OP is using this machine as a router/firewall
then the hosts solution is the better solution blocking outbound
traffic to those sites.
John-Paul Stewart
2004-11-15 16:14:22 UTC
Permalink
[Note follow-ups set to comp.os.linux.misc. This sub-thread no longer
seems relevant to comp.unix.shell.]
Post by noi
INE but doesn't the original poster need to generate INPUT and FORWARD
table statements to block incoming ?
Isn't it easier to change the FORWARD policy to DROP instead of ACCEPT and
allow only valid ip addresses or masked ip addresses and also match
states for TCP?
That would depend on what the system is being used for.

If it is a well-protected server meant to be accessible only to select
users, then it probably would make most sense to drop everything by
default and only allow "valid ip addresses" to connect.

OTOH, if it is a publically visible web server (for example), then
anything is a "valid ip address" to begin with. It could be that the OP
is trying to drop abusive users (or portscanners, or...) from a machine
that is otherwise supposed to be wide open. In that case the current
approach of banning specific addresses makes more sense.

IMHO, the best approach would depend on which is larger: the set of
allowed addresses or the set of banned addresses. Set the default
policy for the larger group and handle the rest sperately.
Peteris Krumins
2004-11-15 17:44:13 UTC
Permalink
Post by j***@invalid.address
Putting it in the local host file won't keep an incoming connection
being made. It will only prevent an outgoing connection.
It does not prevent anything. If you think so, rethink.


P.Krumins
Bit Twister
2004-11-13 21:42:50 UTC
Permalink
Post by Luke Robertson
hi,
I am trying to get a script to read a list of banned sites
and ingore comments.
Well if comment is first char on line

while read BANNED; do

char_1=${BANNED:0:1}
if [ $char_1 != '#' ] ; then
iptables -A FORWARD -j DROP -s $BANNED
fi
done < /root/scripts/bannedsites
Floyd L. Davidson
2004-11-13 23:30:01 UTC
Permalink
Post by Bit Twister
Post by Luke Robertson
hi,
I am trying to get a script to read a list of banned sites
and ingore comments.
Well if comment is first char on line
while read BANNED; do
char_1=${BANNED:0:1}
if [ $char_1 != '#' ] ; then
iptables -A FORWARD -j DROP -s $BANNED
fi
done < /root/scripts/bannedsites
Or the same thing if a case statement (which has the small
advantage of also skipping blank lines):

while read BANNED; do

case $BANNED in
[!\#]) iptables -A FORWARD -j DROP -s $BANNED;;
esac

done < /root/scripts/bannedsites
--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) ***@barrow.com
Bit Twister
2004-11-14 00:02:12 UTC
Permalink
Post by Floyd L. Davidson
Or the same thing if a case statement (which has the small
Hey, garbage requirement in; garbage design out. :)

It gave the OP the option to print the comment which comes in handy
after changes and a iptables line upchucks.

This method only uses a line if the first word is iptables.

while read BANNED; do

set - $BANNED
if [ $1 = "iptables" ] ; then
iptables -A FORWARD -j DROP -s $BANNED;;
fi

done < /root/scripts/bannedsites
Alan Connor
2004-11-14 00:07:44 UTC
Permalink
Post by Bit Twister
Post by Luke Robertson
hi,
I am trying to get a script to read a list of banned sites
and ingore comments.
Well if comment is first char on line
while read BANNED; do
char_1=${BANNED:0:1}
if [ $char_1 != '#' ] ; then
iptables -A FORWARD -j DROP -s $BANNED
fi
done < /root/scripts/bannedsites
Nice. Had to look that up in man bash:

man bash #on my box, the man page comes up in the pager less
# the first "/" means "search forward" and the two
# backslashes are to escape the characters they
# precede, which have special meanings in regular
# expressions, which is what less takes as search
# strings.


/\$\{.*:


${parameter:offset}
${parameter:offset:length}
Substring Expansion. Expands to up to length char
acters of parameter starting at the character spec
ified by offset. If length is omitted, expands to
the substring of parameter starting at the charac
ter specified by offset. length and offset are
arithmetic expressions (see ARITHMETIC EVALUATION
below). length must evaluate to a number greater
than or equal to zero. If offset evaluates to a
number less than zero, the value is used as an off
set from the end of the value of parameter. If
parameter is @, the result is length positional
parameters beginning at offset. If parameter is an
array name indexed by @ or *, the result is the
length members of the array beginning with ${param_
eter[offset]}. Substring indexing is zero-based
unless the positional parameters are used, in which
case the indexing starts at 1.



Also for the OP: "BANNED" could be any string like that. I
usually use "line" because that's what's really going on.


AC
Lawrence D¹Oliveiro
2004-11-15 00:33:29 UTC
Permalink
Post by Bit Twister
Post by Luke Robertson
hi,
I am trying to get a script to read a list of banned sites
and ingore comments.
Well if comment is first char on line
while read BANNED; do
char_1=${BANNED:0:1}
if [ $char_1 != '#' ] ; then
iptables -A FORWARD -j DROP -s $BANNED
fi
done < /root/scripts/bannedsites
bash is very clever and all that, but my instinct is to do this in a
Perl script. That would scale more gracefully to more complicated
requirements (which seem to come along sooner or later):

while (<BANNED>)
{
if (!/^\s*\#/)
{
...
} # if
} # while
Bit Twister
2004-11-15 00:57:46 UTC
Permalink
Post by Lawrence D¹Oliveiro
bash is very clever and all that, but my instinct is to do this in a
Perl script. That would scale more gracefully to more complicated
True, true, and I did feel a little like you do, but your script would be
slower and the example does not contain all the code to open/close and
execute iptables. :)

I do like the KISS methodology, especially when I have had to fight the
system admins when trying to get a newer release of perl installed. :(
Ed Morton
2004-11-13 23:02:28 UTC
Permalink
Post by Luke Robertson
hi,
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses,
one per line.
I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.
there's several other ways to approach this, but if you want to use sed:

sed 's/#.*$//' /root/scripts/bannedsites |
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done

Ed.
Post by Luke Robertson
Any ideas?
Bob
2004-11-14 00:25:36 UTC
Permalink
Post by Luke Robertson
hi,
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses, one
per line.
I would like to be able to put comments in, but I am not sure how to get
the system to ignore those lines. I was thinking of using sed in the
script, but I couldn't get it to work.
Any ideas?
===================================================================
#!/bin/bash
while read BANNED; do
BANNED=${BANNED## } # drop leading spaces
[ "${BANNED#\#}" = "$BANNED" ] || continue # skip comment
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
===================================================================

Line in /root/scripts/bannedsites beginning with # will be treated as
comments.

WARNING!!! Only minimal testing.
Chris F.A. Johnson
2004-11-14 01:59:19 UTC
Permalink
Post by Luke Robertson
hi,
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses,
one per line.
I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.
while read BANNED; do
case $BANNED in \#*) continue ;; esac
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites

Or:

while read BANNED; do
case $BANNED in
\#*) ;;
*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
===================================================================
My code (if any) in this post is copyright 2004, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License
Stachu 'Dozzie' K.
2004-11-14 02:57:28 UTC
Permalink
Zawartość nagłówka ["Followup-To:" comp.unix.shell.]
Post by Luke Robertson
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses,
one per line.
I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.
grep -v '^ *#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
--
Stanislaw Klekot
rakesh sharma
2004-11-14 22:41:40 UTC
Permalink
Post by Luke Robertson
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses,
one per line.
I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.
exec 3< /root/scripts/bannedsites
while IFS= read -r 0<&3 BANNED; do
case $BANNED in
\#*) :;;


[! \t]*)
iptables -A FORWARD -j DROP -s "$BANNED";;


*)
IFS=" \t"

set x $BANNED;shift
case $1 in
\#*):;;
*) iptables -A FORWARD -j DROP -s "$BANNED";;
esac

IFS=;;
esac
done
exec 3<&-
Stephane CHAZELAS
2004-11-15 11:02:04 UTC
Permalink
Post by Luke Robertson
while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done < /root/scripts/bannedsites
At the moment, the bannedsites file just has a list of IP addresses,
one per line.
I would like to be able to put comments in, but I am not sure how to
get the system to ignore those lines. I was thinking of using sed in
the script, but I couldn't get it to work.
[...]

< /root/scripts/bannedsites grep -v '^[[:blank:]]*#' |
xargs -n1 iptables -A FORWARD -j DROP -s

This way, it also ommits blanks, tabs, and empty lines (if have
arguments with blanks, then you can use quotes in the
bannedsites file).
--
Stephane
Luke Robertson
2004-11-15 21:01:42 UTC
Permalink
Thanks everyone. Your response has been astounding!

This is what I eventually went with:


while read BANNED; do
char_1=${BANNED:0:1}
case "$char_1" in
[!\#])
iptables -A FORWARD -j DROP -s $BANNED;;
esac
done < /root/scripts/bannedsites


Thanks to everyone who sent some code in.

PS, to the guy who suggested that I put the IP addresses in
/etc/hosts, and point them to 127.0.0.1:

I didn't do it this way, because /etc/hosts is only used for
resolution, not blocking the sites. This means that if anything tried
to connect directly to the IP address, nothing would have to be
resolved, and therefore the hosts file would not even come in to it.
Secondly, I don't think you can map an IP address to another IP
address in /etc/hosts.
Thirdly, this would only protect one computer. It is supposed to be a
firewall protecting my LAN.
Chris F.A. Johnson
2004-11-15 21:12:34 UTC
Permalink
Post by Luke Robertson
Thanks everyone. Your response has been astounding!
while read BANNED; do
char_1=${BANNED:0:1}
case "$char_1" in
[!\#])
iptables -A FORWARD -j DROP -s $BANNED;;
esac
done < /root/scripts/bannedsites
You can save a line and make the script portable to other shells:

while read BANNED; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
===================================================================
My code (if any) in this post is copyright 2004, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License
Stachu 'Dozzie' K.
2004-11-15 23:06:22 UTC
Permalink
Zawartość nagłówka ["Followup-To:" comp.unix.shell.]
Post by Luke Robertson
Post by Luke Robertson
Thanks everyone. Your response has been astounding!
while read BANNED; do
char_1=${BANNED:0:1}
case "$char_1" in
[!\#])
iptables -A FORWARD -j DROP -s $BANNED;;
esac
done < /root/scripts/bannedsites
while read BANNED; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
Or even two more lines with grep:

grep -v '^#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
--
Stanislaw Klekot
Chris F.A. Johnson
2004-11-15 23:17:23 UTC
Permalink
Zawarto¶æ nag³ówka ["Followup-To:" comp.unix.shell.]
Post by Luke Robertson
Post by Luke Robertson
Thanks everyone. Your response has been astounding!
while read BANNED; do
char_1=${BANNED:0:1}
case "$char_1" in
[!\#])
iptables -A FORWARD -j DROP -s $BANNED;;
esac
done < /root/scripts/bannedsites
while read BANNED; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
grep -v '^#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
At the cost of an external command.

And it doesn't remove comment lines that begin with whitespace.
--
Chris F.A. Johnson http://cfaj.freeshell.org/shell
===================================================================
My code (if any) in this post is copyright 2004, Chris F.A. Johnson
and may be copied under the terms of the GNU General Public License
Floyd L. Davidson
2004-11-16 01:52:11 UTC
Permalink
Zawarto¶æ nag³ówka ["Followup-To:" comp.unix.shell.]
Post by Luke Robertson
Post by Luke Robertson
Thanks everyone. Your response has been astounding!
while read BANNED; do
char_1=${BANNED:0:1}
case "$char_1" in
[!\#])
iptables -A FORWARD -j DROP -s $BANNED;;
esac
done < /root/scripts/bannedsites
while read BANNED; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
grep -v '^#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
You are saving a line or two, but at the expense of adding at
least one subshell, not to mention that your script doesn't
do exactly what either of the other two scripts do.

There is are subtle differences in what is rejected, and the
middle script (which I originally proposed, and Chris one that
was virtually identical) is perhaps more appropriate. Using the
grep version rejects only lines with '#' in column 0. The OP's
script will reject any line where the text begins with a '#',
regardless of which column.

The variations that Chris and I have suggested will reject
a line with a '#' anywhere in the line, and will reject
blank lines. That has one negative side effect, which is
that inline comments are not allowed.

A moment's thought provides a method to solve that
problem too:

while read BANNED JUNK; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites

Note that the bannedsites file *must* have a newline following
the late entry.
--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) ***@barrow.com
rakesh sharma
2004-11-16 10:12:57 UTC
Permalink
***@barrow.com (Floyd L. Davidson) wrote in message news:
... [snipped]
Post by Floyd L. Davidson
There is are subtle differences in what is rejected, and the
middle script (which I originally proposed, and Chris one that
was virtually identical) is perhaps more appropriate. Using the
grep version rejects only lines with '#' in column 0. The OP's
script will reject any line where the text begins with a '#',
regardless of which column.
The variations that Chris and I have suggested will reject
a line with a '#' anywhere in the line, and will reject
blank lines. That has one negative side effect, which is
that inline comments are not allowed.
A moment's thought provides a method to solve that
while read BANNED JUNK; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
Note that the bannedsites file *must* have a newline following
the late entry.
What purpose does the JUUNK variable serve? You don't seem to use it anywhere.

How about this method (which invokes no externals):

exec 3< /root/scripts/bannedsites
while IFS= read -r 0<&3 BANNED; do
IFS=" " # i.e., IFS='<space><TAB>'
set x $BANNED;shift
case $BANNED in
''|\#*) # either an empty line/or full of blanks only/or first non-blank
# char is a '#' => a comment line.
:;;
*) iptables -A FORWARD -j DROP -s "$BANNED";;
esac
done
exec 3<&-
Floyd L. Davidson
2004-11-16 12:45:58 UTC
Permalink
Post by rakesh sharma
Post by Floyd L. Davidson
while read BANNED JUNK; do
case $BANNED in
[!\#]*) iptables -A FORWARD -j DROP -s $BANNED ;;
esac
done < /root/scripts/bannedsites
Note that the bannedsites file *must* have a newline following
the late entry.
What purpose does the JUUNK variable serve? You don't seem to use it anywhere.
Doing "help read" in bash provides, on the first line, an execellent
descripton of what the second variable does:

"One line is read from the standard input, and the first
word is assigned to the first NAME, the second word to the
second NAME, and so on, with leftover words assigned to the
last NAME."

Hence, without JUNK, the variable BANNED is the entire line. But with
two variables, BANNED gets the first word, and JUNK is the rest of the
line (which in this case are unused). It allows an entry in the
bannedsites file that looks like this:

xxx.xxx.xxx.xxx # this site has nothing of value to us

Without the JUNK variable, that line would be rejected, and the
only way to format it would be with a two line statement,

# this site has nothing of value to us
xxx.xxx.xxx.xxx
Post by rakesh sharma
exec 3< /root/scripts/bannedsites
while IFS= read -r 0<&3 BANNED; do
IFS=" " # i.e., IFS='<space><TAB>'
set x $BANNED;shift
case $BANNED in
''|\#*) # either an empty line/or full of blanks only/or first non-blank
# char is a '#' => a comment line.
:;;
*) iptables -A FORWARD -j DROP -s "$BANNED";;
esac
done
exec 3<&-
I don't understand why you are doing a number of things in that
script:

1) What is the advantage to exec'ing the 3rd fd, as opposed
to just redirecting to stdin?

2) Why use the -r options to read?

3) What is the purpose of resetting IFS ?

4) What value is the "set x $BANNED;shift" statement?

5) The description of what is caught by the first regular
expression in the case statement does not match what it
actually does. I.e., it won't catch empty lines.

6) What is the purpose of the ':' in the first case instance?

Your script rejects *only* lines that begin with the # character
in column 0. And it includes white space in the BANNED
variable. BANNED also becomes the entire line; hence, inlined
comments cannot be done, and formatting of each entry in the
list is very strict, with no white space allowed.

It is very instructive to set up a little test file to determine
exactly what happens with variously formatted lines, and change
the script being studied to echo results to the screen. I do
that rather than assume that what I think is going to happen
happens, because I am almost always /very/ surprised by what
does happen!

Here is how I tested your script:

#!/bin/bash
exec 3< ./foo.test
while IFS= read -r 0<&3 BANNED; do
IFS=" " # i.e., IFS='<space><TAB>'
set x $BANNED;shift
case $BANNED in
' '|\#*) # either an empty line/or full of blanks only/or first non-blank
# char is a '#' => a comment line.
echo "ignored: <$BANNED>"
:;;
*) echo "selected: <$BANNED>"
esac
done
exec 3<&-
exit 0

Below is the data file (edited to indicate TABs, and with
Post by rakesh sharma
# comment 1
#comment 2
# comment space 3
<TAB># comment tab 4
<TAB>#comment tab 5
#comment space 6
<TAB><TAB>
<TAB>
<TAB>127.00.00.1 # comment 7
127.00.00.2 # comment 8
127.00.00.3 # comment 9
127.00.00.4 #comment10
127.00.00.5 #comment 11
--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) ***@barrow.com
rakesh sharma
2004-11-18 11:25:52 UTC
Permalink
Post by Floyd L. Davidson
Without the JUNK variable, that line would be rejected, and the
only way to format it would be with a two line statement,
I don't understand why you are doing a number of things in that
1) What is the advantage to exec'ing the 3rd fd, as opposed
to just redirecting to stdin?
2) Why use the -r options to read?
3) What is the purpose of resetting IFS ?
4) What value is the "set x $BANNED;shift" statement?
Hi Floyd,

I see your point now about that JUNK variable. It is a very neat trick
to dump what you don't need into a null variable.

Please ignore my script as it is nothing but a roundabout way to do
what you've done using that JUNK variable. Thanks for the elaborate
explanations and thank you for your time.
I don't think there's any need for
those questions of yours about my code!

---
Rakesh
Lawrence D¹Oliveiro
2004-11-18 04:00:10 UTC
Permalink
Post by Stachu 'Dozzie' K.
grep -v '^#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
Using the grep version rejects only lines with '#' in column 0.
So change the grep expression to '^[[:space:]]*#'.
Floyd L. Davidson
2004-11-18 04:41:37 UTC
Permalink
Post by Lawrence D¹Oliveiro
Post by Stachu 'Dozzie' K.
grep -v '^#' /root/scripts/bannedsites | while read BANNED; do
iptables -A FORWARD -j DROP -s $BANNED
done
Using the grep version rejects only lines with '#' in column 0.
So change the grep expression to '^[[:space:]]*#'.
Okay...

1) it invokes a second, unnecesary, process.

2) it doesn't allow inlined comments

3) it is unnecessarily sensitive to formatting,
particularly white space.

The point being that using the shell's builtin case statement
not only eliminates the unnecessary process, but also provides
significant other benefits. For example, entries configured
like these will work (but won't with the above script),

127.0.0.2 #this site has nothing but trash
127.0.0.3 this one too.
--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) ***@barrow.com
iON
2004-12-01 13:13:00 UTC
Permalink
Hi Luke,

Much of this discussion seems to circle around the whether its
nessecary or not. The solutions i've seen so far has the disadvatage
of cancel the whole line containing then # character!

Here is a quick'n'dirty awk solution that will remove all comments
AFTER the # character and ALSO skip empty lines.

(awk -F# {'print $1'} |grep -v "^$" |while read X; do iptables -A
FORWARD -j DROP -s $X; done)<bannade.txt

Best of luck



- Jon Bjoerkebaeck
Floyd L. Davidson
2004-12-01 13:29:42 UTC
Permalink
Post by iON
Hi Luke,
Much of this discussion seems to circle around the whether its
nessecary or not. The solutions i've seen so far has the disadvatage
of cancel the whole line containing then # character!
Here is a quick'n'dirty awk solution that will remove all comments
AFTER the # character and ALSO skip empty lines.
(awk -F# {'print $1'} |grep -v "^$" |while read X; do iptables -A
FORWARD -j DROP -s $X; done)<bannade.txt
Best of luck
- Jon Bjoerkebaeck
Not bad. It has one redeeming feature! It will process a file
with no newline at the end of the last line. If I remember
right, none of the other examples would do that.

However, it doesn't quite do what you claim either! It will
send an empty X variable to the iptables command if there is an
otherwise empty line that does contain whitespace (spaces or
tabs), and that will also happen if a comment has leading
whitespace. I'd say both of those are fatal errors, because it
cannot process this data file:

left margin -->
# This line is deleted
# This line is not correctly handled

# ^ EOL on previous line
# The line above with only whitespace is not
# correctly handled
127.0.0.2 # valid data is handled correctly
--
Floyd L. Davidson <http://web.newsguy.com/floyd_davidson>
Ukpeagvik (Barrow, Alaska) ***@barrow.com
Peter T. Breuer
2004-12-01 14:38:06 UTC
Permalink
Post by iON
Much of this discussion seems to circle around the whether its
nessecary or not.
What's "necessary or not"? Would you mind quoting enough context for
whatever you are saying to make sense?
Post by iON
The solutions i've seen so far has the disadvatage
of cancel the whole line containing then # character!
Then they would not be "solutions" if their objective were not to do
that. Is it?
Post by iON
Here is a quick'n'dirty awk solution that will remove all comments
AFTER the # character and ALSO skip empty lines.
(awk -F# {'print $1'} |grep -v "^$" |while read X; do iptables -A
FORWARD -j DROP -s $X; done)<bannade.txt
You forgotto skip lines containing only white space.

What on earth is your problem with

| egrep -v '^ *$' | sed -e 's/#.*//' | ...

(apart from the obvious problem that it messes up lines that contain a
quoted '#', like this one does - but then do does your longer
non-"solution").

Personally, I would be parsing, like so:

while read line; do
[ -n "$line" ] || continue
set -- $line
line=""
[ -n "$1" ] || continue
for word; do
case "$word" in
#*) break; ;;
*) line="${line}${line:+' '}$word" ;;
esac
done
echo "$line"
done | ...

Since this has the advantage of getting (or being able to get, if you
wanted it to) all embedded interpretations right.

Peter
Stephane CHAZELAS
2004-12-01 14:53:37 UTC
Permalink
2004-12-1, 15:38(+01), Peter T. Breuer:
[...]
Post by Peter T. Breuer
while read line; do
Beware of the special parsing of read regarding leading and
trailing blanks and regarding backslashes if you leave IFS at
its default value and don't use the -r option.
Post by Peter T. Breuer
[ -n "$line" ] || continue
set -- $line
You need to disable word splitting (or, for instance a * on the
line would be expanded to the list of files in the current
directory).

set -f
set -- $line
Post by Peter T. Breuer
line=""
[ -n "$1" ] || continue
$1 can't be empty there as $line is not empty and doesn't start
nor ends with blanks.
--
Stephane
Peter T. Breuer
2004-12-01 15:13:09 UTC
Permalink
Post by Stephane CHAZELAS
[...]
Post by Peter T. Breuer
while read line; do
Beware of the special parsing of read regarding leading and
trailing blanks and regarding backslashes if you leave IFS at
its default value and don't use the -r option.
Hmm .. I intended white space to be squeezed and normalized (and
leading and trailing white space removed), since this is a parser. All
I want to do is get the tokens in the line. I think you are saying that
if I do this ("read line") then an escaped end of line will be read as a
line continuation, which may not be what I want.

Fortunately I don't care for myself, but personally my preference would
indeed be for \ to be interpreted as NOT having special signficance, so
that I only get one physical line when I expect to get one line here. I
would indeed rather regard a trailing escape as an error. So I think -r
is appropriate.

But it's a matter of taste.
Post by Stephane CHAZELAS
Post by Peter T. Breuer
[ -n "$line" ] || continue
set -- $line
You need to disable word splitting (or, for instance a * on the
Oh - yes. set -o noglob is probably a good idea at the start.
Post by Stephane CHAZELAS
line would be expanded to the list of files in the current
directory).
set -f
set -- $line
Post by Peter T. Breuer
line=""
[ -n "$1" ] || continue
$1 can't be empty there as $line is not empty and doesn't start
nor ends with blanks.
Defensive programming. I agree. I didn't want to think about the case
when $1 is a single space because of something read has done.


Peter
Stephane CHAZELAS
2004-12-01 15:39:50 UTC
Permalink
Post by Peter T. Breuer
Post by Stephane CHAZELAS
[...]
Post by Peter T. Breuer
while read line; do
Beware of the special parsing of read regarding leading and
trailing blanks and regarding backslashes if you leave IFS at
its default value and don't use the -r option.
Hmm .. I intended white space to be squeezed and normalized (and
leading and trailing white space removed), since this is a parser. All
I want to do is get the tokens in the line. I think you are saying that
if I do this ("read line") then an escaped end of line will be read as a
line continuation, which may not be what I want.
Not only

If the input is "foo\b\\ar", then line will be "foob\ar".

[...]
Post by Peter T. Breuer
Post by Stephane CHAZELAS
Post by Peter T. Breuer
[ -n "$line" ] || continue
set -- $line
You need to disable word splitting (or, for instance a * on the
Oh - yes. set -o noglob is probably a good idea at the start.
Or

set -f

for better portability.

[...]
Post by Peter T. Breuer
Post by Stephane CHAZELAS
Post by Peter T. Breuer
line=""
[ -n "$1" ] || continue
$1 can't be empty there as $line is not empty and doesn't start
nor ends with blanks.
Defensive programming. I agree. I didn't want to think about the case
when $1 is a single space because of something read has done.
[...]

I can only agree with that. But that may have indicated that you
wasn't aware of read special behavior.
--
Stephane
Peter T. Breuer
2004-12-01 15:52:31 UTC
Permalink
Post by Stephane CHAZELAS
Post by Peter T. Breuer
Post by Stephane CHAZELAS
[...]
Post by Peter T. Breuer
while read line; do
Beware of the special parsing of read regarding leading and
trailing blanks and regarding backslashes if you leave IFS at
its default value and don't use the -r option.
Hmm .. I intended white space to be squeezed and normalized (and
leading and trailing white space removed), since this is a parser. All
I want to do is get the tokens in the line. I think you are saying that
if I do this ("read line") then an escaped end of line will be read as a
line continuation, which may not be what I want.
Not only
If the input is "foo\b\\ar", then line will be "foob\ar".
Well, yes, escapes will be interpreted. It is a question of whether you
want that or not. I don't know whether it is wanted or not. It's only a
question of taste for me from where I sit.
Post by Stephane CHAZELAS
Post by Peter T. Breuer
Post by Stephane CHAZELAS
You need to disable word splitting (or, for instance a * on the
Oh - yes. set -o noglob is probably a good idea at the start.
Or
set -f
for better portability.
Probably.
Post by Stephane CHAZELAS
Post by Peter T. Breuer
Post by Stephane CHAZELAS
Post by Peter T. Breuer
line=""
[ -n "$1" ] || continue
$1 can't be empty there as $line is not empty and doesn't start
nor ends with blanks.
Defensive programming. I agree. I didn't want to think about the case
when $1 is a single space because of something read has done.
[...]
I can only agree with that. But that may have indicated that you
wasn't aware of read special behavior.
Indeed, I am not clear on precisely what read does in the case there is
a line consisting only of spaces and tabs and stuff like that or other
even weirder stuff. I would have to check.

And you will note I forgot to discard lines that were empty after
having their comment discarded. Shrug. Fix for you.


Peer

Loading...