Discussion:
lex / yacc grammar for ksh and/or POSIX sh
(too old to reply)
Kevin Collins
2004-03-04 17:10:46 UTC
Permalink
Hi,

does anyone have lex and yacc grammar files for ksh and/or POSIX
sh? I am trying to build a tool for "indexing" shell scripts/functions
and what other commands, scripts and functions they call.

It seems the best bet I have is to write my own "shell interpreter"
which I can "run" each script with to capture any commands that should
be executed.

I've started this on my own already, but as I am a complete newbie to
lex/yacc, it is a painstaking process :)

Any help would be appreciated - especially if anyone can point me to a
tool that already does some or all of what I want to do.

Thanks,

Kevin
Icarus Sparry
2004-03-05 07:24:59 UTC
Permalink
Post by Kevin Collins
Hi,
does anyone have lex and yacc grammar files for ksh and/or POSIX
sh? I am trying to build a tool for "indexing" shell scripts/functions
and what other commands, scripts and functions they call.
It seems the best bet I have is to write my own "shell interpreter"
which I can "run" each script with to capture any commands that should
be executed.
I've started this on my own already, but as I am a complete newbie to
lex/yacc, it is a painstaking process :)
Any help would be appreciated - especially if anyone can point me to a
tool that already does some or all of what I want to do.
One of the reasons that Plan9 abandoned the traditional shell was that
almost no one knew exactly what it did. In particular it didn't use
YACC/LEX. Its replacement 'rc' is built using YACC.

I am not clear exactly what you are expecting the index to actually do.
Perhaps your best bet would be to take an existing shell source and add
extra print statements to it.

You also should look at the '-n' and '-x' arguments to sh/ksh.

What do you want the following to put into the index

if : ; then echo true
else halt_and_catch_fire
fi

If you want it to work out, in the general case, that halt_and_catch_fire
is never called, then you had better be prepared for a very long wait.
This is a variant on the 'halting problem'. Other things are also
troublesome, for instance if the output of 'date' indicates that it is
Friday the 13th, then you define a function called 'ls' which attempts to
start a nuclear war. Later in the script you have the line 'ls'.

OK, these are extreme examples, but it does point out that this is not an
obvious trivial task.
Kevin Collins
2004-03-05 17:24:00 UTC
Permalink
Post by Icarus Sparry
Post by Kevin Collins
Hi,
does anyone have lex and yacc grammar files for ksh and/or POSIX
sh? I am trying to build a tool for "indexing" shell scripts/functions
and what other commands, scripts and functions they call.
It seems the best bet I have is to write my own "shell interpreter"
which I can "run" each script with to capture any commands that should
be executed.
I've started this on my own already, but as I am a complete newbie to
lex/yacc, it is a painstaking process :)
Any help would be appreciated - especially if anyone can point me to a
tool that already does some or all of what I want to do.
One of the reasons that Plan9 abandoned the traditional shell was that
almost no one knew exactly what it did. In particular it didn't use
YACC/LEX. Its replacement 'rc' is built using YACC.
I am not clear exactly what you are expecting the index to actually do.
Perhaps your best bet would be to take an existing shell source and add
extra print statements to it.
You also should look at the '-n' and '-x' arguments to sh/ksh.
What do you want the following to put into the index
if : ; then echo true
else halt_and_catch_fire
fi
If you want it to work out, in the general case, that halt_and_catch_fire
is never called, then you had better be prepared for a very long wait.
This is a variant on the 'halting problem'. Other things are also
troublesome, for instance if the output of 'date' indicates that it is
Friday the 13th, then you define a function called 'ls' which attempts to
start a nuclear war. Later in the script you have the line 'ls'.
OK, these are extreme examples, but it does point out that this is not an
obvious trivial task.
Good points (I am well versed in -n, -x and -v) - but I am not
expecting to execute any code, only parse. I want to simply collect
the names of all commands executed by a particular script. So, as an
example, this script:

#!/usr/bin/ksh

function Dummy
{
fdate="$(/usr/local/utils/bin/format_date)"

/usr/local/bin/im_a_dummy $fdate
}

eval ./test_cmd \$stuff$(Dummy)

if grep dumb ./more_dumb_stuff
then
cat <<EOF
dumb
dumber
$(cat ./more_dumb_stuff)
EOF

fi

/home/dummy/another_dumb_cmd


would produce the following list:

/usr/local/utils/bin/format_date
/usr/local/bin/im_a_dummy
./test_cmd
Dummy()
grep
cat
cat <-- intentionally there a 2nd time
/home/dummy/another_dumb_cmd

Of course, this script is nonsensical, but it illustrates many of the
possible places you would find a "command" being executed.

The point being that when I need to update "/usr/local/bin/im_a_dummy
", I can (at a glance) get an idea of how many other potential scripts
are affected by generating a report with a "dependency tree".

Now, I can write a Perl script to do all the various parsing logic (in
fact, I had already started that), but I thought it might be simpler
to use lex/yacc since they are designed explicitly for doing this sort
of parsing.

Thanks,

Kevin

Markus Gyger
2004-03-05 09:38:07 UTC
Permalink
Post by Kevin Collins
It seems the best bet I have is to write my own "shell interpreter"
which I can "run" each script with to capture any commands that should
be executed.
ksh93 comes with a command shcomp that creates a binary format file
of a shell script, maybe that would be easier to parse. AFAIK, it's
not documented (other than "shcomp --man") so you would need to get
the ast-ksh source from http://www.research.att.com/sw/download/
(see in src/cmd/ksh93 the files DESIGN, README and sh/shcomp.c).

OTOH, you might want to look at the flat database output of ksh93's
-R option that can be used with command cql (also part of ast-open).


Markus
Loading...