Subject: A note on using RE's matching the emty string
From: Sreenivasa Rao Viswanadha <sreeni@cs.albany.edu>
Date: Sat, 11 Jan 1997 10:43:58 -0500 (EST)


In the last couple of days, we had seen a couple of users facing problems with
regular expressions that match "". There is a minor bug in the way it is
implemented in 0.6.-9. We will fix it.

But the purpose of this mail is to suggest you should be careful when you use
RE's that match the "" string. Consider the following example of string literals
where two consecutive "" are interpreted as the literal " (equivalent to \" in
Java).

< STRING_LITERAL: ( "\"" (~["\""])* "\"" )* >

This will work in general. But, if this a part of a lot of other lexical rules,
then if there a lexical error, say a char is given that cannot be the first one
of any token, then, the lexer decides to use the empty string "" and match it
as STRING_LITERAL without actually giving the lexical error. And since this is
the empty string, no character will be consumed and you will start getting the
same STRING_LITERAL token (with "" as the image) infinte number of times. In
fact, if this was the only lexical rule, then if you give a input that starts
with any char other than the ", you will get into an infinite loop.

So a better alternative is to use the + operator which will not match the empty
string. As a matter of fact, I don't know any practical grammar where matching
"" is useful.

In version 0.5, the lexer generated implicitly treated it as + (which is not
totally right). But in 0.6.-9, it does it right and so there is a chance that
your grammar that used to work with 0.5 will not work with 0.6.-9. So if you
have any top-level lexical rule with ? or *, please change those rules so that
they don't match the empty string "".

Sreeni.