Regexes: best practices and gotchas
Some tips on regexes and grammars
To help with robust regexes and grammars, here are some best practices for code layout and readability, what to actually match, and avoiding common pitfalls.
:sigspace adverb, whitespace is not significant in Raku regexes. Use that to your own advantage and insert whitespace where it increases readability. Also, insert comments where necessary.
Compare the very compact
to the more readable
As a rule of thumb, use whitespace around atoms and inside groups; put quantifiers directly after the atom; and vertically align opening and closing square brackets and parentheses.
When you use a list of alternations inside parentheses or square brackets, align the vertical bars:
Keep it small
Regexes are often more compact than regular code. Because they do so much with so little, keep regexes short.
When you can name a part of a regex, it's usually best to put it into a separate, named regex.
For example, you could take the float regex from earlier:
And decompose it into parts:
That helps, especially when the regex becomes more complicated. For example, you might want to make the decimal point optional in the presence of an exponent.
What to match
Often the input data format has no clear-cut specification, or the specification is not known to the programmer. Then, it's good to be liberal in what you expect, but only so long as there are no possible ambiguities.
For example, in
What can be inside the section header? Allowing only a word might be too restrictive. Somebody might write
[two words], or use dashes, etc. Instead of asking what's allowed on the inside, it might be worth asking instead: what's not allowed?
Clearly, closing square brackets are not allowed, because
[a]b] would be ambiguous. By the same argument, opening square brackets should be forbidden. This leaves us with
which is fine if you are only processing one line. But if you're processing a whole file, suddenly the regex parses
[with anewline in between]
which might not be a good idea. A compromise would be
and then, in the post-processing, strip leading and trailing spaces and tabs from the section header.
:sigspace adverb (or using the
rule declarator instead of
regex) is very handy for implicitly parsing whitespace that can appear in many places.
Going back to the example of parsing
ini files, we have
which is probably not as liberal as we want it to be, since the user might put spaces around the equals sign. So, then we may try this:
But that's looking unwieldy, so we try something else:
But wait! The implicit whitespace matching after the value uses up all whitespace, including newline characters, so the
\n+ doesn't have anything left to match (and
rule also disables backtracking, so no luck there).
Therefore, it's important to redefine your definition of implicit whitespace to whitespace that is not significant in the input format.
This works by redefining the token
ws; however, it only works for grammars:
my =say so IniFormat.parse();
Besides putting all regexes into a grammar and turning them into tokens (because they don't need to backtrack anyway), the interesting new bit is
which gets called for implicit whitespace parsing. It matches when it's not between two word characters (
<!ww> , negated "within word" assertion), and zero or more horizontal space characters. The limitation to horizontal whitespace is important, because newlines (which are vertical whitespace) delimit records and shouldn't be matched implicitly.
Still, there's some whitespace-related trouble lurking. The regex
\n+ won't match a string like
"\n \n", because there's a blank between the two newlines. To allow such input strings, replace