Difference between revisions of "Regular expressions"

Revision as of 11:05, 1 July 2012

Regular Expressions (regex) are essentially a search engine for finding patterns in a text. While the syntax is a bit tricky to learn, regex will save tons of time and effort in the long run. Many of you are probably familiar with regex, even if only through the use of wildcards. Wildcard notation, such as *.html, matches to all html files in the given search directory. Regex takes this idea and expands on it dramatically, allowing for very complicated search patterns. A regular expression to find all html files in a given directory would be .*\.html$

It's important to note that regular expressions rely very heavily on context. Each character has the potential to mean something entirely different depending on it's position in the expression. This is a relatively hard concept for beginners to understand. Just keep at it and, with practice, you'll be a keyboard cowboy in no time.

Syntax

The following information pertains to the Perl 5 Regex engine. Different engines have slightly different syntax

Characters

Literals

The most basic regex is a literal character. A literal character, such as a matches a in the string alex. However, in a string such as adam, it will only match the first a, before the 'd', unless you tell the regex engine otherwise. Most text editors that have a 'find' function, also have a 'find next' function.

Similarly, a regex search for hat in the string Blackhat Academy will return 'hat' from the end of the first word. This is merely a string of literal characters, and the regex engine handles it the same way as it handles a single literal character.

Metacharacters

Regex wouldn't really be that useful if we just wanted to match literal strings of text. That's where special characters come in to play. Each metacharacter has it's own use, and in order to use any of them as literal characters, you need to escape them using the \ (backslash) key.

There are 11 metacharacters we'll use:

• Opening square bracket [
• Backslash \
• Caret ^
• Dollar Sign $
• Dot .
• Pipe |
• Question mark ?
• Asterik *
• Plus +
• Open round bracket (
• Close round bracket )

Take 1+1=2 for example. The correct regex to match this string is 1\+1=2. It's important to note that 1+1=2 is still valid regex, but it wouldn't match 1+1=2. Instead, it would match 111=2 in the string 111=25*4+11. This is the a great example of how to properly escape special characters when you want to find their literal character counterpart, but how it might not always throw an error, even if you don't escape your special characters.

Non-Printable

Non-printable characters are any ASCII or ANSI character not represented in the standard A-z0-9 character set. Popular non-printable characters include the tab character, represented by \t or \x09, the carriage return (\r or \x0D), and the new line (\n or \0A). These are by no means the only non printable characters you can use in regular expressions. In fact, you can use any ASCII or ANSI character code for the character set you are working with.

Being able to use the ASCII character codes means that even if you're trying to match a character that isn't on your keyboard, you can still easily match it.

Character Classes (Sets)

Character classes, put simply, allow regex engines to match only one out of several provided characters. A character class is denoted by square brackets around the character variations that you're looking for. For instance, if you want to find all instances of the word grey, both American and British, in a document, you could search for gr[ae]y. This would match either gray or grey. It will not, however, match graay or graey.

You can also use hyphens inside a character class to represent a range of characters, such as [0-9] to represent any digit. Similarly, [a-z] would match any single, lowercase letter, and [A-Z] would match any single, uppercase letter. More importantly, perhaps, is that you can chain these together. For instance, if I wanted to match any AlphaNumeric character, I could search for [a-zA-Z]. If I wanted to find any single hexadecimal digit, I could search for [a-fA-F0-9].

Negated Character Classes

Negating a character class is done by typing a caret ^ after the opening bracket. Negating a character class means it will match any character that's not in the defined character class. It's important to note that even negated character classes have to match some character. A regular expression foo[^b] does not mean "match any instance of foo not followed by b," but rather "match any instance of foo followed by any character that isn't a b.

Metacharacters

The only metacharacters inside character classes are the closing bracket ], the backslash \, the caret ^ and the hyphen -. All the other metacharacters are treated as literals inside a character class and do not need to be escaped with a backslash. For instance, if you wanted to find a + or a *, you could use regex that looks like [+*].

If you want to include a backslash in your character class, you need to escape it with another backslash, [\\+] matches a backslash or a plus sign. The other metacharacters that character classes use can be included as a literal inside the class as long as they are in a position that doesn't match it's purpose. For instance, to search for a closing bracket, a hyphen, or a caret, you could write a regular expression that looks like []^-]. This is a little confusing for some, and if you'd prefer, you can write it by escaping the metacharacters that you want to search for. This decreases readability for those less skilled, but increases consistency. To search for a closing bracket, a hyphen, or a caret, using escapes, you could write [\^\-\]]. Both methods are entirely valid, and it's entirely up to the user to decide how to write it.

The exception to using the backslash to escape metacharacters inside character classes occurs when using POSIX regular expressions. POSIX treats backslashes as a literal inside character classes, so to use the other metacharacters, you would want to use the ASCII method as mentioned earlier.

Shorthand

Character classes are made easier by the shorthand that has been developed for them. Since many character classes use used frequently, such as [0-9], a shorter notation is used like [\d]. The \d shorthand stands for the digits, 0-9. The \w shorthand stands for word, and contains the character class [a-zA-Z0-9_]. The shorthand \s contains the whitespace characters. This shorthand varies from flavor to flavor, but in the Perl engine, it matches [ \t\r\n].

Shorthand can be used both inside and outside of character classes. \s\d matches a space followed by a digit, while [\s\d] will match either a whitespace character or a digit.

Negated Shorthand

Shorthand also has negated versions. Just like you can negate a character class by using ^, as in [^a-z], you can negate the shorthand outside of character classes using \D (negates \d), \W (negates \w), and \S (negates \s). Be careful not to mix up [^\s\d] and [\S\D]. These are not the same things. [^\s\d] will match any character that is not whitespace and not a digit. But [\S\D] will match any character that's not whitespace or not a digit. Since whitespace is not a digit, and digits are not whitespace, this regex will match any character, whether it's digit, whitespace, or otherwise.

Repeating Character Classes

Repeating character classes can be useful depending on the situation. You can repeat a character class by using ?, * or +. Say I wanted to isolate a set of numbers that are not 0 or 1 inside a larger file, without caring how long the string of numbers may be. The best way to do this is to repeat a character class, [2-9]+.

Dot

The dot (.) is the "Almost Anything" character of regular expressions. The dot matches a single character, including anything other than newlines. By default, the dot is essentially shorthand for the negated character class [^\n]. It is easily one of the most used metacharacters in regular expressions.

Since the dot doesn't normally match the new line character, modern tools and languages have an option to allow the dot to match new lines. The reason that the dot doesn't include new line characters by default is because in the original regex engines, files were read line-by-line to find your search, causing the engine to never encounter a newline character, as it only applied the regex to each individual line.

In Perl, you can match new lines with the dot using a feature called single-line mode. This is not to be confused with multi-line mode, which only effects anchors.

Anchors

Anchors are a whole new concept in regular expressions. Instead of matching characters, they match positions. ^ matches the position before the first character in the line, while $ matches the position after the last character in the line. For instance, searching for ^f in the string foobar will match "f", but not any further. Applying ^b to the same string would return no results, since b is not the first character in the string. Similarly, r$ matches "r", but o$ applied to the same string doesn't match anything.

Permanent Anchors

While ^ and $ match to the start and end of lines, respectively. \A matches the very start of the string, and \Z matches the very end of the string.

Word Boundaries

Alternation

Quantifiers

Tools

Utilities

• grep
• grepWin
• RegexPal

Programming Languages

• Gnulib
• Java
• JavaScript
• .NET
• Perl
• PHP
• PowerShell
• Python
• Ruby

Databases

• MySQL
• Oracle
• PostgreSQL

chart

there's a little chart on the sqli page of regex stuff at Sqli#Using_Regular_Expressions_for_Boolean_enumeration