Perl compatible regular expression (PCRE) in Python

I have to parse some strings based on PCRE in Python, and I've no idea how to do that.

Strings I want to parse looks like:

match mysql m/^.n(4.[-.w]+).../s p/MySQL/ i/$1/

In this example, I have to get this different items:

"m/^.n(4.[-.w]+).../s" ; "p/MySQL/" ; "i/$1/"

The only thing I've found relating to PCRE manipulation in Python is this module: http://pydoc.org/2.2.3/pcre.html (but it's written it's a .so file ...)

Do you know if some Python module exists to parse this kind of string?

Just to be clear: You want to match PCREs, not use them?
– Tim Pietzcker
Aug 15 '11 at 9:37

I have to use them
– Steeve
Aug 15 '11 at 9:41

OK, so in other words, you need someone/something to translate PCRE regexes into Python regexes? Take a look at RegexBuddy.
– Tim Pietzcker
Aug 15 '11 at 9:43

Then you're out of luck. There is nothing like RegexBuddy out there that's free. Think about your time. Is it free?
– Tim Pietzcker
Aug 15 '11 at 9:52

There are some really subtle issues with how Python deals with, or fails to deal with, non-ASCII in regex patterns and strings, which vary substantially according to the version of Python you are using and whether you have a “wide build”. Python3 w/a wide build works best and Python2 w/a narrow build works worst, but all are still pretty far from Perl regexes vis‐à‐vis Unicode. Matthew Barnett’s regex module for python 2&3 alike is much, much better and will prolly replace re eventually. See also my 3rd OSCON talk.
– tchrist
Aug 15 '11 at 13:24

re

3 Answers
3

Be Especially Careful with non‐ASCII in Python

There are some really subtle issues with how Python deals with, or fails to deal with, non-ASCII in patterns and strings. Worse, these disparities vary substantially according, not just to which version of Python you are using, but also whether you have a “wide build”.

In general, when you’re doing Unicode stuff, Python 3 with a wide build works best and Python 2 with a narrow build works worst, but all combinations are still a pretty far cry far from how Perl regexes work vis‐à‐vis Unicode. If you’re looking for ᴘᴄʀᴇ patterns in Python, you may have to look a bit further afield than its old re module.

re

The vexing “wide-build” issues have finally been fixed once and for all — provided you use a sufficiently advanced release of Python. Here’s an excerpt from the v3.3 release notes:

Changes introduced by PEP 393 are the following:

len()

len('U0010FFFF') == 1

'uDBFFuDFFF' != 'U0010FFFF'

'U0010FFFF'[0]

'U0010FFFF'

'uDBFF'

sys.maxunicode

PyUnicode_GetMax()

The ./configure

--with-wide-unicode

In contrast to what’s currently available in the standard Python distribution’s re library, Matthew Barnett’s regex module for both Python 2 and Python 3 alike is much, much better in pretty much all possible ways and will quite probably replace re eventually. Its particular relevance to your question is that his regex library is far more ᴘᴄʀᴇ (i.e. it’s much more Perl‐compatible) in every way than re now is, which will make porting Perl regexes to Python easier for you. Because it is a ground‐up rewrite (as in from‐scratch, not as in hamburger :), it was written with non-ASCII in mind, which re was not.

re

regex

re

regex

re

The regex library therefore much more closely follows the (current) recommendations of UTS#18: Unicode Regular Expressions in how it approaches things. It meets or exceeds the UTS#18 Level 1 requirements in most if not all regards, something you normally have to use the ICU regex library or Perl itself for — or if you are especially courageous, the new Java 7 update to its regexes, as that also conforms to the Level One requirements from UTS#18.

regex

Beyond meeting those Level One requirements, which are all absolutely essential for basic Unicode support, but which are not met by Python’s current re library, the awesome regex library also meets the Level Two requirements for RL2.5 Named Characters (N{...})), RL2.2 Extended Grapheme Clusters (X), and the new RL2.7 on Full Properties from revision 14 of UTS#18.

re

regex

N{...})

X

Matthew’s regex module also does Unicode casefolding so that case insensitive matches work reliably on Unicode, which re does not.

regex

re

The following is no longer true, because regex now supports full Unicode casefolding, like Perl and Ruby.

regex

One super‐tiny difference is that for now, Perl’s case‐insensitive patterns use full string‐oriented casefolds while his regex module still uses simple single‐char‐oriented casefolds, but this is something he’s looking into. It’s actually a very hard problem, one which apart from Perl, only Ruby even attempts.

regex

Under full casefolding, this means that (for example) "ß" now correct matches "SS", "ss", "ſſ", "ſs" (etc.) when case-insensitive matching is selected. ^{_{(This is admittedly more important in the Greek script than the Latin one.)}}

"ß"

"SS"

"ss"

"ſſ"

"ſs"

See also the slides or doc source code from my third OSCON2011 talk entitled “Unicode Support Shootout: The Good, the Bad, and the (mostly) Ugly” for general issues in Unicode support across JavaScript, PHP, Go, Ruby, Python, Java, and Perl. If can’t use either Perl regexes or possibly the ICU regex library (which doesn’t have named captures, alas!), then Matthew’s regex for Python is probably your best shot.

regex

Nᴏᴛᴀ Bᴇɴᴇ s.ᴠ.ᴘ. (= ^{_{s’il vous plaît, et même s’il ne vous plaît pas}} :) The following unsolicited noncommercial nonadvertisement was not actually put here by the author of the Python regex library. :)

regex

The Python regex library has a cornucopeia of superneat features, some of which are found in no other regex system anywhere. These make it very much worth checking out no matter whether you happen to be using it for its ᴘᴄʀᴇ‐ness or its stellar Unicode support.

regex

A few of this module’s outstanding features of interest are:

ismx

(?i:foo)

(?-i:foo)

agrep

glimpse

L<list>

m

M

R

(w+s+)+

@+

@-

(?|...|...|...|)

w

b

s

X

G

re

Ok, that’s enough hype. :)

Yet Another Fine Alternate Regex Engine

One final alternative that is worth looking at if you are a regex geek is the Python library bindings to Russ Cox’s awesome RE2 library. It also supports Unicode natively, including simple char‐based casefolding, and unlike re it notably provides for both the Unicode General Category and the Unicode Script character properties, which are the two key properties you most often need for the simpler kinds of Unicode processing.

re

Although RE2 misses out on a few Unicode features like N{...} named character support found in ICU, Perl, and Python, it has extremely serious computational advantages that make it the regex engine of choice whenever you’re concern with starvation‐based denial‐of‐service attacks through regexes in web queries and such. It manages this by forbidding backreferences, which cause a regex to stop being regular and risk super‐exponential explosions in time and space.

N{...}

Library bindings for RE2 are available not just for C/C++ and Python, but also for Perl and most especially for Go, where it is slated to very shortly replace the standard regex library there.

You're looking for '(w/[^/]+/w*)'.

'(w/[^/]+/w*)'

Used like so,

import re x = re.compile('(w/[^/]+/w*)') s = 'match mysql m/^.n(4.[-.w]+).../s p/MySQL/ i/$1/' y = x.findall(s) # y = ['m/^.x00x00x00n(4.[-.w]+)x00...x00/s', 'p/MySQL/', 'i/$1/']

Found it while playing with Edi Weitz's Regex Coach, so thanks to the comments to the question which made me remember its existence.

Since you want to run PCRE regexes, and Python's re module has diverged from its original PCRE origins, you may also want to check out Arkadiusz Wahlig's Python bindings for PCRE. That way you'll have access to native PCRE and won't need to translate between regex flavors.

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Search This Blog

Mgiyuk