Emulation of lex like functionality in Perl or Python

Here's the deal. Is there a way to have strings tokenized in a line based on multiple regexes?

One example:

I have to get all href tags, their corresponding text and some other text based on a different regex. So I have 3 expressions and would like to tokenize the line and extract tokens of text matching every expression.

I have actually done this using flex (not to be confused with Adobe), which is an implementation of the good old lex. lex provides an elegant way to do this by executing "actions" based on expressions. One can control the way lex reading a file too (block / line based read).

The problem is that flex actually produces C/ C++ code which actually does the tokenizing job. I have a make file which wraps all these things. I was wondering if perl /python can in some way do the same thing. Its just that I would like to do everything I like in a single programming language itself.

Tokenizing is just one of the things that I want to do as part of my application.

Apart from perl or python can any language (functional also) do this?

I did read about PLY and ANTLR here (Parsing, where can I learn about it).

But is there a way to do it naturally in python itself? pardon my ignorance, but are these tools used in any popular products / services?

Thank you.

Asked by: Patrick886 | Posted: 28-01-2022

Answer 1

Look at documentation for following modules on CPAN





I've used these modules to process quite large and complex web-pages.

Answered by: Lydia312 | Posted: 01-03-2022

Answer 2

If you're specifically after parsing links out of web-pages, then Perl's WWW::Mechanize module will figure things out for you in a very elegant fashion. Here's a sample program that grabs the first page of Stack Overflow and parses out all the links, printing their text and corresponding URLs:

use strict;
use warnings;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new;


$mech->success or die "Oh no! Couldn't fetch stackoverflow.com";

foreach my $link ($mech->links) {
    print "* [",$link->text, "] points to ", $link->url, "\n";

In the main loop, each $link is a WWW::Mechanize::Link object, so you're not just constrained to getting the text and URL.

All the best,


Answered by: Jack216 | Posted: 01-03-2022

Answer 3

Sounds like you really just want to parse HTML, I recommend looking at any of the wonderful packages for doing so:

Or! You can use a parser like one of the following:

  • PyParsing
  • DParser - A GLR parser with good python bindings.
  • ANTLR - A recursive decent parser generator that can generate python code.

This example is from the BeautifulSoup Documentation:

from BeautifulSoup import BeautifulSoup, SoupStrainer
import re

links = SoupStrainer('a')
[tag for tag in BeautifulSoup(doc, parseOnlyThese=links)]
# [<a href="http://www.bob.com/">success</a>, 
#  <a href="http://www.bob.com/plasma">experiments</a>, 
#  <a href="http://www.boogabooga.net/">BoogaBooga</a>]

linksToBob = SoupStrainer('a', href=re.compile('bob.com/'))
[tag for tag in BeautifulSoup(doc, parseOnlyThese=linksToBob)]
# [<a href="http://www.bob.com/">success</a>, 
#  <a href="http://www.bob.com/plasma">experiments</a>]

Answered by: Roland214 | Posted: 01-03-2022

Answer 4

Have you looked at PyParsing?

From their homepage:

Here is a program to parse "Hello, World!" (or any greeting of the form ", !"):

from pyparsing import Word, alphas
greet = Word( alphas ) + "," + Word( alphas ) + "!" # <-- grammar defined here
hello = "Hello, World!"
print hello, "->", greet.parseString( hello )

The program outputs the following:

Hello, World! -> ['Hello', ',', 'World', '!']

Answered by: Owen169 | Posted: 01-03-2022

Answer 5

If your problem has anything at all to do with web scraping, I recommend looking at Web::Scraper , which provides easy element selection via XPath respectively CSS selectors. I have a (German) talk on Web::Scraper , but if you run it through babelfish or just look at the code samples, that can help you to get a quick overview of the syntax.

Hand-parsing HTML is onerous and won't give you much over using one of the premade HTML parsers. If your HTML is of very limited variation, you can get by by using clever regular expressions, but if you're already breaking out hard-core parser tools, it sounds as if your HTML is far more regular than what is sane to parse with regular expressions.

Answered by: Anna235 | Posted: 01-03-2022

Answer 6

Also check out pQuery it as a really nice Perlish way of doing this kind of stuff....

use pQuery;

pQuery( 'http://www.perl.com' )->find( 'a' )->each( 
    sub {
        my $pQ = pQuery( $_ ); 
        say $pQ->text, ' -> ', $pQ->toHtml;

# prints all HTML anchors on www.perl.com
# =>  link text -> anchor HTML

However if your requirement is beyond HTML/Web then here is the earlier "Hello World!" example in Parse::RecDescent...

use strict;
use warnings;
use Parse::RecDescent;

my $grammar = q{
    alpha : /\w+/
    sep   : /,|\s/
    end   : '!'
    greet : alpha sep alpha end { shift @item; return \@item }

my $parse = Parse::RecDescent->new( $grammar );
my $hello = "Hello, World!";
print "$hello -> @{ $parse->greet( $hello ) }";

# => Hello, World! -> Hello , World !

Probably too much of a large hammer to crack this nut ;-)

Answered by: Arnold656 | Posted: 01-03-2022

Answer 7

From perlop:

A useful idiom for lex -like scanners is /\G.../gc . You can combine several regexps like this to process a string part-by-part, doing different actions depending on which regexp matched. Each regexp tries to match where the previous one leaves off.

      print(" digits"),       redo LOOP if /\G\d+\b[,.;]?\s*/gc;
      print(" lowercase"),    redo LOOP if /\G[a-z]+\b[,.;]?\s*/gc;
      print(" UPPERCASE"),    redo LOOP if /\G[A-Z]+\b[,.;]?\s*/gc;
      print(" Capitalized"),  redo LOOP if /\G[A-Z][a-z]+\b[,.;]?\s*/gc;
      print(" MiXeD"),        redo LOOP if /\G[A-Za-z]+\b[,.;]?\s*/gc;
      print(" alphanumeric"), redo LOOP if /\G[A-Za-z0-9]+\b[,.;]?\s*/gc;
      print(" line-noise"),   redo LOOP if /\G[^A-Za-z0-9]+/gc;
      print ". That's all!\n";

Answered by: Darcy756 | Posted: 01-03-2022

Answer 8

Modifying Bruno's example to include error checking:

my $input = "...";
while (1) {
    if ($input =~ /\G(\w+)/gc) { print "word: '$1'\n"; next }
    if ($input =~ /\G(\s+)/gc) { print "whitespace: '$1'\n"; next }

    if ($input !~ /\G\z/gc)  { print "tokenizing error at character " . pos($input) . "\n" }
    print "done!\n"; last;

(Note that using scalar //g is unfortunately the one place where you really can't avoid using the $1, etc. variables.)

Answered by: Maddie775 | Posted: 01-03-2022

Similar questions

path - mkdir -p functionality in Python

This question already has answers here:

python - Giving anonymous users the same functionality as registered ones

I'm working on an online store in Django (just a basic shopping cart right now), and I'm planning to add functionality for users to mark items as favorite (just like in stackoverflow). Models for the cart look something like this: class Cart(models.Model): user = models.OneToOneField(User) class CartItem(models.Model): cart = models.ForeignKey(Cart) product = models.ForeignKey(Product, verbose_...

python - Why does declaring a descriptor class in the __init__ function break the descriptor functionality?

In class B below I wanted the __set__ function in class A to be called whenever you assign a value to B().a . Instead, setting a value to B().a overwrites B().a with the value. Class C assigning to C().a works correctly, but I wanted to have a separate instance of A for each user class, i.e. I don't want changing 'a' in one instance of C() to change 'a' in al...

python - Remove the "Add" functionality in Django admin

This question already has answers here:

Java equivalent of a Python functionality -> set(string)

I want to mimic a Python functionality in Java. In Python if I want unique characters in a string I can do just, text = "i am a string" print set(text) # o/p is set(['a', ' ', 'g', 'i', 'm', 'n', 's', 'r', 't']) How can I do this in Java trivially or directly?

openssl - Does keyczar python library provide functionality to verify signatures signed using x509 PEM certificates?

I could not find a method to parse x509 pem files. I tried using ParseX509 of utils module which threw up.

heap - How can I implement decrease-key functionality in Python's heapq?

I know it is possible to realize decrease-key functionality in O(log n) but I don't know how?

Easiest way to write a Python program with access to Django database functionality

I have a website which fetches information from RSS feeds periodically (well, currently manually, and this is my problem). This is currently implemented as a normal Django view, which isn't very nice in my opinion. I'd like to have a Python program which is run using a cronjob instead of manually visiting the correct URL to update the information. What is the easiest way to make a Python program have access to my...

How to generate basic CRUD functionality in python given a database tables

I want to develop a desktop application using python with basic crud operation. Is there any library in python that can generate a code for CRUD functionality and user interface given a database table.

python - How to extend the Turbogears 2.1 login functionality

I'm using Turbogears 2.1 and repoze.who/what and am having trouble figuring out how to extend the basic authentication functionality. I am essentially attempting to require users to activate their account via an emailed link before they can login. If they try to login without activating their account, I want to display an appropriate error message. The default Turbogears functionality simply displays one message for all er...

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python