Answer 1

Look at documentation for following modules on CPAN

HTML::TreeBuilder

HTML::TableExtract

and

Parse::RecDescent

I've used these modules to process quite large and complex web-pages.

Answered by: Lydia312 | Posted: 01-03-2022

Answer 2

If you're specifically after parsing links out of web-pages, then Perl's WWW::Mechanize module will figure things out for you in a very elegant fashion. Here's a sample program that grabs the first page of Stack Overflow and parses out all the links, printing their text and corresponding URLs:

#!/usr/bin/perl
use strict;
use warnings;
use WWW::Mechanize;

my $mech = WWW::Mechanize->new;

$mech->get("http://stackoverflow.com/");

$mech->success or die "Oh no! Couldn't fetch stackoverflow.com";

foreach my $link ($mech->links) {
    print "* [",$link->text, "] points to ", $link->url, "\n";
}

In the main loop, each $link is a WWW::Mechanize::Link object, so you're not just constrained to getting the text and URL.

All the best,

Paul

Answered by: Jack216 | Posted: 01-03-2022

Answer 3

Sounds like you really just want to parse HTML, I recommend looking at any of the wonderful packages for doing so:

• BeautifulSoup
• lxml.html
• html5lib

Or! You can use a parser like one of the following:

• PyParsing
• DParser - A GLR parser with good python bindings.
• ANTLR - A recursive decent parser generator that can generate python code.

This example is from the BeautifulSoup Documentation:

from BeautifulSoup import BeautifulSoup, SoupStrainer
import re

links = SoupStrainer('a')
[tag for tag in BeautifulSoup(doc, parseOnlyThese=links)]
# [<a href="http://www.bob.com/">success</a>, 
#  <a href="http://www.bob.com/plasma">experiments</a>, 
#  <a href="http://www.boogabooga.net/">BoogaBooga</a>]

linksToBob = SoupStrainer('a', href=re.compile('bob.com/'))
[tag for tag in BeautifulSoup(doc, parseOnlyThese=linksToBob)]
# [<a href="http://www.bob.com/">success</a>, 
#  <a href="http://www.bob.com/plasma">experiments</a>]

Answered by: Roland214 | Posted: 01-03-2022

Answer 4

Have you looked at PyParsing?

From their homepage:

Here is a program to parse "Hello, World!" (or any greeting of the form ", !"):

from pyparsing import Word, alphas
greet = Word( alphas ) + "," + Word( alphas ) + "!" # <-- grammar defined here
hello = "Hello, World!"
print hello, "->", greet.parseString( hello )

The program outputs the following:

Hello, World! -> ['Hello', ',', 'World', '!']

Answered by: Owen169 | Posted: 01-03-2022

Answer 5

If your problem has anything at all to do with web scraping, I recommend looking at Web::Scraper , which provides easy element selection via XPath respectively CSS selectors. I have a (German) talk on Web::Scraper , but if you run it through babelfish or just look at the code samples, that can help you to get a quick overview of the syntax.

Hand-parsing HTML is onerous and won't give you much over using one of the premade HTML parsers. If your HTML is of very limited variation, you can get by by using clever regular expressions, but if you're already breaking out hard-core parser tools, it sounds as if your HTML is far more regular than what is sane to parse with regular expressions.

Answered by: Anna235 | Posted: 01-03-2022

Answer 6

Also check out pQuery it as a really nice Perlish way of doing this kind of stuff....

use pQuery;

pQuery( 'http://www.perl.com' )->find( 'a' )->each( 
    sub {
        my $pQ = pQuery( $_ ); 
        say $pQ->text, ' -> ', $pQ->toHtml;
    }
);

# prints all HTML anchors on www.perl.com
# =>  link text -> anchor HTML

However if your requirement is beyond HTML/Web then here is the earlier "Hello World!" example in Parse::RecDescent...

use strict;
use warnings;
use Parse::RecDescent;

my $grammar = q{
    alpha : /\w+/
    sep   : /,|\s/
    end   : '!'
    greet : alpha sep alpha end { shift @item; return \@item }
};

my $parse = Parse::RecDescent->new( $grammar );
my $hello = "Hello, World!";
print "$hello -> @{ $parse->greet( $hello ) }";

# => Hello, World! -> Hello , World !

Probably too much of a large hammer to crack this nut ;-)

Answered by: Arnold656 | Posted: 01-03-2022

Answer 7

From perlop:

A useful idiom for lex -like scanners is /\G.../gc . You can combine several regexps like this to process a string part-by-part, doing different actions depending on which regexp matched. Each regexp tries to match where the previous one leaves off.

 LOOP:
    {
      print(" digits"),       redo LOOP if /\G\d+\b[,.;]?\s*/gc;
      print(" lowercase"),    redo LOOP if /\G[a-z]+\b[,.;]?\s*/gc;
      print(" UPPERCASE"),    redo LOOP if /\G[A-Z]+\b[,.;]?\s*/gc;
      print(" Capitalized"),  redo LOOP if /\G[A-Z][a-z]+\b[,.;]?\s*/gc;
      print(" MiXeD"),        redo LOOP if /\G[A-Za-z]+\b[,.;]?\s*/gc;
      print(" alphanumeric"), redo LOOP if /\G[A-Za-z0-9]+\b[,.;]?\s*/gc;
      print(" line-noise"),   redo LOOP if /\G[^A-Za-z0-9]+/gc;
      print ". That's all!\n";
    }

Answered by: Darcy756 | Posted: 01-03-2022

Answer 8

Modifying Bruno's example to include error checking:

my $input = "...";
while (1) {
    if ($input =~ /\G(\w+)/gc) { print "word: '$1'\n"; next }
    if ($input =~ /\G(\s+)/gc) { print "whitespace: '$1'\n"; next }

    if ($input !~ /\G\z/gc)  { print "tokenizing error at character " . pos($input) . "\n" }
    print "done!\n"; last;
}

(Note that using scalar //g is unfortunately the one place where you really can't avoid using the $1, etc. variables.)

Answered by: Maddie775 | Posted: 01-03-2022

Similar questions

path - mkdir -p functionality in Python

This question already has answers here:

---

python - Giving anonymous users the same functionality as registered ones

I'm working on an online store in Django (just a basic shopping cart right now), and I'm planning to add functionality for users to mark items as favorite (just like in stackoverflow). Models for the cart look something like this: class Cart(models.Model): user = models.OneToOneField(User) class CartItem(models.Model): cart = models.ForeignKey(Cart) product = models.ForeignKey(Product, verbose_...

---

python - Why does declaring a descriptor class in the __init__ function break the descriptor functionality?

In class B below I wanted the __set__ function in class A to be called whenever you assign a value to B().a . Instead, setting a value to B().a overwrites B().a with the value. Class C assigning to C().a works correctly, but I wanted to have a separate instance of A for each user class, i.e. I don't want changing 'a' in one instance of C() to change 'a' in al...

---

python - Remove the "Add" functionality in Django admin

This question already has answers here:

---

Java equivalent of a Python functionality -> set(string)

I want to mimic a Python functionality in Java. In Python if I want unique characters in a string I can do just, text = "i am a string" print set(text) # o/p is set(['a', ' ', 'g', 'i', 'm', 'n', 's', 'r', 't']) How can I do this in Java trivially or directly?

---

openssl - Does keyczar python library provide functionality to verify signatures signed using x509 PEM certificates?

I could not find a method to parse x509 pem files. I tried using ParseX509 of utils module which threw up.

---

heap - How can I implement decrease-key functionality in Python's heapq?

I know it is possible to realize decrease-key functionality in O(log n) but I don't know how?

---

Easiest way to write a Python program with access to Django database functionality

I have a website which fetches information from RSS feeds periodically (well, currently manually, and this is my problem). This is currently implemented as a normal Django view, which isn't very nice in my opinion. I'd like to have a Python program which is run using a cronjob instead of manually visiting the correct URL to update the information. What is the easiest way to make a Python program have access to my...

---

How to generate basic CRUD functionality in python given a database tables

I want to develop a desktop application using python with basic crud operation. Is there any library in python that can generate a code for CRUD functionality and user interface given a database table.

---

python - How to extend the Turbogears 2.1 login functionality

I'm using Turbogears 2.1 and repoze.who/what and am having trouble figuring out how to extend the basic authentication functionality. I am essentially attempting to require users to activate their account via an emailed link before they can login. If they try to login without activating their account, I want to display an appropriate error message. The default Turbogears functionality simply displays one message for all er...

---

Still can't find your answer? Check out these communities...

PySlackers | Full Stack Python | NHS Python | Pythonist Cafe | Hacker Earth | Discord Python