Beginning Web Development With Perl : From Novice to ... - Nabo

CHAPTER 5 ■ LWP MODULES 103For this example, assume that you have a browser object called $browser and proxy servercalled proxy.example.com. If you want to set the HTTP proxy server for use within the program,the invocation of the proxy() method looks like this:$browser->proxy("http","http://proxy.example.com");It’s quite common for a proxy server to be used for URLs that are outside the local network.Inside the network, a proxy server should not be used. For these cases, the LWPincludes a no_proxy() method that accepts a comma-separated list of domains for which noproxy server should be used. Assume that you have a server located at local.example.comfor which you want direct access, as opposed to access through the proxy. The no_proxy()method call looks like this:$browser->no_proxy("local.example.com");Calling no_proxy() with an empty list clears out the list of hosts:$browser->no_proxy();Removing HTML Tags from a PageAs you’ve undoubtedly seen if you’ve followed the examples in this chapter, the content thatcomes back from a GET request is the raw, uncensored HTML (and other language) contentfrom the web server. To say that this is difficult for a human to read and interpret is an understatement.Unfortunately, there is no surefire method for extracting the useful text from a webpage. However, you have some options for retrieving the text from a page.For example, Listing 5-6 shows the Get.pl example shown earlier in the chapter, but modifiedto use HTML::FormatText to produce output that is more human-friendly.Listing 5-6. Using HTML::FormatText to Retrieve the Text from a Page!/usr/bin/perl -wuse strict;use HTML::TreeBuilder;use HTML::FormatText;use LWP::Simple;my $webpage = get("http://www.braingia.org/");my $htmltree = HTML::TreeBuilder->new->parse($webpage);my $output = HTML::FormatText->new();print $output->format($htmltree);