the-web-application-hackers-handbook

74 Chapter 4 n Mapping the Application
Enumerating Content and Functionality
In a typical application, the majority of the content and functionality can be
identified via manual browsing. The basic approach is to walk through the
application starting from the main initial page, following every link, and navigating
through all multistage functions (such as user registration or password
resetting). If the application contains a “site map,” this can provide a useful
starting point for enumerating content.
However, to perform a rigorous inspection of the enumerated content, and
to obtain a comprehensive record of everything identified, you must employ
more advanced techniques than simple browsing.
Web Spidering
Various tools can perform automated spidering of websites. These tools work
by requesting a web page, parsing it for links to other content, requesting these
links, and continuing recursively until no new content is discovered.
Building on this basic function, web application spiders attempt to achieve
a higher level of coverage by also parsing HTML forms and submitting these
back to the application using various preset or random values. This can enable
them to walk through multistage functionality and to follow forms-based navigation
(such as where drop-down lists are used as content menus). Some tools
also parse client-side JavaScript to extract URLs pointing to further content.
Numerous free tools are available that do a decent job of enumerating application
content and functionality, including Burp Suite, WebScarab, Zed Attack
Proxy, and CAT (see Chapter 20 for more details).
TIP Many web servers contain a file named robots.txt in the web root that
contains a list of URLs that the site does not want web spiders to visit or search
engines to index. Sometimes, this file contains references to sensitive functionality,
which you are certainly interested in spidering. Some spidering tools
designed for attacking web applications check for the robots.txt file and use
all URLs within it as seeds in the spidering process. In this case, the robots.txt
file may be counterproductive to the security of the web application.
This chapter uses a fictional application, Extreme Internet Shopping (EIS), to
provide examples of common application mapping actions. Figure 4-1 shows
Burp Spider running against EIS. Without logging on, it is possible to map out the
/shop directory and two news articles in the /media directory. Also note that
the robots.txt file shown in the figure references the directories /mdsecportal
and /site-old. These are not linked from anywhere in the application and would
not be indexed by a web spider that only followed links from published content.
TIP Applications that employ REST-style URLs use portions of the URL file
path to uniquely identify data and other resources used within the application