the-web-application-hackers-handbook

Chapter 4 n Mapping the Application 89
n Testing for alternative file extensions on identified resources
n Spidering from discovered content
n Automatic fingerprinting of valid and invalid responses to reduce false
positives
All exercises are carried out recursively, with new discovery tasks being
scheduled as new application content is discovered. Figure 4-8 shows a content
discovery session in progress against the EIS application.
Figure 4-8: A content discovery session in progress against the EIS application
TIP The DirBuster project from OWASP is also a useful resource when performing
automated content discovery tasks. It includes large lists of directory
names that have been found in the wild, ordered by frequency of occurrence.
Use of Public Information
The application may contain content and functionality that are not presently linked
from the main content but that have been linked in the past. In this situation,
it is likely that various historical repositories will still contain references to the
hidden content. Two main types of publicly available resources are useful here:
n Search engines such as Google, Yahoo, and MSN. These maintain a finegrained
index of all content that their powerful spiders have discovered,
and also cached copies of much of this content, which persists even after
the original content has been removed.
n Web archives such as the WayBack Machine, located at www.archive.org/.
These archives maintain a historical record of a large number of websites.
In many cases they allow users to browse a fully replicated snapshot of a
given site as it existed at various dates going back several years.