13.06.2015 Views

Introduction to the Apache Web Server - ApacheCon

Introduction to the Apache Web Server - ApacheCon

Introduction to the Apache Web Server - ApacheCon

SHOW MORE
SHOW LESS

Create successful ePaper yourself

Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.

Section 18<br />

Spiders<br />

18.1 <strong>Introduction</strong><br />

Spiders, also known as robots, or au<strong>to</strong>mated user agents, or a variety of o<strong>the</strong>r things, are any software which<br />

au<strong>to</strong>matically fetches content from <strong>the</strong> web. This may be done for a variety of different purposes.<br />

• Indexing<br />

• Searching<br />

• Offline browsing<br />

• Testing<br />

• Link checking<br />

• Performance testing (like ab)<br />

18.2 Potential problems<br />

• High server load<br />

• Black holes<br />

• DOS<br />

18.3 Spiders in <strong>the</strong> logs<br />

• altavista.com<br />

• yahoo.com<br />

• google.com<br />

111

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!