13.06.2015 Views

Introduction to the Apache Web Server - ApacheCon

Introduction to the Apache Web Server - ApacheCon

Introduction to the Apache Web Server - ApacheCon

SHOW MORE
SHOW LESS

You also want an ePaper? Increase the reach of your titles

YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.

• etc<br />

• Also, names like ’emailsiphon’<br />

18.4 Excluding spiders from your site<br />

There are a number of ways <strong>to</strong> exclude robots from your site.<br />

18.4.1 robots.txt<br />

Place a file called robots.txt in your DocumentRoot direc<strong>to</strong>ry.<br />

User-agent: *<br />

Disallow: /cgi-bin/<br />

Disallow: /datafiles/<br />

or<br />

User-agent: Scooter<br />

Disallow: /dont-index/<br />

18.4.2 ROBOTS metatag<br />

<br />

• INDEX<br />

• NOINDEX<br />

• FOLLOW<br />

• NOFOLLOW<br />

18.4.3 Yell at <strong>the</strong> opera<strong>to</strong>r<br />

Look up <strong>the</strong> IP address that it is coming from, and email <strong>the</strong> admin at that location.<br />

112

Hooray! Your file is uploaded and ready to be published.

Saved successfully!

Ooh no, something went wrong!