Download - Future of the Internet – And how to stop it.

224
Solutions
received: “It is not an official standard backed by a standards body, or owned by
any commercial organisation. It is not enforced by anybody, and there [sic] no
guarantee that all current and future robots will use it. Consider it a common
facility the majority of robot authors offer the WWW community to protect
WWW server [sic] against unwanted accesses by their robots.” 107
Today, nearly all Web programmers know robots.txt is the way in which sites
can signal their intentions to robots, and these intentions are respected by every
major search engine across differing cultures and legal jurisdictions. 108 On this
potentially contentious topic—search engines might well be more valuable if
they indexed everything, especially content marked as something to avoid—
harmony was reached without any application of law. The robots.txt standard
did not address the legalities of search engines and robots; it merely provided a
way to defuse many conflicts before they could even begin. The apparent legal
vulnerabilities of robots.txt—its lack of ownership or backing of a large private
standards setting organization, and the absence of private enforcement devices—may
in fact be essential to its success. 109 Law professor Jody Freeman
and others have written about the increasingly important role played by private
organizations in the formation of standards across a wide range of disciplines
and the ways in which some organizations incorporate governmental notions
of due process in their activities. 110 Many Internet standards have been forged
much less legalistically but still cooperatively. 111
The questions not preempted or settled by such cooperation tend to be
clashes between firms with some income stream in dispute—and where the law
has then partially weighed in. For example, eBay sued data aggregator Bidder’s
Edge for using robots to scrape its site even after eBay clearly objected both in
person and through robots.txt. eBay won in a case that has made it singularly
into most cyberlaw casebooks and even into a few general property casebooks—a
testament to how rarely such disputes enter the legal system. 112
Similarly, the safe harbors of the U.S. Digital Millennium Copyright Act of
1998 give some protection to search engines that point customers to material
that infringes copyright, 113 but they do not shield the actions required to create
the search database in the first place. The act of creating a search engine, like
the act of surfing itself, is something so commonplace that it would be difficult
to imagine deeming it illegal—but this is not to say that search engines rest on
any stronger of a legal basis than the practice of using robots.txt to determine
when it is and is not appropriate to copy and archive a Web site. 114 Only recently,
with Google’s book scanning project, have copyright holders really begun
to test this kind of question. 115 That challenge has arisen over the scanning