untangling_the_web

DID: 4046925
UNCLASSIFIEOMj;QR Qj;j;IGIAL l:JSE or.LY
pages may be any number of things, including pages with robots.txt command or
!.§.g. Unindexed pages are identifiable by what they lack: no summary, no page size,
and no cached copy.
WI/WII. statvcuedufrobots.Dd
§i!:r.'L§!J~~9!!~
javangeli st.sni psnap orgfspacefSnipSnapfconfi gfrob...
.~jLr.~U:§}:...r:..9..9.?.9.
WI/WII.atmoswashi ngton .edufrobots .Dd
~jJ.t.!.:.§.r .. E.~{t.~.~
Google Orphans-no cached
copy, no summary, no page size
fichier indiquant auxrobots les endroltsinterdits # # voir http 'n - I T:~nsl"te this 03q8 ]
fichier indiquant aux robots les endroits interdits # # voir
http://info. webcrawler. com/mak/projectslrobots/norobots.html User-agent: • Disallow: n.
www.ann.jussieu.fr/robots.txt - 1k - Ci