# 20050826 SDM - Updated to end spidering of non-content dirs. # Repeated, like cnn.com does, for specific robots. User-agent: * Disallow: / User-agent: GoogleBot Disallow: / User-agent: Slurp Disallow: / User-agent: Scooter Disallow: / User-agent: Ultraseek Disallow: / User-agent: smallbear Disallow: /