# All robots will spider the domain User-agent: * Disallow: # Disallow directory /icms/ User-agent: * Disallow: /icms/ Disallow: /pdf/ Disallow: /art/ # You can add more rules here.