There is a non-standard robots.txt tag Crawl-delay in the non-standard robots.txt file that limits the rate at which some bots crawl a site. At least yahoo and msft honour this. We should set this in the robots.txt that ships with foswiki to prevent problems with resource usage by crawlers hitting a foswiki site.
is this completed for trunk?? -- SvenDowideit - 10 May 2009 any reason not to put this in 1.0.6? -- WillNorris - 10 May 2009 It is in trunk and was there before we created the branch. Closing. -- KennethLavrsen - 10 May 2009
is this completed for trunk?? -- SvenDowideit - 10 May 2009 any reason not to put this in 1.0.6? -- WillNorris - 10 May 2009 It is in trunk and was there before we created the branch. Closing. -- KennethLavrsen - 10 May 2009
The copyright of the content on this website is held by the contributing authors, except where stated elsewhere. see CopyrightStatement. 
