summaryrefslogtreecommitdiff
path: root/htroot/robots.txt
AgeCommit message (Expand)Author
2020-12-11turned HostBrowser into a admin-only page, now called IndexBrowserMichael Peter Christen
2013-11-26disallow remote robots to crawl the HostBrowser servletMichael Peter Christen
2012-08-17local robots.txt: disallow external crawlers to follow the URL proxycominch
2012-08-16changed local robots.txt to prevent external crawlers to submit randomcominch
2010-09-06- added a missing entry in YaCy interface robots.txt for bookmarksorbiter
2007-06-12- added ViewProfile / Impressum (default on) to local peer's robots.txtkarlchenofhell
2007-03-02- fix for last commitkarlchenofhell
2007-03-02- improved version of robots.txt (delete your old htroot/robots.txt before up...karlchenofhell