yacy/htroot/robots.txt, branch master YaCy search server with some extra patches https://git.fennell.dev/yacy/atom?h=master 2026-07-11T09:11:06Z cleaned up user authentication 2026-07-11T09:11:06Z Michael Peter Christen mc@yacy.net 2026-07-11T09:11:06Z urn:sha1:b1380c4a9f479de4fc456b6305bee57bc752ad03 this removes the userDB-exception in the normal authentication process which affected mostly wiki and blog which are not required any more. We also cleaned up translation messages as this function was not used at all. This is a required step forward a renovation of the jetty server. turned HostBrowser into a admin-only page, now called IndexBrowser 2020-12-10T23:50:52Z Michael Peter Christen mc@yacy.net 2020-12-10T23:50:52Z urn:sha1:c0d9a3e9a782607acbe90f70abb5c09eb83fea41 This was required because spiders and bots crawled through this page and created load on the peer without use for the user or the YaCy network. disallow remote robots to crawl the HostBrowser servlet 2013-11-26T06:06:25Z Michael Peter Christen mc@yacy.net 2013-11-26T06:06:25Z urn:sha1:3d4b5e66cefaedded443f9c08e2d29c08956e3d2 local robots.txt: disallow external crawlers to follow the URL proxy 2012-08-17T09:47:39Z cominch dominic@luetjohann.com 2012-08-17T09:47:39Z urn:sha1:8a91f4fa4287658193f5ce30db978390f2fb0322 changed local robots.txt to prevent external crawlers to submit random 2012-08-16T15:38:10Z cominch dominic@luetjohann.com 2012-08-16T15:38:10Z urn:sha1:6456a1656a62ca257b217eedb0f367a2a1ba1276 search queries - added a missing entry in YaCy interface robots.txt for bookmarks 2010-09-06T09:58:54Z orbiter orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542 2010-09-06T09:58:54Z urn:sha1:0ab6a462ee4bdca684aa24b19872f2fcc1f53b34 - changed default robots.txt deny list to include some more interface pages because the loading of such pages are a peer load issue for YaCy when crawlers come by and information on these pages are not useful for public search. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7112 6c8d7289-2bf4-0310-a012-ef5d649a1542 - added ViewProfile / Impressum (default on) to local peer's robots.txt 2007-06-12T13:48:27Z karlchenofhell karlchenofhell@6c8d7289-2bf4-0310-a012-ef5d649a1542 2007-06-12T13:48:27Z urn:sha1:669f840eabdd66a9d8c60c904fc90cdd9115b19e git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542 - fix for last commit 2007-03-02T01:50:21Z karlchenofhell karlchenofhell@6c8d7289-2bf4-0310-a012-ef5d649a1542 2007-03-02T01:50:21Z urn:sha1:93e1ad2bca12a69bb6a104e40fd4baeb2a43485c git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3424 6c8d7289-2bf4-0310-a012-ef5d649a1542 - improved version of robots.txt (delete your old htroot/robots.txt before updating): 2007-03-02T01:19:38Z karlchenofhell karlchenofhell@6c8d7289-2bf4-0310-a012-ef5d649a1542 2007-03-02T01:19:38Z urn:sha1:88245e44d86e5612be13376cfaacbc2a9e907cf6 - robots.txt is a servlet now - no need to rewrite the whole file each time a section is added or removed - user-defined disallows, added manually, won't be overwritten anymore - new config-setting: httpd.robots.txt, holding names of the disallowed sections git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542