| Age | Commit message (Collapse) | Author |
|---|
|
this removes the userDB-exception in the normal
authentication process which affected mostly wiki
and blog which are not required any more. We also
cleaned up translation messages as this function was
not used at all. This is a required step forward
a renovation of the jetty server.
|
|
This was required because spiders and bots crawled through this page and
created load on the peer without use for the user or the YaCy network.
|
|
|
|
|
|
search queries
|
|
- changed default robots.txt deny list to include some more interface pages because the loading of such pages are a peer load issue for YaCy when crawlers come by and information on these pages are not useful for public search.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7112 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
|
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3874 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
|
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3424 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
|
updating):
- robots.txt is a servlet now
- no need to rewrite the whole file each time a section is added or removed
- user-defined disallows, added manually, won't be overwritten anymore
- new config-setting: httpd.robots.txt, holding names of the disallowed sections
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3423 6c8d7289-2bf4-0310-a012-ef5d649a1542
|
