YaCy search server with some extra patches https://git.fennell.dev/yacy/atom?h=master 2023-01-16T13:50:30Z 2023-01-16T13:50:30Z Michael Peter Christen mc@yacy.net 2023-01-16T13:50:30Z urn:sha1:9fcd8f1bdac38beb18e00e7ed37c6532ea5bd416 attention: this is on by default! (it should do the right thing) 2019-05-01T06:54:19Z luccioman luccioman@users.noreply.github.com 2019-05-01T06:54:19Z urn:sha1:6b45cd579922574059e5385153b84be3ca07533b For finer control over which parsed documents can trigger an addition of their links to the crawl stack, complementary to the existing crawl depth parameter. 2018-10-25T08:42:12Z luccioman luccioman@users.noreply.github.com 2018-10-25T08:42:12Z urn:sha1:fcf6b16db4fb064a05a0594d83f4dc5d19255a3b New "Media Type detection" section in the advanced crawl start page allow to choose between : - not loading URLs with unknown or unsupported file extension without checking the actual Media Type (relying Content-Type header for now). This was the old default behavior, faster, but not really accurate. - always cross check URL file extension against the actual Media Type. This lets properly parse URLs ending with an apparently odd file extension, but which have actually a supported Media Type such as text/html. Sample URLs with misleading file extensions added as documentation in the crawl start page. fixes issue #244 2018-06-19T08:12:20Z luccioman luccioman@users.noreply.github.com 2018-06-19T08:12:20Z urn:sha1:cced94298ab946125bc29e58583431ac4dd6a426 This makes possbile to set up much more advanced document crawl filters, by filtering on one or more document indexed fields before inserting in the index. 2018-03-23T09:28:19Z luccioman luccioman@users.noreply.github.com 2018-03-23T09:28:19Z urn:sha1:fb3032c530e5e531a4167691f911d85d7edcf2ad 2014-10-09T11:27:20Z Michael Peter Christen mc@yacy.net 2014-10-09T11:27:20Z urn:sha1:d8beafba3ae1044c87b66b4d339afdfc7323c506 is available in the xml. 2014-10-08T16:48:57Z Michael Peter Christen mc@yacy.net 2014-10-08T16:48:57Z urn:sha1:ec95dfa2e6318b9ea8b7801b564b0cafcf4b6321 status. 2014-10-01T13:34:43Z Michael Peter Christen mc@yacy.net 2014-10-01T13:34:43Z urn:sha1:329262231f30d6d0a23f27c802c8c44c4fa3e08e 2010-09-30T12:50:34Z orbiter orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542 2010-09-30T12:50:34Z urn:sha1:f6eebb6f99ef8fd0874be7bc29065c37fec90621 - nobody understand the auto-dom filter without a lenghtly introduction about the function of a crawler - nobody ever used the auto-dom filter other than with a crawl depth of 1 - the auto-dom filter was buggy since the filter did not survive a restart and then a search index contained waste - the function of the auto-dom filter was in fact to just load a link list from the given start url and then start separate crawls for all these urls restricted by their domain - the new Site Link-List option shows the target urls in real-time during input of the start url (like the robots check) and gives a transparent feed-back what it does before it can be used - the new option also fits into the easy site-crawl start menu git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7213 6c8d7289-2bf4-0310-a012-ef5d649a1542 2008-11-14T09:58:56Z orbiter orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542 2008-11-14T09:58:56Z urn:sha1:dba7ef51440ae3992260759f58478334af73e8af - removed never-used secondary crawl depth - added a must-not-match filter that can be used to exclude urls from a crawl - added stub for crawl tags which will be used to identify search results that had been produced from specific crawls please update the yacybar: replace property name 'crawlFilter' with 'mustmatch'. Additionally, a new parameter named 'mustnotmatch' can be used, which should be by default the empty sring (match-never) git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@5342 6c8d7289-2bf4-0310-a012-ef5d649a1542