diff options
| author | orbiter <orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542> | 2010-09-30 12:50:34 +0000 |
|---|---|---|
| committer | orbiter <orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542> | 2010-09-30 12:50:34 +0000 |
| commit | f6eebb6f99ef8fd0874be7bc29065c37fec90621 (patch) | |
| tree | b8555d3a9b5ecb5003986dda0e6d59ff6aebc976 /htroot/CrawlProfileEditor_p.xml | |
| parent | 63e387508ccacb89d43d8e83f074aa21597ffd6c (diff) | |
replaced auto-dom filter with easy-to-understand Site Link-List crawler option
- nobody understand the auto-dom filter without a lenghtly introduction about the function of a crawler
- nobody ever used the auto-dom filter other than with a crawl depth of 1
- the auto-dom filter was buggy since the filter did not survive a restart and then a search index contained waste
- the function of the auto-dom filter was in fact to just load a link list from the given start url and then start separate crawls for all these urls restricted by their domain
- the new Site Link-List option shows the target urls in real-time during input of the start url (like the robots check) and gives a transparent feed-back what it does before it can be used
- the new option also fits into the easy site-crawl start menu
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7213 6c8d7289-2bf4-0310-a012-ef5d649a1542
Diffstat (limited to 'htroot/CrawlProfileEditor_p.xml')
| -rw-r--r-- | htroot/CrawlProfileEditor_p.xml | 1 |
1 files changed, 0 insertions, 1 deletions
diff --git a/htroot/CrawlProfileEditor_p.xml b/htroot/CrawlProfileEditor_p.xml index 5b5f54bee..67a254261 100644 --- a/htroot/CrawlProfileEditor_p.xml +++ b/htroot/CrawlProfileEditor_p.xml @@ -9,7 +9,6 @@ <mustmatch>#[mustmatch]#</mustmatch> <mustnotmatch>#[mustnotmatch]#</mustnotmatch> <crawlingIfOlder>#[crawlingIfOlder]#</crawlingIfOlder> - <crawlingDomFilterDepth>#[crawlingDomFilterDepth]#</crawlingDomFilterDepth> <crawlingDomFilterContent> #{crawlingDomFilterContent}# <item>#[item]#</item> |