enhanced crawler

a main problem when crawling is long waiting time cuased by crawl-delay
values from robots.txt entries. that attribute is not supported by
google and interpreted by yandex and bing in different ways. In large
crawls there is always one host which blocks the whole crawl with
extreme large values. YaCy now still obeys crawl-delay but limits them
to 10 seconds.
Additionally the blocking logic when loading new robots.txt was analyzed
and a deadlock was removed. Furthermore the construction of new queue
lists was redesigned and it was ensured that always a large list of
different hosts for host-balancing is provided for the loader.

1 files changed, 3 insertions, 3 deletions

diff --git a/defaults/yacy.init b/defaults/yacy.init
index d35d6de8c..bc1992310 100644
--- a/defaults/yacy.init
+++ b/defaults/yacy.init
@@ -699,12 +699,12 @@ collection=user
 50_localcrawl_idlesleep=2000

 50_localcrawl_busysleep=10

 50_localcrawl_memprereq=25165824

-50_localcrawl_loadprereq=6.0

+50_localcrawl_loadprereq=8.0

 50_localcrawl_isPaused=false

 55_autocrawl_idlesleep=10000

 55_autocrawl_busysleep=10000

 55_autocrawl_memprereq=25165824

-55_autocrawl_loadprereq=6.0

+55_autocrawl_loadprereq=8.0

 60_remotecrawlloader_idlesleep=4000

 60_remotecrawlloader_busysleep=800

 60_remotecrawlloader_memprereq=12582912

@@ -740,7 +740,7 @@ collection=user
 reindexSolr_idlesleep=1000

 reindexSolr_busysleep=1

 reindexSolr_memprereq=10485760

-reindexSolr_loadprereq=9.0

+reindexSolr_loadprereq=16.0

 

 # additional attributes:

 # performanceIO is a percent-value. a value of 10 means, that 10% of the busysleep time