diff options
| author | Michael Peter Christen <mc@yacy.net> | 2021-08-17 15:23:21 +0200 |
|---|---|---|
| committer | Michael Peter Christen <mc@yacy.net> | 2021-08-17 15:23:21 +0200 |
| commit | e6a87e0426a68f25207db9403ffec5a9a700da20 (patch) | |
| tree | c5244811f730c58726ff6083cced6f7024f0a9d5 /.classpath | |
| parent | e9c5e78868d8185843adad0ca0c943b88914c084 (diff) | |
enhanced crawler
a main problem when crawling is long waiting time cuased by crawl-delay
values from robots.txt entries. that attribute is not supported by
google and interpreted by yandex and bing in different ways. In large
crawls there is always one host which blocks the whole crawl with
extreme large values. YaCy now still obeys crawl-delay but limits them
to 10 seconds.
Additionally the blocking logic when loading new robots.txt was analyzed
and a deadlock was removed. Furthermore the construction of new queue
lists was redesigned and it was ensured that always a large list of
different hosts for host-balancing is provided for the loader.
Diffstat (limited to '.classpath')
| -rw-r--r-- | .classpath | 2 |
1 files changed, 1 insertions, 1 deletions
diff --git a/.classpath b/.classpath index 3f4b3e9bc..36153ff3e 100644 --- a/.classpath +++ b/.classpath @@ -9,7 +9,6 @@ <attribute name="module" value="true"/> </attributes> </classpathentry> - <classpathentry kind="lib" path="lib/J7Zip-modified.jar"/> <classpathentry kind="lib" path="lib/apache-mime4j-0.6.jar"/> <classpathentry kind="lib" path="lib/avatica-core-1.13.0.jar"/> <classpathentry kind="lib" path="lib/bcmail-jdk15on-1.60.jar"/> @@ -128,5 +127,6 @@ <classpathentry kind="src" path="htroot/yacy"/> <classpathentry kind="src" path="source"/> <classpathentry kind="src" path="test/java"/> + <classpathentry kind="lib" path="lib/J7Zip-modified.jar"/> <classpathentry kind="output" path="gen"/> </classpath> |