YaCy search server with some extra patches https://git.fennell.dev/yacy/atom?h=master 2022-01-09T15:06:14Z 2022-01-09T15:06:14Z Andreas otteresk@users.noreply.github.com 2022-01-09T15:06:14Z urn:sha1:590f39b4037ca0146e944bc35d4d18e1a07aac3a Allow to sort for count and host 2014-04-16T19:34:28Z Michael Peter Christen mc@yacy.net 2014-04-16T19:34:28Z urn:sha1:da86f150ab8339a1ae516b732a40220578979069 This organizes all urls to be loaded in separate queues for each host. Each host separates the crawl depth into it's own queue. The primary rule for urls taken from any queue is, that the crawl depth is minimal. This produces a crawl depth which is identical to the clickdepth. Furthermorem the crawl is able to create a much better balancing over all hosts which is fair to all hosts that are in the queue. This process will create a very large number of files for wide crawls in the QUEUES folder: for each host a directory, for each crawl depth a file inside the directory. A crawl with maxdepth = 4 will be able to create 10.000s of files. To be able to use that many file readers, it was necessary to implement a new index data structure which opens the file only if an access is wanted (OnDemandOpenFileIndex). The usage of such on-demand file reader shall prevent that the number of file pointers is over the system limit, which is usually about 10.000 open files. Some parts of YaCy had to be adopted to handle the crawl depth number correctly. The logging and the IndexCreateQueues servlet had to be adopted to show the crawl queues differently, because the host name is attached to the port on the host to differentiate between http, https, and ftp services. 2014-03-30T23:15:35Z Michael Peter Christen mc@yacy.net 2014-03-30T23:15:35Z urn:sha1:656e2ce62ac0667e700dab8cb12fbe476ce2c7b7 2014-03-18T12:42:31Z Michael Peter Christen mc@yacy.net 2014-03-18T12:42:31Z urn:sha1:92655c7fd96fc5a356631a1924ce8a7faf88215b - adopted all YaCy administration pages to new framework - created new search page layout (working, but still work in progress) - old skin files are fully appliable! (and looking good) - target is a new style based on bootstrap examples, see /test.html - icons in YaCy may be replaced by glyphicons (to be done) 2014-02-10T20:40:42Z reger reger18@arcor.de 2014-02-10T20:40:42Z urn:sha1:365f77ea8c7aa07eb501095e9987f55dd5cd8a0b note also http://bugs.yacy.net/view.php?id=106 2013-08-14T22:05:32Z reger reger18@arcor.de 2013-08-14T22:05:32Z urn:sha1:43348a98a9562f99ed0dea1b6b8943d47213327a 2012-11-05T14:23:03Z Michael Peter Christen mc@yacy.net 2012-11-05T14:23:03Z urn:sha1:5e77801aac1f9181668bff63666f9f6dc6ae8be7 2012-05-06T22:54:00Z Michael Peter Christen mc@yacy.net 2012-05-06T22:54:00Z urn:sha1:dcccbe0be875e392202a20da090726b7198f8792 2012-02-02T20:33:42Z Michael Peter Christen mc@yacy.net 2012-02-02T20:33:42Z urn:sha1:9ad1d8dde25326632f01270f57db54b80803fcb3 ready-prepared crawl list but at the stacks of the domains that are stored for balanced crawling. This affects also the balancer since that does not need to prepare the pre-selected crawl list for monitoring. As a effect: - it is no more possible to see the correct order of next to-be-crawled links, since that depends on the actual state of the balancer stack the next time another url is requested for loading - the balancer works better since the next url can be selected according to the current situation and not according to a pre-selected order.