| Age | Commit message (Collapse) | Author |
|
peer-statistic iam parameter, transfer-IP fallback, and matching tests/docs
|
|
reverse proxy
|
|
this removes the userDB-exception in the normal
authentication process which affected mostly wiki
and blog which are not required any more. We also
cleaned up translation messages as this function was
not used at all. This is a required step forward
a renovation of the jetty server.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- DNS errors (NXDOMAIN, SERVFAIL, UnknownHostException) now treated as permanent failures
- Updated default permanentStatus from '404,410' to '404,410,-1'
- Added documentation explaining -1 status code represents DNS/network failures
- Updated UI description in IndexFederated_p.html to reflect DNS error handling
- Affects both transferURL and transferRWI DHT operations
|
|
- Implement proactive DHT error URL rejection for both URL and RWI transfers
- Add configurable opt-out via indexReceiveBlockErrors setting (default: true)
- Introduce retry window for temporary errors (default: 30 days)
- Permanent errors (404, 410) always blocked, configurable via permanentStatus
- Add web UI controls in IndexFederated_p.html under Peer-to-Peer section
- Bidirectional feedback: receivers reject and report error URLs to senders
- Detailed logging shows blocked vs error-blocked counts separately
- Uses load_date_dt field to calculate error age for retry decisions
|
|
outdated LLMs with new ones
|
|
|
|
https://github.com/yacy/yacy_search_server/issues/499
|
|
menu in case that was wanted by the user. Also added the AILab to the main menu.
|
|
|
|
|
|
"PACK" concept.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
see also https://github.com/yacy/yacy_search_server/issues/510
zip is not on the list because it can be parsed
|
|
This was not used at all (as I know) and was blocking a smooth
integration of ivy in the context of an existing JSON parser.
|
|
|
|
|
|
that was created but never used
|
|
|
|
a main problem when crawling is long waiting time cuased by crawl-delay
values from robots.txt entries. that attribute is not supported by
google and interpreted by yandex and bing in different ways. In large
crawls there is always one host which blocks the whole crawl with
extreme large values. YaCy now still obeys crawl-delay but limits them
to 10 seconds.
Additionally the blocking logic when loading new robots.txt was analyzed
and a deadlock was removed. Furthermore the construction of new queue
lists was redesigned and it was ensured that always a large list of
different hosts for host-balancing is provided for the loader.
|
|
We will use the default value for now on.
This is much better for resource economy and fits better into a
container/docker/kubernetes strategy.
Furthermore, a small memory footprint is essential for the usage on
small devices like RaspberryPi.
|
|
variables
To use that feature, set an environment variable with prefix "yacy." and
suffix identical to the yacy configuration attribute name.
Additionaly we implemented a way to set a peer name using the setting
"network.unit.agent". This can therefore now be used to set a peer name
with the java call parameter
-Dyacy.network.unit.agent=anonymous
The purpose for this feature is the ability to set peer names in
mass-deployed kubernetes clusters to the same name to prevent that we
are flooding peer name statistics with auto-deployment-generated names.
|
|
|
|
|
|
This was required because spiders and bots crawled through this page and
created load on the peer without use for the user or the YaCy network.
|
|
|
|
|
|
|
|
|
|
TLS Server Name Indication (SNI) extension activation can now be
configured with the new Settings_p.html?page=httpClient administration
page.
SNI extension is also now enabled by default, as in 2019 the
unrecognized_name(112) alert is more properly handled by major web
servers TLS implementations, following the RFC 6066 standard.
Related YaCy issues : #153 #189 and #272
JDK 1.7 bug :
https://bugs.java.com/bugdatabase/view_bug.do?bug_id=7127374
Apache httpd issue :
https://bz.apache.org/bugzilla/show_bug.cgi?id=56241
RFC 6066 : https://tools.ietf.org/html/rfc6066#section-3
|
|
Acces rate limitations to this search mode by unauthenticated users are
set low by default to prevent unwanted server overload but can be
customized through the SearchAccessRate_p.html configuration page
Fixes #291
|
|
With a new admin page at /SearchAccessRate_p.html in menu Network Access
> Local Search > Access Rate Limitations
|
|
Previously search navigators/facets elements were sorted only by counts.
Now from the ConfigSearchPage_p.html admin page, sort direction
(ascending/descending) and type (on counts or labels) can be customized
independently for each navigator.
|
|
Necessary to prevent blocking the indexing workflow when some
wkhtmltopdf renderings fail without terminating
|
|
|
|
For a finer control over the maximum simultaneously active outgoing
connections.
|
|
Enabled as default, but can be disabled using the "Server Access
Settings" admin page.
|
|
If not interested in displaying this on your search results and notably
on a peer with limited resources this can help saving some CPU and
outgoing network connections.
|