| Age | Commit message (Collapse) | Author |
|---|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- using exactly the same search method as normal search
- removed special search method
|
|
|
|
|
|
|
|
|
|
|
|
|
|
localStorage any more
|
|
|
|
|
|
them configurable in the YaCy GUI.
|
|
|
|
|
|
|
|
RAGProxyServlet; also provided a large set of basic tools to handle calculations, date understanding, unit conversion, web fetch, number parsing, self reflection and more
|
|
Persist recrawl job to restart on peer boot
|
|
HTML5 <meta charset="..."> support
|
|
|
|
Fixes mojibake when crawling some websites.
|
|
- DNS errors (NXDOMAIN, SERVFAIL, UnknownHostException) now treated as permanent failures
- Updated default permanentStatus from '404,410' to '404,410,-1'
- Added documentation explaining -1 status code represents DNS/network failures
- Updated UI description in IndexFederated_p.html to reflect DNS error handling
- Affects both transferURL and transferRWI DHT operations
|
|
- Implement proactive DHT error URL rejection for both URL and RWI transfers
- Add configurable opt-out via indexReceiveBlockErrors setting (default: true)
- Introduce retry window for temporary errors (default: 30 days)
- Permanent errors (404, 410) always blocked, configurable via permanentStatus
- Add web UI controls in IndexFederated_p.html under Peer-to-Peer section
- Bidirectional feedback: receivers reject and report error URLs to senders
- Detailed logging shows blocked vs error-blocked counts separately
- Uses load_date_dt field to calculate error age for retry decisions
|
|
Problem:
Failed URLs (404, DNS errors, timeouts) are continuously redistributed via DHT,
causing infinite recrawl loops and network-wide index pollution. No mechanism
exists in YaCy to communicate error status across peers.
Solution - Layer 1 (Proactive Rejection):
- Receiver checks local Solr index for httpstatus_i != 200 BEFORE accepting RWI entries
- Rejects URL immediately if marked as failed previously
- Adds rejected URL hash to errorURL response list
- Prevents index pollution at ingestion time
- Works even if sender doesn't support errorURL protocol (backward compatible)
Solution - Layer 2 (Error Feedback):
- Receiver reports rejected error URLs back to sender via errorURL response parameter
- Sender receives errorURL list and marks those URLs locally as failed
- Sender stops re-distributing these URLs to other peers
- Network-wide error propagation prevents repeated distribution cycles
Implementation Details:
- transferURL.java: Implements proactive rejection + error reporting
* Checks incoming URL against Solr error status before storing
* Collects rejected URL hashes in errorURLs StringBuilder
* Returns errorURL list to sender in response
- Protocol.java: Processes error URL feedback from receiver
* Extracts errorURL from response
* Marks reported URLs locally via crawlQueues.errorURL.push()
* Logs DHT error reports for monitoring
Benefits:
- Dramatically reduces network traffic of broken URLs
- Prevents wasted crawl resources on unreachable targets
- Maintains clean, usable index across distributed network
- Defense-in-depth: two independent layers work together
- Backward compatible: old peers ignore errorURL parameter
Testing:
- Log monitoring shows 'DHT: Received X rejected error URL reports from peer Y'
- Proactive rejection shows 'blocked X URLs' in transfer logs
- Error URLs automatically removed from circulation
|
|
|
|
outdated LLMs with new ones
|
|
|
|
|
|
Use the remote metadata list directly (via addNodes) after
storing/committing remote search results. This bypasses the flaky
RWI→metadata lookup that can drop all results and yield a ZERO COUNT
even when peers return references.
|
|
https://github.com/yacy/yacy_search_server/issues/438
Use separate one‑time licenses for image thumbnails and full previews to
prevent preview failures after thumbnail loads, and set the Images tab
to request 20 results by default via the content‑domain switch form.
|
|
https://github.com/yacy/yacy_search_server/issues/684
YaCy’s peer queries use HTTP POST and were not following redirects, so
HTTP→HTTPS redirections (e.g., 8090→8443) failed and peers never
switched protocol/port. This adds a redirect strategy that allows POST
redirects (preserving method/body, except for 303), letting remote peer
connections follow HTTPS redirects correctly.
|
|
https://github.com/yacy/yacy_search_server/issues/499
|
|
|
|
menu in case that was wanted by the user. Also added the AILab to the main menu.
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
buttons to chat
|
|
|
