| Age | Commit message (Collapse) | Author |
|---|
|
fix: scale nodeStack capacity with query offset so late pages aren't …
|
|
fix: HTML-escape reflected user input and crawled document metadata to prevent XSS (#401, #630)
|
|
Fix JS syntax error when system prompt contains single quotes
|
|
WeakPriorityBlockingQueue evicts lowest-scoring entries when full. With a
fixed 150-slot nodeStack, page 2+ results compete against page 1 results
and high-quality late-page candidates get evicted before ranking completes.
Adding query.offset + query.itemsPerPage() to the initial capacity ensures
enough slots exist for the requested page's candidates to survive.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
|
|
The add operation iterated over all selected items, parsed them, and wrote to
disk inline on the servlet thread. For large blacklists this blocked the whole
request. Now: read all item strings from post before returning (thread safety),
hand off the parse+write work to a daemon thread, and redirect immediately.
Failures are logged via ConcurrentLog.warn instead of silently dropped.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
prevent XSS (#630)
dc_title, dc_creator, dc_subject, dc_description, dc_publisher, dc_format, and
dc_identifier were all written with prop.put() and rendered unescaped in HTML
<dd> elements. A malicious page title like <script>... would execute in the
viewer's browser. Switch to prop.putHTML() for all dc_* metadata fields.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
XSS (#401)
In the early return path (no index / search not allowed), prefermaskfilter was
reflected into value="#[prefermaskfilter]#" in yacysearch.html using prop.put(),
allowing attribute-breaking XSS. The normal search path already used putHTML;
apply the same to the early-return path.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
|
thinking across LLM calls
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
|
- using exactly the same search method as normal search
- removed special search method
|
|
|
|
|
|
|
|
|
|
|
|
|
|
localStorage any more
|
|
|
|
|
|
them configurable in the YaCy GUI.
|
|
|
|
|
|
|
|
RAGProxyServlet; also provided a large set of basic tools to handle calculations, date understanding, unit conversion, web fetch, number parsing, self reflection and more
|
|
Persist recrawl job to restart on peer boot
|
|
HTML5 <meta charset="..."> support
|
|
|
|
Fixes mojibake when crawling some websites.
|
|
- DNS errors (NXDOMAIN, SERVFAIL, UnknownHostException) now treated as permanent failures
- Updated default permanentStatus from '404,410' to '404,410,-1'
- Added documentation explaining -1 status code represents DNS/network failures
- Updated UI description in IndexFederated_p.html to reflect DNS error handling
- Affects both transferURL and transferRWI DHT operations
|
|
- Implement proactive DHT error URL rejection for both URL and RWI transfers
- Add configurable opt-out via indexReceiveBlockErrors setting (default: true)
- Introduce retry window for temporary errors (default: 30 days)
- Permanent errors (404, 410) always blocked, configurable via permanentStatus
- Add web UI controls in IndexFederated_p.html under Peer-to-Peer section
- Bidirectional feedback: receivers reject and report error URLs to senders
- Detailed logging shows blocked vs error-blocked counts separately
- Uses load_date_dt field to calculate error age for retry decisions
|
|
Problem:
Failed URLs (404, DNS errors, timeouts) are continuously redistributed via DHT,
causing infinite recrawl loops and network-wide index pollution. No mechanism
exists in YaCy to communicate error status across peers.
Solution - Layer 1 (Proactive Rejection):
- Receiver checks local Solr index for httpstatus_i != 200 BEFORE accepting RWI entries
- Rejects URL immediately if marked as failed previously
- Adds rejected URL hash to errorURL response list
- Prevents index pollution at ingestion time
- Works even if sender doesn't support errorURL protocol (backward compatible)
Solution - Layer 2 (Error Feedback):
- Receiver reports rejected error URLs back to sender via errorURL response parameter
- Sender receives errorURL list and marks those URLs locally as failed
- Sender stops re-distributing these URLs to other peers
- Network-wide error propagation prevents repeated distribution cycles
Implementation Details:
- transferURL.java: Implements proactive rejection + error reporting
* Checks incoming URL against Solr error status before storing
* Collects rejected URL hashes in errorURLs StringBuilder
* Returns errorURL list to sender in response
- Protocol.java: Processes error URL feedback from receiver
* Extracts errorURL from response
* Marks reported URLs locally via crawlQueues.errorURL.push()
* Logs DHT error reports for monitoring
Benefits:
- Dramatically reduces network traffic of broken URLs
- Prevents wasted crawl resources on unreachable targets
- Maintains clean, usable index across distributed network
- Defense-in-depth: two independent layers work together
- Backward compatible: old peers ignore errorURL parameter
Testing:
- Log monitoring shows 'DHT: Received X rejected error URL reports from peer Y'
- Proactive rejection shows 'blocked X URLs' in transfer logs
- Error URLs automatically removed from circulation
|
|
|
|
outdated LLMs with new ones
|
|
|
|
|
|
Use the remote metadata list directly (via addNodes) after
storing/committing remote search results. This bypasses the flaky
RWI→metadata lookup that can drop all results and yield a ZERO COUNT
even when peers return references.
|
