diff options
| author | Michael Peter Christen <mc@yacy.net> | 2026-07-10 00:20:29 +0200 |
|---|---|---|
| committer | Michael Peter Christen <mc@yacy.net> | 2026-07-10 00:20:29 +0200 |
| commit | dc84868b8b32e106dd97672040f251bb78f7d9ab (patch) | |
| tree | 2b566a7fc8d6f63d1e3c89cec38c25aefe2e8ec6 | |
| parent | ae8941b245f0e4b6e3c5f04bcac7a3ceb53ac370 (diff) | |
added help files for YaCy UI web pages
152 files changed, 11979 insertions, 0 deletions
diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 000000000..180115c43 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,10 @@ +# Repository Instructions + +## Web UI, API, Help, And Localization + +When changing YaCy web pages or API endpoints, update all matching user-facing and tool-facing artifacts in the same change. + +- For `htroot/**/*.html` changes, update the corresponding localization files under `locales/` when visible text, labels, form controls, messages, or navigation text changes. +- For `htroot/**/*.html` changes, update the corresponding Markdown help file under `help/`. +- For API or servlet behavior changes under `source/net/yacy/htroot/**`, update the related `help/**/*.md` file with changed endpoints, access requirements, parameters, side effects, response fields, and automation guidance. +- Treat `locales` and `help` updates as required checklist items for HTML, servlet, and API changes. Do not leave them for a follow-up unless the change is explicitly internal and has no user-visible page, request parameter, response, or behavior impact. diff --git a/help/AILab.md b/help/AILab.md new file mode 100644 index 000000000..fbd99b695 --- /dev/null +++ b/help/AILab.md @@ -0,0 +1,56 @@ +--- +page: htroot/AILab.html +help: help/AILab.md +title: AI Lab +package: ranking-ai-analysis +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/AILab.java +--- + +# AI Lab + +## Purpose + +AI Lab is a workspace for experimenting with AI-assisted retrieval on top of YaCy's index. + +Use it to test how indexed documents can support model-assisted answers before making the behavior part of a production workflow. + +## What You Can Do Here + +- AI Lab is a workspace for experimenting with AI-assisted retrieval on top of YaCy's index. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/AILab.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- `LLMSelection_p.html` +- `CrawlStartSite.html` +- `IndexPackDownloader_p.html` +- `RAGConfig_p.html` +- `yacychat.html` +- `ToolsConfig_p.html` +- `LogReports_p.html` +- `AIShield_p.html` diff --git a/help/AIShield_p.md b/help/AIShield_p.md new file mode 100644 index 000000000..982fcf443 --- /dev/null +++ b/help/AIShield_p.md @@ -0,0 +1,90 @@ +--- +page: htroot/AIShield_p.html +help: help/AIShield_p.md +title: AI Shield +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/AIShield_p.java +--- + +# AI Shield + +## Purpose + +AI Shield configures safeguards around AI-assisted features. + +Use it to decide which model-assisted operations are allowed and how aggressively the peer should protect prompts, responses, and retrieved context. + +## What You Can Do Here + +- AI Shield configures safeguards around AI-assisted features. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `ai.shield.limit-all` | Limit for all requests, including localhost. | Limit for all requests, including localhost | +| `ai.shield.all.per-minute` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.all.per-hour` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.all.per-day` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.allow-nonlocalhost` | Allow non-localhost clients to access the chat interface. | Allow non-localhost clients to access the chat interface | +| `ai.shield.rate.per-minute` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.rate.per-hour` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.rate.per-day` | Per minute. | Text value; use the page label and surrounding context to choose the exact content. | +| `ai.shield.show-chat-link` | Show a link to yacychat.html on the search front page. | Show a link to yacychat.html on the search front page | + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/AIShield_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/AIShield_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/AIShield_p.html` | `POST` | admin | `source/net/yacy/htroot/AIShield_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `ai.shield.limit-all` | Limit for all requests, including localhost. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.all.per-minute` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.all.per-hour` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.all.per-day` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.allow-nonlocalhost` | Allow non-localhost clients to access the chat interface. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.rate.per-minute` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.rate.per-hour` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.rate.per-day` | Per minute. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.shield.show-chat-link` | Show a link to yacychat.html on the search front page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /AIShield_p.html +Content-Type: application/x-www-form-urlencoded + +ai.shield.limit-all=...&ai.shield.all.per-minute=...&ai.shield.all.per-hour=...&ai.shield.all.per-day=...&ai.shield.allow-nonlocalhost=... +``` + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- Related quality work usually continues on ranking settings, content analysis, LLM selection, RAG configuration, or a representative search result page. diff --git a/help/AccessGrid_p.md b/help/AccessGrid_p.md new file mode 100644 index 000000000..0cb60de2f --- /dev/null +++ b/help/AccessGrid_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/AccessGrid_p.html +help: help/AccessGrid_p.md +title: YaCy Network Access +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/AccessGrid_p.java +--- + +# YaCy Network Access + +## Purpose + +YaCy Network Access visualizes or lists network-facing access information. + +Use it to understand how peer access appears across the YaCy network. + +## What You Can Do Here + +- YaCy Network Access visualizes or lists network-facing access information. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/AccessGrid_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/AccessTracker_p.md b/help/AccessTracker_p.md new file mode 100644 index 000000000..8b3158593 --- /dev/null +++ b/help/AccessTracker_p.md @@ -0,0 +1,67 @@ +--- +page: htroot/AccessTracker_p.html +help: help/AccessTracker_p.md +title: Access Tracker +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/AccessTracker_p.java +--- + +# Access Tracker + +## Purpose + +Access Tracker shows recent requests handled by the peer. + +Use it to understand who is using the interface or API and which paths are active. + +## What You Can Do Here + +- Access Tracker shows recent requests handled by the peer. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/AccessTracker_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/AccessTracker_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/AccessTracker_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/AccessTracker_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `host` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /AccessTracker_p.html?host=...&page=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `yacysearch.html` diff --git a/help/Autocrawl_p.md b/help/Autocrawl_p.md new file mode 100644 index 000000000..50a35d1b6 --- /dev/null +++ b/help/Autocrawl_p.md @@ -0,0 +1,78 @@ +--- +page: htroot/Autocrawl_p.html +help: help/Autocrawl_p.md +title: Crawl Start +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Autocrawl_p.java +--- + +# Crawl Start + +## Purpose + +Autocrawl starts discovery from configured sources without a person submitting every URL manually. + +Use it when YaCy should keep finding new content within controlled boundaries. + +## What You Can Do Here + +- Autocrawl starts discovery from configured sources without a person submitting every URL manually. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `autocrawlEnable` | Enables the named feature. | Text value; use the page label and surrounding context to choose the exact content. | +| `autocrawlShallow` | Enables the named feature. | Text value; use the page label and surrounding context to choose the exact content. | +| `save` | Saves settings. | `Save` | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Autocrawl_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Autocrawl_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Autocrawl_p.html` | `POST` | admin | `source/net/yacy/htroot/Autocrawl_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `autocrawlEnable` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `autocrawlShallow` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `save` | Saves settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /Autocrawl_p.html +Content-Type: application/x-www-form-urlencoded + +autocrawlEnable=...&autocrawlRatio=...&autocrawlRows=...&autocrawlDays=...&autocrawlQuery=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/Automation_p.md b/help/Automation_p.md new file mode 100644 index 000000000..529aa4715 --- /dev/null +++ b/help/Automation_p.md @@ -0,0 +1,93 @@ +--- +page: htroot/Automation_p.html +help: help/Automation_p.md +title: Automation +package: core-search-public +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Automation_p.java +--- + +# Automation + +## Purpose + +Automation lists API calls YaCy recorded while administrators used the web interface. It is the bridge between “I clicked this once” and “I want to repeat this operation deliberately.” + +Use it to inspect, schedule, execute, or delete recorded actions. It is powerful because the stored rows may represent crawls, imports, deletions, configuration changes, or other servlet calls. + +## What You Can Do Here + +- Review API calls recorded from previous YaCy actions. +- Use recorded calls as examples for repeatable automation, not as commands to replay blindly. +- Check endpoint, method, access requirement, and destructive parameters before reusing a call. + +## Page Architecture + +Automation is built around YaCy's recorded API-call table. Each row represents an action that was captured from a servlet request. The page lets an administrator select rows, execute them, delete them, and edit scheduling fields such as next execution date, event trigger, repetition interval, and run frequency. + +## Correct Use + +Treat recorded actions as powerful examples, not as harmless history. Read the endpoint and parameters before executing a row. Pay special attention to actions that delete data, start crawls, change settings, or contact other peers. For scheduled execution, check both the next execution date and the repeat/event fields so the action does not run more often than intended. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Automation_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Automation_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Automation_p.html` | `POST` | admin | `source/net/yacy/htroot/Automation_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `submitNextExecDates` | Result of API execution. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `item_#[count]#` | Choice value. Options: `mark_`. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `date_next_exec_#[pk]#` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `event_select_#[pk]#` | Choice value. Options: `off` = no event, `on` = activate event. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `event_kind_#[pk]#` | Choice value. Options: `off`, `once` = run once, `regular` = run regular. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `event_action_#[pk]#` | Choice value. Options: `startup` = after start-up, `0000` = at 00:00h, `0100` = at 01:00h, `0200` = at 02:00h, `0300` = at 03:00h, `0400` = at 04:00h, `0500` = at 05:00h, `0600` = at 06:00h, `0700` = at 07:00h, `0800` = at 08:00h, `0900` = at 09:00h, `1000` = at 10:00h, `1100` = at 11:00h, `1200` = at 12:00h, `1300` = at 13:00h, `1400` = at 14:00h, `1500` = at 15:00h, `1600` = at 16:00h, `1700` = at 17:00h, `1800` = at 18:00h. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_select_#[pk]#` | Choice value. Options: `off` = no repetition, `on` = activate scheduler. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_time_#[pk]#` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_unit_#[pk]#` | Choice value. Options: `selminutes` = minutes, `selhours` = hours, `seldays` = days. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `execrows` | Result of API execution. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleterows` | Result of API execution. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteoldtime` | Result of API execution. Options: `1` = 1 day, `2` = 2 days, `3` = 3 days, `4` = 4 days, `5` = 5 days, `6` = 6 days, `7` = 1 week, `14` = 2 weeks, `21` = 3 weeks, `30` = 1 month, `60` = 2 months, `90` = 3 months, `180` = 6 months, `270` = 9 months, `365` = 1 year, `730` = 2 years. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `current_pk` | Result of API execution. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `filter` | Filter text or expression used to narrow the displayed records. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `inline` | Result of API execution. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `repeat_time_` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `scheduleeventaction` | Result of API execution. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sort` | Result of API execution. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Automation_p.html +Content-Type: application/x-www-form-urlencoded + +query=...&maximumRecords=...&allswitch=...&submitNextExecDates=...&item_#[count]#=... +``` + +## What To Expect + +Successful changes update the automation table: selected rows may run immediately, disappear after deletion, or receive new scheduling metadata. If an executed action starts a crawl, import, export, or deletion, follow the related monitor page to see the real operational result. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/BlacklistCleaner_p.md b/help/BlacklistCleaner_p.md new file mode 100644 index 000000000..66758e9ca --- /dev/null +++ b/help/BlacklistCleaner_p.md @@ -0,0 +1,86 @@ +--- +page: htroot/BlacklistCleaner_p.html +help: help/BlacklistCleaner_p.md +title: Blacklist Cleaner +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/BlacklistCleaner_p.java +--- + +# Blacklist Cleaner + +## Purpose + +Blacklist Cleaner finds invalid or dangerous blacklist entries. + +Use it after importing or editing lists so malformed patterns do not silently break filtering. + +## What You Can Do Here + +- Blacklist Cleaner finds invalid or dangerous blacklist entries. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `listNames` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Text value; use the page label and surrounding context to choose the exact content. | +| `allowRegex` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex | +| `list` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | `Check` | +| `select#[entry]#` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex | +| `entry#[entry]#` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Text value; use the page label and surrounding context to choose the exact content. | +| `alter` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | `Change Selected` | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete Selected` | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/BlacklistCleaner_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/BlacklistCleaner_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/BlacklistCleaner_p.html` | `POST` | admin | `source/net/yacy/htroot/BlacklistCleaner_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `listNames` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `allowRegex` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `list` | Requests a list view for the selected host or path scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `select#[entry]#` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `entry#[entry]#` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `alter` | Two wildcards in host-part :: ::Either subdomain or wildcard ::Path is invalid Regex ::Wildcard not on begin or end ::Host contains illegal chars ::Double ::Host is invalid Regex. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /BlacklistCleaner_p.html +Content-Type: application/x-www-form-urlencoded + +listNames=...&allowRegex=...&list=...&select#[entry]#=...&entry#[entry]#=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/BlacklistImpExp_p.md b/help/BlacklistImpExp_p.md new file mode 100644 index 000000000..c8d6ae7e8 --- /dev/null +++ b/help/BlacklistImpExp_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/BlacklistImpExp_p.html +help: help/BlacklistImpExp_p.md +title: Blacklist Import +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/BlacklistImpExp_p.java +--- + +# Blacklist Import + +## Purpose + +Blacklist Import/Export moves blacklist rules between files, peers, or installations. + +Use it when maintaining shared policy rather than editing one rule at a time. + +## What You Can Do Here + +- Blacklist Import/Export moves blacklist rules between files, peers, or installations. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Exact YaCy hash value. | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `file` | Selected or uploaded file. | Text value; use the page label and surrounding context to choose the exact content. | +| `listname` | Choice value. Options: all. | all | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/sharedBlacklist_p.html`, `/api/blacklists_p.xml`, `/api/blacklists_p.txt`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/BlacklistImpExp_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/sharedBlacklist_p.html` | `GET` | admin | `source/net/yacy/htroot/sharedBlacklist_p.java` | +| `/api/blacklists_p.xml` | `GET` | admin | `source/net/yacy/htroot/api/blacklists_p.java` | +| `/api/blacklists_p.txt` | `GET` | admin | `source/net/yacy/htroot/api/blacklists_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `listname` | Choice value. Options: all. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /sharedBlacklist_p.html?url=...&hash=...&file=...&listname=...&add=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/BlacklistTest_p.md b/help/BlacklistTest_p.md new file mode 100644 index 000000000..36dd81337 --- /dev/null +++ b/help/BlacklistTest_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/BlacklistTest_p.html +help: help/BlacklistTest_p.md +title: Blacklist Test +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/BlacklistTest_p.java +--- + +# Blacklist Test + +## Purpose + +Blacklist Test checks whether a URL matches the active blacklist rules. + +Use it before blaming the crawler or search page for a missing result. + +## What You Can Do Here + +- Blacklist Test checks whether a URL matches the active blacklist rules. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/BlacklistTest_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/BlacklistTest_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/BlacklistTest_p.html` | `POST` | admin | `source/net/yacy/htroot/BlacklistTest_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /BlacklistTest_p.html +Content-Type: application/x-www-form-urlencoded + +testurl=...&testList=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/Blacklist_p.md b/help/Blacklist_p.md new file mode 100644 index 000000000..c8e08a570 --- /dev/null +++ b/help/Blacklist_p.md @@ -0,0 +1,82 @@ +--- +page: htroot/Blacklist_p.html +help: help/Blacklist_p.md +title: Blacklist Administration +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Blacklist_p.java +--- + +# Blacklist Administration + +## Purpose + +Blacklist Administration edits the URL and host patterns YaCy blocks. + +Use it to prevent crawling, indexing, or displaying unwanted hosts and paths. + +## What You Can Do Here + +- Blacklist Administration edits the URL and host patterns YaCy blocks. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `selectedListName` | Choice value. Options: [not shared::shared]. | [not shared::shared] | +| `action` | Choice value. Options: `editBlacklistEntry` = Edit selected pattern(s), `deleteBlacklistEntry` = Delete selected pattern(s), `moveBlacklistEntry` = Move selected pattern(s) to. | `editBlacklistEntry` = Edit selected pattern(s), `deleteBlacklistEntry` = Delete selected pattern(s), `moveBlacklistEntry` = Move selected pattern(s) to | +| `offset` | Zero-based result offset for pagination. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteList` | Deletion or termination action. Use only with explicit intent. | `Delete this list` | +| `activateList4#[blTypeName]#` | Choice value. Options: `on`. | `on` | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Blacklist_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Blacklist_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Blacklist_p.html` | `POST` | admin | `source/net/yacy/htroot/Blacklist_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `selectedListName` | Choice value. Options: [not shared::shared]. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `action` | Choice value. Options: `editBlacklistEntry` = Edit selected pattern(s), `deleteBlacklistEntry` = Delete selected pattern(s), `moveBlacklistEntry` = Move selected pattern(s) to. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control; use it to request later result pages. | +| `deleteList` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `activateList4#[blTypeName]#` | Choice value. Options: `on`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Blacklist_p.html +Content-Type: application/x-www-form-urlencoded + +selectedListName=...&newListName=...&createNewList=...&newEntry=...&addBlacklistEntry=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- `https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html` diff --git a/help/Blog.md b/help/Blog.md new file mode 100644 index 000000000..406a01d9a --- /dev/null +++ b/help/Blog.md @@ -0,0 +1,93 @@ +--- +page: htroot/Blog.html +help: help/Blog.md +title: Blog +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Blog.java +--- + +# Blog + +## Purpose + +Blog provides a local publication area inside YaCy. + +Use it for peer-local announcements or notes when that feature is enabled. + +## What You Can Do Here + +- Blog provides a local publication area inside YaCy. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `author` | Author. | Text value; use the page label and surrounding context to choose the exact content. | +| `subject` | Subject. | Text value; use the page label and surrounding context to choose the exact content. | +| `content` | Text. | Text value; use the page label and surrounding context to choose the exact content. | +| `commentMode` | Comments. Options: `0` = deactivated, `1` = activated, `2` = moderated. | `0` = deactivated, `1` = activated, `2` = moderated | +| `submit` | Submits the form. | `Submit` | +| `preview` | Author. | `Preview` | +| `discard` | Author. | `Discard` | +| `xmlfile` | Author. | Text value; use the page label and surrounding context to choose the exact content. | +| `importxml` | Author. | `Import` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Blog.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Blog.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/Blog.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `author` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `subject` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `content` | Text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `commentMode` | Comments. Options: `0` = deactivated, `1` = activated, `2` = moderated. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `preview` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discard` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `xmlfile` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `importxml` | Author. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `page` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Blog.html +Content-Type: application/x-www-form-urlencoded + +author=...&subject=...&content=...&commentMode=...&submit=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `BlogComments.html` +- `WikiHelp.html` diff --git a/help/BlogComments.md b/help/BlogComments.md new file mode 100644 index 000000000..474b03327 --- /dev/null +++ b/help/BlogComments.md @@ -0,0 +1,88 @@ +--- +page: htroot/BlogComments.html +help: help/BlogComments.md +title: Blog +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/BlogComments.java +--- + +# Blog + +## Purpose + +Blog Comments stores discussion around local blog posts. + +Use it to review or moderate comments attached to peer-local posts. + +## What You Can Do Here + +- Blog Comments stores discussion around local blog posts. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `author` | Author. | Text value; use the page label and surrounding context to choose the exact content. | +| `subject` | Subject. | Text value; use the page label and surrounding context to choose the exact content. | +| `content` | Text. | Text value; use the page label and surrounding context to choose the exact content. | +| `submit` | Submits the form. | `Submit` | +| `preview` | Author. | `Preview` | +| `view` | Author. | `Discard` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/BlogComments.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/BlogComments.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/BlogComments.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `author` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `subject` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `content` | Text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `preview` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `view` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `allow` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `page` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /BlogComments.html +Content-Type: application/x-www-form-urlencoded + +author=...&subject=...&content=...&submit=...&preview=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `Blog.html` +- `WikiHelp.html` diff --git a/help/Bookmarks.md b/help/Bookmarks.md new file mode 100644 index 000000000..d79f6ead6 --- /dev/null +++ b/help/Bookmarks.md @@ -0,0 +1,102 @@ +--- +page: htroot/Bookmarks.html +help: help/Bookmarks.md +title: Bookmarks +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Bookmarks.java +--- + +# Bookmarks + +## Purpose + +Bookmarks stores useful URLs known to the peer or user. + +Use it to save, organize, and later re-use important search or crawl targets. + +## What You Can Do Here + +- Bookmarks stores useful URLs known to the peer or user. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `title` | Title. | Text value; use the page label and surrounding context to choose the exact content. | +| `description` | Description. | Text value; use the page label and surrounding context to choose the exact content. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | +| `path` | URL path, URL prefix, or host/path scope. In index tools it decides which part of the stored URL tree is inspected or changed. | URL, host/path prefix, or path scope shown by the page. | +| `tags` | Tags (comma separated). | `imported` | +| `public` | Public. Options: `public` = yes, `private` = no. | `public` = yes, `private` = no | +| `feed` | Bookmark is a newsfeed. | `true` = Bookmark is a newsfeed | +| `add` | URL. | `create`, `Save` | +| `xmlfile` | File. | Text value; use the page label and surrounding context to choose the exact content. | +| `importxml` | File. | `import` | +| `htmlfile` | File. | Text value; use the page label and surrounding context to choose the exact content. | +| `importbookmarks` | File. | `import` | +| `tag` | URL. Options: All (), (). | All (), () | +| `startautosearch` | URL. | `start it` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Bookmarks.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Bookmarks.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/Bookmarks.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `title` | Title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `description` | Description. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `path` | URL path, URL prefix, or host/path scope. In index tools it decides which part of the stored URL tree is inspected or changed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `tags` | Tags (comma separated). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `public` | Public. Options: `public` = yes, `private` = no. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `feed` | Bookmark is a newsfeed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `add` | URL. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `xmlfile` | File. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `importxml` | File. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `htmlfile` | File. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `importbookmarks` | File. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `tag` | URL. Options: All (), (). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `startautosearch` | URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `user` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /Bookmarks.html?query=...&url=...&title=...&description=...&path=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `yacysearch.html` diff --git a/help/CacheResource_p.md b/help/CacheResource_p.md new file mode 100644 index 000000000..5e368bd79 --- /dev/null +++ b/help/CacheResource_p.md @@ -0,0 +1,68 @@ +--- +page: htroot/CacheResource_p.html +help: help/CacheResource_p.md +title: CacheResource p +package: core-search-public +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CacheResource_p.java +--- + +# CacheResource p + +## Purpose + +Cache Resource serves cached resource data. + +Use it to retrieve a document or asset from YaCy's cache when direct loading is not desired or possible. + +## What You Can Do Here + +- Cache Resource serves cached resource data. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CacheResource_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CacheResource_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CacheResource_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/CacheResource_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `load` | Loads the selected URL, host, or path detail view. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /CacheResource_p.html?url=...&load=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/Collage.md b/help/Collage.md new file mode 100644 index 000000000..4c6b51f0d --- /dev/null +++ b/help/Collage.md @@ -0,0 +1,66 @@ +--- +page: htroot/Collage.html +help: help/Collage.md +title: Image Collage +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Collage.java +--- + +# Image Collage + +## Purpose + +Image Collage displays image search results as a visual collection. + +Use it when browsing pictures is more useful than reading a ranked text list. + +## What You Can Do Here + +- Image Collage displays image search results as a visual collection. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Collage.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Collage.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/Collage.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET /Collage.html?emb=...&height=...&max=...&width=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/ConfigAccountList_p.md b/help/ConfigAccountList_p.md new file mode 100644 index 000000000..42e309842 --- /dev/null +++ b/help/ConfigAccountList_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/ConfigAccountList_p.html +help: help/ConfigAccountList_p.md +title: User Accounts +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigAccountList_p.java +--- + +# User Accounts + +## Purpose + +User Accounts lists configured accounts and their role-related state. + +Use it to audit who can sign in before changing permissions or exposing the peer. + +## What You Can Do Here + +- User Accounts lists configured accounts and their role-related state. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigAccountList_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ConfigUser_p.html` diff --git a/help/ConfigAccounts_p.md b/help/ConfigAccounts_p.md new file mode 100644 index 000000000..8ed1897bc --- /dev/null +++ b/help/ConfigAccounts_p.md @@ -0,0 +1,115 @@ +--- +page: htroot/ConfigAccounts_p.html +help: help/ConfigAccounts_p.md +title: User Accounts +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigAccounts_p.java +--- + +# User Accounts + +## Purpose + +User Accounts defines administrator and user credentials. + +Use it early, especially before the peer is reachable from another machine. + +## What You Can Do Here + +- User Accounts defines administrator and user credentials. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `access` | Access from localhost without account / Access only with qualified account. Options: `localhost` = Access from localhost without account, `account` = Access only with qualified account. | `localhost` = Access from localhost without account, `account` = Access only with qualified account | +| `adminuser` | Peer User. | Text value; use the page label and surrounding context to choose the exact content. | +| `adminpw1` | New Peer Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `adminpw2` | Repeat Peer Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `setAdmin` | Peer User. | `Define Administrator` | +| `adminAccountAllPages` | Access from localhost without account. | Access from localhost without account | +| `setAccess` | Access from localhost without account. | `Set Access Rules` | +| `user` | Select user. | `newuser` = New user | +| `change_user` | Select user. | `Edit User` | +| `delete_user` | Select user. | `Delete User` | +| `username` | Username. | Checkbox/boolean; present usually means enabled. | +| `password` | Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `password2` | Repeat password. | Text value; use the page label and surrounding context to choose the exact content. | +| `firstname` | First name. | Text value; use the page label and surrounding context to choose the exact content. | +| `lastname` | Last name. | Text value; use the page label and surrounding context to choose the exact content. | +| `address` | Address. | Text value; use the page label and surrounding context to choose the exact content. | +| `#[name]#` | right. | right | +| `timelimit` | Timelimit. | Text value; use the page label and surrounding context to choose the exact content. | +| `timeused` | Time used. | Text value; use the page label and surrounding context to choose the exact content. | +| `change` | Username. | `Save User` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigAccounts_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigAccounts_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigAccounts_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigAccounts_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `access` | Access from localhost without account / Access only with qualified account. Options: `localhost` = Access from localhost without account, `account` = Access only with qualified account. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `adminuser` | Peer User. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `adminpw1` | New Peer Password. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `adminpw2` | Repeat Peer Password. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `setAdmin` | Peer User. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `adminAccountAllPages` | Access from localhost without account. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `setAccess` | Access from localhost without account. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `user` | Select user. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `change_user` | Select user. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete_user` | Select user. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `username` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `password` | Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `password2` | Repeat password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `firstname` | First name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lastname` | Last name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `address` | Address. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `#[name]#` | right. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timelimit` | Timelimit. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timeused` | Time used. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `change` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `current_user` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /ConfigAccounts_p.html +Content-Type: application/x-www-form-urlencoded + +access=...&adminuser=...&adminpw1=...&adminpw2=...&setAdmin=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ConfigAccountList_p.html` diff --git a/help/ConfigAppearance_p.md b/help/ConfigAppearance_p.md new file mode 100644 index 000000000..81da74281 --- /dev/null +++ b/help/ConfigAppearance_p.md @@ -0,0 +1,81 @@ +--- +page: htroot/ConfigAppearance_p.html +help: help/ConfigAppearance_p.md +title: Appearance and Integration +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigAppearance_p.java +--- + +# Appearance and Integration + +## Purpose + +Appearance and Integration changes the visual identity of the YaCy interface and embedded presentation. + +Use it to adapt a peer to a public portal, intranet, or local administration style. + +## What You Can Do Here + +- Appearance and Integration changes the visual identity of the YaCy interface and embedded presentation. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `skin` | Available Skins. | `generic_pd.css` | +| `use_button` | Current skin. | `Use` | +| `delete_button` | Current skin. | `Delete` | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `use_skin` | Use this skin. | `on` = Use this skin | +| `install_button` | Install new skin from URL. | `Install` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigAppearance_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigAppearance_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigAppearance_p.html` | `GET` | admin | `source/net/yacy/htroot/ConfigAppearance_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `skin` | Available Skins. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `use_button` | Current skin. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete_button` | Current skin. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `use_skin` | Use this skin. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `install_button` | Install new skin from URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /ConfigAppearance_p.html?url=...&skin=...&use_button=...&delete_button=...&color_background=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ConfigPortal_p.html` diff --git a/help/ConfigBasic.md b/help/ConfigBasic.md new file mode 100644 index 000000000..f4796a1b7 --- /dev/null +++ b/help/ConfigBasic.md @@ -0,0 +1,100 @@ +--- +page: htroot/ConfigBasic.html +help: help/ConfigBasic.md +title: Access Configuration +package: configuration-administration +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/ConfigBasic.java +--- + +# Access Configuration + +## Purpose + +Basic Configuration is the first meaningful setup page after installation. It sets the language of the web interface, the use case of the peer, the peer name, and the network port. These are not cosmetic details: they decide who can comfortably use YaCy, whether the peer joins the community network, whether it behaves as a private portal, and how other browsers or peers reach it. + +The language selector shows that YaCy is built for an international audience. The value `default` uses English. The two-letter values select specific translations, for example `de` for German, `fr` for French, `es` for Spanish, `zh` for Chinese, `ja` for Japanese, and `ko` for Korean. + +## What You Can Do Here + +- Choose the interface language used by YaCy pages. +- Choose the operating mode: community search, independent portal, or intranet indexing. +- Set a recognizable peer name for this YaCy instance. +- Change the HTTP port and optionally enable HTTPS. +- Ask YaCy to configure router port mapping through UPnP when available. +- Continue to the next setup steps: account security, search, crawling, peer profile, and network monitoring. + +## Page Architecture + +The page is organized as a setup checklist. Language is first because it affects the whole user interface. Use case comes next because it changes YaCy's network and index-sharing behavior. Peer name and port define how this peer identifies itself and how it is reached. + +| Control | Meaning | Valid values | +| --- | --- | --- | +| `language` | Interface language. `default` selects English, and two-letter language codes select translations. | `default`, `de`, `fr`, `pl`, `el`, `it`, `es`, `tr`, `sk`, `uk`, `ru`, `zh`, `hi`, `ja`, `ko` | +| `usecase` | Peer operating mode. | `freeworld` = join the community-based search network; `portal` = independent search portal for your own pages; `intranet` = private/local indexing for intranet, file, FTP, SMB, or local-domain content | +| `peername` | Human-readable peer name. Java accepts names matching `[A-Za-z0-9\-_]{3,80}` after spaces are converted to hyphens. | Example: `research-peer`, `library_search`, `OfficeNode01` | +| `port` | HTTP port where YaCy listens. The form accepts ports above 1023; changing it redirects the browser to the new address. | Default is commonly `8090` | +| `withssl` | Enables HTTPS support. | Checkbox: present means enabled | +| `enableUpnp` | Enables router port mapping through UPnP when the environment supports it. | Checkbox: present means enabled | +| `set` | Applies the configuration. | Submit button | + +## Correct Use + +Pick the language first. If translated pages for the selected language are not generated yet, YaCy can generate them and reload the page. Then choose the use case deliberately. `freeworld` enables community network participation and index exchange. `portal` behaves independently and is suited for a topic or site search. `intranet` is for private/local material and must be handled carefully to avoid exposing private documents. + +Changing the port or SSL setting changes how the browser reaches YaCy. Expect a reconnect or redirect. Change the administrator password on `ConfigAccounts_p.html` before exposing the peer to other machines. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/ConfigBasic.html`. + +Backend checks: administrator authentication, transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigBasic.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigBasic.html` | `POST` | mixed | `source/net/yacy/htroot/ConfigBasic.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `language` | Interface language. Values are `default` for English, or a language code such as `de`, `fr`, `es`, `zh`, `ja`, or `ko`. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `usecase` | Operating mode for the peer: `freeworld` joins the community search network, `portal` creates an independent search portal, and `intranet` is for local/private indexing. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `peername` | Public peer name. Use 3 to 80 letters, digits, hyphen, or underscore; spaces are converted to hyphens. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `port` | HTTP port where YaCy listens. Values below 1024 are ignored by this form; changing the port triggers reconnect/redirect behavior. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `withssl` | Enables HTTPS support for the peer. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `enableUpnp` | Asks YaCy to configure router port mapping through UPnP when available. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ConfigBasic.html +Content-Type: application/x-www-form-urlencoded + +language=...&usecase=...&peername=...&port=...&withssl=... +``` + +## What To Expect + +Changing `language` can immediately reload the interface in the selected translation. Changing `port` or SSL can trigger a reconnect to the new address. Changing `usecase` changes network/index-sharing behavior and may be blocked or warned when remote Solr data could expose private or irrelevant documents. + +## Related Pages + +- `ConfigAccounts_p.html` +- `IndexControlURLs_p.html` +- `IndexFederated_p.html` +- `Settings_p.html` +- `index.html` +- `CrawlStartSite.html` +- `ConfigProfile_p.html` +- `Network.html` diff --git a/help/ConfigHTCache_p.md b/help/ConfigHTCache_p.md new file mode 100644 index 000000000..9bc7d194d --- /dev/null +++ b/help/ConfigHTCache_p.md @@ -0,0 +1,88 @@ +--- +page: htroot/ConfigHTCache_p.html +help: help/ConfigHTCache_p.md +title: Hypertext Cache Configuration +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigHTCache_p.java +--- + +# Hypertext Cache Configuration + +## Purpose + +Hypertext Cache Configuration controls whether fetched documents are stored for later reuse. + +Use it to balance speed, disk use, freshness, and privacy. + +## What You Can Do Here + +- Hypertext Cache Configuration controls whether fetched documents are stored for later reuse. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `HTCachePath` | The path where the cache is stored. | Text value; use the page label and surrounding context to choose the exact content. | +| `maxCacheSize` | The maximum size of the cache. | Integer value. | +| `compressionLevel` | Compression level. | Text value; use the page label and surrounding context to choose the exact content. | +| `lockTimeout` | Concurrent access timeout. | Text value; use the page label and surrounding context to choose the exact content. | +| `set` | Submits and applies the basic configuration. | `Set` | +| `deleteCache` | The path where the cache is stored. | The path where the cache is stored | +| `deleteRobots` | The path where the cache is stored. | The path where the cache is stored | +| `deletecomplete` | The path where the cache is stored. | `Delete` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigHTCache_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigHTCache_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigHTCache_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigHTCache_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `HTCachePath` | The path where the cache is stored. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maxCacheSize` | The maximum size of the cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `compressionLevel` | Compression level. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lockTimeout` | Concurrent access timeout. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `deleteCache` | The path where the cache is stored. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteRobots` | The path where the cache is stored. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deletecomplete` | The path where the cache is stored. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /ConfigHTCache_p.html +Content-Type: application/x-www-form-urlencoded + +HTCachePath=...&maxCacheSize=...&compressionLevel=...&lockTimeout=...&set=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/ConfigHeuristics_p.md b/help/ConfigHeuristics_p.md new file mode 100644 index 000000000..d75dc194e --- /dev/null +++ b/help/ConfigHeuristics_p.md @@ -0,0 +1,85 @@ +--- +page: htroot/ConfigHeuristics_p.html +help: help/ConfigHeuristics_p.md +title: Heuristics Configuration +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigHeuristics_p.java +--- + +# Heuristics Configuration + +## Purpose + +Heuristics Configuration controls optional search heuristics that can broaden or improve result discovery beyond the most direct index lookup. + +Use it when the peer should apply helper strategies during search, for example to enrich result discovery or compensate for sparse local index coverage. Heuristics can make search feel smarter, but they should be enabled deliberately because every extra strategy can also add cost or change result expectations. + +## What You Can Do Here + +- Enable or disable the heuristic sources that YaCy may consult during search. +- Add or remove OpenSearch systems that can supplement local results. +- Compare search behavior before and after changes so extra result sources do not hide the quality of the local index. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `site_check` | Choice value. Options: `site`. | `site` | +| `searchresult_check` | Choice value. Options: `searchresult`. | `searchresult` | +| `searchresultglobal_check` | Choice value. Options: `siteresultglobal`. | `siteresultglobal` | +| `opensearch_check` | Choice value. Options: `opensearch`. | `opensearch` | +| `ossys_#[title]#` | Choice value. Options: `checked`. | `checked` | +| `ossys_url_#[title]#` | Human-readable title. | URL or URL-derived value; use the exact format shown by the page. | +| `ossys_del_#[title]#` | Choice value. Options: `checked`. | `checked` | +| `ossys_newtitle` | Human-readable title. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigHeuristics_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigHeuristics_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigHeuristics_p.html` | `GET` | admin | `source/net/yacy/htroot/ConfigHeuristics_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `site_check` | Choice value. Options: `site`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `searchresult_check` | Choice value. Options: `searchresult`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `searchresultglobal_check` | Choice value. Options: `siteresultglobal`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `opensearch_check` | Choice value. Options: `opensearch`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ossys_#[title]#` | Choice value. Options: `checked`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ossys_url_#[title]#` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ossys_del_#[title]#` | Choice value. Options: `checked`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ossys_newtitle` | Human-readable title. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /ConfigHeuristics_p.html?site_check=...&searchresult_check=...&searchresultglobal_check=...&opensearch_check=...&ossys_#[title]#=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `IndexSchema_p.html` diff --git a/help/ConfigLanguage_p.md b/help/ConfigLanguage_p.md new file mode 100644 index 000000000..21d065e33 --- /dev/null +++ b/help/ConfigLanguage_p.md @@ -0,0 +1,79 @@ +--- +page: htroot/ConfigLanguage_p.html +help: help/ConfigLanguage_p.md +title: Language selection +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigLanguage_p.java +--- + +# Language selection + +## Purpose + +Language selection changes the interface translation. + +Use it when operators or users need YaCy pages in another language. + +## What You Can Do Here + +- Language selection changes the interface translation. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `language` | Interface language. Values are `browser` for the browser-preferred language, `default` for English, or a language code such as `de`, `fr`, `es`, `zh`, `ja`, or `ko`. | Text value; use the page label and surrounding context to choose the exact content. | +| `use_button` | Current language. | `Use` | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete` | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `use_lang` | Use this language. | `on` = Use this language | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigLanguage_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigLanguage_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigLanguage_p.html` | `GET` | admin | `source/net/yacy/htroot/ConfigLanguage_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `language` | Interface language. Values are `browser` for the browser-preferred language, `default` for English, or a language code such as `de`, `fr`, `es`, `zh`, `ja`, or `ko`. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `use_button` | Current language. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `use_lang` | Use this language. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /ConfigLanguage_p.html?language=...&url=...&use_button=...&delete=...&use_lang=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `Translator_p.html` diff --git a/help/ConfigNetwork_p.md b/help/ConfigNetwork_p.md new file mode 100644 index 000000000..85bd3982a --- /dev/null +++ b/help/ConfigNetwork_p.md @@ -0,0 +1,106 @@ +--- +page: htroot/ConfigNetwork_p.html +help: help/ConfigNetwork_p.md +title: Network Configuration +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigNetwork_p.java +--- + +# Network Configuration + +## Purpose + +Network Configuration decides how the peer participates in YaCy's network. + +Use it to choose between private operation and communication with other peers. + +## What You Can Do Here + +- Network Configuration decides how the peer participates in YaCy's network. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `networkDefinition` | Network and Domain Specification. | Text value; use the page label and surrounding context to choose the exact content. | +| `networkDefinitionURL` | Network and Domain Specification. | URL or URL-derived value; use the exact format shown by the page. | +| `changeNetwork` | Network and Domain Specification. | `Change Network` | +| `network` | Peer-to-Peer Mode / Robinson Mode. Options: `p2p` = Peer-to-Peer Mode, `robinson` = Robinson Mode. | `p2p` = Peer-to-Peer Mode, `robinson` = Robinson Mode | +| `indexDistribute` | Index Distribution. | Index Distribution | +| `indexDistributeWhileCrawling` | enabled / disabled during crawling. Options: `on` = enabled, `off` = disabled during crawling. | `on` = enabled, `off` = disabled during crawling | +| `indexDistributeWhileIndexing` | enabled / disabled during indexing. Options: `on` = enabled, `off` = disabled during indexing. | `on` = enabled, `off` = disabled during indexing | +| `indexReceive` | Index Receive. | Index Receive | +| `indexReceiveBlockBlacklist` | reject / accept transmitted URLs that match your blacklist. Options: `on` = reject, `off` = accept transmitted URLs that match your blacklist. | `on` = reject, `off` = accept transmitted URLs that match your blacklist | +| `indexReceiveSearch` | allow / deny remote search. Options: `on` = allow, `off` = deny remote search. | `on` = allow, `off` = deny remote search | +| `save` | Saves settings. | `Save` | +| `cluster.mode` | Private Peer / Public Peer / Public Cluster. Options: `privatepeer` = Private Peer, `publicpeer` = Public Peer, `publiccluster` = Public Cluster. | `privatepeer` = Private Peer, `publicpeer` = Public Peer, `publiccluster` = Public Cluster | +| `cluster.peers.yacydomain` | Private Peer. | Text value; use the page label and surrounding context to choose the exact content. | +| `peertags` | Peer Tags. | Text value; use the page label and surrounding context to choose the exact content. | +| `network.unit.protocol.https.preferred` | Prefer HTTPS for outgoing connexions to remote peers.. | `true` = Prefer HTTPS for outgoing connexions to remote peers. | +| `setEncryption` | Prefer HTTPS for outgoing connexions to remote peers.. | `Save` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigNetwork_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigNetwork_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigNetwork_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigNetwork_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `networkDefinition` | Network and Domain Specification. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `networkDefinitionURL` | Network and Domain Specification. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `changeNetwork` | Network and Domain Specification. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `network` | Peer-to-Peer Mode / Robinson Mode. Options: `p2p` = Peer-to-Peer Mode, `robinson` = Robinson Mode. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexDistribute` | Index Distribution. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexDistributeWhileCrawling` | enabled / disabled during crawling. Options: `on` = enabled, `off` = disabled during crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexDistributeWhileIndexing` | enabled / disabled during indexing. Options: `on` = enabled, `off` = disabled during indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexReceive` | Index Receive. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexReceiveBlockBlacklist` | reject / accept transmitted URLs that match your blacklist. Options: `on` = reject, `off` = accept transmitted URLs that match your blacklist. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexReceiveSearch` | allow / deny remote search. Options: `on` = allow, `off` = deny remote search. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `save` | Saves settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `cluster.mode` | Private Peer / Public Peer / Public Cluster. Options: `privatepeer` = Private Peer, `publicpeer` = Public Peer, `publiccluster` = Public Cluster. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cluster.peers.yacydomain` | Private Peer. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `peertags` | Peer Tags. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `network.unit.protocol.https.preferred` | Prefer HTTPS for outgoing connexions to remote peers.. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `setEncryption` | Prefer HTTPS for outgoing connexions to remote peers.. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ConfigNetwork_p.html +Content-Type: application/x-www-form-urlencoded + +networkDefinition=...&networkDefinitionURL=...&changeNetwork=...&network=...&indexDistribute=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ConfigPortal_p.html` diff --git a/help/ConfigParser_p.md b/help/ConfigParser_p.md new file mode 100644 index 000000000..09a201dc8 --- /dev/null +++ b/help/ConfigParser_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/ConfigParser_p.html +help: help/ConfigParser_p.md +title: Advanced Settings +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigParser_p.java +--- + +# Advanced Settings + +## Purpose + +Parser settings decide which document formats YaCy can understand. + +Use it when crawled files arrive but are not converted into searchable text as expected. + +## What You Can Do Here + +- Parser settings decide which document formats YaCy can understand. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigParser_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigParser_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigParser_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigParser_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /ConfigParser_p.html +Content-Type: application/x-www-form-urlencoded + +allswitch=...&extension_#[extension]#=...&mimename_#[mimetype]#=...&parserSettings=...&extension_=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ViewFile.html` diff --git a/help/ConfigPortal_p.md b/help/ConfigPortal_p.md new file mode 100644 index 000000000..6c7e9fef3 --- /dev/null +++ b/help/ConfigPortal_p.md @@ -0,0 +1,102 @@ +--- +page: htroot/ConfigPortal_p.html +help: help/ConfigPortal_p.md +title: Integration of a Search Portal +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigPortal_p.java +--- + +# Integration of a Search Portal + +## Purpose + +Integration of a Search Portal controls the public search portal around the result page. + +Use it to make YaCy feel like a site search or topic portal instead of a raw administration interface. + +## What You Can Do Here + +- Configure the public search portal that users see before and after searching. +- Adjust branding, navigation, and result-page behavior so the portal fits the intended audience. +- Check the public search page after saving, not only the administration form. + +## Page Architecture + +The portal configuration page controls the public shell around YaCy search: names, navigation, visible features, and integration details. It separates administration of the portal from the search index itself; changing the portal changes how users approach search, not which documents exist. + +## Correct Use + +Configure the portal for the audience first: public web search, site search, intranet search, or a specialized collection. After saving, open the public search page as a normal user would and check whether the labels, links, and available search options make sense. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigPortal_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigPortal_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigPortal_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigPortal_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `promoteSearchPageGreeting` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `promoteSearchPageGreeting.homepage` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `promoteSearchPageGreeting.smallImage` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `promoteSearchPageGreeting.largeImage` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `promoteSearchPageGreeting.imageAlt` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `publicSearchpage` | Extended. Options: `true` = Extended, `false` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `publicTopmenu` | Extended. Options: `true` = Extended, `false` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.options` | Extended. Options: `true` = Extended, `false` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.strictContentDom` | Extended. Options: `false` = Extended, `true` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.jsresort` | On demand, server-side. Options: `false` = On demand, server-side, `true` = On demand, server-side. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `remotesearch.https.preferred` | Prefer https for search queries on remote peers.. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.verify` | Extended. Options: `nocache` = Extended, `iffresh` = Extended, `ifexist` = Extended, `cacheonly` = Extended, `false` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.verify.delete` | Extended. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `greedylearning.active` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `remotesearch.result.store` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `remotesearch.result.store.maxsize` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `popup` | Extended. Options: `status` = Extended, `front` = Extended, `search` = Extended, `interactive` = Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexForward` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `target` | Extended. Options: `_blank` = "_blank" (new window), `_self` = "_self" (same window), `_parent` = "_parent" (the parent frame of a frameset), `_top` = "_top" (top of all frames), `searchresult` = "searchresult" (a default custom page name for search results). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `target_special` | Extended. Options: `_blank` = "_blank" (new window), `_self` = "_self" (same window), `_parent` = "_parent" (the parent frame of a frameset), `_top` = "_top" (top of all frames), `searchresult` = "searchresult" (a default custom page name for search results). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `target_special_pattern` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.excludehosts` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `about.headline` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `about.body` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `searchpage_set` | Extended. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `searchpage_default` | Extended. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /ConfigPortal_p.html +Content-Type: application/x-www-form-urlencoded + +maximumRecords=...&promoteSearchPageGreeting=...&promoteSearchPageGreeting.homepage=...&promoteSearchPageGreeting.smallImage=...&promoteSearchPageGreeting.largeImage=... +``` + +## What To Expect + +A successful change alters the public-facing search portal or its generated integration code. Existing indexed documents are not recrawled by this page. + +## Related Pages + +- `ConfigAppearance_p.html` +- `SearchAccessRate_p.html` +- `Settings_p.html` +- `ConfigHeuristics_p.html` +- `ConfigAccounts_p.html` diff --git a/help/ConfigProfile_p.md b/help/ConfigProfile_p.md new file mode 100644 index 000000000..5a06e1d00 --- /dev/null +++ b/help/ConfigProfile_p.md @@ -0,0 +1,93 @@ +--- +page: htroot/ConfigProfile_p.html +help: help/ConfigProfile_p.md +title: Your Personal Profile +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigProfile_p.java +--- + +# Your Personal Profile + +## Purpose + +Your Personal Profile describes this peer operator to other users or peers when profile sharing is enabled. + +Use it to publish only information that should be visible in the YaCy environment. + +## What You Can Do Here + +- Your Personal Profile describes this peer operator to other users or peers when profile sharing is enabled. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `name` | Name. | Text value; use the page label and surrounding context to choose the exact content. | +| `nickname` | Nick Name. | Text value; use the page label and surrounding context to choose the exact content. | +| `homepage` | Homepage (appears on every Supporter Page as long as your peer is online). | Text value; use the page label and surrounding context to choose the exact content. | +| `email` | eMail. | Text value; use the page label and surrounding context to choose the exact content. | +| `icq` | ICQ. | Text value; use the page label and surrounding context to choose the exact content. | +| `jabber` | Jabber. | Text value; use the page label and surrounding context to choose the exact content. | +| `yahoo` | Yahoo!. | Text value; use the page label and surrounding context to choose the exact content. | +| `msn` | MSN. | Text value; use the page label and surrounding context to choose the exact content. | +| `skype` | Skype. | Text value; use the page label and surrounding context to choose the exact content. | +| `comment` | Comment. | Text value; use the page label and surrounding context to choose the exact content. | +| `set` | Submits and applies the basic configuration. | `Save` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigProfile_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigProfile_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigProfile_p.html` | `GET` | admin | `source/net/yacy/htroot/ConfigProfile_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `name` | Name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `nickname` | Nick Name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `homepage` | Homepage (appears on every Supporter Page as long as your peer is online). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `email` | eMail. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `icq` | ICQ. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `jabber` | Jabber. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `yahoo` | Yahoo!. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `msn` | MSN. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `skype` | Skype. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `comment` | Comment. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /ConfigProfile_p.html?name=...&nickname=...&homepage=...&email=...&icq=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `ViewProfile.html` +- `Supporter.html` +- `WikiHelp.html` diff --git a/help/ConfigProperties_p.md b/help/ConfigProperties_p.md new file mode 100644 index 000000000..c12ddaa2e --- /dev/null +++ b/help/ConfigProperties_p.md @@ -0,0 +1,74 @@ +--- +page: htroot/ConfigProperties_p.html +help: help/ConfigProperties_p.md +title: Advanced Config +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigProperties_p.java +--- + +# Advanced Config + +## Purpose + +Advanced Config exposes low-level configuration properties. + +Use it for precise repair or expert tuning when a named setting is known. + +## What You Can Do Here + +- Advanced Config exposes low-level configuration properties. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigProperties_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigProperties_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigProperties_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigProperties_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /ConfigProperties_p.html +Content-Type: application/x-www-form-urlencoded + +key=...&value=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/ConfigRobotsTxt_p.md b/help/ConfigRobotsTxt_p.md new file mode 100644 index 000000000..d976fea55 --- /dev/null +++ b/help/ConfigRobotsTxt_p.md @@ -0,0 +1,96 @@ +--- +page: htroot/ConfigRobotsTxt_p.html +help: help/ConfigRobotsTxt_p.md +title: Local robots.txt +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigRobotsTxt_p.java +--- + +# Local robots.txt + +## Purpose + +Local robots.txt publishes crawl instructions for YaCy's own embedded web server. + +Use it to tell external crawlers which YaCy-local paths should or should not be fetched. + +## What You Can Do Here + +- Local robots.txt publishes crawl instructions for YaCy's own embedded web server. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `all` | Entire Peer. | Entire Peer | +| `status` | Status page. | Status page | +| `network` | Network pages. | Network pages | +| `surftips` | Surftips. | Surftips | +| `news` | News pages. | News pages | +| `blog` | Blog. | Blog | +| `wiki` | Wiki. | Wiki | +| `bookmarks` | Public bookmarks. | Public bookmarks | +| `homepage` | Home Page. | Home Page | +| `fileshare` | File Share. | File Share | +| `profile` | Impressum. | Impressum | +| `save` | Saves settings. | `Save restrictions` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigRobotsTxt_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigRobotsTxt_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigRobotsTxt_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigRobotsTxt_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `all` | Entire Peer. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `status` | Status page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `network` | Network pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `surftips` | Surftips. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `news` | News pages. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `blog` | Blog. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `wiki` | Wiki. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `bookmarks` | Public bookmarks. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `homepage` | Home Page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `fileshare` | File Share. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `profile` | Impressum. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `save` | Saves settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ConfigRobotsTxt_p.html +Content-Type: application/x-www-form-urlencoded + +all=...&status=...&network=...&surftips=...&news=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/ConfigSearchBox.md b/help/ConfigSearchBox.md new file mode 100644 index 000000000..012d3c900 --- /dev/null +++ b/help/ConfigSearchBox.md @@ -0,0 +1,94 @@ +--- +page: htroot/ConfigSearchBox.html +help: help/ConfigSearchBox.md +title: Integration of a Search Box +package: configuration-administration +access: mixed +kind: search-page +backend_java: source/net/yacy/htroot/ConfigSearchBox.java +--- + +# Integration of a Search Box + +## Purpose + +Integration of a Search Box generates embeddable search-box code for another website. It helps turn this YaCy peer into the search backend for a page that lives somewhere else. + +Use it when visitors should type a query on an external site and receive results from `yacysearch.html` on this peer. + +## What You Can Do Here + +- Generate or inspect the HTML snippet for an external search box. +- Understand that the generated form submits to `yacysearch.html`, so the real search parameters are the same as the search page. +- Use the snippet only after choosing the target peer address and expected result behavior. + +## Page Architecture + +This page is an integration helper. It shows the HTML form code for a search box and demonstrates the target request that the form will send to `yacysearch.html`. The controls on this page are not a separate search engine; they are a compact front door into the normal YaCy search endpoint. + +## Correct Use + +Use the generated code as a template, then adapt the surrounding site HTML outside YaCy. Keep the form action pointed at the YaCy peer that should answer searches, and keep the query field named `query`. If the external page should search only a collection or site, add the same search parameters documented for `yacysearch.html`. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/yacysearch.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigSearchBox.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearch.html` | `GET` | mixed | `source/net/yacy/htroot/yacysearch.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `constraint` | Encoded YaCy bitfield constraint. Leave it empty unless you are replaying a URL generated by YaCy itself. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleteref` | URL hash/reference selected for deletion from results. Use only with administrator intent and an exact known reference. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `focus` | Browser presentation flag controlling whether the search input receives focus. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexof` | Interprets the query as an index-of style request when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lr` | Language filter, usually a two-letter code such as `de`, `fr`, or `en` when the page supports language-restricted search. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `nav` | Navigator/facet selection for search results. `all` requests the normal set; narrower values reduce displayed facets. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control; use it to request later result pages. | +| `prefermaskfilter` | Regular-expression URL preference. Matching results are favored without excluding all others. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `recommendref` | URL hash/reference selected for recommendation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resortCachedResults` | Requests resorting of a cached search event instead of starting a fully fresh result event. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `resource` | Search source. `local` searches this peer index; `global` may use the YaCy network when the peer and configuration allow it. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resource-switch` | Browser control for switching between local and global search. For direct calls use `resource=local` or `resource=global`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `rows` | Rows requested by API-style clients. It is another result-count alias on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `strictContentDom` | When true, YaCy enforces the selected content domain more strictly instead of using it mainly for presentation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `urlmaskfilter` | Regular-expression URL filter for search results. Use `.*` for no restriction; use a precise host/path expression to search inside a site or section. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `verify` | Snippet/cache verification strategy. Typical values are `iffresh`, `ifexist`, `cacheonly`, `nocache`, and `false`; choose according to whether freshness or speed matters more. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacysearch.html?query=...&maximumRecords=...&resource=...&contentdom=...&Enter=... +``` + +## What To Expect + +The page provides an embeddable snippet or demonstration form. The actual user search happens later when a visitor submits the form and YaCy receives a `yacysearch.html` request. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/ConfigSearchPage_p.md b/help/ConfigSearchPage_p.md new file mode 100644 index 000000000..0dc55fb02 --- /dev/null +++ b/help/ConfigSearchPage_p.md @@ -0,0 +1,113 @@ +--- +page: htroot/ConfigSearchPage_p.html +help: help/ConfigSearchPage_p.md +title: Search Page +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigSearchPage_p.java +--- + +# Search Page + +## Purpose + +Search Page settings control how the result page looks and which search features are offered. + +Use it to shape the default search experience before users begin relying on the portal. + +## What You Can Do Here + +- Configure default search-page behavior and presentation. +- Decide which navigators, result types, and search options users should see. +- Test a normal query after saving so ranking and presentation can be checked together. + +## Page Architecture + +This page configures the result page: which result controls, navigators, media views, and presentation defaults users see after they submit a query. It sits between the raw search backend and the human search experience. + +## Correct Use + +Change result-page options with a real query in mind. After saving, run representative searches for text, media, and restricted collections so you can see whether the page still guides users clearly. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigSearchPage_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigSearchPage_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigSearchPage_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigSearchPage_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `publicTopmenu` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.publicTopNavBar.login` | Choice value. Options: `true`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.protocol` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.topics` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.location` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `del.nav` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.#[name]#.navSort` | Sort by Descending counts Ascending counts Descending labels Ascending labels. Options: `count:desc` = Descending counts, `count:asc` = Ascending counts, `label:desc` = Descending labels, `label:asc` = Ascending labels. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.text` | Text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.image` | Images. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.audio` | Audio. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.video` | Video. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.app` | Applications. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.options` | more options. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.date` | Date Navigation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.dates.maxcount` | Date Navigation. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.result.show.favicon` | Show websites favicon. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.keywords` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.keywords.firstMaxCount` | Max. tags initially displayed (remaining can then be expanded). | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.result.show.date` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.size` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.metadata` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.parser` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.citation` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.pictures` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.cache` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.proxy` | Max. tags initially displayed (remaining can then be expanded). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.result.show.indexbrowser` | Max. tags initially displayed (remaining can then be expanded). | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.navigation.navname` | append #{search.navigation.list}# #{/search.navigation.list}#. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `add.nav` | append #{search.navigation.list}# #{/search.navigation.list}#. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `search.navigation.maxcount` | max. items. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `searchpage_set` | append #{search.navigation.list}# #{/search.navigation.list}#. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `searchpage_default` | append #{search.navigation.list}# #{/search.navigation.list}#. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.navigation.active` | Sort by Descending counts Ascending counts Descending labels Ascending labels. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /ConfigSearchPage_p.html +Content-Type: application/x-www-form-urlencoded + +query=...&publicTopmenu=...&search.publicTopNavBar.login=...&search.navigation.protocol=...&search.navigation.topics=... +``` + +## What To Expect + +A successful change affects future result-page rendering. It does not itself add documents to the index or change crawler behavior. + +## Related Pages + +- `ConfigAppearance_p.html` +- `ConfigPortal_p.html` +- `yacysearch.html` +- `ViewFile.html` +- `api/citation.html` +- `CacheResource_p.html` +- `proxy.html` +- `IndexBrowser_p.html` +- `Settings_p.html` diff --git a/help/ConfigUpdate_p.md b/help/ConfigUpdate_p.md new file mode 100644 index 000000000..a6725df80 --- /dev/null +++ b/help/ConfigUpdate_p.md @@ -0,0 +1,88 @@ +--- +page: htroot/ConfigUpdate_p.html +help: help/ConfigUpdate_p.md +title: System Update +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigUpdate_p.java +--- + +# System Update + +## Purpose + +System Update checks and applies YaCy software updates. + +Use it when maintaining the peer version, but read the current state before starting an update. + +## What You Can Do Here + +- System Update checks and applies YaCy software updates. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `releasedownload` | Choice value. Options: (unsigned)::(signed). | (unsigned)::(signed) | +| `update` | Date/time value for filtering, display, or scheduling. | `1`, `Install Release` | +| `releaseinstall` | Choice value. Options: (no signature)::(signed). | (no signature)::(signed) | +| `deleteRelease` | Deletion or termination action. Use only with explicit intent. | `Delete Release` | +| `autoUpdate` | Date/time value for filtering, display, or scheduling. | `Check + Download + Install Release Now` | +| `updateMode` | Choice value. Options: `manual`, `auto`. | `manual`, `auto` | +| `releaseType` | Choice value. Options: `main`, `any`. | `main`, `any` | +| `onlySignedFiles` | Choice value. Options: `true`. | `true` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigUpdate_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigUpdate_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigUpdate_p.html` | `GET` | admin | `source/net/yacy/htroot/ConfigUpdate_p.java` | +| `/Steering.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/Steering.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `releasedownload` | Choice value. Options: (unsigned)::(signed). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `update` | Submit action that refreshes, updates, or applies the selected setting depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `releaseinstall` | Choice value. Options: (no signature)::(signed). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteRelease` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `autoUpdate` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `updateMode` | Choice value. Options: `manual`, `auto`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `releaseType` | Choice value. Options: `main`, `any`. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `onlySignedFiles` | Choice value. Options: `true`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /ConfigUpdate_p.html?releasedownload=...&downloadRelease=...&checkRelease=...&update=...&releaseinstall=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/ConfigUser_p.md b/help/ConfigUser_p.md new file mode 100644 index 000000000..877c63d7c --- /dev/null +++ b/help/ConfigUser_p.md @@ -0,0 +1,97 @@ +--- +page: htroot/ConfigUser_p.html +help: help/ConfigUser_p.md +title: User Editor +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ConfigUser_p.java +--- + +# User Editor + +## Purpose + +User Editor changes one account. + +Use it to create or repair a named user without changing unrelated access settings. + +## What You Can Do Here + +- User Editor changes one account. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `username` | Username. | Checkbox/boolean; present usually means enabled. | +| `password` | Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `password2` | Repeat password. | Text value; use the page label and surrounding context to choose the exact content. | +| `firstname` | First name. | Text value; use the page label and surrounding context to choose the exact content. | +| `lastname` | Last name. | Text value; use the page label and surrounding context to choose the exact content. | +| `address` | Address. | Text value; use the page label and surrounding context to choose the exact content. | +| `#[name]#` | right. | right | +| `timelimit` | Timelimit. | Text value; use the page label and surrounding context to choose the exact content. | +| `timeused` | Time used. | Text value; use the page label and surrounding context to choose the exact content. | +| `change` | Username. | `Save User` | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete User` | +| `cancel` | Username. | `ConfigAccountList_p.html` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ConfigUser_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ConfigUser_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ConfigUser_p.html` | `POST` | admin | `source/net/yacy/htroot/ConfigUser_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `username` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `password` | Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `password2` | Repeat password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `firstname` | First name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lastname` | Last name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `address` | Address. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `#[name]#` | right. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timelimit` | Timelimit. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timeused` | Time used. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `change` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `cancel` | Username. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `user` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /ConfigUser_p.html +Content-Type: application/x-www-form-urlencoded + +username=...&password=...&password2=...&firstname=...&lastname=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/Connections_p.md b/help/Connections_p.md new file mode 100644 index 000000000..b965f1841 --- /dev/null +++ b/help/Connections_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/Connections_p.html +help: help/Connections_p.md +title: Server Connection Tracking +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Connections_p.java +--- + +# Server Connection Tracking + +## Purpose + +Server Connection Tracking shows open and recent connections. + +Use it when requests appear stuck, slow, or unexpectedly numerous. + +## What You Can Do Here + +- Server Connection Tracking shows open and recent connections. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Connections_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/ContentAnalysis_p.md b/help/ContentAnalysis_p.md new file mode 100644 index 000000000..4eabdd5dd --- /dev/null +++ b/help/ContentAnalysis_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/ContentAnalysis_p.html +help: help/ContentAnalysis_p.md +title: Content Analysis +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ContentAnalysis_p.java +--- + +# Content Analysis + +## Purpose + +Content Analysis inspects document text for duplicate or near-duplicate patterns. + +Use it to understand whether repeated content, boilerplate, or very similar pages are polluting the index. + +## What You Can Do Here + +- Content Analysis inspects document text for duplicate or near-duplicate patterns. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `minTokenLen` | minTokenLen. | Text value; use the page label and surrounding context to choose the exact content. | +| `quantRate` | quantRate. | Text value; use the page label and surrounding context to choose the exact content. | +| `EnterDoublecheck` | minTokenLen. | `Set` | +| `ResetDoublecheck` | minTokenLen. | `Re-Set to default` | + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ContentAnalysis_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ContentAnalysis_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ContentAnalysis_p.html` | `POST` | admin | `source/net/yacy/htroot/ContentAnalysis_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `minTokenLen` | minTokenLen. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `quantRate` | quantRate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `EnterDoublecheck` | minTokenLen. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ResetDoublecheck` | minTokenLen. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ContentAnalysis_p.html +Content-Type: application/x-www-form-urlencoded + +minTokenLen=...&quantRate=...&EnterDoublecheck=...&ResetDoublecheck=...&ResetRanking=... +``` + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- `https://lucene.apache.org/solr/5_5_2/solr-core/org/apache/solr/update/processor/TextProfileSignature.html` diff --git a/help/ContentIntegrationPHPBB3_p.md b/help/ContentIntegrationPHPBB3_p.md new file mode 100644 index 000000000..032df3309 --- /dev/null +++ b/help/ContentIntegrationPHPBB3_p.md @@ -0,0 +1,73 @@ +--- +page: htroot/ContentIntegrationPHPBB3_p.html +help: help/ContentIntegrationPHPBB3_p.md +title: Content Integration: Retrieval from phpBB3 Databases +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ContentIntegrationPHPBB3_p.java +--- + +# Content Integration: Retrieval from phpBB3 Databases + +## Purpose + +phpBB3 content integration imports forum data into the index. + +Use it to make discussions searchable without relying only on crawler traversal. + +## What You Can Do Here + +- phpBB3 content integration imports forum data into the index. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `content.phpbb3.dbhost` | Host or domain scope. | Text value; use the page label and surrounding context to choose the exact content. | +| `content.phpbb3.dbuser` | User or account value. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ContentIntegrationPHPBB3_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ContentIntegrationPHPBB3_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ContentIntegrationPHPBB3_p.html` | `GET` | admin | `source/net/yacy/htroot/ContentIntegrationPHPBB3_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `content.phpbb3.dbhost` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `content.phpbb3.dbuser` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /ContentIntegrationPHPBB3_p.html?content.phpbb3.urlstub=...&content.phpbb3.dbtype=...&content.phpbb3.dbhost=...&content.phpbb3.dbport=...&content.phpbb3.dbname=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/CookieMonitorIncoming_p.md b/help/CookieMonitorIncoming_p.md new file mode 100644 index 000000000..805b62f22 --- /dev/null +++ b/help/CookieMonitorIncoming_p.md @@ -0,0 +1,76 @@ +--- +page: htroot/CookieMonitorIncoming_p.html +help: help/CookieMonitorIncoming_p.md +title: Incoming Cookies Monitor +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CookieMonitorIncoming_p.java +--- + +# Incoming Cookies Monitor + +## Purpose + +Incoming Cookies Monitor shows cookies received by the peer. + +Use it to inspect what browsers or clients send before diagnosing session or privacy behavior. + +## What You Can Do Here + +- Incoming Cookies Monitor shows cookies received by the peer. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `enableCookieMonitoring` | Enables the named feature. | `Enable Cookie Monitoring` | +| `disableCookieMonitoring` | Disables the named feature. | `Disable Cookie Monitoring` | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CookieMonitorIncoming_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CookieMonitorIncoming_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CookieMonitorIncoming_p.html` | `POST` | admin | `source/net/yacy/htroot/CookieMonitorIncoming_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `enableCookieMonitoring` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `disableCookieMonitoring` | Disables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /CookieMonitorIncoming_p.html +Content-Type: application/x-www-form-urlencoded + +enableCookieMonitoring=...&disableCookieMonitoring=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/CookieMonitorOutgoing_p.md b/help/CookieMonitorOutgoing_p.md new file mode 100644 index 000000000..69e6ed053 --- /dev/null +++ b/help/CookieMonitorOutgoing_p.md @@ -0,0 +1,76 @@ +--- +page: htroot/CookieMonitorOutgoing_p.html +help: help/CookieMonitorOutgoing_p.md +title: Outgoing Cookies Monitor +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CookieMonitorOutgoing_p.java +--- + +# Outgoing Cookies Monitor + +## Purpose + +Outgoing Cookies Monitor shows cookies YaCy sends back. + +Use it to inspect session, authentication, and browser-facing state. + +## What You Can Do Here + +- Outgoing Cookies Monitor shows cookies YaCy sends back. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `enableCookieMonitoring` | Enables the named feature. | `Enable Cookie Monitoring` | +| `disableCookieMonitoring` | Disables the named feature. | `Disable Cookie Monitoring` | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CookieMonitorOutgoing_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CookieMonitorOutgoing_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CookieMonitorOutgoing_p.html` | `POST` | admin | `source/net/yacy/htroot/CookieMonitorOutgoing_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `enableCookieMonitoring` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `disableCookieMonitoring` | Disables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /CookieMonitorOutgoing_p.html +Content-Type: application/x-www-form-urlencoded + +enableCookieMonitoring=...&disableCookieMonitoring=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/CrawlCheck_p.md b/help/CrawlCheck_p.md new file mode 100644 index 000000000..0ed414c9d --- /dev/null +++ b/help/CrawlCheck_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/CrawlCheck_p.html +help: help/CrawlCheck_p.md +title: Crawl Check +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CrawlCheck_p.java +--- + +# Crawl Check + +## Purpose + +Crawl Check tests how YaCy would treat a URL before a crawl is started. + +Use it to catch malformed URLs, blocked targets, parser issues, or rule conflicts before work enters the queue. + +## What You Can Do Here + +- Paste one or more candidate start URLs and ask YaCy how it would handle them. +- Confirm that URL syntax, crawler user-agent choice, and crawl rules allow the target. +- Use the result to repair the crawl plan before creating queue work. + +## Page Architecture + +Crawl Check is a preflight page. It submits candidate URLs to the same kind of URL-normalization and crawler-decision logic used later by crawl start pages, but the goal is diagnosis rather than building a full crawl queue. + +## Correct Use + +Paste exact URLs, one per line when checking several targets. Use the same crawler agent that the real crawl would use. Fix rejected or surprising URLs before starting a crawl, because bad start URLs create empty queues or noisy crawl failures. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CrawlCheck_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlCheck_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CrawlCheck_p.html` | `POST` | admin | `source/net/yacy/htroot/CrawlCheck_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlingURLs` | List of possible crawl start URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlcheck` | List of possible crawl start URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /CrawlCheck_p.html +Content-Type: application/x-www-form-urlencoded + +crawlingURLs=...&crawlcheck=...&agentName=... +``` + +## What To Expect + +The response should explain how YaCy interprets each submitted URL and whether it is suitable as a crawl target. It should not be treated as proof that the full site has been indexed. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/CrawlMonitorRemoteStart.md b/help/CrawlMonitorRemoteStart.md new file mode 100644 index 000000000..ab5dd456c --- /dev/null +++ b/help/CrawlMonitorRemoteStart.md @@ -0,0 +1,49 @@ +--- +page: htroot/CrawlMonitorRemoteStart.html +help: help/CrawlMonitorRemoteStart.md +title: Monitor for remotely started global crawls +package: crawler +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/CrawlMonitorRemoteStart.java +--- + +# Monitor for remotely started global crawls + +## Purpose + +Remote Crawl Monitor shows crawls that were started remotely through the YaCy network. + +Use it to audit externally triggered crawl activity before trusting or continuing it. + +## What You Can Do Here + +- Remote Crawl Monitor shows crawls that were started remotely through the YaCy network. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlMonitorRemoteStart.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/CrawlProfileEditor_p.md b/help/CrawlProfileEditor_p.md new file mode 100644 index 000000000..679f118dc --- /dev/null +++ b/help/CrawlProfileEditor_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/CrawlProfileEditor_p.html +help: help/CrawlProfileEditor_p.md +title: Crawl Profile Editor +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CrawlProfileEditor_p.java +--- + +# Crawl Profile Editor + +## Purpose + +Crawl Profile Editor changes the rules of an existing crawl profile. + +Use it when a running or reusable crawl needs different limits, filters, or indexing behavior. + +## What You Can Do Here + +- Crawl Profile Editor changes the rules of an existing crawl profile. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Text value; use the page label and surrounding context to choose the exact content. | +| `terminate` | Deletion or termination action. Use only with explicit intent. | `Terminate` | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete` | +| `deleteTerminatedProfiles` | Deletion or termination action. Use only with explicit intent. | `Delete finished crawls` | +| `submit` | Submits the form. | `Submit changes` | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CrawlProfileEditor_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlProfileEditor_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CrawlProfileEditor_p.html` | `GET` | admin | `source/net/yacy/htroot/CrawlProfileEditor_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteTerminatedProfiles` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /CrawlProfileEditor_p.html?handle=...&terminate=...&delete=...&deleteTerminatedProfiles=...&edit=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/CrawlResults.md b/help/CrawlResults.md new file mode 100644 index 000000000..f1f72242a --- /dev/null +++ b/help/CrawlResults.md @@ -0,0 +1,79 @@ +--- +page: htroot/CrawlResults.html +help: help/CrawlResults.md +title: Crawl Results +package: crawler +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/CrawlResults.java +--- + +# Crawl Results + +## Purpose + +Crawl Results summarizes what crawls produced. + +Use it after a crawl to understand successes, failures, discovered URLs, and indexed material. + +## What You Can Do Here + +- Crawl Results summarizes what crawls produced. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `deletedomain` | Deletion or termination action. Use only with explicit intent. | `delete all` | +| `deleteentry` | Deletion or termination action. Use only with explicit intent. | `delete` | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlResults.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CrawlResults.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/CrawlResults.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `deletedomain` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteentry` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `domain` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /CrawlResults.html?selectedblacklist=...&deletedomain=...&delandaddtoblacklist=...&clearlist=...&deleteentry=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- `CrawlStartExpert.html` +- `RemoteCrawl_p.html` +- `IndexImportMediawiki_p.html` +- `IndexImportOAIPMH_p.html` +- `ViewFile.html` diff --git a/help/CrawlStartExpert.md b/help/CrawlStartExpert.md new file mode 100644 index 000000000..06c4a38cb --- /dev/null +++ b/help/CrawlStartExpert.md @@ -0,0 +1,196 @@ +--- +page: htroot/CrawlStartExpert.html +help: help/CrawlStartExpert.md +title: Crawl Start +package: crawler +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/CrawlStartExpert.java +--- + +# Crawl Start + +## Purpose + +Expert Crawl Start exposes the full crawler model: start sources, depth, filters, media handling, recrawl policy, cache behavior, and indexing rules. + +Use it when the simple site crawl page is too limited for a careful crawl plan. + +## What You Can Do Here + +- Expert Crawl Start exposes the full crawler model: start sources, depth, filters, media handling, recrawl policy, cache behavior, and indexing rules. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `crawlingMode` | Crawl source mode, such as URL, site list, or sitemap. | `url` = Add Crawl result to collection (important for Index Pack generation), `sitelist` = Add Crawl result to collection (important for Index Pack generation), `sitemap` = Add Crawl result to collection (important for Index Pack generation), `file` = Add Crawl result to collection (important for Index Pack generation) | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Absolute URL, for example `https://example.org/docs/`. | +| `bookmarkTitle` | Add Crawl result to collection (important for Index Pack generation). | Text value; use the page label and surrounding context to choose the exact content. | +| `expandSiteListBtn` | Add Crawl result to collection (important for Index Pack generation). | Text value; use the page label and surrounding context to choose the exact content. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Absolute URL to an XML sitemap. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Text value; use the page label and surrounding context to choose the exact content. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Collection name such as `user`, `docs`, or a project-specific name. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Text value; use the page label and surrounding context to choose the exact content. | +| `indexText` | Index extracted text. | index text | +| `indexMedia` | Index media resources. | index media | +| `crawlOrder` | Do Remote Indexing. | Do Remote Indexing | +| `intention` | Optional human label describing why the crawl was started. | Text value; use the page label and surrounding context to choose the exact content. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Integer link depth. | +| `directDocByURL` | index text. | index text | +| `crawlingDepthExtension` | index text. | Integer value. | +| `crawlingDomMaxCheck` | Enables maximum page count for the crawl. | Use | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Integer page limit. | +| `crawlingQ` | index text. | index text | +| `obeyHtmlRobotsNoindex` | Respect HTML robots noindex. | index text | +| `obeyHtmlRobotsNofollow` | Respect HTML robots nofollow. | index text | +| `crawlerAlwaysCheckMediaType` | Do not load URLs with an unsupported file extension. Options: `false` = Do not load URLs with an unsupported file extension, `true` = Do not load URLs with an unsupported file extension. | `false` = Do not load URLs with an unsupported file extension, `true` = Do not load URLs with an unsupported file extension | +| `range` | Crawl boundary, usually domain-wide or limited to the start path. | `domain` = index text, `subpath` = index text, `wide` = index text | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Regular expression; default allow-all behavior is equivalent to matching everything. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Regular expression; use a never-match pattern to disable exclusion. | +| `crawlerOriginURLMustMatch` | index text. | URL or URL-derived value; use the exact format shown by the page. | +| `crawlerOriginURLMustNotMatch` | index text. | URL or URL-derived value; use the exact format shown by the page. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Pattern or filter expression; test narrow expressions before broad use. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Pattern or filter expression; test narrow expressions before broad use. | +| `countryMustMatchSwitch` | index text. Options: `0` = index text, `1` = index text. | `0` = index text, `1` = index text | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Regular expression for URLs allowed into the index. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Regular expression for URLs excluded from the index. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | URL or URL-derived value; use the exact format shown by the page. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexMediaTypeMustMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexMediaTypeMustNotMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexSolrQueryMustMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Pattern or filter expression; test narrow expressions before broad use. | +| `indexSolrQueryMustNotMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Pattern or filter expression; test narrow expressions before broad use. | +| `default_valency` | Add Crawl result to collection (important for Index Pack generation). Options: `EVAL` = Add Crawl result to collection (important for Index Pack generation), `IGNORE` = Add Crawl result to collection (important for Index Pack generation). | `EVAL` = Add Crawl result to collection (important for Index Pack generation), `IGNORE` = Add Crawl result to collection (important for Index Pack generation) | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Text value; use the page label and surrounding context to choose the exact content. | +| `cleanSearchCache` | Clean up search events cache. | Clean up search events cache | +| `deleteold` | Clean up search events cache. Options: `off` = Clean up search events cache, `on` = Clean up search events cache, `age` = Clean up search events cache. | `off` = Clean up search events cache, `on` = Clean up search events cache, `age` = Clean up search events cache | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Integer value. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | `year`, `month`, `day`, or `hour`. | +| `recrawl` | Whether and how existing documents are refreshed. | `nodoubles` = Add Crawl result to collection (important for Index Pack generation), `reload` = Add Crawl result to collection (important for Index Pack generation) | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Integer value. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | `year`, `month`, `day`, or `hour`. | +| `storeHTCache` | Store to Web Cache. | Store to Web Cache | +| `cachePolicy` | How YaCy may use cached content while loading. | `nocache` = Store to Web Cache, `iffresh` = Store to Web Cache, `ifexist` = Store to Web Cache, `cacheonly` = Store to Web Cache | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | One of the user-agent names offered by the page. | +| `vocabulary_#[name]#_class` | Scraping Fields. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Protected related endpoint(s): `/Crawler_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlStartExpert.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Crawler_p.html` | `POST` | admin | `source/net/yacy/htroot/Crawler_p.java` | +| `/CrawlStartExpert.html` | `GET or POST` | public or page-dependent | `source/net/yacy/htroot/CrawlStartExpert.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlingMode` | Crawl source mode. `url` starts from one URL, `sitemap` reads sitemap entries, `file`/list modes submit many URLs, and page-specific modes may prepare these values for `Crawler_p.html`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `bookmarkTitle` | Add Crawl result to collection (important for Index Pack generation). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlOrder` | Do Remote Indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `intention` | Optional human label describing why the crawl was started. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `directDocByURL` | Treats the submitted URL as a direct document target rather than mainly as a link-discovery seed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepthExtension` | index text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxCheck` | Enables the maximum-page safeguard for a domain crawl. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlerAlwaysCheckMediaType` | Do not load URLs with an unsupported file extension. Options: `false` = Do not load URLs with an unsupported file extension, `true` = Do not load URLs with an unsupported file extension. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `range` | Boundary for a simple site crawl. `domain` stays in the same domain; `subpath` stays below the start path; expert pages may expose wider policies. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlerOriginURLMustMatch` | index text. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlerOriginURLMustNotMatch` | index text. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchSwitch` | Enables country-code filtering for crawl targets. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexMediaTypeMustMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexMediaTypeMustNotMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexSolrQueryMustMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexSolrQueryMustNotMatch` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `default_valency` | Default link valency used by the crawler when deciding how links contribute to discovery and indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cleanSearchCache` | Clears cached search results so newly crawled material can appear without stale search state. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `recrawl` | Recrawl policy for already known URLs, such as avoiding duplicates, reloading, or using scheduler rules depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Store to Web Cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cachePolicy` | Cache strategy for fetching documents, for example whether cached material may be reused when fresh enough. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `vocabulary_#[name]#_class` | Scraping Fields. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile$file` | Multipart file content containing crawl URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingPerformance` | Crawler speed preset. Use slower presets for shared servers; use custom only when you understand the load impact. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingstart` | Submit action that creates a crawl job. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hidewebstructuregraph` | Hide the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `latencyFactor` | Politeness multiplier for crawl delay. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `pause` | Pauses the selected queue or crawl process. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `queues_terminate_all` | Stops all crawler queues. This is a broad emergency control, not a normal crawl setting. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /Crawler_p.html +Content-Type: application/x-www-form-urlencoded + +crawlingURL=...&range=...&crawlingDepth=...&crawlingDomMaxPages=...&crawlingMode=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- `IndexFederated_p.html` +- `RemoteCrawl_p.html` +- `https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html` +- `RegexTest.html` +- `IndexSchema_p.html` +- `https://lucene.apache.org/solr/guide/6_6/the-standard-query-parser.html` diff --git a/help/CrawlStartScanner_p.md b/help/CrawlStartScanner_p.md new file mode 100644 index 000000000..d79640572 --- /dev/null +++ b/help/CrawlStartScanner_p.md @@ -0,0 +1,83 @@ +--- +page: htroot/CrawlStartScanner_p.html +help: help/CrawlStartScanner_p.md +title: Network Scanner +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/CrawlStartScanner_p.java +--- + +# Network Scanner + +## Purpose + +Network Scanner discovers services in an IP range and can turn discoveries into crawl targets. + +Use it for intranet discovery when the content landscape is not already known. + +## What You Can Do Here + +- Network Scanner discovers services in an IP range and can turn discoveries into crawl targets. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `source` | Choice value. Options: `hosts`, `intranet`, `all`. | `hosts`, `intranet`, `all` | +| `scanhosts` | Host or domain scope. | Text value; use the page label and surrounding context to choose the exact content. | +| `subnet` | Choice value. Options: `31`, `24`, `20`, `16`. | `31`, `24`, `20`, `16` | +| `timeout` | Date/time value for filtering, display, or scheduling. | `2000` | +| `rescan` | Choice value. Options: `off`, `scheduler`. | `off`, `scheduler` | +| `repeat_time` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `15`, `21`, `28`, `30`. | `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `15`, `21`, `28`, `30` | +| `repeat_unit` | Choice value. Options: `selminutes` = minutes, `selhours` = hours, `seldays` = days. | `selminutes` = minutes, `selhours` = hours, `seldays` = days | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/CrawlStartScanner_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlStartScanner_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CrawlStartScanner_p.html` | `GET` | admin | `source/net/yacy/htroot/CrawlStartScanner_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `source` | Choice value. Options: `hosts`, `intranet`, `all`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `scanhosts` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `subnet` | Choice value. Options: `31`, `24`, `20`, `16`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timeout` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `rescan` | Choice value. Options: `off`, `scheduler`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_time` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `15`, `21`, `28`, `30`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_unit` | Choice value. Options: `selminutes` = minutes, `selhours` = hours, `seldays` = days. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /CrawlStartScanner_p.html?source=...&scanhosts=...&subnet=...&timeout=...&accumulatescancache=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/CrawlStartSite.md b/help/CrawlStartSite.md new file mode 100644 index 000000000..3eb0ebabc --- /dev/null +++ b/help/CrawlStartSite.md @@ -0,0 +1,154 @@ +--- +page: htroot/CrawlStartSite.html +help: help/CrawlStartSite.md +title: Crawl Start +package: crawler +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/CrawlStartSite.java +--- + +# Crawl Start + +## Purpose + +Site Crawling is the beginner-friendly way to teach YaCy a website. The operator gives a start URL, chooses the crawl boundary, optionally limits the number of documents, and starts a crawl job that feeds the index. + +Use it for small, deliberate crawls before moving to expert crawling. + +The page is organized around a simple site-crawl form and a short hint section. The form defines the start URL, the crawl boundary, an optional page limit, and the target collection. + +## What You Can Do Here + +- Start a small crawl from one website or one section of a website. +- Set boundaries before starting or changing a crawl, so the crawler loads the intended part of the web and no more. +- Put the discovered documents into a named collection when the crawl belongs to a project or topic. +- Check the resulting queues, errors, and indexed documents before widening a crawl or making it faster. + +## Page Architecture + +This page is a simplified front end for `Crawler_p.html`. It prepares a normal crawl-start request with a start URL, a boundary, an optional page limit, and a collection. YaCy then hands the request to the crawler backend, which creates a crawl profile and queues the first URL. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `crawlingMode` | Crawl source mode. On this page `url` is the normal start-URL crawl; `sitelist` and `sitemap` are prepared modes that become available when YaCy detects a link list or sitemap. | `url`, `sitelist`, `sitemap` | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Absolute URL, for example `https://example.org/docs/`. | +| `bookmarkTitle` | Optional title discovered for the submitted URL; normally read-only on the simple crawl page. | Usually omitted by scripts. | +| `expandSiteListBtn` | Browser-only button that expands a discovered link list; do not send it in API calls. | Browser action only. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Absolute URL to an XML sitemap. | +| `range` | Crawl boundary. `domain` allows the crawl to stay within the same domain; `subpath` limits it to URLs below the start path. | `domain`, `subpath` | +| `crawlingDomMaxCheck` | Enables the page-count limit. Use it for first crawls and for sites where the full size is unknown. | checkbox | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Integer page limit. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Collection name such as `user`, `docs`, or a project-specific name. | +| `crawlingstart` | Submit action that creates the crawl job. | `Start New Crawl` | + +## Correct Use + +Start narrow. Enter a full URL including `http://` or `https://`, choose `subpath` when you only want a section of a site, and enable a page limit for first crawls. A website crawl is iterative: YaCy loads the start page, extracts links, keeps the links that match the boundary rules, queues them, loads those pages, and repeats the process until the queue is empty or a limit stops it. + +After submitting, watch `Crawler_p.html` and verify indexed results before widening depth or limits. If the user only asked for one known document, use the expert/direct-document options instead of a site crawl. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Protected related endpoint(s): `/Crawler_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/CrawlStartSite.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Crawler_p.html` | `POST` | admin | `source/net/yacy/htroot/Crawler_p.java` | + +### Parameter Guide + +For this simple page, an agent normally needs only the first group of parameters below. The remaining parameters belong to the shared `Crawler_p.html` backend and are included because they may appear in recorded API calls or advanced crawl automation. + +Simple site-crawl request: + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlingMode` | Crawl source mode. `url` starts from one URL, `sitemap` reads sitemap entries, `file`/list modes submit many URLs, and page-specific modes may prepare these values for `Crawler_p.html`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `bookmarkTitle` | Optional title discovered for the submitted URL; normally read-only on the simple crawl page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `range` | Boundary for a simple site crawl. `domain` stays in the same domain; `subpath` stays below the start path; expert pages may expose wider policies. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxCheck` | Enables the maximum-page safeguard for a domain crawl. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingstart` | Submit action that creates a crawl job. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Advanced `Crawler_p.html` controls: + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cachePolicy` | Cache strategy for fetching documents, for example whether cached material may be reused when fresh enough. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cleanSearchCache` | Clears cached search results so newly crawled material can appear without stale search state. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchSwitch` | Enables country-code filtering for crawl targets. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlOrder` | Crawl ordering strategy, such as balanced host scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlerAlwaysCheckMediaType` | Check media type before deciding parser/indexing behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepthExtension` | Additional depth behavior for special crawl modes. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile$file` | Multipart file content containing crawl URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingPerformance` | Crawler speed preset. Use slower presets for shared servers; use custom only when you understand the load impact. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `default_valency` | Default link valency used by the crawler when deciding how links contribute to discovery and indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `directDocByURL` | Treats the submitted URL as a direct document target rather than mainly as a link-discovery seed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hidewebstructuregraph` | Hide the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `intention` | Optional human label describing why the crawl was started. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `latencyFactor` | Politeness multiplier for crawl delay. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `pause` | Pauses the selected queue or crawl process. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `queues_terminate_all` | Stops all crawler queues. This is a broad emergency control, not a normal crawl setting. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `recrawl` | Recrawl policy for already known URLs, such as avoiding duplicates, reloading, or using scheduler rules depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Stores fetched documents in YaCy hypertext cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Crawler_p.html +Content-Type: application/x-www-form-urlencoded + +crawlingstart=Start+New+Crawl&crawlingMode=url&crawlingURL=https%3A%2F%2Fexample.org%2Fdocs%2F&range=subpath&crawlingDomMaxCheck=on&crawlingDomMaxPages=100 +``` + +## What To Expect + +Successful crawl actions usually create or modify crawl jobs. Verify progress on `Crawler_p.html`, then confirm indexed content through search or index browsing. + +## Related Pages + +- `CrawlStartExpert.html` +- `Automation_p.html` diff --git a/help/Crawler_p.md b/help/Crawler_p.md new file mode 100644 index 000000000..d8b421e8d --- /dev/null +++ b/help/Crawler_p.md @@ -0,0 +1,138 @@ +--- +page: htroot/Crawler_p.html +help: help/Crawler_p.md +title: Crawler +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Crawler_p.java +--- + +# Crawler + +## Purpose + +Crawler is the control room for running crawl jobs and queues. + +Use it to watch progress, pause or continue work, terminate jobs, and understand what the crawler is doing now. + +## What You Can Do Here + +- Crawler is the control room for running crawl jobs and queues. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Integer pages per minute. | +| `latencyFactor` | LF. | Text value; use the page label and surrounding context to choose the exact content. | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Integer value. | +| `crawlingPerformance` | PPM. | `set` | +| `terminate` | Deletion or termination action. Use only with explicit intent. | `Terminate` | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | `show link structure` | +| `hidewebstructuregraph` | PPM. | `hide graphic` | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Crawler_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Crawler_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Crawler_p.html` | `GET` | admin | `source/net/yacy/htroot/Crawler_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `latencyFactor` | LF. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingPerformance` | Crawler speed preset. Use slower presets for shared servers; use custom only when you understand the load impact. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hidewebstructuregraph` | PPM. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cachePolicy` | Cache strategy for fetching documents, for example whether cached material may be reused when fresh enough. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cleanSearchCache` | Clears cached search results so newly crawled material can appear without stale search state. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchSwitch` | Enables country-code filtering for crawl targets. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlOrder` | Crawl ordering strategy, such as balanced host scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlerAlwaysCheckMediaType` | Check media type before deciding parser/indexing behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepthExtension` | Additional depth behavior for special crawl modes. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxCheck` | Enables the maximum-page safeguard for a domain crawl. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile$file` | Multipart file content containing crawl URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingMode` | Crawl source mode. `url` starts from one URL, `sitemap` reads sitemap entries, `file`/list modes submit many URLs, and page-specific modes may prepare these values for `Crawler_p.html`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingstart` | Submit action that creates a crawl job. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `default_valency` | Default link valency used by the crawler when deciding how links contribute to discovery and indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `directDocByURL` | Treats the submitted URL as a direct document target rather than mainly as a link-discovery seed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `intention` | Optional human label describing why the crawl was started. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `pause` | Pauses the selected queue or crawl process. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `queues_terminate_all` | Stops all crawler queues. This is a broad emergency control, not a normal crawl setting. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `range` | Boundary for a simple site crawl. `domain` stays in the same domain; `subpath` stays below the start path; expert pages may expose wider policies. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `recrawl` | Recrawl policy for already known URLs, such as avoiding duplicates, reloading, or using scheduler rules depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Stores fetched documents in YaCy hypertext cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /Crawler_p.html?crawlingURL=...&range=...&crawlingDepth=...&crawlingDomMaxPages=...&customPPM=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- `IndexCreateQueues_p.html` +- `IndexCreateLoaderQueue_p.html` +- `PerformanceQueues_p.html` +- `IndexFederated_p.html` +- `RemoteCrawl_p.html` diff --git a/help/DictionaryLoader_p.md b/help/DictionaryLoader_p.md new file mode 100644 index 000000000..c2b6a6a81 --- /dev/null +++ b/help/DictionaryLoader_p.md @@ -0,0 +1,132 @@ +--- +page: htroot/DictionaryLoader_p.html +help: help/DictionaryLoader_p.md +title: Knowledge Loader +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/DictionaryLoader_p.java +--- + +# Knowledge Loader + +## Purpose + +Knowledge Loader imports word lists or dictionaries. + +Use it when vocabulary data should support search, classification, or local knowledge features. + +## What You Can Do Here + +- Knowledge Loader imports word lists or dictionaries. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `geon0Load` | Download from. | `Load` | +| `geon0Deactivate` | Download from. | `Deactivate` | +| `geon0Remove` | Download from. | `Remove` | +| `geon0Activate` | Download from. | `Activate` | +| `geon1Load` | Download from. | `Load` | +| `geon1Deactivate` | Download from. | `Deactivate` | +| `geon1Remove` | Download from. | `Remove` | +| `geon1Activate` | Download from. | `Activate` | +| `geon2Load` | Download from. | `Load` | +| `geon2Deactivate` | Download from. | `Deactivate` | +| `geon2Remove` | Download from. | `Remove` | +| `geon2Activate` | Download from. | `Activate` | +| `geo0Deactivate` | Downloaded from. | `Deactivate` | +| `geo0Remove` | Downloaded from. | `Remove` | +| `geo1Deactivate` | Downloaded from. | `Deactivate` | +| `geo1Remove` | Downloaded from. | `Remove` | +| `geo2Load` | Downloaded from. | `Load` | +| `geo2Deactivate` | Downloaded from. | `Deactivate` | +| `geo2Remove` | Downloaded from. | `Remove` | +| `geo2Activate` | Downloaded from. | `Activate` | +| `drw0Load` | Download from. | `Load` | +| `drw0Deactivate` | Download from. | `Deactivate` | +| `drw0Remove` | Download from. | `Remove` | +| `drw0Activate` | Download from. | `Activate` | +| `syn0Activate` | Status. | `Activate` | +| `syn0Deactivate` | Status. | `Deactivate` | +| `syn1Activate` | Status. | `Activate` | +| `syn1Deactivate` | Status. | `Deactivate` | +| `syn2Activate` | Status. | `Activate` | +| `syn2Deactivate` | Status. | `Deactivate` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/DictionaryLoader_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/DictionaryLoader_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/DictionaryLoader_p.html` | `POST` | admin | `source/net/yacy/htroot/DictionaryLoader_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `geon0Load` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon0Deactivate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon0Remove` | Download from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geon0Activate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon1Load` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon1Deactivate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon1Remove` | Download from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geon1Activate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon2Load` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon2Deactivate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geon2Remove` | Download from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geon2Activate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geo0Deactivate` | Downloaded from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geo0Remove` | Downloaded from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geo1Deactivate` | Downloaded from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geo1Remove` | Downloaded from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geo2Load` | Downloaded from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geo2Deactivate` | Downloaded from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `geo2Remove` | Downloaded from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `geo2Activate` | Downloaded from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `drw0Load` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `drw0Deactivate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `drw0Remove` | Download from. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `drw0Activate` | Download from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn0Activate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn0Deactivate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn1Activate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn1Deactivate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn2Activate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `syn2Deactivate` | Status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /DictionaryLoader_p.html +Content-Type: application/x-www-form-urlencoded + +geon0Load=...&geon0Deactivate=...&geon0Remove=...&geon0Activate=...&geon1Load=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `http://www1.ids-mannheim.de/kl/projekte/methoden/derewo.html` diff --git a/help/Help.md b/help/Help.md new file mode 100644 index 000000000..5b6aee181 --- /dev/null +++ b/help/Help.md @@ -0,0 +1,49 @@ +--- +page: htroot/Help.html +help: help/Help.md +title: Tutorial +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Help.java +--- + +# Tutorial + +## Purpose + +Tutorial links users to YaCy help and learning material. + +Use it as an entry point when the user is lost in the interface. + +## What You Can Do Here + +- Tutorial links users to YaCy help and learning material. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Help.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/IndexBrowser_p.md b/help/IndexBrowser_p.md new file mode 100644 index 000000000..42f990d2c --- /dev/null +++ b/help/IndexBrowser_p.md @@ -0,0 +1,93 @@ +--- +page: htroot/IndexBrowser_p.html +help: help/IndexBrowser_p.md +title: Index Browser +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexBrowser_p.java +--- + +# Index Browser + +## Purpose + +The Index Browser is the diagnostic view for known URLs and hosts. It helps answer whether YaCy has stored a document, which host/path is involved, and what can be inspected or removed. + +Use it after crawling when search results do not match expectations. + +The page has three layers: a host/path browser for finding stored URLs, analysis views for link and host structure, and administration actions for removing or reloading selected data. + +## What You Can Do Here + +- Inspect whether a host, path, or exact URL is present in YaCy's URL database. +- Work from the narrowest host, path, query, or field scope that answers the question. +- Open link-structure views to understand why a document was discovered and how it connects to other pages. +- Reload failed documents such as stored 404/load errors when the failure may have been temporary. +- Confirm destructive actions carefully because index maintenance can remove documents or start expensive reprocessing. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `path` | URL path, URL prefix, or host/path scope. In index tools it decides which part of the stored URL tree is inspected or changed. | URL, host/path prefix, or path scope shown by the page. | +| `list` | Requests a list view for the selected host or path scope. | Usually sent with `path`; the value can be left as the button value generated by the form. | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete Subpath` | +| `reload404` | Queues failed documents such as 404/load-error entries for another load attempt. | `Re-load load-failure docs (404s etc)` | +| `addtoblacklist` | Adds the selected URL or host pattern to the configured blacklist. | Submit only after confirming that the selected host/path should be blocked. | +| `showlinkstructure` | Shows incoming and outgoing link structure for the selected host or path. | `show link graph` | +| `deleteLoadErrors` | Deletes stored load-error records for the selected scope. | `Delete Load Errors` | + +## Correct Use + +Inspect before changing. Start with a host or path lookup, then use `complete` only when the shortened view hides the information you need. Use `facetcount` carefully: larger values give a broader host/path analysis but force YaCy to calculate more facets. + +For deletion, blacklisting, and load-error cleanup, copy the target from the page rather than inventing it. A path such as `example.org/docs/` and a broader host such as `example.org/` can have very different effects. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexBrowser_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexBrowser_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexBrowser_p.html` | `GET` | admin | `source/net/yacy/htroot/IndexBrowser_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `path` | URL path, URL prefix, or host/path scope. In index tools it decides which part of the stored URL tree is inspected or changed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `list` | Requests a list view for the selected host or path scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `reload404` | Queues failed documents such as 404/load-error entries for another load attempt. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `addtoblacklist` | Adds the selected URL or host pattern to the configured blacklist. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `showlinkstructure` | Shows incoming and outgoing link structure for the selected host or path. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteLoadErrors` | Deletes stored load-error records for the selected scope. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `complete` | Requests a complete listing instead of YaCy’s shortened default view. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `facetcount` | Maximum number of host/path facets to calculate. Higher values give broader analysis but cost more work. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `hosts` | Host list filter. Values such as `crawling` or `error` restrict the host overview to active crawl hosts or error hosts when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `load` | Loads the selected URL, host, or path detail view. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /IndexBrowser_p.html?path=example.org/docs/&list=1&facetcount=50 +``` + +## What To Expect + +The response shows the selected URL tree, host overview, or link graph. Destructive actions re-render the page after changing stored index metadata; verify the resulting scope immediately and use search to confirm whether user-visible results changed. + +## Related Pages + +- `ViewFile.html` diff --git a/help/IndexControlRWIs_p.md b/help/IndexControlRWIs_p.md new file mode 100644 index 000000000..6b6e2360e --- /dev/null +++ b/help/IndexControlRWIs_p.md @@ -0,0 +1,92 @@ +--- +page: htroot/IndexControlRWIs_p.html +help: help/IndexControlRWIs_p.md +title: Reverse Word Index Administration +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexControlRWIs_p.java +--- + +# Reverse Word Index Administration + +## Purpose + +Reverse Word Index Administration manages the word-to-document index. + +Use it for low-level diagnosis when search terms do not connect to documents as expected. + +## What You Can Do Here + +- Reverse Word Index Administration manages the word-to-document index. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `maxReferencesRadio` | Choice value. Options: `off`, `on`. | `off`, `on` | +| `description` | Description text. | Text value; use the page label and surrounding context to choose the exact content. | +| `title` | Human-readable title. | Text value; use the page label and surrounding context to choose the exact content. | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `indexof` | Interprets the query as an index-of style request when supported. | Checkbox/boolean; present usually means enabled. | +| `keyhashdelete` | Deletion or termination action. Use only with explicit intent. | `Delete reference to selected URLs` | +| `blacklistdomains` | Host or domain scope. | `Add selected domains to blacklist` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexControlRWIs_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexControlRWIs_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexControlRWIs_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexControlRWIs_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `maxReferencesRadio` | Choice value. Options: `off`, `on`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `description` | Description text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `title` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexof` | Interprets the query as an index-of style request when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `keyhashdelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `blacklistdomains` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `host` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hostHash` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `keyhashdeleteall` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /IndexControlRWIs_p.html +Content-Type: application/x-www-form-urlencoded + +url=...&keystring=...&keystringsearch=...&keyhash=...&keyhashsearch=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- `IndexControlURLs_p.html` diff --git a/help/IndexControlURLs_p.md b/help/IndexControlURLs_p.md new file mode 100644 index 000000000..ed3dc73de --- /dev/null +++ b/help/IndexControlURLs_p.md @@ -0,0 +1,107 @@ +--- +page: htroot/IndexControlURLs_p.html +help: help/IndexControlURLs_p.md +title: URL Database Administration +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexControlURLs_p.java +--- + +# URL Database Administration + +## Purpose + +URL Database Administration manages the URL records stored by YaCy. + +Use it when a document is known by URL and must be inspected, exported, deleted, or repaired. + +## What You Can Do Here + +- URL Database Administration manages the URL records stored by YaCy. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `urlhash` | YaCy URL hash identifying an indexed document. | URL or URL-derived value; use the exact format shown by the page. | +| `deleteIndex` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteRemoteSolr` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteRWI` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteCitation` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteFirstSeen` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteCache` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteCrawlQueues` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deleteRobots` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `deletecomplete` | Deletion or termination action. Use only with explicit intent. | `Delete` | +| `deletedomain` | Deletion or termination action. Use only with explicit intent. | `delete all` | +| `urlhashdelete` | Deletion or termination action. Use only with explicit intent. | `Delete URL` | +| `urlhashdeleteall` | Deletion or termination action. Use only with explicit intent. | `Delete URL and remove all references from words` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexControlURLs_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexControlURLs_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexControlURLs_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexControlURLs_p.java` | +| `/ViewFile.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/ViewFile.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `urlhash` | YaCy URL hash identifying an indexed document. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteIndex` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteRemoteSolr` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteRWI` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteCitation` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteFirstSeen` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteCache` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteCrawlQueues` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteRobots` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deletecomplete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deletedomain` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `urlhashdelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `urlhashdeleteall` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domain` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `urldelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /IndexControlURLs_p.html +Content-Type: application/x-www-form-urlencoded + +query=...&url=...&urlhash=...&urlstring=...&urlstringsearch=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- Related index work is usually reached through `IndexBrowser_p.html`, `IndexControlURLs_p.html`, queue pages, deletion pages, or re-indexing monitors. diff --git a/help/IndexCreateLoaderQueue_p.md b/help/IndexCreateLoaderQueue_p.md new file mode 100644 index 000000000..b8a20cc54 --- /dev/null +++ b/help/IndexCreateLoaderQueue_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/IndexCreateLoaderQueue_p.html +help: help/IndexCreateLoaderQueue_p.md +title: Loader Queue +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexCreateLoaderQueue_p.java +--- + +# Loader Queue + +## Purpose + +Loader Queue shows documents waiting to be loaded or parsed. + +Use it when crawler work exists but fetching or parsing appears delayed. + +## What You Can Do Here + +- Loader Queue shows documents waiting to be loaded or parsed. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexCreateLoaderQueue_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- Related index work is usually reached through `IndexBrowser_p.html`, `IndexControlURLs_p.html`, queue pages, deletion pages, or re-indexing monitors. diff --git a/help/IndexCreateParserErrors_p.md b/help/IndexCreateParserErrors_p.md new file mode 100644 index 000000000..a4ec7fb95 --- /dev/null +++ b/help/IndexCreateParserErrors_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/IndexCreateParserErrors_p.html +help: help/IndexCreateParserErrors_p.md +title: Parser Errors +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexCreateParserErrors_p.java +--- + +# Parser Errors + +## Purpose + +Parser Errors lists documents YaCy could fetch but could not parse cleanly. + +Use it to find format, charset, parser, or document-quality problems. + +## What You Can Do Here + +- Parser Errors lists documents YaCy could fetch but could not parse cleanly. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexCreateParserErrors_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexCreateParserErrors_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexCreateParserErrors_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexCreateParserErrors_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /IndexCreateParserErrors_p.html +Content-Type: application/x-www-form-urlencoded + +moreRejected=...&clearRejected=...&showRejected=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- Related index work is usually reached through `IndexBrowser_p.html`, `IndexControlURLs_p.html`, queue pages, deletion pages, or re-indexing monitors. diff --git a/help/IndexCreateQueues_p.md b/help/IndexCreateQueues_p.md new file mode 100644 index 000000000..5e1e1fb3b --- /dev/null +++ b/help/IndexCreateQueues_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/IndexCreateQueues_p.html +help: help/IndexCreateQueues_p.md +title: '' Crawl Queue +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexCreateQueues_p.java +--- + +# '' Crawl Queue + +## Purpose + +Crawl Queue shows pending and failed indexing work. + +Use it to understand why documents have not reached the searchable index yet. + +## What You Can Do Here + +- Crawl Queue shows pending and failed indexing work. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `pattern` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Text value; use the page label and surrounding context to choose the exact content. | +| `option` | Choice value. Options: `5` = Initiator, `3` = Profile, `4` = Depth, `6` = Modified Date, `2` = Anchor Name, `1` = URL. | `5` = Initiator, `3` = Profile, `4` = Depth, `6` = Modified Date, `2` = Anchor Name, `1` = URL | +| `delete` | Deletes the selected object or scope. Use only with explicit confirmation. | `Delete` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexCreateQueues_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexCreateQueues_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexCreateQueues_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexCreateQueues_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `pattern` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `option` | Choice value. Options: `5` = Initiator, `3` = Profile, `4` = Depth, `6` = Modified Date, `2` = Anchor Name, `1` = URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `sort` | Sort field or sort direction. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `urlsPerHost` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexCreateQueues_p.html +Content-Type: application/x-www-form-urlencoded + +pattern=...&option=...&delete=...&embed=...&sort=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- Related index work is usually reached through `IndexBrowser_p.html`, `IndexControlURLs_p.html`, queue pages, deletion pages, or re-indexing monitors. diff --git a/help/IndexDeletion_p.md b/help/IndexDeletion_p.md new file mode 100644 index 000000000..f83e66a31 --- /dev/null +++ b/help/IndexDeletion_p.md @@ -0,0 +1,107 @@ +--- +page: htroot/IndexDeletion_p.html +help: help/IndexDeletion_p.md +title: Index Deletion +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexDeletion_p.java +--- + +# Index Deletion + +## Purpose + +Index Deletion removes selected documents from the index. + +Use it only with a clear query, host, collection, or URL scope. + +## What You Can Do Here + +- Index Deletion removes selected documents from the index. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `urldelete` | Deletion or termination action. Use only with explicit intent. | URL or URL-derived value; use the exact format shown by the page. | +| `urldelete-mm` | Choice value. Options: `subpath`, `regexp`. | `subpath`, `regexp` | +| `simulate-urldelete` | Deletion or termination action. Use only with explicit intent. | `Simulate Deletion` | +| `engage-urldelete` | Deletion or termination action. Use only with explicit intent. | `Engage Deletion` | +| `timedelete-number` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `24`, `28`, `30`, `60`, `90`. | `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `24`, `28`, `30` | +| `timedelete-unit` | Choice value. Options: `year` = years, `month` = months, `day` = days, `hour` = hours. | `year` = years, `month` = months, `day` = days, `hour` = hours | +| `timedelete-source` | Choice value. Options: `loaddate`, `lastmodified`. | `loaddate`, `lastmodified` | +| `simulate-timedelete` | Deletion or termination action. Use only with explicit intent. | `Simulate Deletion` | +| `engage-timedelete` | Deletion or termination action. Use only with explicit intent. | `Engage Deletion` | +| `collectiondelete-mode` | Choice value. Options: `unassigned`, `assigned`. | `unassigned`, `assigned` | +| `collectiondelete` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `simulate-collectiondelete` | Deletion or termination action. Use only with explicit intent. | `Simulate Deletion` | +| `engage-collectiondelete` | Deletion or termination action. Use only with explicit intent. | `Engage Deletion` | +| `querydelete` | Deletion or termination action. Use only with explicit intent. | Text value; use the page label and surrounding context to choose the exact content. | +| `simulate-querydelete` | Deletion or termination action. Use only with explicit intent. | `Simulate Deletion` | +| `engage-querydelete` | Deletion or termination action. Use only with explicit intent. | `Engage Deletion` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexDeletion_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexDeletion_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexDeletion_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexDeletion_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `urldelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `urldelete-mm` | Choice value. Options: `subpath`, `regexp`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `simulate-urldelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `engage-urldelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timedelete-number` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `24`, `28`, `30`, `60`, `90`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timedelete-unit` | Choice value. Options: `year` = years, `month` = months, `day` = days, `hour` = hours. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timedelete-source` | Choice value. Options: `loaddate`, `lastmodified`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `simulate-timedelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `engage-timedelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `collectiondelete-mode` | Choice value. Options: `unassigned`, `assigned`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `collectiondelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `simulate-collectiondelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `engage-collectiondelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `querydelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `simulate-querydelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `engage-querydelete` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +POST /IndexDeletion_p.html +Content-Type: application/x-www-form-urlencoded + +urldelete=...&urldelete-mm=...&simulate-urldelete=...&engage-urldelete=...&timedelete-number=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- `IndexControlURLs_p.html` diff --git a/help/IndexExportImportSolr_p.md b/help/IndexExportImportSolr_p.md new file mode 100644 index 000000000..038684635 --- /dev/null +++ b/help/IndexExportImportSolr_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/IndexExportImportSolr_p.html +help: help/IndexExportImportSolr_p.md +title: URL Database Administration +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexExportImportSolr_p.java +--- + +# URL Database Administration + +## Purpose + +Solr Index Export/Import moves YaCy index data between this peer and Solr dump formats. + +Use it when preserving or restoring the Solr-backed part of the index. + +## What You Can Do Here + +- Solr Index Export/Import moves YaCy index data between this peer and Solr dump formats. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexExportImportSolr_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexExportImportSolr_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexExportImportSolr_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexExportImportSolr_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /IndexExportImportSolr_p.html +Content-Type: application/x-www-form-urlencoded + +indexdump=...&dumpfile=...&indexrestore=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexExport_p.md b/help/IndexExport_p.md new file mode 100644 index 000000000..8cf01dafd --- /dev/null +++ b/help/IndexExport_p.md @@ -0,0 +1,78 @@ +--- +page: htroot/IndexExport_p.html +help: help/IndexExport_p.md +title: URL Database Administration +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexExport_p.java +--- + +# URL Database Administration + +## Purpose + +URL Database Export writes selected index data to an external file. + +Use it for backup, migration, or analysis of a chosen part of the index. + +## What You Can Do Here + +- Export selected URL database records for backup, migration, or offline inspection. +- Choose the output format according to the tool that will read the export. +- Use `exportfilter` or related query controls to avoid exporting more of the index than needed. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `exportfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | `.*.*` | +| `minified` | Choice value. Options: `no`, `yes`. | `no`, `yes` | +| `format` | Choice value. Options: `url-text`, `url-html`, `dom-text`, `dom-html`, `text-text`. | `url-text`, `url-html`, `dom-text`, `dom-html`, `text-text` | + +## Correct Use + +Decide what the export is for before starting. A human-readable URL list, an HTML view, and a text export serve different follow-up tools. Keep filters narrow for experiments; export broad scopes only when disk space and runtime are acceptable. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexExport_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexExport_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexExport_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexExport_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `exportfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `minified` | Choice value. Options: `no`, `yes`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `format` | Choice value. Options: `url-text`, `url-html`, `dom-text`, `dom-html`, `text-text`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexExport_p.html +Content-Type: application/x-www-form-urlencoded + +exportfilepath=...&exportfilter=...&exportquery=...&exportmaxseconds=...&maxchunksize=... +``` + +## What To Expect + +A successful export creates or updates an export file or background export job. Verify the generated file and record count before deleting, moving, or relying on the exported data. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexFederated_p.md b/help/IndexFederated_p.md new file mode 100644 index 000000000..9bdd4ee32 --- /dev/null +++ b/help/IndexFederated_p.md @@ -0,0 +1,78 @@ +--- +page: htroot/IndexFederated_p.html +help: help/IndexFederated_p.md +title: Index Sources & Targets +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexFederated_p.java +--- + +# Index Sources & Targets + +## Purpose + +Index Sources & Targets configures external index connections, especially Solr federation. + +Use it when YaCy should read from or write to another index service. + +## What You Can Do Here + +- Index Sources & Targets configures external index connections, especially Solr federation. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `federated.service.solr.indexing.authenticated.allowSelfSigned` | Enables the named feature. | Text value; use the page label and surrounding context to choose the exact content. | +| `solr.indexing.solrremote.writeenabled` | Enables the named feature. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexFederated_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexFederated_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexFederated_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexFederated_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `federated.service.solr.indexing.authenticated.allowSelfSigned` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `solr.indexing.solrremote.writeenabled` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /IndexFederated_p.html +Content-Type: application/x-www-form-urlencoded + +solr.indexing.lazy=...&core.service.fulltext=...&solr.indexing.solrremote=...&federated.service.solr.indexing.authenticated.allowSelfSigned=...&solr.indexing.url=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- `IndexSchema_p.html` diff --git a/help/IndexImportJsonList_p.md b/help/IndexImportJsonList_p.md new file mode 100644 index 000000000..d3602bdd6 --- /dev/null +++ b/help/IndexImportJsonList_p.md @@ -0,0 +1,77 @@ +--- +page: htroot/IndexImportJsonList_p.html +help: help/IndexImportJsonList_p.md +title: JsonList Import +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportJsonList_p.java +--- + +# JsonList Import + +## Purpose + +JsonList Import reads a structured list of documents or URLs into YaCy. + +Use it when another tool already prepared crawl or document metadata as JSON. + +## What You Can Do Here + +- JsonList Import reads a structured list of documents or URLs into YaCy. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Text value; use the page label and surrounding context to choose the exact content. | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `submit` | Submits the form. | `Import JsonList File` | +| `abort` | File. | `Stop` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportJsonList_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportJsonList_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportJsonList_p.html` | `GET` | admin | `source/net/yacy/htroot/IndexImportJsonList_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `abort` | File. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /IndexImportJsonList_p.html?url=...&file=...&submit=...&abort=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexImportMediawiki_p.md b/help/IndexImportMediawiki_p.md new file mode 100644 index 000000000..5d2d0b601 --- /dev/null +++ b/help/IndexImportMediawiki_p.md @@ -0,0 +1,82 @@ +--- +page: htroot/IndexImportMediawiki_p.html +help: help/IndexImportMediawiki_p.md +title: MediaWiki Dump Import +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportMediawiki_p.java +--- + +# MediaWiki Dump Import + +## Purpose + +MediaWiki Dump Import turns wiki dump content into searchable documents. + +Use it to build a search index for a wiki without crawling every page through HTTP. + +## What You Can Do Here + +- MediaWiki Dump Import turns wiki dump content into searchable documents. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Text value; use the page label and surrounding context to choose the exact content. | +| `iffresh` | Import only when modified since last import. | Import only when modified since last import | +| `submit` | Submits the form. | `Import MediaWiki Dump` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportMediawiki_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportMediawiki_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportMediawiki_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexImportMediawiki_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `iffresh` | Import only when modified since last import. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /IndexImportMediawiki_p.html +Content-Type: application/x-www-form-urlencoded + +file=...&iffresh=...&submit=...&report=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- `https://dumps.wikimedia.org/backup-index-bydb.html` +- `Automation_p.html` +- `CrawlResults.html` diff --git a/help/IndexImportOAIPMHList_p.md b/help/IndexImportOAIPMHList_p.md new file mode 100644 index 000000000..0b026892f --- /dev/null +++ b/help/IndexImportOAIPMHList_p.md @@ -0,0 +1,76 @@ +--- +page: htroot/IndexImportOAIPMHList_p.html +help: help/IndexImportOAIPMHList_p.md +title: OAI-PMH source import list +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportOAIPMHList_p.java +--- + +# OAI-PMH source import list + +## Purpose + +OAI-PMH Source Import List manages known OAI-PMH sources. + +Use it to organize repository endpoints before harvesting records. + +## What You Can Do Here + +- OAI-PMH Source Import List manages known OAI-PMH sources. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `item_#[count]#` | Choice value. Options: `mark_`. | `mark_` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportOAIPMH_p.html`, `/IndexImportOAIPMHList_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportOAIPMHList_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportOAIPMH_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexImportOAIPMH_p.java` | +| `/IndexImportOAIPMHList_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/IndexImportOAIPMHList_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `item_#[count]#` | Choice value. Options: `mark_`. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexImportOAIPMH_p.html +Content-Type: application/x-www-form-urlencoded + +loadrows=...&allswitch=...&item_#[count]#=...&agentName=...&getlist=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexImportOAIPMH_p.md b/help/IndexImportOAIPMH_p.md new file mode 100644 index 000000000..90b4396c6 --- /dev/null +++ b/help/IndexImportOAIPMH_p.md @@ -0,0 +1,75 @@ +--- +page: htroot/IndexImportOAIPMH_p.html +help: help/IndexImportOAIPMH_p.md +title: OAI-PMH Import +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportOAIPMH_p.java +--- + +# OAI-PMH Import + +## Purpose + +OAI-PMH Import harvests records from repositories that expose the Open Archives protocol. + +Use it for libraries, archives, and research repositories that publish metadata through OAI-PMH. + +## What You Can Do Here + +- OAI-PMH Import harvests records from repositories that expose the Open Archives protocol. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `submit` | Submits the form. | `Import OAI-PMH source` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportOAIPMH_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportOAIPMH_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportOAIPMH_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexImportOAIPMH_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexImportOAIPMH_p.html +Content-Type: application/x-www-form-urlencoded + +urlstartone=...&submit=...&urlstart=...&importroot=...&getlist=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- `CrawlResults.html` diff --git a/help/IndexImportWarc_p.md b/help/IndexImportWarc_p.md new file mode 100644 index 000000000..ec362af98 --- /dev/null +++ b/help/IndexImportWarc_p.md @@ -0,0 +1,82 @@ +--- +page: htroot/IndexImportWarc_p.html +help: help/IndexImportWarc_p.md +title: Warc Import +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportWarc_p.java +--- + +# Warc Import + +## Purpose + +WARC Import reads web-archive files into YaCy. + +Use it to search archived crawls or preserved web collections. + +## What You Can Do Here + +- WARC Import reads web-archive files into YaCy. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Text value; use the page label and surrounding context to choose the exact content. | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `collection` | Collection name used to group indexed documents. | `user` | +| `submit` | Submits the form. | `Import Warc File` | +| `abort` | File. | `Stop` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportWarc_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportWarc_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportWarc_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexImportWarc_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `abort` | File. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexImportWarc_p.html +Content-Type: application/x-www-form-urlencoded + +url=...&file=...&collection=...&submit=...&abort=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexImportZim_p.md b/help/IndexImportZim_p.md new file mode 100644 index 000000000..2f3ad04db --- /dev/null +++ b/help/IndexImportZim_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/IndexImportZim_p.html +help: help/IndexImportZim_p.md +title: ZIM File Import +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexImportZim_p.java +--- + +# ZIM File Import + +## Purpose + +ZIM File Import reads offline web/wiki archives into YaCy. + +Use it to make a ZIM collection searchable without external network access. + +## What You Can Do Here + +- ZIM File Import reads offline web/wiki archives into YaCy. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Text value; use the page label and surrounding context to choose the exact content. | +| `collection` | Collection name used to group indexed documents. | `user` | +| `submit` | Submits the form. | `Import ZIM File` | +| `abort` | File. | `Stop` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexImportZim_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexImportZim_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexImportZim_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexImportZim_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `abort` | File. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexImportZim_p.html +Content-Type: application/x-www-form-urlencoded + +file=...&collection=...&submit=...&abort=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexPackDownloader_p.md b/help/IndexPackDownloader_p.md new file mode 100644 index 000000000..ef1819a26 --- /dev/null +++ b/help/IndexPackDownloader_p.md @@ -0,0 +1,66 @@ +--- +page: htroot/IndexPackDownloader_p.html +help: help/IndexPackDownloader_p.md +title: Index Pack Downloader +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexPackDownloader_p.java +--- + +# Index Pack Downloader + +## Purpose + +Index Pack Downloader retrieves packaged YaCy index data. + +Use it when a prepared index package should be installed instead of crawling from scratch. + +## What You Can Do Here + +- Index Pack Downloader retrieves packaged YaCy index data. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexPackDownloader_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexPackDownloader_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexPackDownloader_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/IndexPackDownloader_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /IndexPackDownloader_p.html?dlfile=...&dlrepoid=...&dlsource=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexPackGenerator_p.md b/help/IndexPackGenerator_p.md new file mode 100644 index 000000000..283f37513 --- /dev/null +++ b/help/IndexPackGenerator_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/IndexPackGenerator_p.html +help: help/IndexPackGenerator_p.md +title: Index Pack Generator +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexPackGenerator_p.java +--- + +# Index Pack Generator + +## Purpose + +Index Pack Generator creates a package from selected local index data. + +Use it to distribute or archive a curated part of the index. + +## What You Can Do Here + +- Index Pack Generator creates a package from selected local index data. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `category` | Choice value. Options: `mix` = mix - a mix of document types, for content from wide web crawls, `core` = core - technical documentation, operating systems, computer hardware, open source and free software, manuals, protocol standards, `scroll` = scroll - non-technical documents: knowledge, encyclopedia, linguistic corpora, dictionaries, translation memories, texts, non-fiction books, historical books, `regula` = regula - non-technical standards: industry standards, laws, rules, compliance, `gem` = gem - research, papers, university publications, science, `fiction` = fiction - fictional documents: movies, stories, series, books (fiction, science-fiction), `map` = map - geological data, geolocation-data, earth/world information, `echo` = echo – micro-content (tweets, toots, short headlines, SMS corpora), podcasts, radio archives, audio lectures, spoken-word datasets, logs, incidents, telemetry, `spirit` = spirit – related to non-textual data (possibly only metadata): art, music, game assets, creative-commons media (non-text culture loot), `vault` = vault - sensitive data: secrets, leaks, non-public documents, security advisories. | `mix` = mix - a mix of document types, for content from wide web crawls, `core` = core - technical documentation, operating systems, computer hardware, open source and free software, manuals, protocol standards, `scroll` = scroll - non-technical documents: knowledge, encyclopedia, linguistic corpora, dictionaries, translation memories, texts, non-fiction books, historical books, `regula` = regula - non-technical standards: industry standards, laws, rules, compliance, `gem` = gem - research, papers, university publications, science, `fiction` = fiction - fictional documents: movies, stories, series, books (fiction, science-fiction), `map` = map - geological data, geolocation-data, earth/world information, `echo` = echo – micro-content (tweets, toots, short headlines, SMS corpora), podcasts, radio archives, audio lectures, spoken-word datasets, logs, incidents, telemetry, `spirit` = spirit – related to non-textual data (possibly only metadata): art, music, game assets, creative-commons media (non-text culture loot), `vault` = vault - sensitive data: secrets, leaks, non-public documents, security advisories | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Collection name such as `user`, `docs`, or a project-specific name. | +| `exportfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | `.*.*` | +| `format` | Choice value. Options: `full-elasticsearch`, `full-solr`, `full-rss`. | `full-elasticsearch`, `full-solr`, `full-rss` | + +## Correct Use + +Generate an index pack only from a deliberate collection or filter. Index packs are meant to be reused elsewhere, so include enough content to be useful but avoid private or irrelevant documents. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexPackGenerator_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexPackGenerator_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexPackGenerator_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexPackGenerator_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `category` | Choice value. Options: `mix` = mix - a mix of document types, for content from wide web crawls, `core` = core - technical documentation, operating systems, computer hardware, open source and free software, manuals, protocol standards, `scroll` = scroll - non-technical documents: knowledge, encyclopedia, linguistic corpora, dictionaries, translation memories, texts, non-fiction books, historical books, `regula` = regula - non-technical standards: industry standards, laws, rules, compliance, `gem` = gem - research, papers, university publications, science, `fiction` = fiction - fictional documents: movies, stories, series, books (fiction, science-fiction), `map` = map - geological data, geolocation-data, earth/world information, `echo` = echo – micro-content (tweets, toots, short headlines, SMS corpora), podcasts, radio archives, audio lectures, spoken-word datasets, logs, incidents, telemetry, `spirit` = spirit – related to non-textual data (possibly only metadata): art, music, game assets, creative-commons media (non-text culture loot), `vault` = vault - sensitive data: secrets, leaks, non-public documents, security advisories. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `exportfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `format` | Choice value. Options: `full-elasticsearch`, `full-solr`, `full-rss`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexPackGenerator_p.html +Content-Type: application/x-www-form-urlencoded + +category=...&collection=...&slug=...&exportfilter=...&exportquery=... +``` + +## What To Expect + +A successful run creates a package artifact or package-generation job. Check the package manager or output location before distributing it. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexPackManager_p.md b/help/IndexPackManager_p.md new file mode 100644 index 000000000..e34f63aad --- /dev/null +++ b/help/IndexPackManager_p.md @@ -0,0 +1,68 @@ +--- +page: htroot/IndexPackManager_p.html +help: help/IndexPackManager_p.md +title: Index Pack Manager +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexPackManager_p.java +--- + +# Index Pack Manager + +## Purpose + +Index Pack Manager supervises installed or available index packages. + +Use it to see which packages are present and how they affect local search. + +## What You Can Do Here + +- Index Pack Manager supervises installed or available index packages. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +## Correct Use + +Use the manager to inspect packages before installing or removing them. Package changes can alter what local searches find without running a new crawl. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexPackManager_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexPackManager_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexPackManager_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/IndexPackManager_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `delete` | Deletes the selected URL, path, or index scope. Confirm the scope first. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /IndexPackManager_p.html?delete=...&file=...&mvfrom=...&mvto=... +``` + +## What To Expect + +The page should show package state, installed packages, or available package actions. Search results may change after package installation or removal. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/IndexReIndexMonitor_p.md b/help/IndexReIndexMonitor_p.md new file mode 100644 index 000000000..2537029dc --- /dev/null +++ b/help/IndexReIndexMonitor_p.md @@ -0,0 +1,94 @@ +--- +page: htroot/IndexReIndexMonitor_p.html +help: help/IndexReIndexMonitor_p.md +title: Field Re-Indexing +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexReIndexMonitor_p.java +--- + +# Field Re-Indexing + +## Purpose + +Field Re-Indexing rebuilds selected index fields for documents already stored. + +Use it after schema, parser, or ranking changes when existing records need recalculation. + +## What You Can Do Here + +- Field Re-Indexing rebuilds selected index fields for documents already stored. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `reindexnow` | Solr query. | `start reindex job now` | +| `stopreindex` | Solr query. | `stop reindexing` | +| `recrawlquerytext` | Solr query / Edit Solr Query. | Text value; use the page label and surrounding context to choose the exact content. | +| `simulateRecrawl` | Solr query. | `Simulate` | +| `includefailedurls` | Include failed URLs / Include failed urls. Options: Include failed URLs, Include failed urls. | Include failed URLs, Include failed urls | +| `deleteOnRecrawl` | Delete URLs / Delete urls. Options: Delete URLs, Delete urls. | Delete URLs, Delete urls | +| `recrawlDefaults` | Solr query. | `Set defaults` | +| `recrawlnow` | Solr query. | `start recrawl job now` | +| `updquery` | Edit Solr Query. | `update` | +| `stoprecrawl` | Edit Solr Query. | `stop recrawl job` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexReIndexMonitor_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexReIndexMonitor_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexReIndexMonitor_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexReIndexMonitor_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `reindexnow` | Solr query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `stopreindex` | Solr query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `recrawlquerytext` | Solr query / Edit Solr Query. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `simulateRecrawl` | Solr query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `includefailedurls` | Include failed URLs / Include failed urls. Options: Include failed URLs, Include failed urls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteOnRecrawl` | Delete URLs / Delete urls. Options: Delete URLs, Delete urls. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `recrawlDefaults` | Solr query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `recrawlnow` | Solr query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `updquery` | Edit Solr Query. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `stoprecrawl` | Edit Solr Query. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /IndexReIndexMonitor_p.html +Content-Type: application/x-www-form-urlencoded + +reindexnow=...&stopreindex=...&recrawlquerytext=...&simulateRecrawl=...&includefailedurls=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- Related index work is usually reached through `IndexBrowser_p.html`, `IndexControlURLs_p.html`, queue pages, deletion pages, or re-indexing monitors. diff --git a/help/IndexSchema_p.md b/help/IndexSchema_p.md new file mode 100644 index 000000000..7b86ef6ef --- /dev/null +++ b/help/IndexSchema_p.md @@ -0,0 +1,82 @@ +--- +page: htroot/IndexSchema_p.html +help: help/IndexSchema_p.md +title: Solr Schema Editor +package: index-management +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexSchema_p.java +--- + +# Solr Schema Editor + +## Purpose + +Solr Schema Editor exposes fields used by YaCy's Solr index. + +Use it to inspect field availability and advanced schema behavior before changing ranking or import assumptions. + +## What You Can Do Here + +- Solr Schema Editor exposes fields used by YaCy's Solr index. +- Select the narrowest URL, host, field, query, queue, or collection scope. +- Treat deletion, rebuilding, and reloading as maintenance operations with visible search impact. + +## Page Architecture + +Index pages expose stored documents through URL, host, path, Solr field, or queue views. Read-only inspection and destructive maintenance often share the same page, so the target scope matters more than the button label. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `filter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | `active`, `disabled` | +| `schema_#[key]#` | Choice value. Options: `checked`. | `checked` | +| `set` | Submits and applies the basic configuration. | `Set` | +| `reindexSolr` | Reindex documents. | `reindex Solr` | + +## Correct Use + +Use inspection before maintenance. First identify the exact URL, host, field, queue, collection, or query scope, then choose the action. Deletion and re-indexing can be expensive or irreversible from the user's point of view, so never broaden the scope just to make a command easier. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexSchema_p.html`, `/IndexReIndexMonitor_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexSchema_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexSchema_p.html` | `GET` | admin | `source/net/yacy/htroot/IndexSchema_p.java` | +| `/IndexReIndexMonitor_p.html` | `POST` | admin | `source/net/yacy/htroot/IndexReIndexMonitor_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `filter` | Filter text or expression used to narrow the displayed records. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `schema_#[key]#` | Choice value. Options: `checked`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `reindexSolr` | Reindex documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteOnRecrawl` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `reindexnow` | Reindex documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /IndexSchema_p.html?core=...&filter=...&schema_#[key]#=...&schema_solrfieldname_#[key]#=...&set=... +``` + +## What To Expect + +The response should make the selected index scope clearer: records listed, queues changed, errors shown, fields rebuilt, or deletion confirmed. Verify user-visible impact with search after maintenance actions. + +## Related Pages + +- `IndexReIndexMonitor_p.html` diff --git a/help/IndexShare_p.md b/help/IndexShare_p.md new file mode 100644 index 000000000..c062120c3 --- /dev/null +++ b/help/IndexShare_p.md @@ -0,0 +1,69 @@ +--- +page: htroot/IndexShare_p.html +help: help/IndexShare_p.md +title: Index Sharing +package: import-export-federation +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/IndexShare_p.java +--- + +# Index Sharing + +## Purpose + +Index Sharing controls how this peer shares index information with others. + +Use it to decide what local search knowledge may leave the peer. + +## What You Can Do Here + +- Index Sharing controls how this peer shares index information with others. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/IndexShare_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/IndexShare_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/IndexShare_p.html` | `GET` | admin | `source/net/yacy/htroot/IndexShare_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET /IndexShare_p.html?distribute=...&receive=...&linkfreq=...&wordfreq=...&indexsharesetting=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- Related transfer work is usually reached through the import/export page for the same format, the index-pack pages, or the queue/status page that reports progress. diff --git a/help/LLMSelection_p.md b/help/LLMSelection_p.md new file mode 100644 index 000000000..7194c0896 --- /dev/null +++ b/help/LLMSelection_p.md @@ -0,0 +1,68 @@ +--- +page: htroot/LLMSelection_p.html +help: help/LLMSelection_p.md +title: LLM Selection +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/LLMSelection_p.java +--- + +# LLM Selection + +## Purpose + +LLM Selection chooses the language model provider or model profile used by YaCy's AI and retrieval features. + +Use it before RAG or AI analysis work so later pages know which model endpoint and capability profile to use. + +## What You Can Do Here + +- Choose the LLM service family YaCy should use for AI-assisted workflows. +- Set the host or endpoint stub so later RAG and AI pages know where to send model requests. +- Verify the selected service before troubleshooting model-assisted answers elsewhere. + +## Page Architecture + +The page chooses an LLM service family and base host for model-assisted features. `service` selects the integration style, for example Ollama, LM Studio, OpenAI, or Open Router. `hoststub` points YaCy at the local or remote API base that will receive model requests. + +## Correct Use + +Select the service that matches the running model endpoint. For local tools such as Ollama or LM Studio, verify the local server URL first. For hosted services, treat API keys and host settings as sensitive operational configuration. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/LLMSelection_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/LLMSelection_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/LLMSelection_p.html` | `GET` | admin | `source/net/yacy/htroot/LLMSelection_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `service` | Choice value. Options: `OLLAMA` = Ollama, `LMSTUDIO` = LMStudio, `OPENAI` = OpenAI, `OPENROUTER` = Open Router. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hoststub` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /LLMSelection_p.html?service=...&hoststub=...&apikey=...&llmselection=...&BODY=... +``` + +## What To Expect + +After saving, RAG and AI analysis pages should use the selected provider profile. Model failures after this point usually mean the model server, API key, host URL, or model name needs checking. + +## Related Pages + +- Related quality work usually continues on ranking settings, content analysis, LLM selection, RAG configuration, or a representative search result page. diff --git a/help/Load_MediawikiWiki.md b/help/Load_MediawikiWiki.md new file mode 100644 index 000000000..41d78007f --- /dev/null +++ b/help/Load_MediawikiWiki.md @@ -0,0 +1,133 @@ +--- +page: htroot/Load_MediawikiWiki.html +help: help/Load_MediawikiWiki.md +title: Configuration of a Wiki Search +package: import-export-federation +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Load_MediawikiWiki.java +--- + +# Configuration of a Wiki Search + +## Purpose + +Wiki Search configuration prepares MediaWiki content as a search source. + +Use it when a wiki should become searchable through YaCy with a repeatable setup. + +## What You Can Do Here + +- Wiki Search configuration prepares MediaWiki content as a search source. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Absolute URL, for example `https://example.org/docs/`. | +| `crawlingstart` | Submit action that starts the crawl. | `Get content of Wiki: crawl wiki pages` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Protected related endpoint(s): `/Crawler_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Load_MediawikiWiki.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Crawler_p.html` | `POST` | admin | `source/net/yacy/htroot/Crawler_p.java` | +| `/Load_MediawikiWiki.html` | `GET or POST` | public or page-dependent | `source/net/yacy/htroot/Load_MediawikiWiki.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingstart` | Submit action that creates a crawl job. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cachePolicy` | Cache strategy for fetching documents, for example whether cached material may be reused when fresh enough. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cleanSearchCache` | Clears cached search results so newly crawled material can appear without stale search state. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchSwitch` | Enables country-code filtering for crawl targets. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlOrder` | Crawl ordering strategy, such as balanced host scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlerAlwaysCheckMediaType` | Check media type before deciding parser/indexing behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepthExtension` | Additional depth behavior for special crawl modes. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxCheck` | Enables the maximum-page safeguard for a domain crawl. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile$file` | Multipart file content containing crawl URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingMode` | Crawl source mode. `url` starts from one URL, `sitemap` reads sitemap entries, `file`/list modes submit many URLs, and page-specific modes may prepare these values for `Crawler_p.html`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingPerformance` | Crawler speed preset. Use slower presets for shared servers; use custom only when you understand the load impact. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `default_valency` | Default link valency used by the crawler when deciding how links contribute to discovery and indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `directDocByURL` | Treats the submitted URL as a direct document target rather than mainly as a link-discovery seed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hidewebstructuregraph` | Hide the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `intention` | Optional human label describing why the crawl was started. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `latencyFactor` | Politeness multiplier for crawl delay. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `pause` | Pauses the selected queue or crawl process. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `queues_terminate_all` | Stops all crawler queues. This is a broad emergency control, not a normal crawl setting. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `range` | Boundary for a simple site crawl. `domain` stays in the same domain; `subpath` stays below the start path; expert pages may expose wider policies. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `recrawl` | Recrawl policy for already known URLs, such as avoiding duplicates, reloading, or using scheduler rules depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Stores fetched documents in YaCy hypertext cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Crawler_p.html +Content-Type: application/x-www-form-urlencoded + +crawlingURL=...&range=...&crawlingDepth=...&crawlingDomMaxPages=...&crawlingstart=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- `ConfigLiveSearch.html` diff --git a/help/Load_PHPBB3.md b/help/Load_PHPBB3.md new file mode 100644 index 000000000..9a6fb93a5 --- /dev/null +++ b/help/Load_PHPBB3.md @@ -0,0 +1,134 @@ +--- +page: htroot/Load_PHPBB3.html +help: help/Load_PHPBB3.md +title: Configuration of a phpBB3 Search +package: import-export-federation +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Load_PHPBB3.java +--- + +# Configuration of a phpBB3 Search + +## Purpose + +phpBB3 Search configuration prepares forum content as a search source. + +Use it when a phpBB3 database or forum should be indexed as structured discussion content. + +## What You Can Do Here + +- phpBB3 Search configuration prepares forum content as a search source. +- Define the source, target, format, collection, and expected size before starting. +- Monitor progress because large transfers continue beyond the initial request. + +## Page Architecture + +Import and export pages translate external data formats into YaCy documents or move YaCy index data into another store. Most long-running actions create background work that should be monitored afterward. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Absolute URL, for example `https://example.org/docs/`. | +| `crawlingstart` | Submit action that starts the crawl. | `Get content of phpBB3: crawl forum pages` | + +## Correct Use + +Prepare source and target details before starting: file path or URL, format, collection, credentials if needed, and expected size. Imports and exports can continue in the background, so confirm progress on the related monitor or queue page. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Protected related endpoint(s): `/Crawler_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Load_PHPBB3.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Crawler_p.html` | `POST` | admin | `source/net/yacy/htroot/Crawler_p.java` | +| `/Load_PHPBB3.html` | `GET or POST` | public or page-dependent | `source/net/yacy/htroot/Load_PHPBB3.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlingURL` | Start URL for a crawl. Include `http://` or `https://`; for local files or intranet protocols use the expert crawler where appropriate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingstart` | Submit action that creates a crawl job. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `MaxSameHostInQueue` | Maximum queued URLs allowed for the same host. It prevents one host from occupying too much of the crawl queue. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cachePolicy` | Cache strategy for fetching documents, for example whether cached material may be reused when fresh enough. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `cleanSearchCache` | Clears cached search results so newly crawled material can appear without stale search state. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `countryMustMatchList` | Country-code allow list used when country filtering is enabled. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `countryMustMatchSwitch` | Enables country-code filtering for crawl targets. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawlOrder` | Crawl ordering strategy, such as balanced host scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlerAlwaysCheckMediaType` | Check media type before deciding parser/indexing behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepthExtension` | Additional depth behavior for special crawl modes. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxCheck` | Enables the maximum-page safeguard for a domain crawl. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDomMaxPages` | Maximum number of pages allowed when `crawlingDomMaxCheck` is enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile` | Uploaded file name or submitted URL-list source for multi-URL crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingFile$file` | Multipart file content containing crawl URLs. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingMode` | Crawl source mode. `url` starts from one URL, `sitemap` reads sitemap entries, `file`/list modes submit many URLs, and page-specific modes may prepare these values for `Crawler_p.html`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingPerformance` | Crawler speed preset. Use slower presets for shared servers; use custom only when you understand the load impact. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `customPPM` | Custom pages-per-minute target used with custom crawl performance. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `default_valency` | Default link valency used by the crawler when deciding how links contribute to discovery and indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `deleteIfOlderNumber` | Number part of the age threshold for deleting old documents. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteIfOlderUnit` | Unit for deleting old documents, typically `year`, `month`, `day`, or `hour`. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteold` | Policy for deleting or replacing older index entries during crawl setup. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `directDocByURL` | Treats the submitted URL as a direct document target rather than mainly as a link-discovery seed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `domlistlength` | Number of submitted domain-list entries. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `handle` | Crawl profile or job handle. Use the exact value shown by YaCy for the job you want to control. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hidewebstructuregraph` | Hide the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexcontentmustmatch` | Regular expression that extracted content must match before indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexcontentmustnotmatch` | Regular expression excluding documents by extracted content. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustmatch` | Regular expression that fetched URLs must match before they are indexed. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `indexmustnotmatch` | Regular expression excluding fetched URLs from indexing. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `intention` | Optional human label describing why the crawl was started. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ipMustmatch` | IP address pattern that target hosts must match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `ipMustnotmatch` | IP address pattern that target hosts must not match. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `latencyFactor` | Politeness multiplier for crawl delay. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `noindexWhenCanonicalUnequalURL` | Skips indexing when the document declares a canonical URL different from the fetched URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `pause` | Pauses the selected queue or crawl process. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `queues_terminate_all` | Stops all crawler queues. This is a broad emergency control, not a normal crawl setting. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `range` | Boundary for a simple site crawl. `domain` stays in the same domain; `subpath` stays below the start path; expert pages may expose wider policies. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `recrawl` | Recrawl policy for already known URLs, such as avoiding duplicates, reloading, or using scheduler rules depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderNumber` | Number part of the age threshold for refreshing older documents. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reloadIfOlderUnit` | Unit for the refresh threshold, typically `year`, `month`, `day`, or `hour`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `showwebstructuregraph` | Show the web-structure graph on crawler pages. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sitemapURL` | Sitemap URL used as the crawl source. It should point to a valid XML sitemap or sitemap index. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Stores fetched documents in YaCy hypertext cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `terminate` | Terminates the selected crawl profile or running crawl. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `valency_switch_tag_names` | HTML tag names whose links use switched valency behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Crawler_p.html +Content-Type: application/x-www-form-urlencoded + +crawlingURL=...&range=...&crawlingDepth=...&crawlingDomMaxPages=...&crawlingstart=... +``` + +## What To Expect + +Small operations may finish during the request; large imports, exports, harvests, and package operations usually need monitoring. Expect progress, logs, queue entries, or generated files rather than instant final search quality. + +## Related Pages + +- `ContentIntegrationPHPBB3_p.html` +- `ConfigLiveSearch.html` diff --git a/help/Load_RSS_p.md b/help/Load_RSS_p.md new file mode 100644 index 000000000..17cf8305e --- /dev/null +++ b/help/Load_RSS_p.md @@ -0,0 +1,93 @@ +--- +page: htroot/Load_RSS_p.html +help: help/Load_RSS_p.md +title: Configuration of a RSS Search +package: crawler +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Load_RSS_p.java +--- + +# Configuration of a RSS Search + +## Purpose + +RSS Search configuration turns feeds into crawl or import sources. + +Use it when new documents arrive through RSS or Atom rather than normal website navigation. + +## What You Can Do Here + +- RSS Search configuration turns feeds into crawl or import sources. +- Choose limits, boundaries, and queue actions that match the size of the source. +- Watch the crawler monitor afterward because indexing happens after fetching and parsing. + +## Page Architecture + +Crawler pages separate three decisions: the source to load, the rules that decide which discovered URLs are accepted, and the monitor or control action that follows the job after it starts. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Collection name such as `user`, `docs`, or a project-specific name. | +| `repeat` | Choice value. Options: `off`, `on`. | `off`, `on` | +| `repeat_time` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `28`, `30`. | `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `28`, `30` | +| `repeat_unit` | Choice value. Options: `selminutes` = minutes, `selhours` = hours, `seldays` = days. | `selminutes` = minutes, `selhours` = hours, `seldays` = days | +| `item_#[count]#` | Choice value. Options: `mark_`. | `mark_` | +| `removeSelectedFeedsScheduler` | Deletion or termination action. Use only with explicit intent. | `Remove Selected Feeds from Scheduler` | +| `removeAllFeedsScheduler` | Deletion or termination action. Use only with explicit intent. | `Remove All Feeds from Scheduler` | +| `removeSelectedFeedsNewList` | Deletion or termination action. Use only with explicit intent. | `Remove Selected Feeds from Feed List` | +| `removeAllFeedsNewList` | Deletion or termination action. Use only with explicit intent. | `Remove All Feeds from Feed List` | + +## Correct Use + +Begin with the smallest crawl that proves the idea. Use exact start URLs, prefer restrictive boundaries, set limits for unfamiliar sites, and watch the queue after submission. A crawler is not a magic search box: it creates search results only after documents have been loaded, parsed, and indexed. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Load_RSS_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Load_RSS_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Load_RSS_p.html` | `POST` | admin | `source/net/yacy/htroot/Load_RSS_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat` | Choice value. Options: `off`, `on`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_time` | Choice value. Options: `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`, `9`, `10`, `12`, `14`, `21`, `28`, `30`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_unit` | Choice value. Options: `selminutes` = minutes, `selhours` = hours, `seldays` = days. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `item_#[count]#` | Choice value. Options: `mark_`. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `removeSelectedFeedsScheduler` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `removeAllFeedsScheduler` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `removeSelectedFeedsNewList` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `removeAllFeedsNewList` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Load_RSS_p.html +Content-Type: application/x-www-form-urlencoded + +url=...&showrss=...&collection=...&repeat=...&repeat_time=... +``` + +## What To Expect + +A successful action usually changes crawl state rather than producing finished search results immediately. Expect queued URLs, progress counters, success or error reports, and later changes in search results after indexing catches up. + +## Related Pages + +- Related crawler work is usually reached through `Crawler_p.html`, `CrawlStartSite.html`, `CrawlStartExpert.html`, or crawl result and queue monitors. diff --git a/help/LogReports_p.md b/help/LogReports_p.md new file mode 100644 index 000000000..bd2add9ee --- /dev/null +++ b/help/LogReports_p.md @@ -0,0 +1,73 @@ +--- +page: htroot/LogReports_p.html +help: help/LogReports_p.md +title: Log Reports +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/LogReports_p.java +--- + +# Log Reports + +## Purpose + +Log Reports summarize notable events from YaCy logs. + +Use it to read operational history without scanning raw log files first. + +## What You Can Do Here + +- Log Reports summarize notable events from YaCy logs. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/LogReports_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/LogReports_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/LogReports_p.html` | `POST` | admin | `source/net/yacy/htroot/LogReports_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `deleteReport` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /LogReports_p.html +Content-Type: application/x-www-form-urlencoded + +runReportNow=...&deleteReport=...&report=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `LLMSelection_p.html` diff --git a/help/MessageSend_p.md b/help/MessageSend_p.md new file mode 100644 index 000000000..fd15ab6e0 --- /dev/null +++ b/help/MessageSend_p.md @@ -0,0 +1,84 @@ +--- +page: htroot/MessageSend_p.html +help: help/MessageSend_p.md +title: Send message +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/MessageSend_p.java +--- + +# Send message + +## Purpose + +Send Message composes a peer-local or network message. + +Use it to contact another peer or user when messaging is enabled. + +## What You Can Do Here + +- Send Message composes a peer-local or network message. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `subject` | Subject. | Text value; use the page label and surrounding context to choose the exact content. | +| `message` | Text. | Text value; use the page label and surrounding context to choose the exact content. | +| `new` | Subject. | `Enter` | +| `preview` | Subject. | `Preview` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/MessageSend_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/MessageSend_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/MessageSend_p.html` | `POST` | admin | `source/net/yacy/htroot/MessageSend_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `subject` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `message` | Text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `new` | Subject. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `preview` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `attachmentsize` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `messagesize` | Subject. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /MessageSend_p.html +Content-Type: application/x-www-form-urlencoded + +subject=...&message=...&new=...&preview=...&attachmentsize=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `WikiHelp.html` +- `Network.html` diff --git a/help/Messages_p.md b/help/Messages_p.md new file mode 100644 index 000000000..4f59603d8 --- /dev/null +++ b/help/Messages_p.md @@ -0,0 +1,75 @@ +--- +page: htroot/Messages_p.html +help: help/Messages_p.md +title: Messages +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Messages_p.java +--- + +# Messages + +## Purpose + +Messages lists received or stored messages. + +Use it to read, manage, or inspect peer communication. + +## What You Can Do Here + +- Messages lists received or stored messages. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Exact YaCy hash value. | +| `submit` | Submits the form. | `Compose` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/MessageSend_p.html`, `/Messages_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Messages_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/MessageSend_p.html` | `GET` | admin | `source/net/yacy/htroot/MessageSend_p.java` | +| `/Messages_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/Messages_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /MessageSend_p.html?hash=...&submit=...&action=...&attachmentsize=...&message=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `ViewProfile.html` +- `MessageSend_p.html` diff --git a/help/Network.md b/help/Network.md new file mode 100644 index 000000000..9d70e2498 --- /dev/null +++ b/help/Network.md @@ -0,0 +1,87 @@ +--- +page: htroot/Network.html +help: help/Network.md +title: YaCy Search Network +package: monitoring-performance +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/Network.java +--- + +# YaCy Search Network + +## Purpose + +YaCy Search Network shows known peers and network state. + +Use it to understand whether this peer is isolated, connected, senior, junior, or participating normally. + +## What You Can Do Here + +- YaCy Search Network shows known peers and network state. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `match` | Search for a peername (RegExp allowed). | Pattern or filter expression; test narrow expressions before broad use. | +| `search` | Alternative query parameter. | `Search` | +| `peerHash` | Search for a peername (RegExp allowed). | Text value; use the page label and surrounding context to choose the exact content. | +| `peerIP` | Search for a peername (RegExp allowed). | Text value; use the page label and surrounding context to choose the exact content. | +| `peerPort` | Search for a peername (RegExp allowed). | Text value; use the page label and surrounding context to choose the exact content. | +| `addPeer` | Search for a peername (RegExp allowed). | `add Peer` | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/Network.html`. + +Backend checks: administrator authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Network.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Network.html` | `GET` | mixed | `source/net/yacy/htroot/Network.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `match` | Search for a peername (RegExp allowed). | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `peerHash` | Search for a peername (RegExp allowed). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `peerIP` | Search for a peername (RegExp allowed). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `peerPort` | Search for a peername (RegExp allowed). | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `addPeer` | Search for a peername (RegExp allowed). | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `page` | Search for a peername (RegExp allowed). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sort` | Sort field or sort direction. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET /Network.html?match=...&search=...&peerHash=...&peerIP=...&peerPort=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `MessageSend_p.html` +- `ViewProfile.html` +- `goto_p.html` diff --git a/help/News.md b/help/News.md new file mode 100644 index 000000000..9153ae8e3 --- /dev/null +++ b/help/News.md @@ -0,0 +1,78 @@ +--- +page: htroot/News.html +help: help/News.md +title: News Monitor +package: content-apps +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/News.java +--- + +# News Monitor + +## Purpose + +News Monitor shows YaCy network news and peer events. + +Use it to understand what other peers announced or what the local peer received. + +## What You Can Do Here + +- News Monitor shows YaCy network news and peer events. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `deletespecific` | Deletion or termination action. Use only with explicit intent. | `Process Selected News::Delete Selected News::Abort Publication of Selected News::Delete Selected News` | +| `deleteall` | Deletion or termination action. Use only with explicit intent. | `Process All News::Delete All News::Abort Publication of All News::Delete All News` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/News.html`. + +Backend checks: administrator authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/News.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/News.html` | `POST` | mixed | `source/net/yacy/htroot/News.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `deletespecific` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deleteall` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +POST /News.html +Content-Type: application/x-www-form-urlencoded + +deletespecific=...&deleteall=...&del_#[id]#=...&page=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `Translator_p.html` diff --git a/help/PerformanceConcurrency_p.md b/help/PerformanceConcurrency_p.md new file mode 100644 index 000000000..ae0e312ba --- /dev/null +++ b/help/PerformanceConcurrency_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/PerformanceConcurrency_p.html +help: help/PerformanceConcurrency_p.md +title: Performance of Concurrent Processes +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/PerformanceConcurrency_p.java +--- + +# Performance of Concurrent Processes + +## Purpose + +Concurrent Process Performance shows and tunes parallel work. + +Use it when crawling, indexing, or search competes for CPU and threads. + +## What You Can Do Here + +- Concurrent Process Performance shows and tunes parallel work. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/PerformanceConcurrency_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/PerformanceMemory_p.md b/help/PerformanceMemory_p.md new file mode 100644 index 000000000..2f74a32fa --- /dev/null +++ b/help/PerformanceMemory_p.md @@ -0,0 +1,76 @@ +--- +page: htroot/PerformanceMemory_p.html +help: help/PerformanceMemory_p.md +title: Performance Settings for Memory +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/PerformanceMemory_p.java +--- + +# Performance Settings for Memory + +## Purpose + +Memory Performance shows memory use and related limits. + +Use it when the peer slows down, swaps, or approaches heap limits. + +## What You Can Do Here + +- Memory Performance shows memory use and related limits. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `option` | refresh graph. | refresh graph | +| `simulatedshortmemory` | simulate short memory status. | simulate short memory status | +| `useStandardmemoryStrategy` | use Standard Memory Strategy. | use Standard Memory Strategy | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/PerformanceMemory_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/PerformanceMemory_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/PerformanceMemory_p.html` | `GET` | admin | `source/net/yacy/htroot/PerformanceMemory_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `option` | refresh graph. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `simulatedshortmemory` | simulate short memory status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `useStandardmemoryStrategy` | use Standard Memory Strategy. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `dummy` | simulate short memory status. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /PerformanceMemory_p.html?option=...&simulatedshortmemory=...&useStandardmemoryStrategy=...&dummy=...&gc=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/PerformanceQueues_p.md b/help/PerformanceQueues_p.md new file mode 100644 index 000000000..2edd5b5d3 --- /dev/null +++ b/help/PerformanceQueues_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/PerformanceQueues_p.html +help: help/PerformanceQueues_p.md +title: Performance Settings of Queues and Processes +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/PerformanceQueues_p.java +--- + +# Performance Settings of Queues and Processes + +## Purpose + +Queue Performance shows processing queues and worker behavior. + +Use it to find bottlenecks between crawling, parsing, indexing, and response generation. + +## What You Can Do Here + +- Queue Performance shows processing queues and worker behavior. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `#[name]#_maxActive` | Enables the named feature. | Integer value. | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/PerformanceQueues_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/PerformanceQueues_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/PerformanceQueues_p.html` | `POST` | admin | `source/net/yacy/htroot/PerformanceQueues_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `#[name]#_maxActive` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `Crawler Pool_maxActive` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `Robots.txt Pool_maxActive` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlPauseRemotesearch` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `httpd Session Pool_maxActive` | Enables the named feature. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /PerformanceQueues_p.html +Content-Type: application/x-www-form-urlencoded + +#[threadname]#_idlesleep=...&#[threadname]#_busysleep=...&#[threadname]#_memprereq=...&#[threadname]#_loadprereq=...&submitdelay=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/PerformanceSearch_p.md b/help/PerformanceSearch_p.md new file mode 100644 index 000000000..0313cd5df --- /dev/null +++ b/help/PerformanceSearch_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/PerformanceSearch_p.html +help: help/PerformanceSearch_p.md +title: Performance Settings of Search Sequence +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/PerformanceSearch_p.java +--- + +# Performance Settings of Search Sequence + +## Purpose + +Search Sequence Performance breaks down search timing. + +Use it when queries are slow and you need to know which search step costs time. + +## What You Can Do Here + +- Search Sequence Performance breaks down search timing. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/PerformanceSearch_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/Performance_p.md b/help/Performance_p.md new file mode 100644 index 000000000..d73eaf7b0 --- /dev/null +++ b/help/Performance_p.md @@ -0,0 +1,100 @@ +--- +page: htroot/Performance_p.html +help: help/Performance_p.md +title: Performance Settings +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Performance_p.java +--- + +# Performance Settings + +## Purpose + +Performance Settings collect runtime tuning controls. + +Use it when the peer needs different resource behavior after observing actual load. + +## What You Can Do Here + +- Performance Settings collect runtime tuning controls. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `option` | refresh graph. | refresh graph | +| `Xmx` | Memory reserved for JVM. | Text value; use the page label and surrounding context to choose the exact content. | +| `setStartup` | Memory reserved for JVM. | `Set` | +| `diskFree` | Steady-state minimum. | Text value; use the page label and surrounding context to choose the exact content. | +| `diskFreeHardlimit` | Absolute minimum. | Text value; use the page label and surrounding context to choose the exact content. | +| `diskFreeAutoregulate` | Autoregulate. | Autoregulate | +| `diskUsed` | Steady-state maximum. | Text value; use the page label and surrounding context to choose the exact content. | +| `diskUsedHardlimit` | Absolute maximum. | Text value; use the page label and surrounding context to choose the exact content. | +| `diskUsedAutoregulate` | Autoregulate. | Autoregulate | +| `resetObserver` | Minimum required. | `Reset state` | +| `memoryAcceptDHT` | Minimum required. | Text value; use the page label and surrounding context to choose the exact content. | +| `setObserver` | Steady-state minimum. | `Save` | +| `crawlPauseRemotesearch` | User or account value. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Performance_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Performance_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Performance_p.html` | `GET` | admin | `source/net/yacy/htroot/Performance_p.java` | +| `/Steering.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/Steering.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `option` | refresh graph. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `Xmx` | Memory reserved for JVM. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `setStartup` | Memory reserved for JVM. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `diskFree` | Steady-state minimum. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diskFreeHardlimit` | Absolute minimum. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diskFreeAutoregulate` | Autoregulate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diskUsed` | Steady-state maximum. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diskUsedHardlimit` | Absolute maximum. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diskUsedAutoregulate` | Autoregulate. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resetObserver` | Minimum required. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `memoryAcceptDHT` | Minimum required. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `setObserver` | Steady-state minimum. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `crawlPauseRemotesearch` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `restart` | refresh graph. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `update` | Submit action that refreshes, updates, or applies the selected setting depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /Performance_p.html?option=...&Xmx=...&setStartup=...&diskFree=...&diskFreeHardlimit=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/ProxyIndexingMonitor_p.md b/help/ProxyIndexingMonitor_p.md new file mode 100644 index 000000000..f58440c7f --- /dev/null +++ b/help/ProxyIndexingMonitor_p.md @@ -0,0 +1,91 @@ +--- +page: htroot/ProxyIndexingMonitor_p.html +help: help/ProxyIndexingMonitor_p.md +title: Indexing with Proxy +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ProxyIndexingMonitor_p.java +--- + +# Indexing with Proxy + +## Purpose + +Proxy Indexing Monitor shows documents indexed through proxy use. + +Use it when YaCy is configured as a proxy and browsing activity feeds the index. + +## What You Can Do Here + +- Proxy Indexing Monitor shows documents indexed through proxy use. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `proxyYacyOnly` | .yacy-domains only. | .yacy-domains only | +| `proxyPrefetchDepth` | Prefetch Depth. | Integer value. | +| `proxyStoreHTCache` | Store to Cache. | Store to Cache | +| `proxyIndexingLocalText` | Do Local Text-Indexing. | Do Local Text-Indexing | +| `proxyIndexingLocalMedia` | Do Local Media-Indexing. | Do Local Media-Indexing | +| `proxyIndexingRemote` | Do Remote Indexing. | Do Remote Indexing | +| `proxyCache` | Path. | Text value; use the page label and surrounding context to choose the exact content. | +| `proxyCacheSize` | Size. | Text value; use the page label and surrounding context to choose the exact content. | +| `proxyprofileset` | .yacy-domains only. | `Set proxy profile` | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ProxyIndexingMonitor_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ProxyIndexingMonitor_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ProxyIndexingMonitor_p.html` | `POST` | admin | `source/net/yacy/htroot/ProxyIndexingMonitor_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `proxyYacyOnly` | .yacy-domains only. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyPrefetchDepth` | Prefetch Depth. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyStoreHTCache` | Store to Cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyIndexingLocalText` | Do Local Text-Indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyIndexingLocalMedia` | Do Local Media-Indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyIndexingRemote` | Do Remote Indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyCache` | Path. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyCacheSize` | Size. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `proxyprofileset` | .yacy-domains only. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ProxyIndexingMonitor_p.html +Content-Type: application/x-www-form-urlencoded + +proxyYacyOnly=...&proxyPrefetchDepth=...&proxyStoreHTCache=...&proxyIndexingLocalText=...&proxyIndexingLocalMedia=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `Settings_p.html` +- `CrawlResults.html` diff --git a/help/QuickCrawlLink_p.md b/help/QuickCrawlLink_p.md new file mode 100644 index 000000000..fb7b4b120 --- /dev/null +++ b/help/QuickCrawlLink_p.md @@ -0,0 +1,81 @@ +--- +page: htroot/QuickCrawlLink_p.html +help: help/QuickCrawlLink_p.md +title: Quick Crawl Link +package: core-search-public +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/QuickCrawlLink_p.java +--- + +# Quick Crawl Link + +## Purpose + +Quick Crawl Link is a shortcut for submitting one URL quickly. + +Use it for a small, immediate fetch when a full crawl profile would be unnecessary. + +## What You Can Do Here + +- Quick Crawl Link is a shortcut for submitting one URL quickly. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Use this page for one URL or a very small task. If the request needs depth, filters, collections, recrawl policy, or careful politeness settings, use the normal crawl start pages instead. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/QuickCrawlLink_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/QuickCrawlLink_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/QuickCrawlLink_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/QuickCrawlLink_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlOrder` | Crawl ordering strategy, such as balanced host scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingDepth` | Maximum link depth from the start URL. Depth 0 loads only the submitted document; larger values follow links farther away. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawlingQ` | Queues discovered URLs for crawler processing; sitemap mode normally enables queued crawling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `followFrames` | Allows the crawler to follow frame and iframe sources. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexMedia` | Indexes discovered media resources when enabled. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexText` | Indexes extracted text content when enabled. Disable only for specialized media-only crawls. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mustmatch` | Regular expression that discovered URLs must match before they enter the crawl. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `mustnotmatch` | Regular expression that discovered URLs must not match. Use it to exclude logout URLs, calendars, filters, or unwanted directories. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `obeyHtmlRobotsNofollow` | Honors HTML robots `nofollow` instructions while discovering links. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `obeyHtmlRobotsNoindex` | Honors HTML robots `noindex` instructions while indexing. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `storeHTCache` | Stores fetched documents in YaCy hypertext cache. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `title` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /QuickCrawlLink_p.html?crawlingDepth=...&url=...&collection=...&crawlOrder=...&crawlingQ=... +``` + +## What To Expect + +YaCy should queue or load the submitted URL quickly, but search results appear only after fetching, parsing, and indexing finish. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/RAGConfig_p.md b/help/RAGConfig_p.md new file mode 100644 index 000000000..b627a0749 --- /dev/null +++ b/help/RAGConfig_p.md @@ -0,0 +1,74 @@ +--- +page: htroot/RAGConfig_p.html +help: help/RAGConfig_p.md +title: Wire RAG Retrieval +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/RAGConfig_p.java +--- + +# Wire RAG Retrieval + +## Purpose + +Wire RAG Retrieval configures how YaCy turns indexed documents into context for retrieval-augmented generation. + +Use it to tune the bridge between search results and generated answers: retrieval scope, context shape, and model-facing behavior. + +## What You Can Do Here + +- Wire RAG Retrieval configures how YaCy turns indexed documents into context for retrieval-augmented generation. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `ai.llm-user-prefix` | User or account value. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/RAGConfig_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/RAGConfig_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/RAGConfig_p.html` | `POST` | admin | `source/net/yacy/htroot/RAGConfig_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `ai.llm-user-prefix` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /RAGConfig_p.html +Content-Type: application/x-www-form-urlencoded + +ai.system-prompt=...&ai.llm-user-prefix=...&ai.llm-query-generator-prefix=...&ai.rag.search-document-maxlength=... +``` + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- Related quality work usually continues on ranking settings, content analysis, LLM selection, RAG configuration, or a representative search result page. diff --git a/help/RankingRWI_p.md b/help/RankingRWI_p.md new file mode 100644 index 000000000..3d4138407 --- /dev/null +++ b/help/RankingRWI_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/RankingRWI_p.html +help: help/RankingRWI_p.md +title: RWI Ranking Configuration +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/RankingRWI_p.java +--- + +# RWI Ranking Configuration + +## Purpose + +RWI Ranking Configuration controls ranking for YaCy's reverse word index. + +Use it to tune classic YaCy ranking signals such as word statistics, locality, authority, and document properties. + +## What You Can Do Here + +- RWI Ranking Configuration controls ranking for YaCy's reverse word index. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/RankingRWI_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/RankingRWI_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/RankingRWI_p.html` | `POST` | admin | `source/net/yacy/htroot/RankingRWI_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /RankingRWI_p.html +Content-Type: application/x-www-form-urlencoded + +#[nameorg]#=...&slider_#[nameorg]#=...&EnterRanking=...&ResetRanking=... +``` + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- Related quality work usually continues on ranking settings, content analysis, LLM selection, RAG configuration, or a representative search result page. diff --git a/help/RankingSolr_p.md b/help/RankingSolr_p.md new file mode 100644 index 000000000..4e4593c83 --- /dev/null +++ b/help/RankingSolr_p.md @@ -0,0 +1,96 @@ +--- +page: htroot/RankingSolr_p.html +help: help/RankingSolr_p.md +title: Solr Ranking Configuration +package: ranking-ai-analysis +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/RankingSolr_p.java +--- + +# Solr Ranking Configuration + +## Purpose + +Solr Ranking Configuration controls how Solr fields influence result order. + +Use it when the same documents are found but the order does not match the user's idea of relevance. + +## What You Can Do Here + +- Solr Ranking Configuration controls how Solr fields influence result order. +- Use representative queries or documents when evaluating changes. +- Change one model, field, weight, or threshold at a time so the effect can be explained. + +## Page Architecture + +Ranking and analysis pages expose the signals that influence result order or retrieval augmentation. The visible form usually maps directly to weights, model choices, field selections, or diagnostic thresholds. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `profileNr` | boost= / bq= / fq=. | Text value; use the page label and surrounding context to choose the exact content. | +| `bf` | boost=. | Text value; use the page label and surrounding context to choose the exact content. | +| `EnterBF` | boost=. | `Set Boost Function` | +| `ResetBF` | boost=. | `Re-Set to default` | +| `bq` | bq=. | Text value; use the page label and surrounding context to choose the exact content. | +| `EnterBQ` | bq=. | `Set Boost Query` | +| `ResetBQ` | bq=. | `Re-Set to default` | +| `fq` | fq=. | Text value; use the page label and surrounding context to choose the exact content. | +| `EnterFQ` | fq=. | `Set Filter Query` | +| `ResetFQ` | fq=. | `Re-Set to default` | +| `#[field]#` | boost=. | boost= | +| `EnterBoosts` | boost=. | `Set Field Boosts` | +| `ResetBoosts` | boost=. | `Re-Set to default` | + +## Correct Use + +Use representative test queries or documents. Ranking, analysis, and AI settings are meaningful only when their effect can be compared. Keep notes about changed weights, model choices, fields, or thresholds. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/RankingSolr_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/RankingSolr_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/RankingSolr_p.html` | `GET` | admin | `source/net/yacy/htroot/RankingSolr_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `profileNr` | boost= / bq= / fq=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `bf` | boost=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `EnterBF` | boost=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ResetBF` | boost=. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `bq` | bq=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `EnterBQ` | bq=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ResetBQ` | bq=. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `fq` | fq=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `EnterFQ` | fq=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ResetFQ` | fq=. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `#[field]#` | boost=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `EnterBoosts` | boost=. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ResetBoosts` | boost=. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /RankingSolr_p.html?profileNr=...&bf=...&EnterBF=...&ResetBF=...&bq=... +``` + +## What To Expect + +Expect configuration values, diagnostics, or changed result behavior. The effect may only become visible after running the same query again, re-indexing fields, or using the configured model/RAG workflow. + +## Related Pages + +- `IndexSchema_p.html` +- `https://lucene.apache.org/solr/guide/6_6/function-queries.html` diff --git a/help/RegexTest.md b/help/RegexTest.md new file mode 100644 index 000000000..5de53242e --- /dev/null +++ b/help/RegexTest.md @@ -0,0 +1,73 @@ +--- +page: htroot/RegexTest.html +help: help/RegexTest.md +title: Regex Test +package: blacklist-security-access +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/RegexTest.java +--- + +# Regex Test + +## Purpose + +Regex Test helps validate regular expressions used in crawls, blacklists, and filters. + +Use it before applying a pattern to crawling or deletion, where a broad match can have large effects. + +## What You Can Do Here + +- Regex Test helps validate regular expressions used in crawls, blacklists, and filters. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `text` | Text used as input for a test, preview, or stored content field. | Text value; use the page label and surrounding context to choose the exact content. | +| `regex` | Regular expression to test or apply. Escape special characters deliberately. | Regular expression syntax. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Submit button value generated by the page; direct clients normally send the named action parameter. | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/RegexTest.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/RegexTest.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/RegexTest.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `text` | Text used as input for a test, preview, or stored content field. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `regex` | Regular expression to test or apply. Escape special characters deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /RegexTest.html?text=...®ex=...&submit=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- `https://docs.oracle.com/javase/8/docs/api/java/util/regex/Pattern.html` diff --git a/help/RemoteCrawl_p.md b/help/RemoteCrawl_p.md new file mode 100644 index 000000000..79df4a366 --- /dev/null +++ b/help/RemoteCrawl_p.md @@ -0,0 +1,79 @@ +--- +page: htroot/RemoteCrawl_p.html +help: help/RemoteCrawl_p.md +title: Remote Crawl Configuration +package: core-search-public +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/RemoteCrawl_p.java +--- + +# Remote Crawl Configuration + +## Purpose + +Remote Crawl Configuration controls whether other peers may ask this peer to crawl. + +Use it to decide how much cooperative crawling is acceptable for this installation. + +## What You Can Do Here + +- Remote Crawl Configuration controls whether other peers may ask this peer to crawl. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `crawlResponse` | Accept Remote Crawl Requests. | Accept Remote Crawl Requests | +| `acceptCrawlLimit` | Load with a maximum of. | Text value; use the page label and surrounding context to choose the exact content. | +| `save` | Saves settings. | `Save` | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/RemoteCrawl_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/RemoteCrawl_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/RemoteCrawl_p.html` | `POST` | admin | `source/net/yacy/htroot/RemoteCrawl_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `crawlResponse` | Accept Remote Crawl Requests. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `acceptCrawlLimit` | Load with a maximum of. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `save` | Saves settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /RemoteCrawl_p.html +Content-Type: application/x-www-form-urlencoded + +crawlResponse=...&acceptCrawlLimit=...&save=...&search=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `CrawlResults.html` diff --git a/help/SearchAccessRate_p.md b/help/SearchAccessRate_p.md new file mode 100644 index 000000000..03bc5a7ce --- /dev/null +++ b/help/SearchAccessRate_p.md @@ -0,0 +1,103 @@ +--- +page: htroot/SearchAccessRate_p.html +help: help/SearchAccessRate_p.md +title: Local Search access rate limitations +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/SearchAccessRate_p.java +--- + +# Local Search access rate limitations + +## Purpose + +Local Search access rate limitations protect the search interface from excessive requests. + +Use it to keep a public or shared peer responsive under heavy or automated search traffic. + +## What You Can Do Here + +- Local Search access rate limitations protect the search interface from excessive requests. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `search.public.max.access.3s` | Max searches in 3s. | Integer value. | +| `search.public.max.access.1mn` | Max searches in 1mn. | Integer value. | +| `search.public.max.access.10mn` | Max searches in 10mn. | Integer value. | +| `search.public.max.p2p.access.3s` | Max searches in 3s. | Integer value. | +| `search.public.max.p2p.access.1mn` | Max searches in 1mn. | Integer value. | +| `search.public.max.p2p.access.10mn` | Max searches in 10mn. | Integer value. | +| `search.public.max.p2p.jsresort.access.3s` | Max searches in 3s. | Integer value. | +| `search.public.max.p2p.jsresort.access.1mn` | Max searches in 1mn. | Integer value. | +| `search.public.max.p2p.jsresort.access.10mn` | Max searches in 10mn. | Integer value. | +| `search.public.max.remoteSnippet.access.3s` | Max searches in 3s. | Integer value. | +| `search.public.max.remoteSnippet.access.1mn` | Max searches in 1mn. | Integer value. | +| `search.public.max.remoteSnippet.access.10mn` | Max searches in 10mn. | Integer value. | +| `set` | Submits and applies the basic configuration. | `Submit` | +| `setDefaults` | Max searches in 3s. | `Set defaults` | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/SearchAccessRate_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/SearchAccessRate_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/SearchAccessRate_p.html` | `POST` | admin | `source/net/yacy/htroot/SearchAccessRate_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `search.public.max.access.3s` | Max searches in 3s. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.access.1mn` | Max searches in 1mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.access.10mn` | Max searches in 10mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.p2p.access.3s` | Max searches in 3s. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.p2p.access.1mn` | Max searches in 1mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.p2p.access.10mn` | Max searches in 10mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.p2p.jsresort.access.3s` | Max searches in 3s. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.public.max.p2p.jsresort.access.1mn` | Max searches in 1mn. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.public.max.p2p.jsresort.access.10mn` | Max searches in 10mn. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search.public.max.remoteSnippet.access.3s` | Max searches in 3s. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.remoteSnippet.access.1mn` | Max searches in 1mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search.public.max.remoteSnippet.access.10mn` | Max searches in 10mn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `setDefaults` | Max searches in 3s. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /SearchAccessRate_p.html +Content-Type: application/x-www-form-urlencoded + +search.public.max.access.3s=...&search.public.max.access.1mn=...&search.public.max.access.10mn=...&search.public.max.p2p.access.3s=...&search.public.max.p2p.access.1mn=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- `ConfigAccounts_p.html` +- `ConfigPortal_p.html` diff --git a/help/ServerScannerList.md b/help/ServerScannerList.md new file mode 100644 index 000000000..9f8183825 --- /dev/null +++ b/help/ServerScannerList.md @@ -0,0 +1,82 @@ +--- +page: htroot/ServerScannerList.html +help: help/ServerScannerList.md +title: Network Scanner Monitor +package: monitoring-performance +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/ServerScannerList.java +--- + +# Network Scanner Monitor + +## Purpose + +Network Scanner Monitor helps diagnose the running peer. + +Monitoring pages turn the running peer into something observable. They show queues, logs, network contacts, memory, threads, access patterns, and other evidence needed to diagnose a search server. + +## What You Can Do Here + +- Network Scanner Monitor helps diagnose the running peer. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `allswitch` | Available server within the given IP range. | Available server within the given IP range | +| `item_#[count]#` | Available server within the given IP range. | `mark_` = Available server within the given IP range | +| `crawl` | Available server within the given IP range. | `Add Selected Servers to Crawler` | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Protected related endpoint(s): `/CrawlStartScanner_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ServerScannerList.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/CrawlStartScanner_p.html` | `POST` | admin | `source/net/yacy/htroot/CrawlStartScanner_p.java` | +| `/ServerScannerList.html` | `GET or POST` | public or page-dependent | `source/net/yacy/htroot/ServerScannerList.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `allswitch` | Available server within the given IP range. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `item_#[count]#` | Available server within the given IP range. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `crawl` | Available server within the given IP range. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `repeat_time` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `scanhosts` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timeout` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /CrawlStartScanner_p.html +Content-Type: application/x-www-form-urlencoded + +allswitch=...&item_#[count]#=...&crawl=...&accumulatescancache=...&edit=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/SettingsAck_p.md b/help/SettingsAck_p.md new file mode 100644 index 000000000..3b5134447 --- /dev/null +++ b/help/SettingsAck_p.md @@ -0,0 +1,86 @@ +--- +page: htroot/SettingsAck_p.html +help: help/SettingsAck_p.md +title: Settings Acknowledge +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/SettingsAck_p.java +--- + +# Settings Acknowledge + +## Purpose + +Settings Acknowledge is the confirmation page after settings are applied. + +Use it to verify what YaCy accepted and what may still require a restart or follow-up check. + +## What You Can Do Here + +- Settings Acknowledge is the confirmation page after settings are applied. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/SettingsAck_p.html`. + +Backend checks: transaction token for protected POST. + +## Automation And API + +Page backend: `source/net/yacy/htroot/SettingsAck_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/SettingsAck_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/SettingsAck_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `adminaccount` | User or account value. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `adminuser` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `crawler.clientTimeout` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `fileHost` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `msgForwardingEnabled` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `port` | HTTP port where YaCy listens. Values below 1024 are ignored by this form; changing the port triggers reconnect/redirect behavior. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `proxyaccount` | User or account value. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `proxyfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `remoteProxyHost` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `remoteProxyUser` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `serveraccount` | User or account value. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `serverfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `serveruser` | User or account value. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `urlproxydomains` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `urlproxyenabled` | Enables the named feature. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `urlproxyfilter` | Filter expression. It decides which records are included, excluded, displayed, exported, or processed on this page. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `use_proxyaccounts` | User or account value. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET or POST /SettingsAck_p.html?port=...&adminaccount=...&adminpw1=...&adminpw2=...&adminuser=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- `Status.html` +- `ConfigBasic.html` diff --git a/help/Settings_p.md b/help/Settings_p.md new file mode 100644 index 000000000..68cd915c0 --- /dev/null +++ b/help/Settings_p.md @@ -0,0 +1,68 @@ +--- +page: htroot/Settings_p.html +help: help/Settings_p.md +title: Advanced Settings +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Settings_p.java +--- + +# Advanced Settings + +## Purpose + +Advanced Settings gathers technical peer settings that do not fit the first-run setup pages. + +Use it for expert configuration after understanding the operational effect of each option. + +## What You Can Do Here + +- Advanced Settings gathers technical peer settings that do not fit the first-run setup pages. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Settings_p.html`. + +Backend checks: transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Settings_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Settings_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/Settings_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /Settings_p.html?page=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/Status.md b/help/Status.md new file mode 100644 index 000000000..e8ef1c1b6 --- /dev/null +++ b/help/Status.md @@ -0,0 +1,74 @@ +--- +page: htroot/Status.html +help: help/Status.md +title: Console Status +package: monitoring-performance +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/Status.java +--- + +# Console Status + +## Purpose + +Console Status is the operational dashboard for the peer. + +Use it first to see identity, uptime, network position, indexing state, and important warnings. + +## What You Can Do Here + +- Console Status is the operational dashboard for the peer. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/Status.html`. + +Backend checks: administrator authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Status.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Status.html` | `GET` | mixed | `source/net/yacy/htroot/Status.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET /Status.html?aquirerelease=...&ResetTraffic=...&continueCrawlJob=...&jobType=...&login=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `ConfigAccounts_p.html` +- `index.html` +- `CrawlStartSite.html` +- `Crawler_p.html` diff --git a/help/Steering.md b/help/Steering.md new file mode 100644 index 000000000..9fb30741b --- /dev/null +++ b/help/Steering.md @@ -0,0 +1,73 @@ +--- +page: htroot/Steering.html +help: help/Steering.md +title: Steering +package: monitoring-performance +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Steering.java +--- + +# Steering + +## Purpose + +Steering is a navigation and operations hub. + +Use it to reach major administration areas when diagnosing or operating a peer. + +## What You Can Do Here + +- Steering is a navigation and operations hub. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: transaction token for protected POST, transaction token issued for forms, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Steering.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Steering.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/Steering.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `update` | Submit action that refreshes, updates, or applies the selected setting depending on the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Steering.html +Content-Type: application/x-www-form-urlencoded + +releaseinstall=...&restart=...&shutdown=...&update=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- `Settings_p.html` +- `ConfigAccounts_p.html` +- `ConfigUpdate_p.html` +- `ViewLog_p.html` diff --git a/help/Supporter.md b/help/Supporter.md new file mode 100644 index 000000000..55d512e4c --- /dev/null +++ b/help/Supporter.md @@ -0,0 +1,71 @@ +--- +page: htroot/Supporter.html +help: help/Supporter.md +title: Supporter +package: core-search-public +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/Supporter.java +--- + +# Supporter + +## Purpose + +Supporter presents ways to support the YaCy project. + +Use it as informational project context, not as a search or administration tool. + +## What You Can Do Here + +- Supporter presents ways to support the YaCy project. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/Supporter.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Supporter.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Supporter.html` | `GET or POST` | mixed | `source/net/yacy/htroot/Supporter.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `description` | Description text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `title` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /Supporter.html?url=...&comment=...&description=...&display=...&refid=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `Bookmarks.html` diff --git a/help/Surftips.md b/help/Surftips.md new file mode 100644 index 000000000..98e5da4c0 --- /dev/null +++ b/help/Surftips.md @@ -0,0 +1,75 @@ +--- +page: htroot/Surftips.html +help: help/Surftips.md +title: Surftips +package: core-search-public +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/Surftips.java +--- + +# Surftips + +## Purpose + +Surftips displays shared or suggested links. + +Use it to browse peer-provided recommendations when the feature is enabled. + +## What You Can Do Here + +- Surftips displays shared or suggested links. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/Surftips.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Surftips.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Surftips.html` | `GET` | mixed | `source/net/yacy/htroot/Surftips.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `description` | Description text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `title` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /Surftips.html?url=...&publicPage=...&comment=...&description=...&display=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `Bookmarks.html` +- `Supporter.html` diff --git a/help/Table_RobotsTxt_p.md b/help/Table_RobotsTxt_p.md new file mode 100644 index 000000000..5340752dd --- /dev/null +++ b/help/Table_RobotsTxt_p.md @@ -0,0 +1,49 @@ +--- +page: htroot/Table_RobotsTxt_p.html +help: help/Table_RobotsTxt_p.md +title: Table Viewer +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Table_RobotsTxt_p.java +--- + +# Table Viewer + +## Purpose + +Robots table view inspects stored robots.txt information. + +Use it to understand crawl permissions YaCy learned for hosts. + +## What You Can Do Here + +- Robots table view inspects stored robots.txt information. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Table_RobotsTxt_p.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/Tables_p.md b/help/Tables_p.md new file mode 100644 index 000000000..e3eac4977 --- /dev/null +++ b/help/Tables_p.md @@ -0,0 +1,79 @@ +--- +page: htroot/Tables_p.html +help: help/Tables_p.md +title: Table Viewer +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Tables_p.java +--- + +# Table Viewer + +## Purpose + +Table Viewer displays stored table data. + +Use it to inspect structured records saved by YaCy features or APIs. + +## What You Can Do Here + +- Table Viewer displays stored table data. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `count` | Number of records requested. | `10`, `100`, `1000`, `all` | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Text value; use the page label and surrounding context to choose the exact content. | +| `item_#[count]#` | Choice value. Options: `pk_`. | `pk_` | +| `deleterows` | Deletion or termination action. Use only with explicit intent. | `Delete Selected Rows` | +| `deletetable` | Deletion or termination action. Use only with explicit intent. | `Delete Table` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Tables_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Tables_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Tables_p.html` | `GET` | admin | `source/net/yacy/htroot/Tables_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `item_#[count]#` | Choice value. Options: `pk_`. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleterows` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deletetable` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | + +Example request shape: + +```http +GET /Tables_p.html?table=...&count=...&reverse=...&search=...&edittable=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- Related content work usually continues on the matching wiki, blog, bookmark, message, table, translation, vocabulary, or file page. diff --git a/help/Threaddump_p.md b/help/Threaddump_p.md new file mode 100644 index 000000000..9cfd24ef3 --- /dev/null +++ b/help/Threaddump_p.md @@ -0,0 +1,71 @@ +--- +page: htroot/Threaddump_p.html +help: help/Threaddump_p.md +title: Debugging: Thread Dump +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Threaddump_p.java +--- + +# Debugging: Thread Dump + +## Purpose + +Thread Dump captures Java thread state. + +Use it for debugging deadlocks, stalls, or high CPU situations. + +## What You Can Do Here + +- Thread Dump captures Java thread state. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Integer value. | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Threaddump_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Threaddump_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Threaddump_p.html` | `GET` | admin | `source/net/yacy/htroot/Threaddump_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET /Threaddump_p.html?singleThreaddump=...&count=...&multipleThreaddump=...&plain=...&sleep=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/ToolsConfig_p.md b/help/ToolsConfig_p.md new file mode 100644 index 000000000..850e65e7b --- /dev/null +++ b/help/ToolsConfig_p.md @@ -0,0 +1,77 @@ +--- +page: htroot/ToolsConfig_p.html +help: help/ToolsConfig_p.md +title: Tools Config +package: configuration-administration +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ToolsConfig_p.java +--- + +# Tools Config + +## Purpose + +Tools Config controls helper tools available inside the YaCy interface. + +Use it to enable or adjust optional administration conveniences. + +## What You Can Do Here + +- Tools Config controls helper tools available inside the YaCy interface. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `ai.tools.#[name]#.maxCallsPerTurn` | maxCallsPerTurn. | Integer value. | +| `ai.tools.#[name]#.description` | maxCallsPerTurn. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ToolsConfig_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ToolsConfig_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ToolsConfig_p.html` | `POST` | admin | `source/net/yacy/htroot/ToolsConfig_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `ai.tools.#[name]#.maxCallsPerTurn` | maxCallsPerTurn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `ai.tools.#[name]#.description` | maxCallsPerTurn. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `save` | Saves settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /ToolsConfig_p.html +Content-Type: application/x-www-form-urlencoded + +ai.tools.#[name]#.maxCallsPerTurn=...&ai.tools.#[name]#.description=...&save=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/Trails.md b/help/Trails.md new file mode 100644 index 000000000..3a36bf449 --- /dev/null +++ b/help/Trails.md @@ -0,0 +1,49 @@ +--- +page: htroot/Trails.html +help: help/Trails.md +title: CyTag Trails +package: monitoring-performance +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Trails.java +--- + +# CyTag Trails + +## Purpose + +CyTag Trails shows navigation or tag trails recorded by YaCy. + +Use it to understand browsing or tagging traces when this feature is active. + +## What You Can Do Here + +- CyTag Trails shows navigation or tag trails recorded by YaCy. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Trails.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/TransNews_p.md b/help/TransNews_p.md new file mode 100644 index 000000000..d300415a1 --- /dev/null +++ b/help/TransNews_p.md @@ -0,0 +1,74 @@ +--- +page: htroot/TransNews_p.html +help: help/TransNews_p.md +title: Translation News +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/TransNews_p.java +--- + +# Translation News + +## Purpose + +Translation News shows translation-related changes and notices. + +Use it when maintaining or reviewing interface translations. + +## What You Can Do Here + +- Translation News shows translation-related changes and notices. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/TransNews_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/TransNews_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/TransNews_p.html` | `POST` | admin | `source/net/yacy/htroot/TransNews_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /TransNews_p.html +Content-Type: application/x-www-form-urlencoded + +publishtranslation=...&filename=...&source=...&target=...&voteNegative=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `News.html` +- `Translator_p.html` +- `ConfigBasic.html` diff --git a/help/Translator_p.md b/help/Translator_p.md new file mode 100644 index 000000000..b1ca2aa89 --- /dev/null +++ b/help/Translator_p.md @@ -0,0 +1,85 @@ +--- +page: htroot/Translator_p.html +help: help/Translator_p.md +title: Translation Editor +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Translator_p.java +--- + +# Translation Editor + +## Purpose + +Translation Editor edits YaCy interface translations. + +Use it to improve localized text that users see in the web interface. + +## What You Can Do Here + +- Translation Editor edits YaCy interface translations. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `sourcefile` | UI Translation. | Text value; use the page label and surrounding context to choose the exact content. | +| `filteruntranslated` | filter untranslated. | `true` = filter untranslated | +| `editapproved` | UI Translation. | Text value; use the page label and surrounding context to choose the exact content. | +| `approve` | UI Translation. | Text value; use the page label and surrounding context to choose the exact content. | +| `savetranslationlist` | UI Translation. | `Save translation` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Translator_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Translator_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Translator_p.html` | `POST` | admin | `source/net/yacy/htroot/Translator_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `sourcefile` | UI Translation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `filteruntranslated` | filter untranslated. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `editapproved` | UI Translation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `approve` | UI Translation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `savetranslationlist` | UI Translation. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST /Translator_p.html +Content-Type: application/x-www-form-urlencoded + +sourcefile=...&filteruntranslated=...&targettxt#[tokenid]#=...&editapproved=...&approve=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `ConfigBasic.html` +- `TransNews_p.html` diff --git a/help/User.md b/help/User.md new file mode 100644 index 000000000..d6148d18b --- /dev/null +++ b/help/User.md @@ -0,0 +1,87 @@ +--- +page: htroot/User.html +help: help/User.md +title: User Page +package: configuration-administration +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/User.java +--- + +# User Page + +## Purpose + +User Page is the signed-in user's own view of account-related state. + +Use it to inspect the current session and user-facing account information. + +## What You Can Do Here + +- User Page is the signed-in user's own view of account-related state. +- Read the current value before changing it. +- Verify the effect on the public page, status page, or related administration page. + +## Page Architecture + +Configuration pages usually contain persistent settings. A visible form writes values into YaCy configuration, while the backend may reload subsystems such as language files, network listeners, cache handling, or search presentation. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `username` | old Password. | Checkbox/boolean; present usually means enabled. | +| `password` | old Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `logout` | old Password. | `logout` | +| `oldpass` | old Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `newpass` | new Password. | Text value; use the page label and surrounding context to choose the exact content. | +| `newpass2` | new Password(repetition). | Text value; use the page label and surrounding context to choose the exact content. | +| `changepass` | old Password. | `Change` | + +## Correct Use + +Read the current value before changing it. Configuration changes often persist beyond the current request and may affect later crawling, search, network contact, authentication, or resource use. Change one operational idea at a time and verify the result. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/User.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/User.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/User.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `username` | old Password. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `password` | old Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `logout` | old Password. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `oldpass` | old Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `newpass` | new Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `newpass2` | new Password(repetition). | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `changepass` | old Password. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `returnto` | old Password. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /User.html +Content-Type: application/x-www-form-urlencoded + +username=...&password=...&logout=...&oldpass=...&newpass=... +``` + +## What To Expect + +A successful change is visible as a saved value, a confirmation, or changed behavior on a related page. Some settings take effect immediately; others require reconnecting, reloading translations, restarting services, or watching the status page. + +## Related Pages + +- Related configuration work is usually reached from `ConfigBasic.html`, `Settings_p.html`, or the adjacent configuration page in the administration menu. diff --git a/help/VFS.md b/help/VFS.md new file mode 100644 index 000000000..cfab89840 --- /dev/null +++ b/help/VFS.md @@ -0,0 +1,49 @@ +--- +page: htroot/VFS.html +help: help/VFS.md +title: Virtual File System +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/VFS.java +--- + +# Virtual File System + +## Purpose + +Virtual File System exposes files stored in YaCy's virtual storage. + +Use it to inspect or manage stored file-like content. + +## What You Can Do Here + +- Virtual File System exposes files stored in YaCy's virtual storage. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/VFS.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- Related content work usually continues on the matching wiki, blog, bookmark, message, table, translation, vocabulary, or file page. diff --git a/help/ViewFile.md b/help/ViewFile.md new file mode 100644 index 000000000..9aa191b64 --- /dev/null +++ b/help/ViewFile.md @@ -0,0 +1,82 @@ +--- +page: htroot/ViewFile.html +help: help/ViewFile.md +title: View URL Content +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/ViewFile.java +--- + +# View URL Content + +## Purpose + +View URL Content shows a fetched or cached document as YaCy sees it. + +Use it to diagnose parsing, cache, snippet, and indexing problems for one URL. + +## What You Can Do Here + +- View URL Content shows a fetched or cached document as YaCy sees it. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `viewMode` | View as. Options: `iframeWeb` = Original from Web, `iframeCache` = Original from Cache, `plain` = Plain Text, `parsed` = Parsed Text, `sentences` = Parsed Sentences, `words` = Parsed Tokens/Words, `links` = Link List, `schema` = Schema Fields, `iframeCitations` = Citation Report. | `iframeWeb` = Original from Web, `iframeCache` = Original from Cache, `plain` = Plain Text, `parsed` = Parsed Text, `sentences` = Parsed Sentences, `words` = Parsed Tokens/Words, `links` = Link List, `schema` = Schema Fields, `iframeCitations` = Citation Report | +| `url` | URL to inspect, crawl, import, or act on. | URL or URL-derived value; use the exact format shown by the page. | +| `show` | View as. | `Show Metadata`, `Show Snippet`, `Show` | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Use an exact URL or stored reference. Compare the fetched content, cached content, parser output, and headers when diagnosing why a result snippet or indexed text looks wrong. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ViewFile.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ViewFile.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/ViewFile.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `viewMode` | View as. Options: `iframeWeb` = Original from Web, `iframeCache` = Original from Cache, `plain` = Plain Text, `parsed` = Parsed Text, `sentences` = Parsed Sentences, `words` = Parsed Tokens/Words, `links` = Link List, `schema` = Schema Fields, `iframeCitations` = Citation Report. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `show` | View as. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `urlHash` | View as. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `urlhash` | YaCy URL hash identifying an indexed document. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `words` | View as. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /ViewFile.html?query=...&url=...&urlhash=...&viewMode=...&show=... +``` + +## What To Expect + +The page shows YaCy's view of one document. Differences between this view and the live website often explain stale snippets, missing text, parser failures, or cache behavior. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/ViewLog_p.md b/help/ViewLog_p.md new file mode 100644 index 000000000..8c9fb6836 --- /dev/null +++ b/help/ViewLog_p.md @@ -0,0 +1,77 @@ +--- +page: htroot/ViewLog_p.html +help: help/ViewLog_p.md +title: Server Log +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/ViewLog_p.java +--- + +# Server Log + +## Purpose + +Server Log displays raw log lines with filtering. + +Use it when summaries are not enough and the exact sequence of events matters. + +## What You Can Do Here + +- Server Log displays raw log lines with filtering. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `lines` | Maximum number of log or text lines to display. | Integer maximum line count. | +| `mode` | reversed order. | `reversed` = reversed order | +| `filterMode` | Lines (max. ). Options: `regex`, `terms`. | `regex`, `terms` | +| `filter` | Filter text or expression used to narrow the displayed records. | Pattern or filter expression; test narrow expressions before broad use. | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/ViewLog_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ViewLog_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ViewLog_p.html` | `GET` | admin | `source/net/yacy/htroot/ViewLog_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `lines` | Maximum number of log or text lines to display. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `mode` | Display or processing mode selected by the page. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `filterMode` | Filter interpretation mode, for example plain terms or regular expression when available. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `filter` | Filter text or expression used to narrow the displayed records. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET /ViewLog_p.html?lines=...&mode=...&filterMode=...&filter=...&json=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/ViewProfile.md b/help/ViewProfile.md new file mode 100644 index 000000000..de20e0097 --- /dev/null +++ b/help/ViewProfile.md @@ -0,0 +1,69 @@ +--- +page: htroot/ViewProfile.html +help: help/ViewProfile.md +title: Remote Peer Profile +package: content-apps +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/ViewProfile.java +--- + +# Remote Peer Profile + +## Purpose + +Remote Peer Profile displays profile data for a YaCy peer. + +Use it to understand who operates a peer and what information it publishes. + +## What You Can Do Here + +- Remote Peer Profile displays profile data for a YaCy peer. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/ViewProfile.html`. + +Backend checks: administrator authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/ViewProfile.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/ViewProfile.html` | `GET or POST` | mixed | `source/net/yacy/htroot/ViewProfile.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /ViewProfile.html?display=...&hash=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `ConfigProfile_p.html` diff --git a/help/Vocabulary_p.md b/help/Vocabulary_p.md new file mode 100644 index 000000000..795253ada --- /dev/null +++ b/help/Vocabulary_p.md @@ -0,0 +1,127 @@ +--- +page: htroot/Vocabulary_p.html +help: help/Vocabulary_p.md +title: Federated Index +package: content-apps +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/Vocabulary_p.java +--- + +# Federated Index + +## Purpose + +Federated Index Vocabulary manages named vocabulary terms. + +Use it to keep controlled terms that can support structured indexing or search. + +## What You Can Do Here + +- Federated Index Vocabulary manages named vocabulary terms. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `vocabulary` | Vocabulary Name / Objectspace. | Text value; use the page label and surrounding context to choose the exact content. | +| `view` | Vocabulary Name. | `View` | +| `discovername` | Vocabulary Name. | Text value; use the page label and surrounding context to choose the exact content. | +| `discovermethod` | Empty Vocabulary / from file name / from page title. Options: `none` = Empty Vocabulary, `path` = from file name, `title` = from page title, `titlesplitted` = from page title (split), `author` = from page author, `csv` = Import from a csv file. | `none` = Empty Vocabulary, `path` = from file name, `title` = from page title, `titlesplitted` = from page title (split), `author` = from page author, `csv` = Import from a csv file | +| `discoverobjectspace` | Objectspace. | `http://` | +| `discoverpath` | File Path or URL. | Text value; use the page label and surrounding context to choose the exact content. | +| `discoverLineStart` | Start line. | `0` | +| `discovercolumnliteral` | Column for Literals. | `0` | +| `discoversynonymsmethod` | no Synonyms / Auto-Enrich with Synonyms from Stemming Library / Read Column. Options: `none` = no Synonyms, `enrichsynonyms` = Auto-Enrich with Synonyms from Stemming Library, `readcolumn` = Read Column. | `none` = no Synonyms, `enrichsynonyms` = Auto-Enrich with Synonyms from Stemming Library, `readcolumn` = Read Column | +| `discovercolumnsynonyms` | no Synonyms. | `1` | +| `discovercolumnobjectlink` | Column for Object Link (optional). | `1` | +| `charset` | Charset of Import File. | Text value; use the page label and surrounding context to choose the exact content. | +| `columnSeparator` | Column separator. Options: `,` = Comma ',', `;` = Semicolon ';'. | `,` = Comma ',', `;` = Semicolon ';' | +| `create` | Vocabulary Name. | `Create` | +| `objectspace` | Objectspace. | Text value; use the page label and surrounding context to choose the exact content. | +| `isFacet` | Is Facet?. | Is Facet? | +| `vocabularies.matchLinkedData` | Cleartext / Linked data/Semantic web annotations. Options: `false` = Cleartext, `true` = Linked data/Semantic web annotations. | `false` = Cleartext, `true` = Linked data/Semantic web annotations | +| `modify_#[term]#` | add. | `checked` = add | +| `delete_#[term]#` | add. | `checked` = add | +| `synonyms_#[term]#` | add. | Text value; use the page label and surrounding context to choose the exact content. | +| `objectlink_#[term]#` | add. | Text value; use the page label and surrounding context to choose the exact content. | +| `add_new` | add. | `checked` = add | +| `newterm` | add. | Text value; use the page label and surrounding context to choose the exact content. | +| `newsynonyms` | add. | Text value; use the page label and surrounding context to choose the exact content. | +| `newobjectlink` | add. | Text value; use the page label and surrounding context to choose the exact content. | +| `clear_table` | clear table (remove all terms). | `checked` = clear table (remove all terms) | +| `delete_vocabulary` | delete vocabulary. | `checked` = delete vocabulary | +| `set` | Submits and applies the basic configuration. | `Submit` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Vocabulary_p.html`. + +Backend checks: transaction token for protected POST, transaction token issued for forms. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Vocabulary_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Vocabulary_p.html` | `GET` | admin | `source/net/yacy/htroot/Vocabulary_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `vocabulary` | Vocabulary Name / Objectspace. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `view` | Vocabulary Name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discovername` | Vocabulary Name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discovermethod` | Empty Vocabulary / from file name / from page title. Options: `none` = Empty Vocabulary, `path` = from file name, `title` = from page title, `titlesplitted` = from page title (split), `author` = from page author, `csv` = Import from a csv file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discoverobjectspace` | Objectspace. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discoverpath` | File Path or URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discoverLineStart` | Start line. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discovercolumnliteral` | Column for Literals. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discoversynonymsmethod` | no Synonyms / Auto-Enrich with Synonyms from Stemming Library / Read Column. Options: `none` = no Synonyms, `enrichsynonyms` = Auto-Enrich with Synonyms from Stemming Library, `readcolumn` = Read Column. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discovercolumnsynonyms` | no Synonyms. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `discovercolumnobjectlink` | Column for Object Link (optional). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `charset` | Charset of Import File. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `columnSeparator` | Column separator. Options: `,` = Comma ',', `;` = Semicolon ';'. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `create` | Vocabulary Name. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `objectspace` | Objectspace. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `isFacet` | Is Facet?. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `vocabularies.matchLinkedData` | Cleartext / Linked data/Semantic web annotations. Options: `false` = Cleartext, `true` = Linked data/Semantic web annotations. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `modify_#[term]#` | add. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete_#[term]#` | add. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `synonyms_#[term]#` | add. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `objectlink_#[term]#` | add. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `add_new` | add. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `newterm` | add. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `newsynonyms` | add. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `newobjectlink` | add. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `clear_table` | clear table (remove all terms). | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `delete_vocabulary` | delete vocabulary. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `set` | Submit action that saves the page settings. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET /Vocabulary_p.html?vocabulary=...&view=...&discovername=...&discovermethod=...&discoverobjectspace=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- Related content work usually continues on the matching wiki, blog, bookmark, message, table, translation, vocabulary, or file page. diff --git a/help/WatchWebStructure_p.md b/help/WatchWebStructure_p.md new file mode 100644 index 000000000..c559b9263 --- /dev/null +++ b/help/WatchWebStructure_p.md @@ -0,0 +1,74 @@ +--- +page: htroot/WatchWebStructure_p.html +help: help/WatchWebStructure_p.md +title: Web Structure +package: monitoring-performance +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/WatchWebStructure_p.java +--- + +# Web Structure + +## Purpose + +Web Structure visualizes host and link relationships learned by crawling. + +Use it to see the shape of the crawled web rather than individual documents. + +## What You Can Do Here + +- Web Structure visualizes host and link relationships learned by crawling. +- Filter or limit the view to the symptom being investigated. +- Use the observation to decide the next crawler, index, network, or configuration action. + +## Page Architecture + +Monitoring pages read live peer state from queues, logs, network tables, memory counters, or process trackers. They are safest when used first as observation tools and only then as entry points to tuning pages. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `host` | Host or domain scope. | Text value; use the page label and surrounding context to choose the exact content. | +| `time` | Date/time value for filtering, display, or scheduling. | Text value; use the page label and surrounding context to choose the exact content. | + +## Correct Use + +Use monitoring pages as evidence, not as guesses. Capture the current state, then connect it to the user-visible symptom: slow search, missing documents, stuck crawl, memory pressure, network isolation, or unexpected access. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/WatchWebStructure_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/WatchWebStructure_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/WatchWebStructure_p.html` | `GET` | admin | `source/net/yacy/htroot/WatchWebStructure_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `host` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `time` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hosts` | Host list filter. Values such as `crawling` or `error` restrict the host overview to active crawl hosts or error hosts when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /WatchWebStructure_p.html?host=...&depth=...&nodes=...&time=...&width=... +``` + +## What To Expect + +Expect observations: counts, logs, queues, timing, network rows, thread states, or resource values. Monitoring does not fix the issue by itself; it points to the next page or setting to change. + +## Related Pages + +- Related diagnosis usually continues on the status page, log viewer, performance pages, connection tracker, queue monitor, or network view. diff --git a/help/Wiki.md b/help/Wiki.md new file mode 100644 index 000000000..16468bdce --- /dev/null +++ b/help/Wiki.md @@ -0,0 +1,94 @@ +--- +page: htroot/Wiki.html +help: help/Wiki.md +title: Wiki +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/Wiki.java +--- + +# Wiki + +## Purpose + +Wiki provides editable local documentation pages. + +Use it for notes and knowledge that should live inside the YaCy peer. + +## What You Can Do Here + +- Wiki provides editable local documentation pages. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `access` | Grant Write Access to. | `all`, `admin` | +| `author` | Author. | Text value; use the page label and surrounding context to choose the exact content. | +| `content` | Text. | Text value; use the page label and surrounding context to choose the exact content. | +| `submit` | Submits the form. | `Submit` | +| `preview` | Author. | `Preview` | +| `view` | Author. | `Discard` | +| `old` | Compare version from. | Text value; use the page label and surrounding context to choose the exact content. | +| `viewold` | Compare version from. | `Show` | +| `new` | with version from. | Text value; use the page label and surrounding context to choose the exact content. | +| `compare` | Compare version from. | `Compare` | + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/Wiki.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Wiki.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/Wiki.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `access` | Grant Write Access to. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `author` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `content` | Text. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `preview` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `view` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `old` | Compare version from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `viewold` | Compare version from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `new` | with version from. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `compare` | Compare version from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `diff` | Compare version from. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `reason` | Author. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Wiki.html +Content-Type: application/x-www-form-urlencoded + +access=...&author=...&content=...&submit=...&preview=... +``` + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- `WikiHelp.html` diff --git a/help/WikiHelp.md b/help/WikiHelp.md new file mode 100644 index 000000000..46fef7305 --- /dev/null +++ b/help/WikiHelp.md @@ -0,0 +1,49 @@ +--- +page: htroot/WikiHelp.html +help: help/WikiHelp.md +title: Wiki Help +package: content-apps +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/WikiHelp.java +--- + +# Wiki Help + +## Purpose + +Wiki Help explains the local wiki feature. + +Use it before editing wiki content or explaining wiki syntax to users. + +## What You Can Do Here + +- Wiki Help explains the local wiki feature. +- Edit or inspect the specific content object shown on the page. +- Check whether the result is local-only, user-visible, or peer-visible before publishing. + +## Page Architecture + +Content application pages store and render peer-local objects such as bookmarks, messages, wiki text, tables, profiles, or translations. They combine form editing with a rendered view of the stored object. + +## Correct Use + +Edit content deliberately and check the rendered page after saving. For shared or peer-visible features, assume written content may be read by someone else unless the page clearly says otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/WikiHelp.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect stored or rendered content: a message, page, table row, bookmark, profile, translation, or file view. After changes, reload or revisit the object to confirm what was actually saved. + +## Related Pages + +- Related content work usually continues on the matching wiki, blog, bookmark, message, table, translation, vocabulary, or file page. diff --git a/help/YaCySearchPluginFF.md b/help/YaCySearchPluginFF.md new file mode 100644 index 000000000..17ad2c821 --- /dev/null +++ b/help/YaCySearchPluginFF.md @@ -0,0 +1,49 @@ +--- +page: htroot/YaCySearchPluginFF.html +help: help/YaCySearchPluginFF.md +title: Quick Crawl Link +package: core-search-public +access: public +kind: search-page +backend_java: source/net/yacy/htroot/YaCySearchPluginFF.java +--- + +# Quick Crawl Link + +## Purpose + +Search Plugin setup helps browsers add YaCy as a search provider. + +Use it when users should search this peer from the browser search bar. + +## What You Can Do Here + +- Search Plugin setup helps browsers add YaCy as a search provider. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/YaCySearchPluginFF.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/api/citation.md b/help/api/citation.md new file mode 100644 index 000000000..55ce68789 --- /dev/null +++ b/help/api/citation.md @@ -0,0 +1,74 @@ +--- +page: htroot/api/citation.html +help: help/api/citation.md +title: Document Citations for url +package: machine-api-peer +access: public +kind: api-endpoint +backend_java: source/net/yacy/htroot/api/citation.java +--- + +# Document Citations for url + +## Purpose + +Document Citations returns citation information for a URL or document. + +Use it when a tool needs references around one indexed document. + +## What You Can Do Here + +- Retrieve citation data for one URL or document. +- Submit the exact document target. +- Use the response to explain or reference indexed material. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `filter` | List of ::Cited Sentences in. | `true::false` | + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/api/citation.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/api/citation.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/api/citation.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `filter` | Filter text or expression used to narrow the displayed records. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /api/citation.html +Content-Type: application/x-www-form-urlencoded + +url=...&filter=...&ch=...&hash=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/api/push_p.md b/help/api/push_p.md new file mode 100644 index 000000000..55b9d5de0 --- /dev/null +++ b/help/api/push_p.md @@ -0,0 +1,77 @@ +--- +page: htroot/api/push_p.html +help: help/api/push_p.md +title: File Upload +package: machine-api-peer +access: admin +kind: api-endpoint +backend_java: source/net/yacy/htroot/api/push_p.java +--- + +# File Upload + +## Purpose + +File Upload accepts pushed content into YaCy. + +Use it when another tool submits documents directly instead of asking the crawler to fetch them. + +## What You Can Do Here + +- Push document content directly into YaCy. +- Send file or document metadata in the expected upload shape. +- Verify indexing afterward because upload acceptance is not the same as search visibility. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `synchronous` | Choice value. Options: `true`. | `true` | +| `commit` | Choice value. Options: `true`. | `true` | + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/api/push_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/api/push_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/api/push_p.html` | `POST` | admin | `source/net/yacy/htroot/api/push_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `synchronous` | Choice value. Options: `true`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `commit` | Choice value. Options: `true`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +POST /api/push_p.html +Content-Type: application/x-www-form-urlencoded + +synchronous=...&commit=...&data-#[count]#=...&url-#[count]#=...&collection-#[count]#=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/api/share.md b/help/api/share.md new file mode 100644 index 000000000..81d70afe1 --- /dev/null +++ b/help/api/share.md @@ -0,0 +1,70 @@ +--- +page: htroot/api/share.html +help: help/api/share.md +title: File Share +package: machine-api-peer +access: public +kind: api-endpoint +backend_java: source/net/yacy/htroot/api/share.java +--- + +# File Share + +## Purpose + +File Share exposes shared file content through an API-style page. + +Use it when another tool needs to retrieve a shared object from the peer. + +## What You Can Do Here + +- Retrieve or expose shared file content. +- Use exact share identifiers and access parameters. +- Confirm that shared material is intended to be visible. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/api/share.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/api/share.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/api/share.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /api/share.html +Content-Type: application/x-www-form-urlencoded + +data=...&c=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/api/table_p.md b/help/api/table_p.md new file mode 100644 index 000000000..d6e96eef6 --- /dev/null +++ b/help/api/table_p.md @@ -0,0 +1,80 @@ +--- +page: htroot/api/table_p.html +help: help/api/table_p.md +title: Table Viewer +package: machine-api-peer +access: admin +kind: api-endpoint +backend_java: source/net/yacy/htroot/api/table_p.java +--- + +# Table Viewer + +## Purpose + +Table API exposes YaCy table data. + +Use it for scripted reads or writes of structured records when the table name and operation are known. + +## What You Can Do Here + +- Read or modify structured table data through the API surface. +- Specify table names, keys, and operations exactly. +- Check the response before assuming a row was created, changed, or deleted. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `edittable` | Table: ''. | `Edit Table` | + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/Tables_p.html`, `/api/table_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/api/table_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/Tables_p.html` | `POST` | admin | `source/net/yacy/htroot/Tables_p.java` | +| `/api/table_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/api/table_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `edittable` | Table: ''. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleterows` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `deletetable` | Deletion or termination action. Use only with explicit intent. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `table` | Table: ''. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /Tables_p.html +Content-Type: application/x-www-form-urlencoded + +edittable=...&addrow=...&commitrow=...&count=...&deleterows=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/api/yacydoc.md b/help/api/yacydoc.md new file mode 100644 index 000000000..f5c0e2c9c --- /dev/null +++ b/help/api/yacydoc.md @@ -0,0 +1,69 @@ +--- +page: htroot/api/yacydoc.html +help: help/api/yacydoc.md +title: yacydoc +package: machine-api-peer +access: public +kind: api-endpoint +backend_java: source/net/yacy/htroot/api/yacydoc.java +--- + +# yacydoc + +## Purpose + +yacydoc exposes document data for tools. + +Use it when an agent needs a machine-oriented view of one indexed document. + +## What You Can Do Here + +- Retrieve machine-readable information about one indexed document. +- Provide the exact URL, hash, or document identifier expected by the endpoint. +- Use the response as document context for tools or citations. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/api/yacydoc.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/api/yacydoc.html` | `POST` | public or page-dependent | `source/net/yacy/htroot/api/yacydoc.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `urlhash` | YaCy URL hash identifying an indexed document. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /api/yacydoc.html +Content-Type: application/x-www-form-urlencoded + +url=...&urlhash=...&html=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/compare_yacy.md b/help/compare_yacy.md new file mode 100644 index 000000000..3f109c77e --- /dev/null +++ b/help/compare_yacy.md @@ -0,0 +1,73 @@ +--- +page: htroot/compare_yacy.html +help: help/compare_yacy.md +title: Websearch Comparison +package: core-search-public +access: mixed +kind: ui-page +backend_java: source/net/yacy/htroot/compare_yacy.java +--- + +# Websearch Comparison + +## Purpose + +Websearch Comparison compares YaCy results with another search source. + +Use it to evaluate coverage or ranking differences for the same query. + +## What You Can Do Here + +- Websearch Comparison compares YaCy results with another search source. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/compare_yacy.html`. + +Backend checks: administrator authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/compare_yacy.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/compare_yacy.html` | `GET` | mixed | `source/net/yacy/htroot/compare_yacy.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET /compare_yacy.html?query=...&left=...&right=...&display=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/env/grafics/donate.md b/help/env/grafics/donate.md new file mode 100644 index 000000000..eeaeb84ae --- /dev/null +++ b/help/env/grafics/donate.md @@ -0,0 +1,74 @@ +--- +page: htroot/env/grafics/donate.html +help: help/env/grafics/donate.md +title: donate +package: proxy-embedded-static +access: public +kind: embedded-static-page +backend_java: none +--- + +# donate + +## Purpose + +Donate is a static project-support page. + +Use it only to explain the support link embedded in the interface. + +## What You Can Do Here + +- Donate is a static project-support page. +- Use it through the page or proxy context that includes it. +- Treat it as a support view unless the parameters document a direct service call. + +## Page Architecture + +The page is built around a visible form and backend parameters. The visible controls are the starting point; hidden fields usually carry defaults, transaction data, or browser state. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `os0` | Choice value. Options: `beneficial` = beneficial: 5 €, `generous` = generous: 25 €, `gracious` = gracious: 50 €. | `beneficial` = beneficial: 5 €, `generous` = generous: 25 €, `gracious` = gracious: 50 € | +| `submit` | Submits the form. | `Donate!` | + +## Correct Use + +Use this page in the context where YaCy displays it. Embedded and static pages usually explain, wrap, or visualize another workflow; direct calls should stay minimal and read-only unless documented otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +No matching Java servlet was found for this template. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `https://www.paypal.com/cgi-bin/webscr` | `POST` | external | external service | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `os0` | Choice value. Options: `beneficial` = beneficial: 5 €, `generous` = generous: 25 €, `gracious` = gracious: 50 €. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `submit` | Submit action for the form. Its meaning depends on the surrounding fields. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +POST https://www.paypal.com/cgi-bin/webscr +Content-Type: application/x-www-form-urlencoded + +os0=...&submit=... +``` + +## What To Expect + +Expect a focused support view, embedded fragment, visual page, or proxy message. The important action usually happens on the page that linked here. + +## Related Pages + +- Related embedded work usually continues in the proxy page, portal page, or visualization that included this fragment. diff --git a/help/goto_p.md b/help/goto_p.md new file mode 100644 index 000000000..46e8f35b4 --- /dev/null +++ b/help/goto_p.md @@ -0,0 +1,68 @@ +--- +page: htroot/goto_p.html +help: help/goto_p.md +title: forwarding +package: core-search-public +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/goto_p.java +--- + +# forwarding + +## Purpose + +Forwarding redirects to a selected result or target URL. + +Use it as a controlled handoff from YaCy to an external document. + +## What You Can Do Here + +- Forwarding redirects to a selected result or target URL. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/goto_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/goto_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/goto_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/goto_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `path` | URL path, URL prefix, or host/path scope. In index tools it decides which part of the stored URL tree is inspected or changed. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /goto_p.html?hash=...&path=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/index.md b/help/index.md new file mode 100644 index 000000000..0c9494ace --- /dev/null +++ b/help/index.md @@ -0,0 +1,110 @@ +--- +page: htroot/index.html +help: help/index.md +title: Search Page +package: core-search-public +access: mixed +kind: search-page +backend_java: source/net/yacy/htroot/index.java +--- + +# Search Page + +## Purpose + +The start page is the usual entry point for searching and navigating YaCy. + +Use it when a user first opens the peer in a browser. + +## What You Can Do Here + +- The start page is the usual entry point for searching and navigating YaCy. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | +| `Enter` | Browser submit button. Omit it from direct API calls; YaCy uses the query and filter parameters, not the button label, to run the search. | Text value; use the page label and surrounding context to choose the exact content. | +| `contentdom` | Content domain filter: text, image, audio, video, application, or all depending on the page. | `text` = Text, `image` = Text, `audio` = Text, `video` = Text, `app` = Text | +| `indexof` | Text / only index pages. | only index pages | +| `resource` | Search resource selection such as local or global/network search. | `global` = the peer-to-peer network, `local` = only the local index | +| `prefermaskfilter` | Regular-expression URL preference. Matching results are favored without excluding all others. | Regular expression for preferred result URLs. | +| `maximumRecords` | Maximum number of records displayed or returned. | `10`, `50`, `100` | +| `prefermask` | Prefer mask. Options: `yes` = Prefer mask, `no` = Prefer mask. | `yes` = Prefer mask, `no` = Prefer mask | +| `strictContentDom` | Require the content domain filter strictly. | `false` = Extended, `true` = Extended | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/yacysearch.html`, `/index.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/index.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearch.html` | `GET` | mixed | `source/net/yacy/htroot/yacysearch.java` | +| `/index.html` | `GET or POST` | mixed | `source/net/yacy/htroot/index.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexof` | Interprets the query as an index-of style request when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resource` | Search source. `local` searches this peer index; `global` may use the YaCy network when the peer and configuration allow it. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `prefermaskfilter` | Regular-expression URL preference. Matching results are favored without excluding all others. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `prefermask` | Prefer mask. Options: `yes` = Prefer mask, `no` = Prefer mask. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `strictContentDom` | When true, YaCy enforces the selected content domain more strictly instead of using it mainly for presentation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `constraint` | Encoded YaCy bitfield constraint. Leave it empty unless you are replaying a URL generated by YaCy itself. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleteref` | URL hash/reference selected for deletion from results. Use only with administrator intent and an exact known reference. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `focus` | Browser presentation flag controlling whether the search input receives focus. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lr` | Language filter, usually a two-letter code such as `de`, `fr`, or `en` when the page supports language-restricted search. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `meanCount` | Text. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `nav` | Navigator/facet selection for search results. `all` requests the normal set; narrower values reduce displayed facets. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control; use it to request later result pages. | +| `recommendref` | URL hash/reference selected for recommendation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resortCachedResults` | Requests resorting of a cached search event instead of starting a fully fresh result event. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `resource-switch` | Browser control for switching between local and global search. For direct calls use `resource=local` or `resource=global`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `rows` | Rows requested by API-style clients. It is another result-count alias on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `urlmaskfilter` | Regular-expression URL filter for search results. Use `.*` for no restriction; use a precise host/path expression to search inside a site or section. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `verify` | Snippet/cache verification strategy. Typical values are `iffresh`, `ifexist`, `cacheonly`, `nocache`, and `false`; choose according to whether freshness or speed matters more. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacysearch.html?query=...&maximumRecords=...&resource=...&contentdom=...&Enter=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `CrawlStartSite.html` +- `ConfigBasic.html` diff --git a/help/jslicense.md b/help/jslicense.md new file mode 100644 index 000000000..b9c2c6759 --- /dev/null +++ b/help/jslicense.md @@ -0,0 +1,50 @@ +--- +page: htroot/jslicense.html +help: help/jslicense.md +title: JavaScript license information +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/jslicense.java +--- + +# JavaScript license information + +## Purpose + +JavaScript license information lists script licensing details. + +Use it to inspect legal and compliance information for browser-side code. + +## What You Can Do Here + +- JavaScript license information lists script licensing details. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/jslicense.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `http://www.freebsd.org/copyright/freebsd-license.html` +- `http://www.gnu.org/licenses/gpl-2.0.html` diff --git a/help/mediawiki_p.md b/help/mediawiki_p.md new file mode 100644 index 000000000..af08fbce0 --- /dev/null +++ b/help/mediawiki_p.md @@ -0,0 +1,67 @@ +--- +page: htroot/mediawiki_p.html +help: help/mediawiki_p.md +title: mediawiki p +package: machine-api-peer +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/mediawiki_p.java +--- + +# mediawiki p + +## Purpose + +MediaWiki endpoint serves MediaWiki-oriented integration data. + +Use it when a MediaWiki-related tool expects this YaCy service surface. + +## What You Can Do Here + +- MediaWiki endpoint serves MediaWiki-oriented integration data. +- Send exact parameter names and encoded values rather than natural-language approximations. +- Inspect the response format before using the endpoint in a larger tool chain. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/mediawiki_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/mediawiki_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/mediawiki_p.html` | `GET or POST` | admin | `source/net/yacy/htroot/mediawiki_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `title` | Human-readable title. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /mediawiki_p.html?dump=...&title=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/portalsearch/yacy-portalsearch.md b/help/portalsearch/yacy-portalsearch.md new file mode 100644 index 000000000..dec11a420 --- /dev/null +++ b/help/portalsearch/yacy-portalsearch.md @@ -0,0 +1,69 @@ +--- +page: htroot/portalsearch/yacy-portalsearch.html +help: help/portalsearch/yacy-portalsearch.md +title: Bookmarks +package: proxy-embedded-static +access: public +kind: embedded-static-page +backend_java: none +--- + +# Bookmarks + +## Purpose + +Portal Search is an embedded search form for a YaCy-backed site portal. + +Use it when a website integrates YaCy search without exposing the full administration interface. + +## What You Can Do Here + +- Portal Search is an embedded search form for a YaCy-backed site portal. +- Use it through the page or proxy context that includes it. +- Treat it as a support view unless the parameters document a direct service call. + +## Page Architecture + +The page is built around a visible form and backend parameters. The visible controls are the starting point; hidden fields usually carry defaults, transaction data, or browser state. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | + +## Correct Use + +Use this page in the context where YaCy displays it. Embedded and static pages usually explain, wrap, or visualize another workflow; direct calls should stay minimal and read-only unless documented otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +No matching Java servlet was found for this template. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/portalsearch/yacysearch.html` | `GET` | public or page-dependent | template/static | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET /portalsearch/yacysearch.html?query=... +``` + +## What To Expect + +Expect a focused support view, embedded fragment, visual page, or proxy message. The important action usually happens on the page that linked here. + +## Related Pages + +- Related embedded work usually continues in the proxy page, portal page, or visualization that included this fragment. diff --git a/help/processing/domaingraph/applet/index.md b/help/processing/domaingraph/applet/index.md new file mode 100644 index 000000000..8a1141eb6 --- /dev/null +++ b/help/processing/domaingraph/applet/index.md @@ -0,0 +1,49 @@ +--- +page: htroot/processing/domaingraph/applet/index.html +help: help/processing/domaingraph/applet/index.md +title: domaingraph : Built with Processing +package: proxy-embedded-static +access: public +kind: embedded-static-page +backend_java: none +--- + +# domaingraph : Built with Processing + +## Purpose + +Domain Graph displays host relationships as a visual graph. + +Use it to teach or inspect how crawled hosts connect to each other. + +## What You Can Do Here + +- Domain Graph displays host relationships as a visual graph. +- Use it through the page or proxy context that includes it. +- Treat it as a support view unless the parameters document a direct service call. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Use this page in the context where YaCy displays it. Embedded and static pages usually explain, wrap, or visualize another workflow; direct calls should stay minimal and read-only unless documented otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +No matching Java servlet was found for this template. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a focused support view, embedded fragment, visual page, or proxy message. The important action usually happens on the page that linked here. + +## Related Pages + +- `http://java.sun.com/products/plugin/downloads/index.html` diff --git a/help/proxymsg/error.md b/help/proxymsg/error.md new file mode 100644 index 000000000..64d151f49 --- /dev/null +++ b/help/proxymsg/error.md @@ -0,0 +1,49 @@ +--- +page: htroot/proxymsg/error.html +help: help/proxymsg/error.md +title: Error Message +package: proxy-embedded-static +access: public +kind: proxy-message +backend_java: none +--- + +# Error Message + +## Purpose + +Proxy Error Message explains a failed proxy request. + +Use it when YaCy acts as a proxy and must tell the browser why a page could not be delivered. + +## What You Can Do Here + +- Proxy Error Message explains a failed proxy request. +- Use it through the page or proxy context that includes it. +- Treat it as a support view unless the parameters document a direct service call. + +## Page Architecture + +The page is built around a visible form and backend parameters. The visible controls are the starting point; hidden fields usually carry defaults, transaction data, or browser state. + +## Correct Use + +Use this page in the context where YaCy displays it. Embedded and static pages usually explain, wrap, or visualize another workflow; direct calls should stay minimal and read-only unless documented otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +No matching Java servlet was found for this template. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a focused support view, embedded fragment, visual page, or proxy message. The important action usually happens on the page that linked here. + +## Related Pages + +- Related embedded work usually continues in the proxy page, portal page, or visualization that included this fragment. diff --git a/help/proxymsg/urlproxyheader.md b/help/proxymsg/urlproxyheader.md new file mode 100644 index 000000000..73cec77c6 --- /dev/null +++ b/help/proxymsg/urlproxyheader.md @@ -0,0 +1,71 @@ +--- +page: htroot/proxymsg/urlproxyheader.html +help: help/proxymsg/urlproxyheader.md +title: urlproxyheader +package: proxy-embedded-static +access: public +kind: proxy-message +backend_java: source/net/yacy/htroot/proxymsg/urlproxyheader.java +--- + +# urlproxyheader + +## Purpose + +URL Proxy Header is an embedded proxy information fragment. + +Use it to understand the header area shown around proxied content. + +## What You Can Do Here + +- URL Proxy Header is an embedded proxy information fragment. +- Use it through the page or proxy context that includes it. +- Treat it as a support view unless the parameters document a direct service call. + +## Page Architecture + +The page is built around a visible form and backend parameters. The visible controls are the starting point; hidden fields usually carry defaults, transaction data, or browser state. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Use this page in the context where YaCy displays it. Embedded and static pages usually explain, wrap, or visualize another workflow; direct calls should stay minimal and read-only unless documented otherwise. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +Backend checks: user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/proxymsg/urlproxyheader.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/proxymsg/urlproxyheader.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/proxymsg/urlproxyheader.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /proxymsg/urlproxyheader.html?url=...&exit=...&addbookmark=...&bookmark=...&hash=... +``` + +## What To Expect + +Expect a focused support view, embedded fragment, visual page, or proxy message. The important action usually happens on the page that linked here. + +## Related Pages + +- Related embedded work usually continues in the proxy page, portal page, or visualization that included this fragment. diff --git a/help/rct_p.md b/help/rct_p.md new file mode 100644 index 000000000..2800a94d6 --- /dev/null +++ b/help/rct_p.md @@ -0,0 +1,72 @@ +--- +page: htroot/rct_p.html +help: help/rct_p.md +title: Index Control +package: machine-api-peer +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/rct_p.java +--- + +# Index Control + +## Purpose + +Index Control is a compact endpoint for index-related remote control. + +Use it only when a tool needs the specific service behavior exposed by this endpoint. + +## What You Can Do Here + +- Index Control is a compact endpoint for index-related remote control. +- Send exact parameter names and encoded values rather than natural-language approximations. +- Inspect the response format before using the endpoint in a larger tool chain. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/rct_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/rct_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/rct_p.html` | `POST` | admin | `source/net/yacy/htroot/rct_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +POST /rct_p.html +Content-Type: application/x-www-form-urlencoded + +peer=...&retrieve=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/rssTerminal.md b/help/rssTerminal.md new file mode 100644 index 000000000..714432b27 --- /dev/null +++ b/help/rssTerminal.md @@ -0,0 +1,49 @@ +--- +page: htroot/rssTerminal.html +help: help/rssTerminal.md +title: rss terminal +package: machine-api-peer +access: public +kind: ui-page +backend_java: none +--- + +# rss terminal + +## Purpose + +RSS Terminal exposes feed-style output for terminal or script use. + +Use it when a caller needs YaCy data as a feed rather than an HTML page. + +## What You Can Do Here + +- RSS Terminal exposes feed-style output for terminal or script use. +- Send exact parameter names and encoded values rather than natural-language approximations. +- Inspect the response format before using the endpoint in a larger tool chain. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +No matching Java servlet was found for this template. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/sharedBlacklist_p.md b/help/sharedBlacklist_p.md new file mode 100644 index 000000000..48ff4e845 --- /dev/null +++ b/help/sharedBlacklist_p.md @@ -0,0 +1,76 @@ +--- +page: htroot/sharedBlacklist_p.html +help: help/sharedBlacklist_p.md +title: Shared Blacklist +package: blacklist-security-access +access: admin +kind: admin-page +backend_java: source/net/yacy/htroot/sharedBlacklist_p.java +--- + +# Shared Blacklist + +## Purpose + +Shared Blacklist exposes blacklist data for sharing between peers. + +Use it when another peer or tool needs the current block policy in a machine-readable form. + +## What You Can Do Here + +- Shared Blacklist exposes blacklist data for sharing between peers. +- Test a concrete URL, request, user, or pattern before applying broad policy. +- Keep rules narrow enough that they block the intended problem without hiding useful content. + +## Page Architecture + +Security and blacklist pages turn names, patterns, credentials, or request properties into allow/block decisions. The architecture is rule-oriented: define the rule, test the rule, then apply it to crawling, search, or access. + +| Control | Meaning | Values or examples | +| --- | --- | --- | + +## Correct Use + +Test with a concrete example. A blacklist, regular expression, rate limit, cookie rule, or access rule is only understandable when checked against a real URL or request. Prefer narrow patterns and document the reason for broad rules. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +Protected related endpoint(s): `/sharedBlacklist_p.html`. + +## Automation And API + +Page backend: `source/net/yacy/htroot/sharedBlacklist_p.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/sharedBlacklist_p.html` | `POST` | admin | `source/net/yacy/htroot/sharedBlacklist_p.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `agentName` | Crawler user-agent profile used for outgoing HTTP requests. Choose a profile that matches the desired identity and politeness behavior. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `file` | Selected or uploaded file. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `hash` | YaCy hash identifier for a peer, URL, row, or stored object. Use exact values copied from YaCy output. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +POST /sharedBlacklist_p.html +Content-Type: application/x-www-form-urlencoded + +url=...¤tBlacklist=...&item#[count]#=...&add=...&agentName=... +``` + +## What To Expect + +Expect a rule list, match test, access record, cookie view, or confirmation. The real proof is behavioral: the same URL, request, or user should now be accepted, blocked, limited, or displayed as intended. + +## Related Pages + +- Related security work is usually reached through blacklist administration, blacklist testing, cookie monitors, robots data, or access-rate settings. diff --git a/help/terminal_p.md b/help/terminal_p.md new file mode 100644 index 000000000..64cfb2716 --- /dev/null +++ b/help/terminal_p.md @@ -0,0 +1,53 @@ +--- +page: htroot/terminal_p.html +help: help/terminal_p.md +title: System Terminal Monitor +package: machine-api-peer +access: admin +kind: admin-page +backend_java: none +--- + +# System Terminal Monitor + +## Purpose + +System Terminal Monitor exposes terminal-style server interaction. + +Use it with care because terminal-like endpoints can reveal or change operational state. + +## What You Can Do Here + +- System Terminal Monitor exposes terminal-style server interaction. +- Send exact parameter names and encoded values rather than natural-language approximations. +- Inspect the response format before using the endpoint in a larger tool chain. + +## Page Architecture + +Machine/API pages are compact endpoints. Their architecture is request-parameter driven: callers provide the smallest needed set of arguments and consume a direct response rather than a guided administration workflow. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +Administrator access is required. YaCy protects `_p` pages as administration pages. + +## Automation And API + +No matching Java servlet was found for this template. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- `index.html` +- `CrawlStartSite.html` +- `Status.html` +- `Steering.html` +- `http://java.sun.com/products/plugin/downloads/index.html` diff --git a/help/yacy/crawlReceipt.md b/help/yacy/crawlReceipt.md new file mode 100644 index 000000000..7767763c3 --- /dev/null +++ b/help/yacy/crawlReceipt.md @@ -0,0 +1,64 @@ +--- +page: htroot/yacy/crawlReceipt.html +help: help/yacy/crawlReceipt.md +title: crawlReceipt +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/crawlReceipt.java +--- + +# crawlReceipt + +## Purpose + +crawlReceipt acknowledges crawl information exchanged between peers. + +Use it as part of peer communication rather than as a manual browser page. + +## What You Can Do Here + +- Acknowledge or inspect crawl-receipt communication between peers. +- Use exact crawl and peer identifiers. +- Treat the response as protocol confirmation. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/crawlReceipt.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/crawlReceipt.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/crawlReceipt.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /yacy/crawlReceipt.html?iam=...&lurlEntry=...&reason=...&result=...&wordh=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/hello.md b/help/yacy/hello.md new file mode 100644 index 000000000..f73593ce5 --- /dev/null +++ b/help/yacy/hello.md @@ -0,0 +1,65 @@ +--- +page: htroot/yacy/hello.html +help: help/yacy/hello.md +title: hello +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/hello.java +--- + +# hello + +## Purpose + +hello is the peer greeting endpoint. + +Use it when another peer or diagnostic tool needs to identify and negotiate with this YaCy peer. + +## What You Can Do Here + +- Identify this peer during YaCy peer-to-peer handshakes. +- Send the peer identity, key, seed, and protocol fields exactly as expected by the backend. +- Use the response to confirm that the peer can be reached and understood by another YaCy component. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/hello.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/hello.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/hello.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | + +Example request shape: + +```http +GET or POST /yacy/hello.html?count=...&iam=...&key=...&magic=...&seed=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/list.md b/help/yacy/list.md new file mode 100644 index 000000000..546f4aa24 --- /dev/null +++ b/help/yacy/list.md @@ -0,0 +1,64 @@ +--- +page: htroot/yacy/list.html +help: help/yacy/list.md +title: list +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/list.java +--- + +# list + +## Purpose + +list returns peer or index lists for YaCy network communication. + +Use it when a tool needs compact list data instead of a rendered page. + +## What You Can Do Here + +- Request compact list data from the peer. +- Choose the list type and limits intentionally. +- Consume the response as protocol data rather than rendered help. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/list.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/list.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/list.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /yacy/list.html?col=...&iam=...&listname=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/message.md b/help/yacy/message.md new file mode 100644 index 000000000..aeeab8bf7 --- /dev/null +++ b/help/yacy/message.md @@ -0,0 +1,64 @@ +--- +page: htroot/yacy/message.html +help: help/yacy/message.md +title: message +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/message.java +--- + +# message + +## Purpose + +message exchanges messages between YaCy peers. + +Use it for peer protocol communication, not casual browser messaging. + +## What You Can Do Here + +- Exchange peer protocol messages. +- Send exact sender, receiver, and message fields. +- Check whether the response confirms delivery or reports a protocol problem. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/message.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/message.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/message.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /yacy/message.html?iam=...&message=...&myseed=...&process=...&subject=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/profile.md b/help/yacy/profile.md new file mode 100644 index 000000000..53d642f76 --- /dev/null +++ b/help/yacy/profile.md @@ -0,0 +1,49 @@ +--- +page: htroot/yacy/profile.html +help: help/yacy/profile.md +title: profile +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/profile.java +--- + +# profile + +## Purpose + +profile returns peer profile information in a compact form. + +Use it when a tool needs metadata about this peer or another peer. + +## What You Can Do Here + +- Retrieve compact peer profile information. +- Use the response to identify a peer in tools or diagnostics. +- Avoid assuming profile data is complete or private. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/profile.java`. + +No request parameters are needed for normal use of this page. + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/query.md b/help/yacy/query.md new file mode 100644 index 000000000..aa785593b --- /dev/null +++ b/help/yacy/query.md @@ -0,0 +1,64 @@ +--- +page: htroot/yacy/query.html +help: help/yacy/query.md +title: query +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/query.java +--- + +# query + +## Purpose + +query is a peer search/query endpoint. + +Use it when another peer or tool submits a search request through YaCy's protocol surface. + +## What You Can Do Here + +- Submit a protocol-level query to a YaCy peer. +- Provide exact query and peer parameters rather than browser-form labels. +- Use the compact response as data for another peer or tool. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/query.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/query.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/query.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /yacy/query.html?env=...&iam=...&key=...&object=...&youare=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/search.md b/help/yacy/search.md new file mode 100644 index 000000000..e8cba7e54 --- /dev/null +++ b/help/yacy/search.md @@ -0,0 +1,75 @@ +--- +page: htroot/yacy/search.html +help: help/yacy/search.md +title: search +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/search.java +--- + +# search + +## Purpose + +search is a machine-oriented search endpoint. + +Use it when the caller needs search data without the full browser interface. + +## What You Can Do Here + +- Run a machine-oriented search request. +- Choose query, count, resource, and filter parameters explicitly. +- Use the returned data in tools that do not need the full browser result page. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/search.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/search.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/search.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `constraint` | Encoded YaCy bitfield constraint. Leave it empty unless you are replaying a URL generated by YaCy itself. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `filter` | Filter text or expression used to narrow the displayed records. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `language` | Interface language. Values are `browser` for the browser-preferred language, `default` for English, or a language code such as `de`, `fr`, `es`, `zh`, `ja`, or `ko`. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `sitehost` | Host or domain scope. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `strictContentDom` | When true, YaCy enforces the selected content domain more strictly instead of using it mainly for presentation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `time` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | + +Example request shape: + +```http +GET or POST /yacy/search.html?language=...&query=...&contentdom=...&abstracts=...&author=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/seedlist.md b/help/yacy/seedlist.md new file mode 100644 index 000000000..a2409ff44 --- /dev/null +++ b/help/yacy/seedlist.md @@ -0,0 +1,66 @@ +--- +page: htroot/yacy/seedlist.html +help: help/yacy/seedlist.md +title: seedlist +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/seedlist.java +--- + +# seedlist + +## Purpose + +seedlist publishes known peer seeds. + +Use it when bootstrapping or inspecting YaCy network peer knowledge. + +## What You Can Do Here + +- Retrieve known peer seed information. +- Use the response for network bootstrap, peer diagnostics, or network-state inspection. +- Do not treat the seed list as user search content. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/seedlist.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/seedlist.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/seedlist.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `peername` | Public peer name. Use 3 to 80 letters, digits, hyphen, or underscore; spaces are converted to hyphens. | Changes stored data, configuration, or a running job. Use the authenticated action flow where required and verify the result. | + +Example request shape: + +```http +GET or POST /yacy/seedlist.html?peername=...&address=...&callback=...&id=...&maxcount=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/transferRWI.md b/help/yacy/transferRWI.md new file mode 100644 index 000000000..3f7fe06cf --- /dev/null +++ b/help/yacy/transferRWI.md @@ -0,0 +1,64 @@ +--- +page: htroot/yacy/transferRWI.html +help: help/yacy/transferRWI.md +title: transferRWI +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/transferRWI.java +--- + +# transferRWI + +## Purpose + +transferRWI exchanges reverse word index data between peers. + +Use it for network index-sharing behavior, not for ordinary user search. + +## What You Can Do Here + +- Exchange reverse-word-index data with another peer. +- Send only the protocol fields needed for the transfer. +- Use it as part of YaCy index sharing, not as a manual search endpoint. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/transferRWI.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/transferRWI.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/transferRWI.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | + +Example request shape: + +```http +GET or POST /yacy/transferRWI.html?entryc=...&iam=...&indexes=...&key=...&wordc=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacy/transferURL.md b/help/yacy/transferURL.md new file mode 100644 index 000000000..204c7184d --- /dev/null +++ b/help/yacy/transferURL.md @@ -0,0 +1,65 @@ +--- +page: htroot/yacy/transferURL.html +help: help/yacy/transferURL.md +title: transferURL +package: machine-api-peer +access: peer-service +kind: peer-endpoint +backend_java: source/net/yacy/htroot/yacy/transferURL.java +--- + +# transferURL + +## Purpose + +transferURL exchanges URL metadata between peers. + +Use it when YaCy peers share crawl or index knowledge. + +## What You Can Do Here + +- Exchange URL metadata with another peer. +- Keep hashes and URL identifiers exact. +- Use the response to support peer index synchronization. + +## Page Architecture + +This is a compact endpoint-style page. Its behavior is driven mainly by request parameters and the selected response template, so callers should send only the fields needed for the specific query or peer-service action. + +## Correct Use + +Call the endpoint as a protocol surface. Use exact parameter names and encoded values, authenticate when required, and inspect the response before relying on it. Avoid sending browser-only submit buttons unless the backend explicitly requires the action key. + +## Access And Safety + +This is a peer-service endpoint for YaCy peer communication, not a normal editing page. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacy/transferURL.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacy/transferURL.html` | `GET or POST` | peer-service | `source/net/yacy/htroot/yacy/transferURL.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `url` | URL to inspect, crawl, import, or act on. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /yacy/transferURL.html?url=...&iam=...&key=...&urlc=...&youare=... +``` + +## What To Expect + +Expect a compact service response rather than a teaching interface. The response may be XML, RSS, JSON-like text, plain text, or a small HTML template depending on the endpoint. + +## Related Pages + +- Related protocol work usually continues through the calling tool, the peer endpoint family under `/yacy/`, or the API page that consumes this response. diff --git a/help/yacychat.md b/help/yacychat.md new file mode 100644 index 000000000..36a5b7ae2 --- /dev/null +++ b/help/yacychat.md @@ -0,0 +1,71 @@ +--- +page: htroot/yacychat.html +help: help/yacychat.md +title: Chat +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/yacychat.java +--- + +# Chat + +## Purpose + +Chat provides a conversational interface around YaCy search or local features. + +Use it when the user wants an interactive dialogue instead of a classic result page. + +## What You Can Do Here + +- Chat provides a conversational interface around YaCy search or local features. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `userInput` | no search, allow attachments. | Checkbox/boolean; present usually means enabled. | +| `sendButton` | no search, allow attachments. | `Send` | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacychat.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacychat.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/yacychat.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `userInput` | no search, allow attachments. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `sendButton` | no search, allow attachments. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacychat.html?userInput=...&searchButton=...&addFileButton=...&fileInput=...&sendButton=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/yacyinteractive.md b/help/yacyinteractive.md new file mode 100644 index 000000000..524bd669a --- /dev/null +++ b/help/yacyinteractive.md @@ -0,0 +1,73 @@ +--- +page: htroot/yacyinteractive.html +help: help/yacyinteractive.md +title: Interactive Search +package: core-search-public +access: public +kind: ui-page +backend_java: source/net/yacy/htroot/yacyinteractive.java +--- + +# Interactive Search + +## Purpose + +Interactive Search offers a dynamic search experience. + +Use it when users expect result updates or interaction beyond a static result page. + +## What You Can Do Here + +- Interactive Search offers a dynamic search experience. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | +| `Enter` | Browser submit button; omit in direct API calls. | `Search` | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacyinteractive.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacyinteractive.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/yacyinteractive.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `focus` | Browser presentation flag controlling whether the search input receives focus. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacyinteractive.html?query=...&maximumRecords=...&Enter=...&focus=...&startRecord=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/yacysearch.md b/help/yacysearch.md new file mode 100644 index 000000000..36181f950 --- /dev/null +++ b/help/yacysearch.md @@ -0,0 +1,102 @@ +--- +page: htroot/yacysearch.html +help: help/yacysearch.md +title: Search Page +package: core-search-public +access: mixed +kind: search-page +backend_java: source/net/yacy/htroot/yacysearch.java +--- + +# Search Page + +## Purpose + +The Search Page is the main result page. It turns a query into local or distributed search results and carries the parameters that define result count, content type, language, pagination, and verification behavior. + +Use it to understand how a browser search maps to an API-style request. + +## What You Can Do Here + +- Search the local YaCy index, or the YaCy network when the peer is configured for global search. +- Choose query, result type, collection, and verification options according to the information need. +- Page through results, restrict results to a URL pattern, request a content type such as images or text, and choose how fresh snippets must be. +- When authenticated, perform result actions such as deleting a reference, recommending a result, bookmarking it, or sending a URL to a blacklist. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | +| `Enter` | Browser submit button. Omit it from direct API calls; YaCy uses the query and filter parameters, not the button label, to run the search. | Browser-generated submit value only. | + +## Correct Use + +For a normal search, send only `query` and the options that express the user intent: `maximumRecords` for result count, `resource` for local versus network search, `contentdom` for text or media, and `urlmaskfilter` when the user asks to search inside a site. Start with `verify=iffresh` when freshness matters; use `cacheonly` only when the user explicitly wants stored snippets without refetching. + +Do not send result-action parameters such as `deleteref`, `bookmarkref`, or `blacklisturl` as part of an ordinary search. Those parameters change state and require an exact result reference from a prior response. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/yacysearch.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacysearch.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearch.html` | `GET` | mixed | `source/net/yacy/htroot/yacysearch.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. Low-level generated parameters are omitted when they are only meaningful inside the rendered YaCy form. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `constraint` | Encoded YaCy bitfield constraint. Leave it empty unless you are replaying a URL generated by YaCy itself. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleteref` | URL hash/reference selected for deletion from results. Use only with administrator intent and an exact known reference. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `focus` | Browser presentation flag controlling whether the search input receives focus. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexof` | Interprets the query as an index-of style request when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lr` | Language filter, usually a two-letter code such as `de`, `fr`, or `en` when the page supports language-restricted search. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `nav` | Navigator/facet selection for search results. `all` requests the normal set; narrower values reduce displayed facets. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control. Use `0` for the first page and increase by the result count for later pages. | +| `prefermaskfilter` | Regular-expression URL preference. Matching results are favored without excluding all others. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `recommendref` | URL hash/reference selected for recommendation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resortCachedResults` | Requests resorting of a cached search event instead of starting a fully fresh result event. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `resource` | Search source. `local` searches this peer index; `global` may use the YaCy network when the peer and configuration allow it. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resource-switch` | Browser control for switching between local and global search. For direct calls use `resource=local` or `resource=global`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `rows` | Rows requested by API-style clients. It is another result-count alias on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `strictContentDom` | When true, YaCy enforces the selected content domain more strictly instead of using it mainly for presentation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only context for date handling; it does not by itself change the index. | +| `urlmaskfilter` | Regular-expression URL filter for search results. Use `.*` for no restriction; use a precise host/path expression to search inside a site or section. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `verify` | Snippet/cache verification strategy. Typical values are `iffresh`, `ifexist`, `cacheonly`, `nocache`, and `false`; choose according to whether freshness or speed matters more. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacysearch.html?query=climate+data&maximumRecords=10&resource=local&contentdom=text&verify=iffresh +``` + +## What To Expect + +The response is an HTML result page. It contains result entries, navigators/facets when enabled, and hidden references used by authenticated result actions. If a query returns too little, first check whether the crawler has indexed the relevant site and whether `urlmaskfilter`, `collection`, or `contentdom` narrowed the request too far. + +## Related Pages + +- `Status.html` diff --git a/help/yacysearch_location.md b/help/yacysearch_location.md new file mode 100644 index 000000000..c4fee87c7 --- /dev/null +++ b/help/yacysearch_location.md @@ -0,0 +1,75 @@ +--- +page: htroot/yacysearch_location.html +help: help/yacysearch_location.md +title: Location Search +package: core-search-public +access: public +kind: search-page +backend_java: source/net/yacy/htroot/yacysearch_location.java +--- + +# Location Search + +## Purpose + +Location Search searches with geographic or location-oriented result handling. + +Use it when place context matters more than a plain text result list. + +## What You Can Do Here + +- Location Search searches with geographic or location-oriented result handling. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Search terms, for example `climate data` or a more specific phrase. | +| `Enter` | Browser submit button; omit in direct API calls. | `search` | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacysearch_location.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearch_location.html` | `GET` | public or page-dependent | `source/net/yacy/htroot/yacysearch_location.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `maximumTime` | Date/time value for filtering, display, or scheduling. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `verify` | Snippet/cache verification strategy. Typical values are `iffresh`, `ifexist`, `cacheonly`, `nocache`, and `false`; choose according to whether freshness or speed matters more. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacysearch_location.html?query=...&maximumRecords=...&contentdom=...&Enter=...&dom=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/yacysearchitem.md b/help/yacysearchitem.md new file mode 100644 index 000000000..ef699641e --- /dev/null +++ b/help/yacysearchitem.md @@ -0,0 +1,74 @@ +--- +page: htroot/yacysearchitem.html +help: help/yacysearchitem.md +title: yacysearchitem +package: core-search-public +access: mixed +kind: search-page +backend_java: source/net/yacy/htroot/yacysearchitem.java +--- + +# yacysearchitem + +## Purpose + +Search item is a result-rendering fragment. + +Use it indirectly: it explains how one result entry is shaped inside the search page. + +## What You Can Do Here + +- Search item is a result-rendering fragment. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/yacysearchitem.html`. + +Backend checks: administrator authentication, user authentication, viewing rights. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacysearchitem.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearchitem.html` | `GET or POST` | mixed | `source/net/yacy/htroot/yacysearchitem.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET or POST /yacysearchitem.html?auth=...&eventID=...&item=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `ViewFile.html` +- `api/citation.html` +- `yacysearch.html` +- `CacheResource_p.html` +- `proxy.html` +- `IndexBrowser_p.html` diff --git a/help/yacysearchpagination.md b/help/yacysearchpagination.md new file mode 100644 index 000000000..4c1cb83a5 --- /dev/null +++ b/help/yacysearchpagination.md @@ -0,0 +1,68 @@ +--- +page: htroot/yacysearchpagination.html +help: help/yacysearchpagination.md +title: yacysearchpagination +package: core-search-public +access: public +kind: search-page +backend_java: source/net/yacy/htroot/yacysearchpagination.java +--- + +# yacysearchpagination + +## Purpose + +Search pagination is a result-navigation fragment. + +Use it indirectly to understand how result pages link to earlier and later result ranges. + +## What You Can Do Here + +- Search pagination is a result-navigation fragment. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page is normally public or read-only, unless the peer is configured to require authentication for all pages. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacysearchpagination.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearchpagination.html` | `GET or POST` | public or page-dependent | `source/net/yacy/htroot/yacysearchpagination.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `jsResort` | Sort field or sort direction. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control; use it to request later result pages. | + +Example request shape: + +```http +GET or POST /yacysearchpagination.html?maximumRecords=...&auth=...&eventID=...&jsResort=...&offset=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- Related search work usually continues on `yacysearch.html`, `index.html`, `ViewFile.html`, quick crawl, or the search integration pages. diff --git a/help/yacysearchtrailer.md b/help/yacysearchtrailer.md new file mode 100644 index 000000000..0d915e57e --- /dev/null +++ b/help/yacysearchtrailer.md @@ -0,0 +1,101 @@ +--- +page: htroot/yacysearchtrailer.html +help: help/yacysearchtrailer.md +title: yacysearchtrailer +package: core-search-public +access: mixed +kind: search-page +backend_java: source/net/yacy/htroot/yacysearchtrailer.java +--- + +# yacysearchtrailer + +## Purpose + +Search trailer is a result-page footer fragment. + +Use it indirectly to understand what is appended after the result list. + +## What You Can Do Here + +- Search trailer is a result-page footer fragment. +- Choose query, target, content type, or integration options according to the user intent. +- Keep ordinary read-only viewing separate from authenticated actions that alter stored data. + +## Page Architecture + +Public search pages turn request parameters into result lists, previews, snippets, feeds, or integration fragments. They are mostly read-oriented, but some result actions can bookmark, recommend, blacklist, or delete references when authenticated. + +| Control | Meaning | Values or examples | +| --- | --- | --- | +| `contentdom` | Content domain filter: text, image, audio, video, application, or all depending on the page. | `text`, `image`, `audio`, `video`, `app` | +| `strictContentDom` | Require the content domain filter strictly. | `false`, `true` | +| `resource` | Search resource selection such as local or global/network search. | `global`, `local` | + +## Correct Use + +Start from the user's information need. Use query text, content type, collection, URL filters, and pagination to narrow results. Do not mix ordinary search with authenticated result actions unless the user explicitly asks to modify stored data. + +## Access And Safety + +The page may be visible, but the backend performs authentication checks for protected actions. + +Protected related endpoint(s): `/yacysearch.html`, `/yacysearchtrailer.html`. + +Backend checks: administrator authentication, user authentication. + +## Automation And API + +Page backend: `source/net/yacy/htroot/yacysearchtrailer.java`. + +| Endpoint | Method | Access | Backend | +| --- | --- | --- | --- | +| `/yacysearch.html` | `GET` | mixed | `source/net/yacy/htroot/yacysearch.java` | +| `/yacysearchtrailer.html` | `GET or POST` | mixed | `source/net/yacy/htroot/yacysearchtrailer.java` | + +### Parameter Guide + +The table explains values that an agent or script must set deliberately. Parameters not relevant to a task should be omitted or left at the page default. + +| Parameter | Meaning and valid values | Care | +| --- | --- | --- | +| `contentdom` | Content domain filter. Common values are `all`, `text`, `image`, `audio`, `video`, and `app`; use it to ask for web pages, media, or application documents deliberately. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `strictContentDom` | When true, YaCy enforces the selected content domain more strictly instead of using it mainly for presentation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resource` | Search source. `local` searches this peer index; `global` may use the YaCy network when the peer and configuration allow it. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `auth` | Requests authentication-aware behavior when the endpoint supports user-specific or protected actions. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `callback` | JSONP callback name for legacy script clients. Leave empty for normal HTML or JSON-style use. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `collection` | Collection name. Use it to group crawled or imported documents and to search or manage that group later. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `constraint` | Encoded YaCy bitfield constraint. Leave it empty unless you are replaying a URL generated by YaCy itself. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `count` | SRU-style result count. It is an alternative to `maximumRecords` on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `deleteref` | URL hash/reference selected for deletion from results. Use only with administrator intent and an exact known reference. | Can remove data, stop work, expose access, or make a broad operational change. Use only with explicit confirmation and an exact target. | +| `focus` | Browser presentation flag controlling whether the search input receives focus. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `indexof` | Interprets the query as an index-of style request when supported. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `lr` | Language filter, usually a two-letter code such as `de`, `fr`, or `en` when the page supports language-restricted search. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `maximumRecords` | Maximum number of results to return on one page. Use a modest value for interactive use; larger values are for controlled scripts. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `nav` | Navigator/facet selection for search results. `all` requests the normal set; narrower values reduce displayed facets. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `offset` | Zero-based result offset for pagination. | Read-only pagination control; use it to request later result pages. | +| `prefermaskfilter` | Regular-expression URL preference. Matching results are favored without excluding all others. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `query` | Search text. Use ordinary search terms, quoted phrases where supported by YaCy query parsing, and optional YaCy modifiers such as collection filters when you intentionally need them. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `recommendref` | URL hash/reference selected for recommendation. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `resortCachedResults` | Requests resorting of a cached search event instead of starting a fully fresh result event. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `resource-switch` | Browser control for switching between local and global search. For direct calls use `resource=local` or `resource=global`. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `rows` | Rows requested by API-style clients. It is another result-count alias on search endpoints. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `search` | Alternative search text parameter accepted by some search endpoints; prefer `query` on browser search pages unless reproducing an existing URL. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `startRecord` | First result record for pagination; accepted as an alternative to `offset` on search endpoints. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | +| `timezoneOffset` | Client timezone offset in minutes. YaCy uses it for date display or schedule calculations. | Read-only request context for date handling. | +| `urlmaskfilter` | Regular-expression URL filter for search results. Use `.*` for no restriction; use a precise host/path expression to search inside a site or section. | Controls the scope or format of the result. Prefer the narrowest value that answers the request. | +| `verify` | Snippet/cache verification strategy. Typical values are `iffresh`, `ifexist`, `cacheonly`, `nocache`, and `false`; choose according to whether freshness or speed matters more. | Set only when this option is part of the intended request; otherwise omit it and let YaCy use the page default. | + +Example request shape: + +```http +GET /yacysearch.html?query=...&maximumRecords=...&resource=...&contentdom=...&strictContentDom=... +``` + +## What To Expect + +Expect rendered search or content output: result lists, snippets, previews, redirects, widgets, or fragments. If output is empty, check whether the index contains matching documents before changing query syntax. + +## Related Pages + +- `yacysearch_location.html` |
