diff options
| author | Michael Peter Christen <mc@yacy.net> | 2023-11-12 22:11:55 +0100 |
|---|---|---|
| committer | Michael Peter Christen <mc@yacy.net> | 2023-11-12 22:11:55 +0100 |
| commit | c20c4b8a21364bf06d1f91c63650fcf3b434ba04 (patch) | |
| tree | 9428949b6082dde87b1a6555ed181d114f08bf66 /htroot | |
| parent | 655d8db80218312593f37b6026fd6dc0db01d23f (diff) | |
modified export: added maximum number of docs per chunk
The export file can now be many files, called chunks.
By default still only one chunk is exported.
This function is required in case that the exported files shall be
imported to an elasticsearch/opensearch index. The bulk import function
of elasticsearch/opensearch is limited to 100MB. To make it possible to
import YaCy files, those must be splitted into chunks. Right now we
cannot estimate the chunk size as bytes, only as number of documents.
The user must do experiments to find out the optimum chunk max size,
like 50000 docs per chunk. Try this as first attempt.
Diffstat (limited to 'htroot')
| -rw-r--r-- | htroot/IndexExport_p.html | 11 |
1 files changed, 7 insertions, 4 deletions
diff --git a/htroot/IndexExport_p.html b/htroot/IndexExport_p.html index eb3e1f188..87ee4b62d 100644 --- a/htroot/IndexExport_p.html +++ b/htroot/IndexExport_p.html @@ -21,13 +21,16 @@ <dd><input type="text" name="exportfilepath" value="#[exportfilepath]#" size="120" maxlength="250" /> </dd> <dt class="TableCellDark">URL Filter</dt> - <dd><input type="text" name="exportfilter" value=".*.*" size="20" maxlength="250" /> + <dd><input type="text" name="exportfilter" value=".*.*" size="20" maxlength="250" /> .*.* (default) is a catch-all; format: java regex </dd> <dt class="TableCellDark">query</dt> - <dd><input type="text" name="exportquery" value="*:*" size="20" maxlength="250" /> + <dd><input type="text" name="exportquery" value="*:*" size="20" maxlength="250" /> *:* (default) is a catch-all; format: <field-name>:<solr-pattern> </dd> - <dt class="TableCellDark">maximum age (seconds, -1 = unlimited)</dt> - <dd><input type="text" name="exportmaxseconds" value="-1" size="20" maxlength="250" /> + <dt class="TableCellDark">maximum age (seconds)</dt> + <dd><input type="text" name="exportmaxseconds" value="-1" size="20" maxlength="250" /> -1 = unlimited -> no document is too old + </dd> + <dt class="TableCellDark">maximum number of records per chunk</dt> + <dd><input type="text" name="maxchunksize" value="-1" size="20" maxlength="250" /> if exceeded: several chunks are stored; -1 = unlimited (makes only one chunk) </dd> <dt class="TableCellDark">Export Format</dt> <dd> |