yacy/htroot/IndexExport_p.html, branch masterYaCy search server with some extra patches
https://git.fennell.dev/yacy/atom?h=master2025-08-01T06:30:46ZYaCy Index Pack Generator - refactoring of the SURROGATE process into a2025-08-01T06:30:46ZMichael Peter Christenmc@yacy.net2025-08-01T06:30:46Zurn:sha1:de8fef9bf14e0c56f0aaac9199dd6071d87f1731
"PACK" concept.added a 'minified' option to YaCy dumps2023-11-13T09:27:50ZMichael Peter Christenmc@yacy.net2023-11-13T09:27:50Zurn:sha1:3268a93019aa00b49fdfb5925d7caac3c1a94274modified export: added maximum number of docs per chunk2023-11-12T21:11:55ZMichael Peter Christenmc@yacy.net2023-11-12T21:11:55Zurn:sha1:c20c4b8a21364bf06d1f91c63650fcf3b434ba04
The export file can now be many files, called chunks.
By default still only one chunk is exported.
This function is required in case that the exported files shall be
imported to an elasticsearch/opensearch index. The bulk import function
of elasticsearch/opensearch is limited to 100MB. To make it possible to
import YaCy files, those must be splitted into chunks. Right now we
cannot estimate the chunk size as bytes, only as number of documents.
The user must do experiments to find out the optimum chunk max size,
like 50000 docs per chunk. Try this as first attempt.detailed directions in index export to explain how the export can be2023-11-12T14:26:18ZMichael Peter Christenmc@yacy.net2023-11-12T14:26:18Zurn:sha1:655d8db80218312593f37b6026fd6dc0db01d23f
imported again using elasticsearch/opensearchcalculating the correct size of an export.2021-09-15T23:05:09ZMichael Peter Christenmc@yacy.net2021-09-15T23:05:09Zurn:sha1:1bab4ffe2075e5941c46bc9e31623bf3a204fe34
This can be seen as a fix for
https://github.com/yacy/yacy_search_server/issues/343
however, the export was not flawed, it is just the impression that
something is wrong, but the export size must be smaller than the index
size because the index also containers error documents.
Now an information line is presented that shows i.e.:
"The local index currently contains 181,319 documents, only 106,887
exportable with status code 200 - the remaining are error documents."Ensure an embedded Solr is available for Solr dump/restore operations2018-04-07T11:42:06Zlucciomanluccioman@users.noreply.github.com2018-04-07T11:42:06Zurn:sha1:0d34034f17922e1c0dcd82bbccf9564f2100d1ef
Otherwise, these operations triggered NullPointerException when only an
external Solr index is attached.added export to elasticsearch. The export dump can easily be imported to2017-03-31T23:04:17ZMichael Peter Christenmc@yacy.net2017-03-31T23:04:17Zurn:sha1:69081bce00864a61fd70b56078e0c4dbfb72a69e
elasticsearch using the command
curl -XPOST localhost:9200/collection1/yacy/_bulk --data-binary
@yacy_dump_XXX.flatjson
0N - added option to generate index export files for a specific number2016-02-23T17:56:20ZMichael Peter Christenmc@yacy.net2016-02-23T17:56:20Zurn:sha1:a6bf0b1649f89de02fa9f2535891911ede38088e
of minutes in the past and reverted latest change. The export file dump
will now contain four data elements: f - first date of index entry write
date, l - last date of index write date, n - now-date of index dump
time, c - count of numbers inside the dump. '0N' denotes a series of
changes which will lead to the opportunity to exchange index data dumps
in a way that is needed to integrate ZeroNet index data. This will be
based on index dump sharing; that causes this commit.added peer hash to export dump name.2016-02-19T18:26:02ZMichael Peter Christenmc@yacy.net2016-02-19T18:26:02Zurn:sha1:5b9030180c2d12dba7d43ab6dbeb37ad537871c2added export option to export the fulltext of the search index text only2015-07-30T01:21:40ZMichael Peter Christenmc@yacy.net2015-07-30T01:21:40Zurn:sha1:de8cfbe1d7fc894e0ae16704e15b5b22785281c6