diff options
| author | Michael Peter Christen <mc@yacy.net> | 2023-11-12 15:26:18 +0100 |
|---|---|---|
| committer | Michael Peter Christen <mc@yacy.net> | 2023-11-12 15:26:18 +0100 |
| commit | 655d8db80218312593f37b6026fd6dc0db01d23f (patch) | |
| tree | 4ece601709ad7c9ffae2acddb978f70fb11d831b /htroot/IndexExport_p.html | |
| parent | 24011dcbcc26f79f80a81a8dae59c6f65c1cee25 (diff) | |
detailed directions in index export to explain how the export can be
imported again using elasticsearch/opensearch
Diffstat (limited to 'htroot/IndexExport_p.html')
| -rw-r--r-- | htroot/IndexExport_p.html | 51 |
1 files changed, 38 insertions, 13 deletions
diff --git a/htroot/IndexExport_p.html b/htroot/IndexExport_p.html index 1aa992716..eb3e1f188 100644 --- a/htroot/IndexExport_p.html +++ b/htroot/IndexExport_p.html @@ -9,11 +9,10 @@ <body id="IndexControl"> #%env/templates/header.template%# #%env/templates/submenuIndexImport.template%# - - + <h2>Index Export</h2> <p>The local index currently contains #[ucount]# documents, only #[ucount200]# exportable with status code 200 - the remaining are error documents.</p> - + #(lurlexport)#:: <form action="IndexExport_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8"> <fieldset><legend>Loaded URL Export</legend> @@ -34,19 +33,45 @@ <dd> <dl> <dt>Full Data Records:</dt> - <dd><input type="radio" name="format" value="full-solr" /> XML (Rich and full-text Solr data, one document per line in one large xml file, can be processed with shell tools, can be imported with DATA/SURROGATE/in/)<br /> - <input type="radio" name="format" value="full-elasticsearch" checked="checked" /> JSON (Rich and full-text Elasticsearch data, one document per line in one flat JSON file, can be bulk-imported to elasticsearch with the command "curl -XPOST localhost:9200/collection1/yacy/_bulk --data-binary @yacy_dump_XXX.flatjson")<br /> - <input type="radio" name="format" value="full-rss" /> XML (RSS)</dd> + <dd><input type="radio" name="format" value="full-elasticsearch" checked="checked" /> + JSON (Rich and full-text Elasticsearch data, one document per line in one flat JSON file, + can be bulk-imported to elasticsearch. Here is an example for opensearch, using docker:<br /> +Start docker container of opensearch:<br /> +<code>docker run --name opensearch -p 9200:9200 -d -e OPENSEARCH_JAVA_OPTS="-Xms2G -Xmx2G" -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -v $(pwd)/opensearch_data:/usr/share/opensearch/data opensearchproject/opensearch:latest</code><br /> +Unblock index creation:<br /> +<code>curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' +{ + "persistent": { + "cluster.blocks.create_index": null + } +}'</code><br /> +Create the search index:<br /> +<code>curl -X PUT "http://localhost:9200/collection1/yacy"</code><br /> +Bulk-upload the index file:<br /> +<code>curl -XPOST "http://localhost:9200/collection1/yacy/_bulk?filter_path=took,errors" -H "Content-Type: application/x-ndjson" --data-binary @yacy_dump_XXX.flatjson</code><br /> +Make a search, get 10 results, search in fields text_t, title, description with boosts:<br /> +<code>curl -X POST "http://localhost:9200/collection1/yacy/_search" -H 'Content-Type: application/json' -d' +{"size": 10, "query": {"multi_match": { + "query": "one two three", + "fields": ["text_t", "title^10", "description^3"], "fuzziness": "AUTO" +}}}'</code><br /> + <input type="radio" name="format" value="full-solr" /> + XML (Rich and full-text Solr data, one document per line in one large xml file, + can be processed with shell tools, can be imported with DATA/SURROGATE/in/) + <br /> + <input type="radio" name="format" value="full-rss" /> + XML (RSS) + </dd> <dt>Full URL List:</dt> <dd><input type="radio" name="format" value="url-text" /> Plain Text List (URLs only)<br /> <input type="radio" name="format" value="url-html" /> HTML (URLs with title)</dd> <dt>Only Domain:</dt> <dd><input type="radio" name="format" value="dom-text" /> Plain Text List (domains only)<br /> <input type="radio" name="format" value="dom-html" /> HTML (domains as URLs, no title)</dd> - <dt>Only Text:</dt> + <dt>Only Text:</dt> <dd><input type="radio" name="format" value="text-text" /> Fulltext of Search Index Text</dd> - </dl> - </dd> + </dl> + </dd> <dt> </dt> <dd><input type="submit" name="lurlexport" value="Export" class="btn btn-primary" style="width:240px;"/> </dd> @@ -55,16 +80,16 @@ </form>:: <div class="alert alert-info" style="text-decoration:blink">Export to file #[exportfile]# is running .. #[urlcount]# Documents so far</div>:: #(/lurlexport)# - - #(lurlexportfinished)#:: + + #(lurlexportfinished)#:: <div class="alert alert-success">Finished export of #[urlcount]# Documents to file <a href="file://#[exportfile]#" target="_">#[exportfile]#</a><br/> <em>Import this file by moving it to DATA/SURROGATES/in</em></div>:: #(/lurlexportfinished)# - + #(lurlexporterror)#:: <div class="alert alert-warning">Export to file #[exportfile]# failed: #[exportfailmsg]#</div>:: #(/lurlexporterror)# - + #(dumprestore)#:: <form action="IndexExport_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8"> <fieldset><legend>Dump and Restore of Solr Index</legend> |