YaCy Index Pack Generator - refactoring of the SURROGATE process into a

"PACK" concept.

11 files changed, 225 insertions, 93 deletions

diff --git a/htroot/ContentIntegrationPHPBB3_p.html b/htroot/ContentIntegrationPHPBB3_p.html
index c22651f80..3c886f36f 100644
--- a/htroot/ContentIntegrationPHPBB3_p.html
+++ b/htroot/ContentIntegrationPHPBB3_p.html
@@ -22,8 +22,8 @@
 		<li>deselect the partial import flag</li>

     </ul>

     <p>

-      When an export is started, surrogate files are generated into DATA/SURROGATE/in which are automatically fetched by an indexer thread.

-      All indexed surrogate files are then moved to DATA/SURROGATE/out and can be re-cycled when an index is deleted.

+      When an export is started, pack files are generated into DATA/PACKS/load which are automatically fetched by an indexer thread.

+      All indexed pack files are then moved to DATA/PACKS/loaded and can be re-cycled when an index is deleted.

     </p>

         

     <form action="ContentIntegrationPHPBB3_p.html" accept-charset="UTF-8">

@@ -53,7 +53,7 @@
           <dt><b>Password</b> for the account of that user given above</dt>

           <dd><input type="text" name="content.phpbb3.dbpw" value="#[content.phpbb3.dbpw]#" size="20" /></dd>

           

-          <dt><b>Posts per file</b><br />in exported surrogates</dt>

+          <dt><b>Posts per file</b><br />in exported packs</dt>

           <dd><input type="text" name="content.phpbb3.ppf" value="#[content.phpbb3.ppf]#" size="20" /></dd>

           

           <dt>&nbsp;</dt>

@@ -62,7 +62,7 @@
           </dd>

           <dt>&nbsp;</dt>

           <dd>

-            <input type="submit" name="export" value="Export Content to Surrogates" class="btn btn-primary" style="width:240px"/>

+            <input type="submit" name="export" value="Export Content to Packs" class="btn btn-primary" style="width:240px"/>

           </dd>

         </dl>

       </fieldset>

@@ -98,7 +98,7 @@
     #(/check)#

     

     #(export)#::

-    <p>Export successful! Wrote #[files]# files in DATA/SURROGATES/in</p>::

+    <p>Export successful! Wrote #[files]# files in DATA/PACKS/load</p>::

     <p>Export failed: #[error]#</p>

     #(/export)#

     

diff --git a/htroot/CrawlResults.html b/htroot/CrawlResults.html
index 3eccec1e8..11ee431dd 100644
--- a/htroot/CrawlResults.html
+++ b/htroot/CrawlResults.html
@@ -17,7 +17,7 @@
     <p>Case (6) is a monitor of the local receipt-generator, the opposed case of (1). It contains also an indexing result monitor but is not considered private

     since it shows crawl requests from other peers.

     </p>

-    <p>Case (7) occurs if surrogate files are imported</p>

+    <p>Case (7) occurs if pack files are imported</p>

     <p><img src="env/grafics/indexmonitor.png" width="600" height="308" alt="An illustration how yacy works" /></p>

     <p>The image above illustrates the data flow initiated by web index acquisition.

     Some processes occur double to document the complex index migration structure.

@@ -64,9 +64,9 @@
     <p><em>Use Case:</em> This list may fill if you check the 'Accept Remote Crawl Requests'-flag on the '<a href="RemoteCrawl_p.html">Remote Crawling</a>' page</p>

     #(remoteCrawlerDisabled)#::<div class="info"><p>The remote crawler is currently disabled<p></div>#(/remoteCrawlerDisabled)#

     ::

-    <h2>(7) Results from surrogates import</h2>

-    <p>These records had been imported from surrogate files in DATA/SURROGATES/in</p>

-    <p><em>Use Case:</em> place files with dublin core metadata content into DATA/SURROGATES/in or use an index import method (i.e. <a href="IndexImportMediawiki_p.html">MediaWiki import</a>, <a href="IndexImportOAIPMH_p.html">OAI-PMH retrieval</a>)</p>

+    <h2>(7) Results from pack import</h2>

+    <p>These records had been imported from pack files in DATA/PACKS/load</p>

+    <p><em>Use Case:</em> place files with dublin core metadata content into DATA/PACKS/load or use an index import method (i.e. <a href="IndexImportMediawiki_p.html">MediaWiki import</a>, <a href="IndexImportOAIPMH_p.html">OAI-PMH retrieval</a>)</p>

     #(/process)#

     

     

diff --git a/htroot/IndexExportImportSolr_p.html b/htroot/IndexExportImportSolr_p.html
new file mode 100644
index 000000000..c6f687557
--- /dev/null
+++ b/htroot/IndexExportImportSolr_p.html
@@ -0,0 +1,51 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
+<!-- This page is only XHTML 1.0 Transitional because target is being used in a links -->
+<html xmlns="http://www.w3.org/1999/xhtml">
+  <head>
+    <title>YaCy '#[clientname]#': URL Database Administration</title>
+    #%env/templates/metas.template%#
+  </head>
+  <body id="IndexControl">
+    #%env/templates/header.template%#
+    #%env/templates/submenuIndexImport.template%#
+
+    <h2>Solr Index Export/Import</h2>
+    <p>The local index currently contains #[ucount]# documents (including non-http-200 error pages).</p>
+
+    #(dumprestore)#::
+    <form action="IndexExportImportSolr_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
+    <fieldset><legend>Dump and Restore of Solr Index</legend>
+      #(dumpRestoreEnabled)#<div class="alert alert-info">This feature is available only when a local embedded Solr is active.</div>::#(/dumpRestoreEnabled)#
+      <dl>
+        <dt>&nbsp;</dt>
+        <dd><input type="submit" name="indexdump" value="Create Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/><br/>
+            (This may take several minutes. Please be patient and wait until the page reloads.)</dd>
+        </dd>
+      </dl>
+      <dl>
+        <dt class="TableCellDark">Dump File (full path)</dt>
+        <dd><input type="text" name="dumpfile" value="#[dumpfile]#" size="80" maxlength="250" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
+        </dd>
+        <dt>&nbsp;</dt>
+        <dd><input type="submit" name="indexrestore" value="Restore Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
+        </dd>
+      </dl>
+    </fieldset>
+    </form>::
+    #(/dumprestore)#
+
+    #(indexdump)#::
+    <div class="alert alert-success" role="alert">Stored a solr dump to file #[dumpfile]#</div>::
+    <div class="alert alert-danger" role="alert">Could not create the Solr dump : no embedded Solr is available.</div>::
+    <div class="alert alert-danger" role="alert">An error occurred while trying to create the Solr dump.</div>
+    #(/indexdump)#
+
+    #(indexRestore)#::
+    <div class="alert alert-success" role="alert">Successfully restored Solr index from dump file!</div>::
+    <div class="alert alert-danger" role="alert">Could not restore the Solr dump : no embedded Solr is available.</div>::
+    <div class="alert alert-danger" role="alert">An error occurred while trying to restore the Solr dump.</div>
+    #(/indexRestore)#
+
+    #%env/templates/footer.template%#
+  </body>
+</html>
diff --git a/htroot/IndexExport_p.html b/htroot/IndexExport_p.html
index df58837c2..feb449643 100644
--- a/htroot/IndexExport_p.html
+++ b/htroot/IndexExport_p.html
@@ -40,38 +40,8 @@
         <dt class="TableCellDark">Export Format</dt>
         <dd>
         <dl>
-        <dt>Full Data Records:</dt>
-        <dd><input type="radio" name="format" value="full-elasticsearch" checked="checked" />
-            JSON (Rich and full-text Elasticsearch data, one document per line in one flat JSON file,
-            can be bulk-imported to elasticsearch. Here is an example for opensearch, using docker:<br />
-Start docker container of opensearch:<br />
-<code>docker run --name opensearch -p 9200:9200 -d -e OPENSEARCH_JAVA_OPTS="-Xms2G -Xmx2G" -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -v $(pwd)/opensearch_data:/usr/share/opensearch/data opensearchproject/opensearch:latest</code><br />
-Unblock index creation:<br />
-<code>curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
-{
-  "persistent": {
-    "cluster.blocks.create_index": null
-  }
-}'</code><br />
-Create the search index:<br />
-<code>curl -X PUT "http://localhost:9200/collection1/yacy"</code><br />
-Bulk-upload the index file:<br />
-<code>curl -XPOST "http://localhost:9200/collection1/yacy/_bulk?filter_path=took,errors" -H "Content-Type: application/x-ndjson" --data-binary @yacy_dump_XXX.flatjson</code><br />
-Make a search, get 10 results, search in fields text_t, title, description with boosts:<br />
-<code>curl -X POST "http://localhost:9200/collection1/yacy/_search" -H 'Content-Type: application/json' -d'
-{"size": 10, "query": {"multi_match": {
-    "query": "one two three",
-    "fields": ["text_t", "title^10", "description^3"], "fuzziness": "AUTO"
-}}}'</code><br />
-            <input type="radio" name="format" value="full-solr" />
-            XML (Rich and full-text Solr data, one document per line in one large xml file,
-            can be processed with shell tools, can be imported with DATA/SURROGATE/in/)
-            <br />
-            <input type="radio" name="format" value="full-rss" />
-            XML (RSS)
-        </dd>
         <dt>Full URL List:</dt>
-        <dd><input type="radio" name="format" value="url-text" /> Plain Text List (URLs only)<br />
+        <dd><input type="radio" name="format" value="url-text" checked="checked"/> Plain Text List (URLs only)<br />
             <input type="radio" name="format" value="url-html" /> HTML (URLs with title)</dd>
         <dt>Only Domain:</dt>
         <dd><input type="radio" name="format" value="dom-text" /> Plain Text List (domains only)<br />
@@ -79,7 +49,6 @@ Make a search, get 10 results, search in fields text_t, title, description with
         <dt>Only Text:</dt>
         <dd><input type="radio" name="format" value="text-text" /> Fulltext of Search Index Text</dd>
         </dl>
-        </dd>
         <dt>&nbsp;</dt>
         <dd><input type="submit" name="lurlexport" value="Export" class="btn btn-primary" style="width:240px;"/>
         </dd>
@@ -91,46 +60,13 @@ Make a search, get 10 results, search in fields text_t, title, description with
 
     #(lurlexportfinished)#::
     <div class="alert alert-success">Finished export of #[urlcount]# Documents to file <a href="file://#[exportfile]#" target="_">#[exportfile]#</a><br/>
-    <em>Import this file by moving it to DATA/SURROGATES/in</em></div>::
+    <em>Import this file by moving it to DATA/PACKS/load</em></div>::
     #(/lurlexportfinished)#
 
     #(lurlexporterror)#::
     <div class="alert alert-warning">Export to file #[exportfile]# failed: #[exportfailmsg]#</div>::
     #(/lurlexporterror)#
 
-    #(dumprestore)#::
-    <form action="IndexExport_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
-    <fieldset><legend>Dump and Restore of Solr Index</legend>
-      #(dumpRestoreEnabled)#<div class="alert alert-info">This feature is available only when a local embedded Solr is active.</div>::#(/dumpRestoreEnabled)#
-      <dl>
-        <dt>&nbsp;</dt>
-        <dd><input type="submit" name="indexdump" value="Create Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
-        </dd>
-      </dl>
-      <dl>
-        <dt class="TableCellDark">Dump File</dt>
-        <dd><input type="text" name="dumpfile" value="#[dumpfile]#" size="80" maxlength="250" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
-        </dd>
-        <dt>&nbsp;</dt>
-        <dd><input type="submit" name="indexrestore" value="Restore Dump" class="btn btn-primary" style="width:240px;" #(dumpRestoreEnabled)#disabled="disabled"::#(/dumpRestoreEnabled)#/>
-        </dd>
-      </dl>
-    </fieldset>
-    </form>::
-    #(/dumprestore)#
-    
-    #(indexdump)#::
-    <div class="alert alert-success" role="alert">Stored a solr dump to file #[dumpfile]#</div>::
-    <div class="alert alert-danger" role="alert">Could not create the Solr dump : no embedded Solr is available.</div>::
-    <div class="alert alert-danger" role="alert">An error occurred while trying to create the Solr dump.</div>
-    #(/indexdump)#
-    
-    #(indexRestore)#::
-    <div class="alert alert-success" role="alert">Successfully restored Solr index from dump file!</div>::
-    <div class="alert alert-danger" role="alert">Could not restore the Solr dump : no embedded Solr is available.</div>::
-    <div class="alert alert-danger" role="alert">An error occurred while trying to restore the Solr dump.</div>
-    #(/indexRestore)#
-    
     #%env/templates/footer.template%#
   </body>
 </html>
diff --git a/htroot/IndexImportMediawiki_p.html b/htroot/IndexImportMediawiki_p.html
index 5af34631b..5d3e5f92c 100644
--- a/htroot/IndexImportMediawiki_p.html
+++ b/htroot/IndexImportMediawiki_p.html
@@ -80,11 +80,11 @@
 &lt;/surrogates&gt;
     </pre>
     </li>
-    <li>Each 10000 wiki records are combined in one output file which is written to /DATA/SURROGATES/in into a temporary file.</li>
+    <li>Each 10000 wiki records are combined in one output file which is written to /DATA/PACKS/load into a temporary file.</li>
     <li>When each of the generated output file is finished, it is renamed to a .xml file</li>
-    <li>Each time a xml surrogate file appears in /DATA/SURROGATES/in, the YaCy indexer fetches the file and indexes the record entries.</li>
-    <li>When a surrogate file is finished with indexing, it is moved to /DATA/SURROGATES/out</li>
-    <li>You can recycle processed surrogate files by moving them from /DATA/SURROGATES/out to /DATA/SURROGATES/in</li>
+    <li>Each time a xml pack file appears in /DATA/PACKS/load, the YaCy indexer fetches the file and indexes the record entries.</li>
+    <li>When a pack file is finished with indexing, it is moved to /DATA/PACKS/loaded</li>
+    <li>You can recycle processed pack files by moving them from /DATA/PACKS/loaded to /DATA/PACKS/load</li>
     </ul>
     <br />
     ::
diff --git a/htroot/IndexImportOAIPMH_p.html b/htroot/IndexImportOAIPMH_p.html
index 84c9da15f..a4f642db6 100644
--- a/htroot/IndexImportOAIPMH_p.html
+++ b/htroot/IndexImportOAIPMH_p.html
@@ -10,7 +10,7 @@
     #%env/templates/header.template%#

     #%env/templates/submenuIndexImport.template%#

     <h2>OAI-PMH Import</h2>

-    <p>Results from the import can be monitored in the <a href="CrawlResults.html?process=7">indexing results for surrogates</a></p>

+    <p>Results from the import can be monitored in the <a href="CrawlResults.html?process=7">indexing results for packs</a></p>

     

     <form action="IndexImportOAIPMH_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">

       <fieldset>

diff --git a/htroot/IndexPackGenerator_p.html b/htroot/IndexPackGenerator_p.html
new file mode 100644
index 000000000..ef5df6f07
--- /dev/null
+++ b/htroot/IndexPackGenerator_p.html
@@ -0,0 +1,137 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN" "DTD/xhtml1-transitional.dtd">
+<!-- This page is only XHTML 1.0 Transitional because target is being used in a links -->
+<html xmlns="http://www.w3.org/1999/xhtml">
+#(reload)#::<meta http-equiv="REFRESH" content="5; url=/IndexPackGenerator_p.html">#(/reload)#
+  <head>
+    <title>YaCy '#[clientname]#': URL Database Administration</title>
+    #%env/templates/metas.template%#
+  </head>
+  <body id="IndexControl">
+    #%env/templates/header.template%#
+    #%env/templates/submenuIndexImport.template%#
+
+    <h2>Index Pack Generator</h2>
+    <p>The local index currently contains #[ucount]# documents, only #[ucount200]# exportable with status code 200 - the remaining are error documents.</p>
+
+    #(lurlexport)#::
+    <form action="IndexPackGenerator_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
+    <fieldset><legend>Index Pack Generator</legend>
+      <dl>
+        <dt class="TableCellDark">Set a Category (this goes into the filename)</dt>
+        <dd>
+            <select name="category" id="category" class="form-control">
+            <option value="core">core - technical documentation, operation systems, computer hardware, open source and free software, manuals, protocol standards</option>
+            <option value="scroll" selected="selected">scroll - non-technical documents: knowledge, encyclopedia, linguistic corpora, dictionaries, translation memories, texts, non-fiction books, historical books</option>
+            <option value="codex">codex - non-technical standards: industry standards, laws, rules, compliance</option>
+            <option value="gem">gem - research, papers, university publications, science</option>
+            <option value="fiction">fiction - fictional documents: movies, stories, series, books (fiction, science-fiction)</option>
+            <option value="map">map - geological data, geolocation-data, earth/world information</option>
+            <option value="echo">echo – micro-content (tweets, toots, short headlines, SMS corpora), podcasts, radio archives, audio lectures, spoken-word datasets, logs, incidents, telemetry</option>
+            <option value="spirit">spirit – related to non-textual data (possibly only metadata): art, music, game assets, creative-commons media (non-text culture loot)</option>
+            <option value="vault">vault - sensitive data: secrets, leaks, non-public documents, security advisories</option>
+            </select>
+        </dd>
+
+        <dt class="TableCellDark">Index Collection</dt>
+        <dd>
+            <select id="collection" name="collection" class="form-control">
+              #{collections}#
+                <option #(selected)#::selected="selected"#(/selected)#>#[collection]#</option>
+              #{/collections}#
+            </select>&nbsp; the collection name is used as part of the filename to describe the content. Exception: if the collection is "user", then you can name the content with a slug.
+        </dd>
+
+        <dt class="TableCellDark">Slug - describe the content<br>(only if collection is "user")</dt>
+        <dd><input type="text" name="slug" id="slug" value="export" size="20" maxlength="40" class="form-control"/>&nbsp;This will become a part of the filename, spaces will be replaced by "-"; must not be empty; should end with a language description, e.g. "-en"
+        </dd>
+
+
+        <script>
+            const collectionSelect = document.getElementById('collection');
+            const slugInput        = document.getElementById('slug');
+
+            function toggleSlug() {
+                const enable = collectionSelect.value === 'user';
+                slugInput.disabled = !enable;
+                slugInput.style.opacity = enable ? 1 : 0.5;    // optional visual hint
+            }
+
+            // run once on load
+            toggleSlug();
+
+            // keep it in sync
+            collectionSelect.addEventListener('change', toggleSlug);
+        </script>
+
+        <dt class="TableCellDark">URL Filter</dt>
+        <dd><input type="text" name="exportfilter" value=".*.*" size="20" maxlength="250" class="form-control"/>&nbsp;.*.* (default) is a catch-all; format: java regex
+        </dd>
+
+        <dt class="TableCellDark">Search Query - </dt>
+        <dd><input type="text" name="exportquery" value="*:*" size="20" maxlength="250"  class="form-control"/>&nbsp;*:* (default) is a catch-all; format: <field-name>:<solr-pattern>
+        </dd>
+        <dt class="TableCellDark">Export Format</dt>
+        <dd><span class="info" style="float:right"><img src="env/grafics/i16.gif" width="16" height="16" alt="info"/><span style="right:0px;">
+            This JSON is an elasticsearch index dump format and can be bulk-imported to elasticsearch. Here is an example for opensearch, using docker:<br />
+            Start docker container of opensearch:<br />
+            <code>docker run --name opensearch -p 9200:9200 -d -e OPENSEARCH_JAVA_OPTS="-Xms2G -Xmx2G" -e discovery.type=single-node -e DISABLE_SECURITY_PLUGIN=true -v $(pwd)/opensearch_data:/usr/share/opensearch/data opensearchproject/opensearch:latest</code><br />
+            Unblock index creation:<br />
+            <code>curl -X PUT "http://localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d'
+            {
+              "persistent": {
+                "cluster.blocks.create_index": null
+              }
+            }'</code><br />
+            Create the search index:<br />
+            <code>curl -X PUT "http://localhost:9200/collection1/yacy"</code><br />
+            Bulk-upload the index file:<br />
+            <code>curl -XPOST "http://localhost:9200/collection1/yacy/_bulk?filter_path=took,errors" -H "Content-Type: application/x-ndjson" --data-binary @yacy_dump_XXX.flatjson</code><br />
+            Make a search, get 10 results, search in fields text_t, title, description with boosts:<br />
+            <code>curl -X POST "http://localhost:9200/collection1/yacy/_search" -H 'Content-Type: application/json' -d'
+            {"size": 10, "query": {"multi_match": {
+                "query": "one two three",
+                "fields": ["text_t", "title^10", "description^3"], "fuzziness": "AUTO"
+            }}}'</code>
+        </span></span>
+            <input type="radio" name="format" value="full-elasticsearch" checked="checked" />
+            JSON (Rich and full-text Elasticsearch data, one document per line in one flat JSON file)
+            <br />
+
+            <input type="radio" name="format" value="full-solr" />
+            XML (Rich and full-text Solr data, one document per line in one large xml file,
+            can be processed with shell tools, can be imported with DATA/PACKS/load/)
+            <br />
+            <input type="radio" name="format" value="full-rss" />
+            XML (RSS)
+        </dd>
+
+        <dt>&nbsp;</dt>
+        <dd><input type="submit" name="lurlexport" value="Generate Data Pack" class="btn btn-primary" style="width:240px;"/>
+        </dd>
+      </dl>
+    </fieldset>
+    </form>::
+    <div class="alert alert-info" style="text-decoration:blink">Export to file #[exportfile]# is running ..  #[urlcount]# Documents so far</div>::
+    #(/lurlexport)#
+
+    #(lurlexportfinished)#::
+    <div class="alert alert-success">Finished export of #[urlcount]# Documents to file <a href="file://#[exportfile]#" target="_">#[exportfile]#</a><br/>
+    <em>Import this file by moving it to DATA/PACKS/load</em></div>::
+    #(/lurlexportfinished)#
+
+    #(lurlexporterror)#::
+    <div class="alert alert-warning">Export to file #[exportfile]# failed: #[exportfailmsg]#</div>::
+    #(/lurlexporterror)#
+
+    <fieldset><legend>Pack List</legend>
+        <table border="0" summary="Pack List Archive">
+            <tr class="TableHeader"><td>Pack</td><td>Process</td><td>Size (KB)</td></tr>
+        #{packs}#
+            <tr class="TableCell#(dark)#Light::Dark#(/dark)#"><td>#[file]#</td><td>#[type]#</td><td>#[size]#</td></tr>
+        #{/packs}#
+        </table>
+    </fieldset>
+
+    #%env/templates/footer.template%#
+  </body>
+</html>
diff --git a/htroot/env/base.css b/htroot/env/base.css
index 0a99d779d..a82e0b998 100644
--- a/htroot/env/base.css
+++ b/htroot/env/base.css
@@ -150,7 +150,6 @@ tt, *.tt {
 }
 
 .info {
-  font-weight:bold;
   color:olive;
 }
 
@@ -1064,8 +1063,9 @@ div#info:hover span {
   display: block;
   position: absolute;
   font-size:1em;
-  width: 200px;
+  width: 600px;
   padding: 3px;
+  margin-right: 10px;
   color: #000000;
   background: #DDDDDD;
   text-align: center;
@@ -1086,8 +1086,9 @@ div#tagcloud {
   display: block;
   position: absolute;
   font-size:1em;
-  width: 200px;
+  width: 600px;
   padding: 3px;
+  margin-right: 10px;
   color: #000000;
   background: #DDDDDD;
   text-align: left;
diff --git a/htroot/env/templates/header.template b/htroot/env/templates/header.template
index c5267f2db..b42d9a2c2 100644
--- a/htroot/env/templates/header.template
+++ b/htroot/env/templates/header.template
@@ -202,7 +202,7 @@
     <ul class="nav nav-sidebar menugroup">
       <li><h3>Production</h3></li>
       <li><a href="CrawlStartExpert.html" class="MenuItemLink">Advanced Crawler</a></li>
-      <li><a href="IndexExport_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Index Export/Import</a></li>
+      <li><a href="IndexPackGenerator_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Index Export/Import</a></li>
       <li><a href="Vocabulary_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Content Semantic</a></li>
       <li><a href="CrawlCheck_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Target Analysis</a></li>      
     </ul>
diff --git a/htroot/env/templates/submenuCrawlMonitor.template b/htroot/env/templates/submenuCrawlMonitor.template
index 4bbcff639..6da2ac3e1 100644
--- a/htroot/env/templates/submenuCrawlMonitor.template
+++ b/htroot/env/templates/submenuCrawlMonitor.template
@@ -45,7 +45,7 @@
         <li><a href="CrawlResults.html?process=4" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">(4) Proxy Use</a></li>
         <li><a href="CrawlResults.html?process=5" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">(5) Local Crawling</a></li>
         #(navigation-p2p)#::<li><a href="CrawlResults.html?process=6" class="MenuItemLink">(6) Global Crawling</a></li>#(/navigation-p2p)#
-        <li><a href="CrawlResults.html?process=7" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">(7) Surrogate Import</a></li>
+        <li><a href="CrawlResults.html?process=7" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">(7) Pack Import</a></li>
       </ul>
   </div>
 </div>
\ No newline at end of file
diff --git a/htroot/env/templates/submenuIndexImport.template b/htroot/env/templates/submenuIndexImport.template
index ccbc49412..8df07e446 100644
--- a/htroot/env/templates/submenuIndexImport.template
+++ b/htroot/env/templates/submenuIndexImport.template
@@ -3,27 +3,34 @@
 </div>
 <div class="SubMenu">
   <div class="SubMenugroup">
+    <h3>Index Packs</h3>
+    <ul class="SubMenu">
+      <li><a href="IndexPackGenerator_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Index Pack Generator</a></li>
+    </ul>
+  </div>
+  <div class="SubMenugroup">
     <h3>Export</h3>
     <ul class="SubMenu">
-      <li><a href="IndexExport_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Internal Index Export</a></li>
+      <li><a href="IndexExport_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Index Export</a></li>
+      <li><a href="IndexExportImportSolr_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Solr Dump Export/Import</a></li>
     </ul>
   </div>
   <div class="SubMenugroup">
     <h3>Import</h3>
     <ul class="SubMenu">
-      <li><a href="Load_RSS_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">RSS Feed Importer</a></li>
-      <li><a href="IndexImportOAIPMH_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">OAI-PMH Importer</a></li>
-      <li><a href="IndexImportWarc_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Warc Importer</a></li>
-      <li><a href="IndexImportZim_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Zim Importer</a></li>
-      <li><a href="IndexImportJsonList_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">JsonList Importer</a></li>
+      <li><a href="Load_RSS_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">RSS</a></li>
+      <li><a href="IndexImportOAIPMH_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">OAI-PMH</a></li>
+      <li><a href="IndexImportWarc_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">WARC</a></li>
+      <li><a href="IndexImportZim_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">ZIM</a></li>
+      <li><a href="IndexImportJsonList_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">JsonList</a></li>
     </ul>
   </div>
   
   <div class="SubMenugroup">
     <h3>Database Reader</h3>
     <ul class="SubMenu">
-      <li><a href="ContentIntegrationPHPBB3_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Database Reader for phpBB3 Forums</a></li>
-      <li><a href="IndexImportMediawiki_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">Dump Reader for MediaWiki dumps</a></li>
+      <li><a href="ContentIntegrationPHPBB3_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">phpBB3 Database</a></li>
+      <li><a href="IndexImportMediawiki_p.html" class="MenuItemLink #(authorized)#lock::unlock#(/authorized)#">MediaWiki Dump</a></li>
     </ul>
   </div>
 </div>
\ No newline at end of file