htroot/IndexFederated_p.html


1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137

<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
  <head>
    <title>YaCy '#[clientname]#': Index Sources &amp; Targets</title>
    #%env/templates/metas.template%#
  </head>
  <body id="IndexFederated_p">
    #%env/templates/header.template%#
    #%env/templates/submenuIndexControl.template%#
    <h2>Index Sources &amp; Targets</h2>
    <p>
      YaCy supports multiple index storage locations.
      As an internal indexing database a deep-embedded multi-core Solr is used and it is possible to attach also a remote Solr.
    </p>
    
    <form id="config" action="IndexFederated_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
      <input type="hidden" name="transactionToken" value="#[transactionToken]#" />
      <fieldset>
      <legend>Solr Search Index</legend>
      Solr stores the main search index. It is the home of two cores, the default 'collection1' core for documents and the 'webgraph' core for a web structure graph. Detailed information about the used Solr fields can be edited in the <a href="IndexSchema_p.html">Schema Editor</a>.
      <dl>
      
      <dt class="TableCellDark">Lazy Value Initialization&nbsp;<input type="checkbox" name="solr.indexing.lazy" id="solr.indexing.lazy" #(solr.indexing.lazy.checked)#:: checked="checked"#(/solr.indexing.lazy.checked)# /></dt>
      <dd>If checked, only non-zero values and non-empty strings are written to Solr fields.</dd>
        
      <dt>&nbsp;</dt><dd>&nbsp;</dd>
      <dt>Use deep-embedded local Solr&nbsp;<input type="checkbox" name="core.service.fulltext" id="core_service_fulltext" #(core.service.fulltext.checked)#:: checked="checked"#(/core.service.fulltext.checked)# onclick="if(!document.getElementById('config').core_service_fulltext.checked) {document.getElementById('config').solr_indexing_solrremote.checked = true;}"/></dt>
      <dd>
      This will write the YaCy-embedded Solr index which is stored within the YaCy DATA directory.<br/>
      The Solr native search interface is accessible at<br/>
      <a href="solr/select?q=*:*&start=0&rows=3&core=collection1">/solr/select?q=*:*&amp;start=0&amp;rows=3&amp;core=collection1</a>
      for the default search index (core: collection1) and at<br/>
      <a href="solr/select?q=*:*&start=0&rows=3&core=webgraph">/solr/select?q=*:*&amp;start=0&amp;rows=3&amp;core=webgraph</a> for the webgraph core.<br/>
      If you switch off this index, a remote Solr must be activated.</dd>
      
      <dt>Use remote Solr server(s)&nbsp;<input type="checkbox" name="solr.indexing.solrremote" id="solr_indexing_solrremote" #(solr.indexing.solrremote.checked)#:: checked="checked"#(/solr.indexing.solrremote.checked)# onclick="if(!document.getElementById('config').solr_indexing_solrremote.checked) {document.getElementById('config').core_service_fulltext.checked = true;}"/></dt>
      <dd>It's easy to <a href="https://wiki.yacy.net/index.php/Dev:Solr" target="_blank">attach an external Solr to YaCy</a>.
      This external Solr can be used instead of the internal Solr. It can also be used additionally to the internal Solr, then both Solr indexes are mirrored. 
      </dd>
      
      <dt>Allow self-signed certificates <input type="checkbox" name="federated.service.solr.indexing.authenticated.allowSelfSigned" #(federated.service.solr.indexing.authenticated.allowSelfSigned.checked)#:: checked="checked"#(/federated.service.solr.indexing.authenticated.allowSelfSigned.checked)#/></dt>
      <dd>Tick this when the remote Solr server is password protected and is requested over HTTPS but provides only a self-signed certificate (not a validated one by an official Certificate Authority). The Solr URL could be for example something like <i>https://user:password@localhost:8984/solr</i>.
      </dd>	
      
      #(table)#::
        <dt class="TableCellDark">&nbsp;</dt>
        <dd>Solr Hosts<br/>
      <div>
      <table class="sortable" border="0">
      <tr class="TableHeader" valign="bottom">
        <td><strong>Solr Host Administration Interface</strong><br/></td>
        <td><strong>Index Size</strong></td>
      </tr>
      #{list}#
      <tr class="TableCell#(dark)#Light::Dark::Summary#(/dark)#">
        <td><a href="#[url]#" target="_blank">#[url]#</a></td>
        <td align="right">#[size]#</td>
      </tr>
      #{/list}#
      </table>
      </div>
        </dd>
      #(/table)#
      
      <dt class="TableCellDark"></dt>
      <dd>Solr URL(s)<br/><textarea rows="2" cols="80" name="solr.indexing.url" id="solr.indexing.url"/>#[solr.indexing.url]#</textarea><br/>
      You can set one or more Solr targets here which are accessed as a shard. For several targets, list them using a ',' (comma) as separator.
      The set of remote targets are used as shards of a complete index. The host part of the url is used as key for a hash function which selects one of the shards (one of your remote servers).
      When a search request is made, all servers are accessed synchronously and the result is combined.</dd>
        
      <dt class="TableCellDark"></dt>
      <dd>Sharding Method<br/>
      <select id="solr.indexing.sharding" name="solr.indexing.sharding">
          #{solr.indexing.sharding.methods}#
            <option value="#[method]#" #(selected)#::selected="selected"#(/selected)#>#[method]#: #[description]#</option>
          #{/solr.indexing.sharding.methods}#
      </select></dd>

      <dt class="TableCellDark"></dt>
      <dd><input type="checkbox" name="solr.indexing.solrremote.writeenabled" id="solr_indexing_solrremote_writeenabled" #(solr.indexing.solrremote.writeenabled.checked)#:: checked="checked"#(/solr.indexing.solrremote.writeenabled.checked)#/> write-enabled (if unchecked, the remote server(s) will only be used as search peers)</dd>

      <dt><input type="submit" name="setsolr" class="btn btn-primary" value="Set" /></dt><dd></dd>

      </dd>
      </dl>
      </fieldset>
    </form>
    
    <form id="config" action="IndexFederated_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
	  <input type="hidden" name="transactionToken" value="#[transactionToken]#" />
      <fieldset>
      <legend>
        Web Structure Index
      </legend>
      The web structure index is used for host browsing (to discover the internal file/folder structure), ranking (counting the number of references) and file search (there are about forty times more links from loaded pages than in documents of the main search index). 
      <dl>
      
      <dt><input type="checkbox" name="core.service.citation.tmp" id="core_service_citation" #(core.service.citation.tmp.checked)#:: checked="checked"#(/core.service.citation.tmp.checked)# /></dt>
      <dd>use citation reference index (lightweight and fast)</dd>
      
      <dt><input type="checkbox" name="core.service.webgraph.tmp" id="core_service_webgraph" #(core.service.webgraph.tmp.checked)#:: checked="checked"#(/core.service.webgraph.tmp.checked)# /></dt>
      <dd>use webgraph search index (rich information in second Solr core)</dd>
      
      <dt><input type="submit" name="setcitation" class="btn btn-primary" value="Set" /></dt><dd></dd>
      </dl>
      </fieldset>
    </form>
      
    <form id="config" action="IndexFederated_p.html" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
      <input type="hidden" name="transactionToken" value="#[transactionToken]#" />
      <fieldset>
      <legend>
        Peer-to-Peer Operation
      </legend>
      The 'RWI' (Reverse Word Index) is necessary for index transmission in distributed mode. For portal or intranet mode this must be switched off.
      <dl>
      
      <dt><input type="checkbox" name="core.service.rwi" id="core_service_rwi" #(core.service.rwi.checked)#:: checked="checked"#(/core.service.rwi.checked)# /></dt>
      <dd>support peer-to-peer index transmission (DHT RWI index)</dd>

      <dt class="TableCellDark">Block known error URLs in DHT&nbsp;<input type="checkbox" name="indexReceiveBlockErrors" #(indexReceiveBlockErrors.checked)#:: checked="checked"#(/indexReceiveBlockErrors.checked)# /></dt>
      <dd>Reject URLs/RWIs with known errors from peers. Disable to opt out.</dd>

      <dt class="TableCellDark">Retry after (days)</dt>
      <dd><input type="text" name="indexReceiveBlockErrors.retryAfterDays" value="#[indexReceiveBlockErrors.retryAfterDays]#" size="4" /> for temporary errors; permanent errors stay blocked.</dd>

      <dt class="TableCellDark">Permanent error statuses</dt>
      <dd><input type="text" name="indexReceiveBlockErrors.permanentStatus" value="#[indexReceiveBlockErrors.permanentStatus]#" size="15" /> comma-separated (default: 404,410,-1; -1=DNS/network errors)</dd>
      
      <dt><input type="submit" name="setrwi" class="btn btn-primary" value="Set" /></dt><dd></dd>
      </dl>
      </fieldset>
    </form>
    
    #%env/templates/footer.template%#
  </body>
</html>