blob: 11ee431dd5f3754c5606727dcc5a0bd2c7afeefb (
plain)
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
|
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<title>YaCy '#[clientname]#': Crawl Results</title>
#%env/templates/metas.template%#
</head>
<body id="CrawlResults">
#%env/templates/header.template%#
#%env/templates/submenuCrawlMonitor.template%#
#(process)#
<h2>Crawl Results Overview</h2>
<p>These are monitoring pages for the different indexing queues.</p>
<p>YaCy knows 5 different ways to acquire web indexes. The details of these processes (1-5) are described within the submenu's listed
above which also will show you a table with indexing results so far. The information in these tables is considered as private,
so you need to log-in with your administration password.</p>
<p>Case (6) is a monitor of the local receipt-generator, the opposed case of (1). It contains also an indexing result monitor but is not considered private
since it shows crawl requests from other peers.
</p>
<p>Case (7) occurs if pack files are imported</p>
<p><img src="env/grafics/indexmonitor.png" width="600" height="308" alt="An illustration how yacy works" /></p>
<p>The image above illustrates the data flow initiated by web index acquisition.
Some processes occur double to document the complex index migration structure.
</p>
::
<h2>(1) Results of Remote Crawl Receipts</h2>
<p>This is the list of web pages that this peer initiated to crawl,
but had been crawled by <em>other</em> peers.
This is the 'mirror'-case of process (6).
</p>
<p><em>Use Case:</em> You get entries here, if you start a local crawl on the '<a href="CrawlStartExpert.html">Advanced Crawler</a>' page and check the
'Do Remote Indexing'-flag, and if you checked the 'Accept Remote Crawl Requests'-flag on the '<a href="RemoteCrawl_p.html">Remote Crawling</a>' page.
</p>
<p>Every page that a remote peer indexes upon this peer's request is reported back and can be monitored here.</p>
#(remoteCrawlerDisabled)#::<div class="info"><p>No remote crawl results can currently been added to the local index as the remote crawler is disabled on this peer.<p></div>#(/remoteCrawlerDisabled)#
::
<h2>(2) Results for Result of Search Queries</h2>
<p>This index transfer was initiated by your peer by doing a search query.
The index was crawled and contributed by other peers.</p>
<p><em>Use Case:</em> This list fills up if you do a search query on the 'Search Page'</p>
::
<h2>(3) Results for Index Transfer</h2>
<p>The url fetch was initiated and executed by other peers.
These links here have been transmitted to you because your peer is the most appropriate for storage according to
the logic of the Global Distributed Hash Table.</p>
<p><em>Use Case:</em> This list may fill if you check the 'Index Receive'-flag on the 'Index Control' page</p>
::
<h2>(4) Results for Proxy Indexing</h2>
<p>These web pages had been indexed as result of your proxy usage.
<strong>No personal or protected page is indexed</strong>;
such pages are detected by Cookie-Use or POST-Parameters (either in URL or as HTTP protocol)
and automatically excluded from indexing.</p>
<p><em>Use Case:</em> You must use YaCy as proxy to fill up this table.
Set the proxy settings of your browser to the same port as given
on the 'Settings'-page in the 'Proxy and Administration Port' field.</p>
::
<h2>(5) Results for Local Crawling</h2>
<p>These web pages had been crawled by your own crawl task.</p>
<p><em>Use Case:</em> start a crawl by setting a crawl start point on the 'Index Create' page.</p>
::
<h2>(6) Results for Global Crawling</h2>
<p>These pages had been indexed by your peer, but the crawl was initiated by a remote peer.
This is the 'mirror'-case of process (1).</p>
<p><em>Use Case:</em> This list may fill if you check the 'Accept Remote Crawl Requests'-flag on the '<a href="RemoteCrawl_p.html">Remote Crawling</a>' page</p>
#(remoteCrawlerDisabled)#::<div class="info"><p>The remote crawler is currently disabled<p></div>#(/remoteCrawlerDisabled)#
::
<h2>(7) Results from pack import</h2>
<p>These records had been imported from pack files in DATA/PACKS/load</p>
<p><em>Use Case:</em> place files with dublin core metadata content into DATA/PACKS/load or use an index import method (i.e. <a href="IndexImportMediawiki_p.html">MediaWiki import</a>, <a href="IndexImportOAIPMH_p.html">OAI-PMH retrieval</a>)</p>
#(/process)#
#(table)#
<p><em>The stack is empty.</em></p>
::
<p><em>Statistics about #[domains]# domains in this stack:</em></p>
<table >
<tr class="TableHeader">
<td align="center"></td>
<td><strong>Domain</strong></td>
<td><strong>URLs</strong></td>
<td>Blacklist to use
<form name="selectblacklistform" action="#[feedbackpage]#">
<select name="selectedblacklist" style="color:black" onchange="forms.selectblacklistform.submit();">
#{blacklists}#
<option #[selected]# value="#[name]#">#[name]#</option>
#{/blacklists}#
</select>
<input type="hidden" name="process" value="#[tabletype]#" />
</form>
</td>
</tr>
#{domains}#
<tr class="TableCell#(dark)#Light::Dark#(/dark)#">
<td align="center">
<form action="#[feedbackpage]#" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
<div>
<input type="hidden" name="process" value="#[tabletype]#" />
<input type="hidden" name="domain" value="#[domain]#" />
<input type="hidden" name="blacklistname" value="#[blacklistname]#" />
<input type="submit" name="deletedomain" value="delete all" class="btn btn-danger btn-xs"/>
</div>
</form>
</td>
<td><a href="http://#[domain]#/" target="_blank">#[domain]#</a></td>
<td>#[count]#</td>
<td>
<form action="#[feedbackpage]#" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
<div align="center">
<input type="hidden" name="process" value="#[tabletype]#" />
<input type="hidden" name="domain" value="#[domain]#" />
<input type="hidden" name="blacklistname" value="#[blacklistname]#" />
<input type="submit" name="delandaddtoblacklist" value="del & blacklist" class="btn btn-danger btn-xs"/>
</div>
</form>
</td>
</tr>
#{/domains}#
</table><br />
<p><em>
#(size)#
Showing all #[all]# entries in this stack.
::
Showing latest #[count]# lines from a stack of #[all]# entries. <a href="CrawlResults.html?process=#[tabletype]#&count=2147483647">Show all</a>
#(/size)#
</em></p>
<table >
<tr class="TableHeader">
<td align="center">
<form action="#[feedbackpage]#" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
<div>
<input type="hidden" name="process" value="#[tabletype]#" />
<input type="submit" name="clearlist" class="btn btn-default btn-xs" value="clear list" />
</div>
</form>
</td>
#(showCollection)#::<td><strong>Collection</strong></td>#(/showCollection)#
#(showInit)#::<td><strong>Initiator</strong></td>#(/showInit)#
#(showExec)#::<td><strong>Executor</strong></td>#(/showExec)#
#(showDate)#::<td><strong>Modified</strong></td>#(/showDate)#
#(showWords)#::<td><strong>Words</strong></td>#(/showWords)#
#(showTitle)#::<td><strong>Title</strong></td>#(/showTitle)#
#(showCountry)#::<td><strong>Country</strong></td>#(/showCountry)#
#(showIP)#::<td><strong>IP of Host</strong></td>#(/showIP)#
#(showURL)#::<td><strong>URL</strong></td>#(/showURL)#
</tr>
#{indexed}#
<tr class="TableCell#(dark)#Light::Dark#(/dark)#">
<td align="center">
<form action="#[feedbackpage]#" method="post" enctype="multipart/form-data" accept-charset="UTF-8">
<div>
<input type="hidden" name="process" value="#[tabletype]#" />
<input type="hidden" name="hash" value="#[urlhash]#" />
<input type="submit" name="deleteentry" value="delete" class="btn btn-danger btn-xs"/>
</div>
</form>
</td>
#(showCollection)#::<td>#[collection]#</td>#(/showCollection)#
#(showInit)#::<td>#[initiatorSeed]#</td>#(/showInit)#
#(showExec)#::<td>#[executorSeed]#</td>#(/showExec)#
#(showDate)#::<td>#[modified]#</td>#(/showDate)#
#(showWords)#::<td>#[count]#</td>#(/showWords)#
#(showTitle)#
::
<td>
#(available)#
<span class="tt">-not cached-</span>
::
<a href="ViewFile.html?action=info&urlHash=#[urlHash]#" class="small" title="#[urltitle]#">#(nodescr)#no title::#[urldescr]##(/nodescr)#</a>
#(/available)#
</td>
#(/showTitle)#
#(showCountry)#::<td>#[country]#</td>#(/showCountry)#
#(showIP)#::<td>#[ip]#</td>#(/showIP)#
#(showURL)#
::
<td>
#(available)#
<span class="tt">-not cached-</span>
::
<a href="ViewFile.html?action=info&urlHash=#[urlHash]#" class="small" title="#[urltitle]#">#[url]#</a>
#(/available)#
</td>
#(/showURL)#
</tr>
#{/indexed}#
</table>
::
#(/table)#
#%env/templates/footer.template%#
</body>
</html>
|
