preparations for mass remote crawls:

two main changes must be implemented to enable mass remote crawls:
- shift control of robots.txt to crawl queue (away from stacker). This is necessary since remote
  crawls can contain unchecked urls. Each peer must check the robots to prevent that it is misused
  as crawl agent for unwanted file retrieval
- implement new index files that control double-check of remotely crawled urls

After removal of robots.txt checking from stacker threads, the multi-threading of this process is void.
Multithreading has been removed. Also the thread pools for the crawl threads had been removed, since
creation of these threads is not resource-consuming, for a detailed explanation see svn 4106

git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4181 6c8d7289-2bf4-0310-a012-ef5d649a1542

1 files changed, 4 insertions, 2 deletions

diff --git a/htroot/IndexCreateLoaderQueue_p.html b/htroot/IndexCreateLoaderQueue_p.html
index ba8727f4a..c68116386 100644
--- a/htroot/IndexCreateLoaderQueue_p.html
+++ b/htroot/IndexCreateLoaderQueue_p.html
@@ -22,13 +22,15 @@
       </colgroup>

       <tr class="TableHeader">

         <th>Initiator</th>

-        <th>Depth</th>

+        <th>Depth</th>
+        <th>Status</th>

         <th>URL</th>

       </tr>

       #{list}#

       <tr class="TableCell#(dark)#Light::Dark#(/dark)#">

         <td>#[initiator]#</td>

-        <td>#[depth]#</td>

+        <td>#[depth]#</td>
+        <td>#[status]#</td>

         <td><a href="#[url]#">#[url]#</a></td>

       </tr>

       #{/list}#