diff options
| author | orbiter <orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542> | 2007-09-28 01:21:31 +0000 |
|---|---|---|
| committer | orbiter <orbiter@6c8d7289-2bf4-0310-a012-ef5d649a1542> | 2007-09-28 01:21:31 +0000 |
| commit | 842308ea979f5841e2b5ae65e6fb08d0bea07fc7 (patch) | |
| tree | a898376a5c0df4caf594a96fb997c77103eb6cab /htroot | |
| parent | 341f7cb327899d093986fa7bccff5c051f2d916d (diff) | |
- redesigned crawl start menu, integrated monitoring pages
- removed web structure picture from indexing menu and grouped it together with htcache monitor
- added a database for terminated crawls, when a crawl is finished it is automatically moved to the new database
- extended crawl profile edit servlet, shows now also terminated crawls
- option that was used to delete profiles is now redesigned to a function that moves the current crawl to the terminated crawls and removes all urls from the current queues!
- fixed here and there problems with indexing queues
- enhances indexing speed by changing cache flush sizes.
- changed behaviour of crawl result servlet: the list of crawled urls is shown if there is one, othevise the overview window is shown
attention: the new profile databases are not compatible with the old one. current crawls will be lost! the web index is not touched.
next steps: the database of terminated crawls can be used to start with them a new crawl. This is useful if one wants to re-crawl specific pages and wants to use a old crawl profile.
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@4113 6c8d7289-2bf4-0310-a012-ef5d649a1542
Diffstat (limited to 'htroot')
24 files changed, 190 insertions, 166 deletions
diff --git a/htroot/CacheAdmin_p.html b/htroot/CacheAdmin_p.html index c04c5f092..0f6e916ab 100644 --- a/htroot/CacheAdmin_p.html +++ b/htroot/CacheAdmin_p.html @@ -5,7 +5,8 @@ #%env/templates/metas.template%#
</head>
<body id="CacheAdmin">
- #%env/templates/header.template%#
+ #%env/templates/header.template%# + #%env/templates/submenuWebStructure.template%#
<h2>Web Cache</h2>
<p>The current cache size is #[cachesize]# KB. The maximum cache size is #[cachemax]# KB.</p>
<p><a
diff --git a/htroot/CrawlProfileEditor_p.html b/htroot/CrawlProfileEditor_p.html index 1eb02bc80..cef1768de 100644 --- a/htroot/CrawlProfileEditor_p.html +++ b/htroot/CrawlProfileEditor_p.html @@ -29,6 +29,7 @@ </colgroup> <tr class="TableHeader"> <td><strong>Crawl Thread</strong></td> + <td><strong>Status</strong></td> <td><strong>Start URL</strong></td> <td><strong>Depth</strong></td> <td><strong>Filter</strong></td> @@ -46,6 +47,7 @@ #{crawlProfiles}# <tr class="TableCell#(dark)#Light::Dark#(/dark)#"> <td>#[name]#</td> + <td>#(status)#terminated::active#(/status)#</td> <td><a href="#[startURL]#">#[startURL]#</a></td> <td>#[depth]#</td> <td>#[filter]#</td> @@ -58,12 +60,18 @@ <td>#(indexText)#no::yes#(/indexText)#</td> <td>#(indexMedia)#no::yes#(/indexMedia)#</td> <td>#(remoteIndexing)#no::yes#(/remoteIndexing)#</td> - <td>#(deleteButton)#:: - <form action="CrawlProfileEditor_p.html" method="get" enctype="multipart/form-data"> - <pre><input type="hidden" name="handle" value="#[handle]#" /></pre> - <pre><input type="submit" name="deleteprofile" value="Delete" /></pre> - #(/deleteButton)# - </form> + <td>#(terminateButton)#:: + <form action="CrawlProfileEditor_p.html" method="get" enctype="multipart/form-data"> + <input type="hidden" name="handle" value="#[handle]#" /> + <input type="submit" name="terminate" value="Terminate" /> + </form> + #(/terminateButton)# + #(deleteButton)#:: + <form action="CrawlProfileEditor_p.html" method="get" enctype="multipart/form-data"> + <input type="hidden" name="handle" value="#[handle]#" /> + <input type="submit" name="delete" value="Delete" /> + </form> + #(/deleteButton)# </td> </tr> #{/crawlProfiles}# diff --git a/htroot/CrawlProfileEditor_p.java b/htroot/CrawlProfileEditor_p.java index ab1eb894b..2e8f26464 100644 --- a/htroot/CrawlProfileEditor_p.java +++ b/htroot/CrawlProfileEditor_p.java @@ -1,11 +1,15 @@ // CrawlProfileEditor_p.java -// ------------------------------- -// part of the AnomicHTTPD caching proxy -// (C) by Michael Peter Christen; mc@anomic.de -// first published on http://www.anomic.de -// Frankfurt, Germany, 2004, 2005 -// last major change: 04.07.2005 +// (C) 2005, by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany +// first published 04.07.2005 on http://yacy.net // +// This is a part of YaCy, a peer-to-peer based web search engine +// +// $LastChangedDate: 2006-04-02 22:40:07 +0200 (So, 02 Apr 2006) $ +// $LastChangedRevision: 1986 $ +// $LastChangedBy: orbiter $ +// +// LICENSE +// // This program is free software; you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by // the Free Software Foundation; either version 2 of the License, or @@ -19,29 +23,6 @@ // You should have received a copy of the GNU General Public License // along with this program; if not, write to the Free Software // Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA -// -// Using this software in any meaning (reading, learning, copying, compiling, -// running) means that you agree that the Author(s) is (are) not responsible -// for cost, loss of data or any harm that may be caused directly or indirectly -// by usage of this softare or this documentation. The usage of this software -// is on your own risk. The installation and usage (starting/running) of this -// software may allow other people or application to access your computer and -// any attached devices and is highly dependent on the configuration of the -// software which must be done by the user of the software; the author(s) is -// (are) also not responsible for proper configuration and usage of the -// software, even if provoked by documentation provided together with -// the software. -// -// Any changes to this file according to the GPL as documented in the file -// gpl.txt aside this file in the shipment you received can be done to the -// lines that follows this copyright notice here, but changes must not be -// done inside the copyright notive above. A re-distribution must contain -// the intact and unchanged copyright notice. -// Contributions and changes to the program code must be marked as such. - -// You must compile this file with -// javac -classpath .:../classes CrawlProfileEditor_p.java -// if the shell's current path is HTROOT import java.io.IOException; import java.util.ArrayList; @@ -103,14 +84,23 @@ public class CrawlProfileEditor_p { // read post for handle String handle = (post == null) ? "" : post.get("handle", ""); - if ((post != null) && (post.containsKey("deleteprofile"))) { - // deletion of a crawl - sb.profiles.removeEntry(handle); + if (post != null) { + if (post.containsKey("terminate")) { + // termination of a crawl: shift the crawl from active to passive + sb.profilesPassiveCrawls.newEntry(sb.profilesActiveCrawls.getEntry(handle).map()); + sb.profilesActiveCrawls.removeEntry(handle); + // delete all entries from the crawl queue that are deleted here + sb.noticeURL.removeByProfileHandle(handle); + } + if (post.containsKey("delete")) { + // deletion of a terminated crawl profile + sb.profilesPassiveCrawls.removeEntry(handle); + } } // generate handle list int count = 0; - Iterator it = sb.profiles.profiles(true); + Iterator it = sb.profilesActiveCrawls.profiles(true); entry selentry; while (it.hasNext()) { selentry = (entry)it.next(); @@ -126,7 +116,7 @@ public class CrawlProfileEditor_p { count++; } prop.put("profiles", count); - selentry = sb.profiles.getEntry(handle); + selentry = sb.profilesActiveCrawls.getEntry(handle); // read post for change submit if ((post != null) && (selentry != null)) { @@ -138,7 +128,7 @@ public class CrawlProfileEditor_p { tee = (eentry) it.next(); String cval = (String) selentry.map().get(tee.name); String val = (tee.type == eentry.BOOLEAN) ? Boolean.toString(post.containsKey(tee.name)) : post.get(tee.name, cval); - if (!cval.equals(val)) sb.profiles.changeEntry(selentry, tee.name, val); + if (!cval.equals(val)) sb.profilesActiveCrawls.changeEntry(selentry, tee.name, val); } } catch (IOException ex) { prop.put("error", 1); @@ -149,47 +139,22 @@ public class CrawlProfileEditor_p { // generate crawl profile table count = 0; + boolean dark = true; int domlistlength = (post == null) ? 160 : post.getInt("domlistlength", 160); - it = sb.profiles.profiles(true); plasmaCrawlProfile.entry profile; - boolean dark = true; + // put active crawls into list + it = sb.profilesActiveCrawls.profiles(true); while (it.hasNext()) { profile = (plasmaCrawlProfile.entry) it.next(); - prop.put("crawlProfiles_"+count+"_dark", ((dark) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_name", profile.name()); - prop.put("crawlProfiles_"+count+"_startURL", profile.startURL()); - prop.put("crawlProfiles_"+count+"_handle", profile.handle()); - prop.put("crawlProfiles_"+count+"_depth", profile.generalDepth()); - prop.put("crawlProfiles_"+count+"_filter", profile.generalFilter()); - prop.put("crawlProfiles_"+count+"_crawlingIfOlder", (profile.recrawlIfOlder() == Long.MAX_VALUE) ? "no re-crawl" : ""+profile.recrawlIfOlder()); - prop.put("crawlProfiles_"+count+"_crawlingDomFilterDepth", (profile.domFilterDepth() == Integer.MAX_VALUE) ? "inactive" : Integer.toString(profile.domFilterDepth())); - - //start contrib [MN] - int i = 0; - String item; - while((i <= domlistlength) && !((item = profile.domName(true, i)).equals(""))){ - if(i == domlistlength){ - item = item + " ..."; - } - prop.put("crawlProfiles_"+count+"_crawlingDomFilterContent_"+i+"_item", item); - i++; - } - - prop.put("crawlProfiles_"+count+"_crawlingDomFilterContent", i); - //end contrib [MN] - - prop.put("crawlProfiles_"+count+"_crawlingDomMaxPages", (profile.domMaxPages() == Integer.MAX_VALUE) ? "unlimited" : ""+profile.domMaxPages()); - prop.put("crawlProfiles_"+count+"_withQuery", ((profile.crawlingQ()) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_storeCache", ((profile.storeHTCache()) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_indexText", ((profile.indexText()) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_indexMedia", ((profile.indexMedia()) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_remoteIndexing", ((profile.remoteIndexing()) ? 1 : 0)); - prop.put("crawlProfiles_"+count+"_deleteButton", (((profile.name().equals("remote")) || - (profile.name().equals("proxy")) || - (profile.name().equals("snippetText")) || - (profile.name().equals("snippetMedia")) ? 0 : 1))); - prop.put("crawlProfiles_"+count+"_deleteButton_handle", profile.handle()); - + putProfileEntry(prop, profile, true, dark, count, domlistlength); + dark = !dark; + count++; + } + // put passive crawls into list + it = sb.profilesPassiveCrawls.profiles(true); + while (it.hasNext()) { + profile = (plasmaCrawlProfile.entry) it.next(); + putProfileEntry(prop, profile, false, dark, count, domlistlength); dark = !dark; count++; } @@ -223,4 +188,44 @@ public class CrawlProfileEditor_p { return prop; } + + private static void putProfileEntry(servletProperties prop, plasmaCrawlProfile.entry profile, boolean active, boolean dark, int count, int domlistlength) { + prop.put("crawlProfiles_" + count + "_dark", ((dark) ? 1 : 0)); + prop.put("crawlProfiles_" + count + "_status", ((active) ? 1 : 0)); + prop.put("crawlProfiles_" + count + "_name", profile.name()); + prop.put("crawlProfiles_" + count + "_startURL", profile.startURL()); + prop.put("crawlProfiles_" + count + "_handle", profile.handle()); + prop.put("crawlProfiles_" + count + "_depth", profile.generalDepth()); + prop.put("crawlProfiles_" + count + "_filter", profile.generalFilter()); + prop.put("crawlProfiles_" + count + "_crawlingIfOlder", (profile.recrawlIfOlder() == Long.MAX_VALUE) ? "no re-crawl" : ""+profile.recrawlIfOlder()); + prop.put("crawlProfiles_" + count + "_crawlingDomFilterDepth", (profile.domFilterDepth() == Integer.MAX_VALUE) ? "inactive" : Integer.toString(profile.domFilterDepth())); + + // start contrib [MN] + int i = 0; + String item; + while ((i <= domlistlength) && !((item = profile.domName(true, i)).equals(""))){ + if(i == domlistlength){ + item = item + " ..."; + } + prop.put("crawlProfiles_"+count+"_crawlingDomFilterContent_"+i+"_item", item); + i++; + } + + prop.put("crawlProfiles_"+count+"_crawlingDomFilterContent", i); + // end contrib [MN] + + prop.put("crawlProfiles_" + count + "_crawlingDomMaxPages", (profile.domMaxPages() == Integer.MAX_VALUE) ? "unlimited" : ""+profile.domMaxPages()); + prop.put("crawlProfiles_" + count + "_withQuery", (profile.crawlingQ()) ? 1 : 0); + prop.put("crawlProfiles_" + count + "_storeCache", (profile.storeHTCache()) ? 1 : 0); + prop.put("crawlProfiles_" + count + "_indexText", (profile.indexText()) ? 1 : 0); + prop.put("crawlProfiles_" + count + "_indexMedia", (profile.indexMedia()) ? 1 : 0); + prop.put("crawlProfiles_" + count + "_remoteIndexing", (profile.remoteIndexing()) ? 1 : 0); + prop.put("crawlProfiles_" + count + "_terminateButton", ((!active) || (profile.name().equals("remote")) || + (profile.name().equals("proxy")) || + (profile.name().equals("snippetText")) || + (profile.name().equals("snippetMedia"))) ? 0 : 1); + prop.put("crawlProfiles_" + count + "_terminateButton_handle", profile.handle()); + prop.put("crawlProfiles_" + count + "_deleteButton", (active) ? 0 : 1); + prop.put("crawlProfiles_" + count + "_deleteButton_handle", profile.handle()); + } } diff --git a/htroot/CrawlResults.java b/htroot/CrawlResults.java index 7b992180e..a8e0a3e24 100644 --- a/htroot/CrawlResults.java +++ b/htroot/CrawlResults.java @@ -69,6 +69,11 @@ public class CrawlResults { tabletype = 0;
}
+ if ((post != null) && (post.containsKey("autoforward")) && (tabletype == 5) && (sb.wordIndex.loadedURL.getStackSize(5) == 0)) {
+ // the main menu does a request to the local crawler page, but in case this table is empty, the overview page is shown
+ tabletype = 0;
+ }
+
// check if authorization is needed and/or given
if (((tabletype > 0) && (tabletype < 6)) ||
(post.containsKey("clearlist")) ||
diff --git a/htroot/CrawlURLFetch_p.java b/htroot/CrawlURLFetch_p.java index f16e714ed..3abacae11 100644 --- a/htroot/CrawlURLFetch_p.java +++ b/htroot/CrawlURLFetch_p.java @@ -83,9 +83,9 @@ public class CrawlURLFetch_p { public static plasmaCrawlProfile.entry getCrawlProfile(serverSwitch env) { if (profile == null) { - profile = ((plasmaSwitchboard)env).profiles.newEntry( + profile = ((plasmaSwitchboard)env).profilesActiveCrawls.newEntry( "URLFetcher", // Name - "", // URL + null, // URL ".*", ".*", // General / specific filter 0, 0, // General / specific depth -1, -1, -1, // Recrawl / Dom-filter depth / Dom-max-pages diff --git a/htroot/IndexCreateIndexingQueue_p.html b/htroot/IndexCreateIndexingQueue_p.html index 2e467b9f1..c127527b0 100644 --- a/htroot/IndexCreateIndexingQueue_p.html +++ b/htroot/IndexCreateIndexingQueue_p.html @@ -6,7 +6,7 @@ </head>
<body id="IndexCreateIndexingQueue">
#%env/templates/header.template%#
- #%env/templates/submenuCrawler.template%#
+ #%env/templates/submenuIndexCreate.template%#
<h2>Indexing Queue</h2>
diff --git a/htroot/IndexCreateIndexingQueue_p.java b/htroot/IndexCreateIndexingQueue_p.java index a99f81890..970c69376 100644 --- a/htroot/IndexCreateIndexingQueue_p.java +++ b/htroot/IndexCreateIndexingQueue_p.java @@ -92,6 +92,7 @@ public class IndexCreateIndexingQueue_p { plasmaHTCache.deleteFile(entry.url());
}
}
+ switchboard.sbQueue.clear(); // reset file to clean up content completely
}
} catch (Exception e) {}
} else if (post.containsKey("deleteEntry")) {
diff --git a/htroot/IndexCreateLoaderQueue_p.html b/htroot/IndexCreateLoaderQueue_p.html index cd15f5c06..ba8727f4a 100644 --- a/htroot/IndexCreateLoaderQueue_p.html +++ b/htroot/IndexCreateLoaderQueue_p.html @@ -6,7 +6,7 @@ </head>
<body id="IndexCreateLoaderQueue">
#%env/templates/header.template%#
- #%env/templates/submenuCrawler.template%#
+ #%env/templates/submenuIndexCreate.template%#
<h2>Loader Queue</h2>
<p>
diff --git a/htroot/IndexCreateWWWGlobalQueue_p.html b/htroot/IndexCreateWWWGlobalQueue_p.html index d7234864b..b2c8d8dec 100644 --- a/htroot/IndexCreateWWWGlobalQueue_p.html +++ b/htroot/IndexCreateWWWGlobalQueue_p.html @@ -5,8 +5,8 @@ #%env/templates/metas.template%#
</head>
<body id="IndexCreateWWWGlobalQueue">
- #%env/templates/header.template%#
- #%env/templates/submenuCrawler.template%#
+ #%env/templates/header.template%# + #%env/templates/submenuIndexCreate.template%#
<h2>Global Crawl Queue</h2>
<p>
This queue stores the urls that shall be sent to other peers to perform a remote crawl.
diff --git a/htroot/IndexCreateWWWGlobalQueue_p.java b/htroot/IndexCreateWWWGlobalQueue_p.java index 1f067130b..d91a72ee4 100644 --- a/htroot/IndexCreateWWWGlobalQueue_p.java +++ b/htroot/IndexCreateWWWGlobalQueue_p.java @@ -94,7 +94,7 @@ public class IndexCreateWWWGlobalQueue_p { prop.put("info_numEntries", c);
} else if (post.containsKey("deleteEntry")) {
String urlHash = (String) post.get("deleteEntry");
- switchboard.noticeURL.remove(urlHash);
+ switchboard.noticeURL.removeByURLHash(urlHash);
prop.put("LOCATION","");
return prop;
}
@@ -118,7 +118,7 @@ public class IndexCreateWWWGlobalQueue_p { if ((urle != null)&&(urle.url()!=null)) {
initiator = yacyCore.seedDB.getConnected(urle.initiator());
profileHandle = urle.profileHandle();
- profileEntry = (profileHandle == null) ? null : switchboard.profiles.getEntry(profileHandle);
+ profileEntry = (profileHandle == null) ? null : switchboard.profilesActiveCrawls.getEntry(profileHandle);
prop.put("crawler-queue_list_"+showNum+"_dark", ((dark) ? 1 : 0) );
prop.put("crawler-queue_list_"+showNum+"_initiator", ((initiator == null) ? "proxy" : htmlTools.encodeUnicode2html(initiator.getName(), true)) );
prop.put("crawler-queue_list_"+showNum+"_profile", ((profileEntry == null) ? "unknown" : htmlTools.encodeUnicode2html(profileEntry.name(), true)));
diff --git a/htroot/IndexCreateWWWLocalQueue_p.html b/htroot/IndexCreateWWWLocalQueue_p.html index ebefb3bb3..fbfd4d714 100644 --- a/htroot/IndexCreateWWWLocalQueue_p.html +++ b/htroot/IndexCreateWWWLocalQueue_p.html @@ -4,9 +4,9 @@ <title>YaCy '#[clientname]#': Local Crawl Queue</title>
#%env/templates/metas.template%#
</head>
- <body id="IndexCreateWWwLocalQueue">
- #%env/templates/header.template%#
- #%env/templates/submenuCrawler.template%#
+ <body id="IndexCreateWWWLocalQueue">
+ #%env/templates/header.template%# + #%env/templates/submenuIndexCreate.template%#
<h2>Local Crawl Queue</h2>
<p>
This queue stores the urls that shall be crawled localy by this peer.
diff --git a/htroot/IndexCreateWWWLocalQueue_p.java b/htroot/IndexCreateWWWLocalQueue_p.java index 049ee5578..d4ecb88c7 100644 --- a/htroot/IndexCreateWWWLocalQueue_p.java +++ b/htroot/IndexCreateWWWLocalQueue_p.java @@ -109,7 +109,7 @@ public class IndexCreateWWWLocalQueue_p { if (option == PROFILE) {
// search and delete the crawl profile (_much_ faster, independant of queue size)
// XXX: what to do about the annoying LOST PROFILE messages in the log?
- Iterator it = switchboard.profiles.profiles(true);
+ Iterator it = switchboard.profilesActiveCrawls.profiles(true);
plasmaCrawlProfile.entry entry;
while (it.hasNext()) {
entry = (plasmaCrawlProfile.entry)it.next();
@@ -119,8 +119,9 @@ public class IndexCreateWWWLocalQueue_p { name.equals(plasmaSwitchboard.CRAWL_PROFILE_SNIPPET_TEXT) ||
name.equals(plasmaSwitchboard.CRAWL_PROFILE_SNIPPET_MEDIA))
continue;
- if (compiledPattern.matcher(name).find())
- switchboard.profiles.removeEntry(entry.handle());
+ if (compiledPattern.matcher(name).find()) {
+ switchboard.profilesActiveCrawls.removeEntry(entry.handle());
+ }
}
} else {
// iterating through the list of URLs
@@ -144,7 +145,7 @@ public class IndexCreateWWWLocalQueue_p { if (value != null) {
Matcher matcher = compiledPattern.matcher(value);
if (matcher.find()) {
- switchboard.noticeURL.remove(entry.url().hash());
+ switchboard.noticeURL.removeByURLHash(entry.url().hash());
}
}
}
@@ -158,7 +159,7 @@ public class IndexCreateWWWLocalQueue_p { prop.put("info_numEntries", c);
} else if (post.containsKey("deleteEntry")) {
String urlHash = (String) post.get("deleteEntry");
- switchboard.noticeURL.remove(urlHash);
+ switchboard.noticeURL.removeByURLHash(urlHash);
prop.put("LOCATION","");
return prop;
}
@@ -182,7 +183,7 @@ public class IndexCreateWWWLocalQueue_p { if ((urle != null)&&(urle.url()!=null)) {
initiator = yacyCore.seedDB.getConnected(urle.initiator());
profileHandle = urle.profileHandle();
- profileEntry = (profileHandle == null) ? null : switchboard.profiles.getEntry(profileHandle);
+ profileEntry = (profileHandle == null) ? null : switchboard.profilesActiveCrawls.getEntry(profileHandle);
prop.put("crawler-queue_list_"+showNum+"_dark", ((dark) ? 1 : 0) );
prop.put("crawler-queue_list_"+showNum+"_initiator", ((initiator == null) ? "proxy" : htmlTools.encodeUnicode2html(initiator.getName(), true)) );
prop.put("crawler-queue_list_"+showNum+"_profile", ((profileEntry == null) ? "unknown" : profileEntry.name()));
diff --git a/htroot/IndexCreateWWWRemoteQueue_p.html b/htroot/IndexCreateWWWRemoteQueue_p.html index cbe2a95c9..cdde8bb54 100644 --- a/htroot/IndexCreateWWWRemoteQueue_p.html +++ b/htroot/IndexCreateWWWRemoteQueue_p.html @@ -6,7 +6,7 @@ </head> <body id="IndexCreateWWWGlobalQueue"> #%env/templates/header.template%# - #%env/templates/submenuCrawler.template%# + #%env/templates/submenuIndexCreate.template%# <h2>Remote Crawl Queue</h2> <p> This queue stores the urls that other peers sent to you in order to perform a remote crawl for them. diff --git a/htroot/IndexCreateWWWRemoteQueue_p.java b/htroot/IndexCreateWWWRemoteQueue_p.java index 01ba6c758..cb30a2cda 100644 --- a/htroot/IndexCreateWWWRemoteQueue_p.java +++ b/htroot/IndexCreateWWWRemoteQueue_p.java @@ -93,7 +93,7 @@ public class IndexCreateWWWRemoteQueue_p { prop.put("info_numEntries", c); } else if (post.containsKey("deleteEntry")) { String urlHash = (String) post.get("deleteEntry"); - sb.noticeURL.remove(urlHash); + sb.noticeURL.removeByURLHash(urlHash); prop.put("LOCATION",""); return prop; } @@ -117,7 +117,7 @@ public class IndexCreateWWWRemoteQueue_p { if (urle != null && urle.url() != null) { initiator = yacyCore.seedDB.getConnected(urle.initiator()); profileHandle = urle.profileHandle(); - profileEntry = (profileHandle == null) ? null : sb.profiles.getEntry(profileHandle); + profileEntry = (profileHandle == null) ? null : sb.profilesActiveCrawls.getEntry(profileHandle); prop.put("crawler-queue_list_" + showNum + "_dark", ((dark) ? 1 : 0) ); prop.put("crawler-queue_list_" + showNum + "_initiator", ((initiator == null) ? "proxy" : initiator.getName())); prop.put("crawler-queue_list_" + showNum + "_profile", ((profileEntry == null) ? "unknown" : profileEntry.name())); diff --git a/htroot/ProxyIndexingMonitor_p.java b/htroot/ProxyIndexingMonitor_p.java index 92589bbc7..3332fb583 100644 --- a/htroot/ProxyIndexingMonitor_p.java +++ b/htroot/ProxyIndexingMonitor_p.java @@ -117,11 +117,11 @@ public class ProxyIndexingMonitor_p { prop.put("info", 1); //delete DATA/PLASMADB/crawlProfiles0.db
} else {
try {
- sb.profiles.changeEntry(sb.defaultProxyProfile, "generalDepth", Integer.toString(newProxyPrefetchDepth));
- sb.profiles.changeEntry(sb.defaultProxyProfile, "storeHTCache", (proxyStoreHTCache) ? "true": "false");
- sb.profiles.changeEntry(sb.defaultProxyProfile, "remoteIndexing",proxyIndexingRemote ? "true":"false");
- sb.profiles.changeEntry(sb.defaultProxyProfile, "indexText",proxyIndexingLocalText ? "true":"false");
- sb.profiles.changeEntry(sb.defaultProxyProfile, "indexMedia",proxyIndexingLocalMedia ? "true":"false");
+ sb.profilesActiveCrawls.changeEntry(sb.defaultProxyProfile, "generalDepth", Integer.toString(newProxyPrefetchDepth));
+ sb.profilesActiveCrawls.changeEntry(sb.defaultProxyProfile, "storeHTCache", (proxyStoreHTCache) ? "true": "false");
+ sb.profilesActiveCrawls.changeEntry(sb.defaultProxyProfile, "remoteIndexing",proxyIndexingRemote ? "true":"false");
+ sb.profilesActiveCrawls.changeEntry(sb.defaultProxyProfile, "indexText",proxyIndexingLocalText ? "true":"false");
+ sb.profilesActiveCrawls.changeEntry(sb.defaultProxyProfile, "indexMedia",proxyIndexingLocalMedia ? "true":"false");
prop.put("info", 2);//new proxyPrefetchdepth
prop.put("info_message", newProxyPrefetchDepth);
diff --git a/htroot/QuickCrawlLink_p.java b/htroot/QuickCrawlLink_p.java index 8f4d463ba..2b8a85ecd 100644 --- a/htroot/QuickCrawlLink_p.java +++ b/htroot/QuickCrawlLink_p.java @@ -141,15 +141,15 @@ public class QuickCrawlLink_p { String urlhash = crawlingStartURL.hash();
switchboard.wordIndex.loadedURL.remove(urlhash);
- switchboard.noticeURL.remove(urlhash);
+ switchboard.noticeURL.removeByURLHash(urlhash);
switchboard.errorURL.remove(urlhash);
// create crawling profile
plasmaCrawlProfile.entry pe = null;
try {
- pe = switchboard.profiles.newEntry(
+ pe = switchboard.profilesActiveCrawls.newEntry(
crawlingStartURL.getHost(),
- crawlingStart,
+ crawlingStartURL,
crawlingFilter,
crawlingFilter,
CrawlingDepth,
diff --git a/htroot/WatchCrawler_p.html b/htroot/WatchCrawler_p.html index 974b7f045..5a935e3c1 100644 --- a/htroot/WatchCrawler_p.html +++ b/htroot/WatchCrawler_p.html @@ -1,5 +1,6 @@ <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN" "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd"> <html xmlns="http://www.w3.org/1999/xhtml"> +#(forwardToCrawlStart)#::<meta http-equiv="REFRESH" content="0;/CrawlStartSimple_p.html">#(/forwardToCrawlStart)# <head> <title>YaCy '#[clientname]#': Crawler Queues</title> #%env/templates/metas.template%# @@ -9,7 +10,7 @@ <script type="text/javascript" src="/js/WatchCrawler.js"></script></head> <body id="watchCrawler"> #%env/templates/header.template%# -#%env/templates/submenuCrawler.template%# +#%env/templates/submenuIndexCreate.template%# <h2>Crawler Queues</h2> <p> Next update in <input type="text" id="nextUpdate" onfocus="changeInterval()" onblur="newInterval()" size="2" /> seconds. <img src="/env/grafics/empty.gif" name="ajax" alt="empty"/> </p> diff --git a/htroot/WatchCrawler_p.java b/htroot/WatchCrawler_p.java index a8f8aa478..52f39d732 100644 --- a/htroot/WatchCrawler_p.java +++ b/htroot/WatchCrawler_p.java @@ -63,6 +63,7 @@ public class WatchCrawler_p { // return variable that accumulates replacements plasmaSwitchboard switchboard = (plasmaSwitchboard) env; serverObjects prop = new serverObjects(); + prop.put("forwardToCrawlStart", 0); if (post == null) { // not a crawl start, only monitoring @@ -70,6 +71,10 @@ public class WatchCrawler_p { } else { prop.put("info", 0); + if ((post.containsKey("autoforward")) && (switchboard.coreCrawlJobSize() == 0)) { + prop.put("forwardToCrawlStart", 1); + } + if (post.containsKey("continue")) { // continue queue String queue = post.get("continue", ""); @@ -158,18 +163,12 @@ public class WatchCrawler_p { if (pos == -1) crawlingStart = "http://" + crawlingStart; // normalizing URL - try {crawlingStart = new yacyURL(crawlingStart, null).toNormalform(true, true);} catch (MalformedURLException e1) {} - - // check if url is proper yacyURL crawlingStartURL = null; - try { - crawlingStartURL = new yacyURL(crawlingStart, null); - } catch (MalformedURLException e) { - crawlingStartURL = null; - } + try {crawlingStartURL = new yacyURL(crawlingStart, null);} catch (MalformedURLException e1) {} + crawlingStart = (crawlingStartURL == null) ? null : crawlingStartURL.toNormalform(true, true); // check if pattern matches - if ((crawlingStartURL == null) /* || (!(crawlingStart.matches(newcrawlingfilter))) */) { + if ((crawlingStart == null) /* || (!(crawlingStart.matches(newcrawlingfilter))) */) { // print error message prop.put("info", 4); //crawlfilter does not match url prop.put("info_newcrawlingfilter", newcrawlingfilter); @@ -183,12 +182,13 @@ public class WatchCrawler_p { // first delete old entry, if exists String urlhash = (new yacyURL(crawlingStart, null)).hash(); switchboard.wordIndex.loadedURL.remove(urlhash); - switchboard.noticeURL.remove(urlhash); + switchboard.noticeURL.removeByURLHash(urlhash); switchboard.errorURL.remove(urlhash); // stack url - plasmaCrawlProfile.entry pe = switchboard.profiles.newEntry( - crawlingStartURL.getHost(), crawlingStart, newcrawlingfilter, newcrawlingfilter, + switchboard.profilesPassiveCrawls.removeEntry(crawlingStartURL.hash()); // if there is an old entry, delete it + plasmaCrawlProfile.entry pe = switchboard.profilesActiveCrawls.newEntry( + crawlingStartURL.getHost(), crawlingStartURL, newcrawlingfilter, newcrawlingfilter, newcrawlingdepth, newcrawlingdepth, crawlingIfOlder, crawlingDomFilterDepth, crawlingDomMaxPages, crawlingQ, @@ -268,7 +268,8 @@ public class WatchCrawler_p { HashMap hyperlinks = (HashMap) scraper.getAnchors(); // creating a crawler profile - plasmaCrawlProfile.entry profile = switchboard.profiles.newEntry(fileName, "file://" + file.toString(), newcrawlingfilter, newcrawlingfilter, newcrawlingdepth, newcrawlingdepth, crawlingIfOlder, crawlingDomFilterDepth, crawlingDomMaxPages, crawlingQ, indexText, indexMedia, storeHTCache, true, crawlOrder, xsstopw, xdstopw, xpstopw); + yacyURL crawlURL = new yacyURL("file://" + file.toString(), null); + plasmaCrawlProfile.entry profile = switchboard.profilesActiveCrawls.newEntry(fileName, crawlURL, newcrawlingfilter, newcrawlingfilter, newcrawlingdepth, newcrawlingdepth, crawlingIfOlder, crawlingDomFilterDepth, crawlingDomMaxPages, crawlingQ, indexText, indexMedia, storeHTCache, true, crawlOrder, xsstopw, xdstopw, xpstopw); // loop through the contained links Iterator interator = hyperlinks.entrySet().iterator(); @@ -325,10 +326,11 @@ public class WatchCrawler_p { try { // getting the sitemap URL sitemapURLStr = post.get("sitemapURL",""); - + yacyURL sitemapURL = new yacyURL(sitemapURLStr, null); + // create a new profile - plasmaCrawlProfile.entry pe = switchboard.profiles.newEntry( - sitemapURLStr, sitemapURLStr, newcrawlingfilter, newcrawlingfilter, + plasmaCrawlProfile.entry pe = switchboard.profilesActiveCrawls.newEntry( + sitemapURLStr, sitemapURL, newcrawlingfilter, newcrawlingfilter, newcrawlingdepth, newcrawlingdepth, crawlingIfOlder, crawlingDomFilterDepth, crawlingDomMaxPages, crawlingQ, diff --git a/htroot/WatchWebStructure_p.html b/htroot/WatchWebStructure_p.html index cb4559aa5..527db4d46 100644 --- a/htroot/WatchWebStructure_p.html +++ b/htroot/WatchWebStructure_p.html @@ -20,7 +20,7 @@ </head> <body id="WebStructure" style="margin:0px;"> #%env/templates/header.template%# -#%env/templates/submenuCrawler.template%# +#%env/templates/submenuWebStructure.template%# <h2>Web Structure</h2> <div id="left"> diff --git a/htroot/WatchWebStructure_p.java b/htroot/WatchWebStructure_p.java index 818c11d4e..ce1331d0f 100644 --- a/htroot/WatchWebStructure_p.java +++ b/htroot/WatchWebStructure_p.java @@ -31,7 +31,7 @@ public class WatchWebStructure_p { if (host.equals("auto")) { // try to find the host from the crawl profiles - Iterator it = sb.profiles.profiles(true); + Iterator it = sb.profilesActiveCrawls.profiles(true); entry e; while (it.hasNext()) { e = (entry)it.next(); diff --git a/htroot/env/templates/header.template b/htroot/env/templates/header.template index 5e3154e67..bb25bc722 100644 --- a/htroot/env/templates/header.template +++ b/htroot/env/templates/header.template @@ -16,7 +16,8 @@ <li class="menugroup" id="menugroupCrawlerControl">
<h3>Web Indexing</h3>
<ul class="menu">
- <li><a href="/CrawlStartSimple_p.html" class="MenuItemLink lock">Start a Web Crawl</a></li>
+ <li><a href="/WatchCrawler_p.html?autoforward=" class="MenuItemLink lock">Crawl Start & Monitoring</a></li>
+ <li><a href="/CrawlResults.html?process=5&autoforward=" class="MenuItemLink">Crawl Results</a></li>
<li><a href="/IndexControl_p.html" class="MenuItemLink lock">Index Administration</a></li>
<li><a href="/Blacklist_p.html" class="MenuItemLink lock">Filter & Blacklists</a></li>
<li><a href="/ProxyIndexingMonitor_p.html" class="MenuItemLink lock">Indexing with Proxy</a></li>
@@ -26,11 +27,9 @@ <h3>Monitoring</h3>
<ul class="menu">
<li><a href="/Network.html" accesskey="w" class="MenuItemLink">YaCy Network</a></li>
- <li><a href="/WatchWebStructure_p.html?host=auto&depth=2&time=1000" class="MenuItemLink lock">Crawl Activity</a></li>
- <li><a href="/CrawlResults.html" class="MenuItemLink">Crawl Results</a></li>
+ <li><a href="/WatchWebStructure_p.html?host=auto&depth=2&time=1000" class="MenuItemLink lock">Web Visualization</a></li>
<li><a href="/AccessTracker_p.html" class="MenuItemLink lock">Access Tracker</a></li>
<li><a href="/ViewLog_p.html" class="MenuItemLink lock">Server Log</a></li>
- <li><a href="/CacheAdmin_p.html" class="MenuItemLink lock">Web Cache</a></li>
<li><a href="/News.html" accesskey="n" class="MenuItemLink">News Monitor</a></li>
<li><a href="/CookieMonitorIncoming_p.html" class="MenuItemLink lock">Cookies in Proxy</a></li>
</ul>
diff --git a/htroot/env/templates/submenuCrawler.template b/htroot/env/templates/submenuCrawler.template deleted file mode 100644 index f3d36e5fb..000000000 --- a/htroot/env/templates/submenuCrawler.template +++ /dev/null @@ -1,36 +0,0 @@ -<div class="SubMenu"> - <h3>Crawler Activity</h3> - - <div class="SubMenugroup"> - <h3>Activity Visualization</h3> - <ul class="SubMenu"> - <li><a href="/WatchWebStructure_p.html" class="MenuItemLink lock">Web Structure</a></li> - <li><a href="/WatchCrawler_p.html" class="MenuItemLink lock">Crawler Queues</a></li> - </ul> - </div> - - <div class="SubMenugroup"> - <h3>Control Queues</h3> - <ul class="SubMenu"> - <li><a href="/IndexCreateIndexingQueue_p.html" class="MenuItemLink lock">Indexing</a></li> - <li><a href="/IndexCreateLoaderQueue_p.html" class="MenuItemLink lock">Loader</a></li> - </ul> - </div> - <div class="SubMenugroup"> - <h3>WWW Crawl Queues</h3> - <ul class="SubMenu"> - <li><a href="/IndexCreateWWWLocalQueue_p.html" class="MenuItemLink lock">Local</a></li> - <li><a href="/IndexCreateWWWGlobalQueue_p.html" class="MenuItemLink lock">Global</a></li> - <li><a href="/IndexCreateWWWRemoteQueue_p.html" class="MenuItemLink lock">Remote</a></li> - <li><!--<a href="/IndexCreateWWWOverhangQueue_p.html" class="MenuItemLink">--><em class="lock">Overhang</em><!--</a>--></li> - </ul> - </div> - <div class="SubMenugroup"> - <h3>Media Crawl Queues</h3> - <ul class="SubMenu"> - <li><!--<a href="/IndexCreateImageQueue_p.html" class="MenuItemLink">--><em class="lock">Images</em><!--</a>--></li> - <li><!--<a href="/IndexCreateMovieQueue_p.html" class="MenuItemLink">--><em class="lock">Movies</em><!--</a>--></li> - <li><!--<a href="/IndexCreateMusicQueue_p.html" class="MenuItemLink">--><em class="lock">Music</em><!--</a>--></li> - </ul> - </div> -</div>
\ No newline at end of file diff --git a/htroot/env/templates/submenuIndexCreate.template b/htroot/env/templates/submenuIndexCreate.template index 2245182ea..fcea0c668 100644 --- a/htroot/env/templates/submenuIndexCreate.template +++ b/htroot/env/templates/submenuIndexCreate.template @@ -4,5 +4,35 @@ <li><a href="/CrawlStartSimple_p.html" class="MenuItemLink lock">Crawl Start (easy)</a></li>
<li><a href="/CrawlStartExpert_p.html" class="MenuItemLink lock">Crawl Start (expert)</a></li>
<li><a href="/CrawlProfileEditor_p.html" class="MenuItemLink lock">Crawl Profile Editor</a></li>
+ <li><a href="/WatchCrawler_p.html" class="MenuItemLink lock">Crawler Queues</a></li>
</ul>
+ <ul class="SubMenu">
+ <li> </li>
+ </ul>
+</div>
+<div class="SubMenu">
+ <div class="SubMenugroup">
+ <h3>Control Queues</h3>
+ <ul class="SubMenu">
+ <li><a href="/IndexCreateIndexingQueue_p.html" class="MenuItemLink lock">Indexing</a></li>
+ <li><a href="/IndexCreateLoaderQueue_p.html" class="MenuItemLink lock">Loader</a></li>
+ </ul>
+ </div>
+ <div class="SubMenugroup">
+ <h3>WWW Crawl Queues</h3>
+ <ul class="SubMenu">
+ <li><a href="/IndexCreateWWWLocalQueue_p.html" class="MenuItemLink lock">Local</a></li>
+ <li><a href="/IndexCreateWWWGlobalQueue_p.html" class="MenuItemLink lock">Global</a></li>
+ <li><a href="/IndexCreateWWWRemoteQueue_p.html" class="MenuItemLink lock">Remote</a></li>
+ <li><!--<a href="/IndexCreateWWWOverhangQueue_p.html" class="MenuItemLink">--><em class="lock">Overhang</em><!--</a>--></li>
+ </ul>
+ </div>
+ <div class="SubMenugroup">
+ <h3>Media Crawl Queues</h3>
+ <ul class="SubMenu">
+ <li><!--<a href="/IndexCreateImageQueue_p.html" class="MenuItemLink">--><em class="lock">Images</em><!--</a>--></li>
+ <li><!--<a href="/IndexCreateMovieQueue_p.html" class="MenuItemLink">--><em class="lock">Movies</em><!--</a>--></li>
+ <li><!--<a href="/IndexCreateMusicQueue_p.html" class="MenuItemLink">--><em class="lock">Music</em><!--</a>--></li>
+ </ul>
+ </div>
</div>
\ No newline at end of file diff --git a/htroot/env/templates/submenuWebStructure.template b/htroot/env/templates/submenuWebStructure.template new file mode 100644 index 000000000..9d46ab7ac --- /dev/null +++ b/htroot/env/templates/submenuWebStructure.template @@ -0,0 +1,7 @@ +<div class="SubMenu"> + <h3>Web Visualization</h3> + <ul class="SubMenu"> + <li><a href="/WatchWebStructure_p.html" class="MenuItemLink lock">Web Structure</a></li> + <li><a href="/CacheAdmin_p.html" class="MenuItemLink lock">Web Cache</a></li> + </ul> +</div>
\ No newline at end of file |