From 1cba31de43238eca646a666b73380baf094a1431 Mon Sep 17 00:00:00 2001 From: orbiter Date: Tue, 6 Mar 2007 22:43:32 +0000 Subject: redesigned ram organization for database caches - each cache can now allocate as much memory as is available - no more fixed limits - replaced old performance memory monitor by new one - added supervision methods as static functions into the classes that provide cache functionality - steering of ram allocation is done with two simple limits that are ram availability-relative git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@3434 6c8d7289-2bf4-0310-a012-ef5d649a1542 --- htroot/PerformanceMemory_p.html | 456 +++++++++++----------------------------- 1 file changed, 118 insertions(+), 338 deletions(-) (limited to 'htroot/PerformanceMemory_p.html') diff --git a/htroot/PerformanceMemory_p.html b/htroot/PerformanceMemory_p.html index 309c7197a..7283cdade 100644 --- a/htroot/PerformanceMemory_p.html +++ b/htroot/PerformanceMemory_p.html @@ -58,355 +58,135 @@ -
-
+

+

Memory Settings -
-
-
MByte
-
-
MByte
-
-
Changes take effect after restart of YaCy
-
+ + + + + + + + +
  MByte
+ #(setStartupCommit)#::Accepted change. This will take effect after restart of YaCy#(/setStartupCommit)# +
- +

-

RAM Cache for Database Files:

-
- - - - - - - - - - - - - - - - - - - - - - - - - - #(useRWICache)#:: - - - - - - - - - - - - - - - - - - #(/useRWICache)# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - #(usePreNURLCache)#:: - - - - - - - - - - - - - - - - - - #(/usePreNURLCache)# - #(useNURLCache)#:: - - - - - - - - - - - - - - - - - - #(/useNURLCache)# - #(useEURLCache)#:: - - - - - - - - - - - - - - - - - - #(/useEURLCache)# - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Database
 		DB Size
 		Node CacheObject CacheMemory Occupation (KBytes)Description
 
Chunk
Size
(bytes)		Empty
(avail.)
Slots		Used
(filled)
Slots		Node-Cache
Hit:Miss
Uniq:Doub
Del:Flush
Chunk
Size
(bytes)		Max
Size
Hit/Miss		 
Hit-Size
Miss-Size		Hit-Cache
Hit:Miss
Uniq:Doub
Del:Flush
Miss-Cache
Hit:Miss
Uniq:Doub
Del:Flush
Used NowAssigned MaxDefault MaxBest Max
RWI Assortment Cluster#[slreqRWI]##[nodszRWI]##[slempRWI]##[slfilRWI]##[slhittmissRWI]#
#[sluniqdoubRWI]#
#[slflushRWI]#		#[ochunksizRWI]##[ochunkmaxRWI]#
#[omisscmaxRWI]#		#[ohitsizeRWI]#
#[omisssizeRWI]#		#[ohittmissRWI]#
#[ouniqdoubRWI]#
#[oflushRWI]#		#[nhittmissRWI]#
#[nuniqdoubRWI]#
#[nflushRWI]#		#[usedRWI]##[dfltRWI]##[bestRWI]#The Assortment Cluster stores most of the page indexes. - Flushing speed of the temporary RWI cache depends on the size of this file cache. Increasing the space of this - cache will speed up crawls with a depth > 3.
HTTP Response Header#[slreqHTTP]##[nodszHTTP]##[slempHTTP]##[slfilHTTP]##[slhittmissHTTP]#
#[sluniqdoubHTTP]#
#[slflushHTTP]#		#[ochunksizHTTP]##[ochunkmaxHTTP]#
#[omisscmaxHTTP]#		#[ohitsizeHTTP]#
#[omisssizeHTTP]#		#[ohittmissHTTP]#
#[ouniqdoubHTTP]#
#[oflushHTTP]#		#[nhittmissHTTP]#
#[nuniqdoubHTTP]#
#[nflushHTTP]#		#[usedHTTP]##[dfltHTTP]##[bestHTTP]#The Response Header database stores the HTTP heades that other servers send when YaCy retrieves web pages - during proxy mode, when performing crawls or if it fetches pages for snippet generation. - Increasing this cache will be most important for a fast proxy mode.
'loaded' URLs#[slreqLURL]##[nodszLURL]##[slempLURL]##[slfilLURL]##[slhittmissLURL]#
#[sluniqdoubLURL]#
#[slflushLURL]#		#[ochunksizLURL]##[ochunkmaxLURL]#
#[omisscmaxLURL]#		#[ohitsizeLURL]#
#[omisssizeLURL]#		#[ohittmissLURL]#
#[ouniqdoubLURL]#
#[oflushLURL]#		#[nhittmissLURL]#
#[nuniqdoubLURL]#
#[nflushLURL]#		#[usedLURL]##[dfltLURL]##[bestLURL]#This is the database that holds the hash/url - relation and properties regarding the url like load date and server date. - This cache is very important for a fast search process. Increasing the cache size will result in more search results and less IO during DHT transfer.
'pre-noticed' URLs#[slreqPreNURL]##[nodszPreNURL]##[slempPreNURL]##[slfilPreNURL]##[slhittmissPreNURL]#
#[sluniqdoubPreNURL]#
#[slflushPreNURL]#		#[ochunksizPreNURL]##[ochunkmaxPreNURL]#
#[omisscmaxPreNURL]#		#[ohitsizePreNURL]#
#[omisssizePreNURL]#		#[ohittmissPreNURL]#
#[ouniqdoubPreNURL]#
#[oflushPreNURL]#		#[nhittmissPreNURL]#
#[nuniqdoubPreNURL]#
#[nflushPreNURL]#		#[usedPreNURL]##[dfltPreNURL]##[bestPreNURL]#
'noticed' URLs#[slreqNURL]##[nodszNURL]##[slempNURL]##[slfilNURL]##[slhittmissNURL]#
#[sluniqdoubNURL]#
#[slflushNURL]#		#[ochunksizNURL]##[ochunkmaxNURL]#
#[omisscmaxNURL]#		#[ohitsizeNURL]#
#[omisssizeNURL]#		#[ohittmissNURL]#
#[ouniqdoubNURL]#
#[oflushNURL]#		#[nhittmissNURL]#
#[nuniqdoubNURL]#
#[nflushNURL]#		#[usedNURL]##[dfltNURL]##[bestNURL]#A noticed URL is one that was discovered during crawling but was not loaded yet. - Increasing the cache size will result in faster double-check during URL recognition when doing crawls.
'error' URLs#[slreqEURL]##[nodszEURL]##[slempEURL]##[slfilEURL]##[slhittmissEURL]#
#[sluniqdoubEURL]#
#[slflushEURL]#		#[ochunksizEURL]##[ochunkmaxEURL]#
#[omisscmaxEURL]#		#[ohitsizeEURL]#
#[omisssizeEURL]#		#[ohittmissEURL]#
#[ouniqdoubEURL]#
#[oflushEURL]#		#[nhittmissEURL]#
#[nuniqdoubEURL]#
#[nflushEURL]#		#[usedEURL]##[dfltEURL]##[bestEURL]#URLs that cannot be loaded are stored in this database. It is also used for double-checked during crawling. - Increasing the cache size will most probably speed up crawling slightly, but not significantly.
DHT Control#[slreqDHT]##[nodszDHT]##[slempDHT]##[slfilDHT]##[slhittmissDHT]#
#[sluniqdoubDHT]#
#[slflushDHT]#		#[ochunksizDHT]##[ochunkmaxDHT]#
#[omisscmaxDHT]#		#[ohitsizeDHT]#
#[omisssizeDHT]#		#[ohittmissDHT]#
#[ouniqdoubDHT]#
#[oflushDHT]#		#[nhittmissDHT]#
#[nuniqdoubDHT]#
#[nflushDHT]#		#[usedDHT]##[dfltDHT]##[bestDHT]#This is simply the cache for the seed-dbs (active, passive, potential). - This cache is divided into three equal parts. - Increasing this cache may speed up many functions, but we need to test this to see the effects.
Messages#[slreqMessage]##[nodszMessage]##[slempMessage]##[slfilMessage]##[slhittmissMessage]#
#[sluniqdoubMessage]#
#[slflushMessage]#		#[ochunksizMessage]##[ochunkmaxMessage]#
#[omisscmaxMessage]#		#[ohitsizeMessage]#
#[omisssizeMessage]#		#[ohittmissMessage]#
#[ouniqdoubMessage]#
#[oflushMessage]#		#[nhittmissMessage]#
#[nuniqdoubMessage]#
#[nflushMessage]#		#[usedMessage]##[dfltMessage]##[bestMessage]#The Message cache for peer-to-peer messages. Less important.
Wiki#[slreqWiki]##[nodszWiki]##[slempWiki]##[slfilWiki]##[slhittmissWiki]#
#[sluniqdoubWiki]#
#[slflushWiki]#		#[ochunksizWiki]##[ochunkmaxWiki]#
#[omisscmaxWiki]#		#[ohitsizeWiki]#
#[omisssizeWiki]#		#[ohittmissWiki]#
#[ouniqdoubWiki]#
#[oflushWiki]#		#[nhittmissWiki]#
#[nuniqdoubWiki]#
#[nflushWiki]#		#[usedWiki]##[dfltWiki]##[bestWiki]#The YaCy-Wiki uses a database to store its pages. - This cache is divided in two parts, one for the wiki database and one for its backup. - Increasing this cache may speed up access to the wiki pages.
Blog#[slreqBlog]##[nodszBlog]##[slempBlog]##[slfilBlog]##[slhittmissBlog]#
#[sluniqdoubBlog]#
#[slflushBlog]#		#[ochunksizBlog]##[ochunkmaxBlog]##[ohitsizeBlog]#
#[omisssizeBlog]#		#[ohittmissBlog]#
#[ouniqdoubBlog]#
#[oflushBlog]#		#[nhittmissBlog]#
#[nuniqdoubBlog]#
#[nflushBlog]#		#[usedBlog]##[dfltBlog]##[bestBlog]#The YaCy-Blog uses a database to store its entries. - Increasing this cache may speed up access to the Blog.
News#[slreqNews]##[nodszNews]##[slempNews]##[slfilNews]##[slhittmissNews]#
#[sluniqdoubNews]#
#[slflushNews]#		#[ochunksizNews]##[ochunkmaxNews]##[ohitsizeNews]#
#[omisssizeNews]#		#[ohittmissNews]#
#[ouniqdoubNews]#
#[oflushNews]#		#[nhittmissNews]#
#[nuniqdoubNews]#
#[nflushNews]#		#[usedNews]##[dfltNews]##[bestNews]#The News-DB stores property-lists for news that are included in seeds. - Increasing this cache may speed up the peer-ping.
robots.txt DB#[slreqRobots]##[nodszRobots]##[slempRobots]##[slfilRobots]##[slhittmissRobots]#
#[sluniqdoubRobots]#
#[slflushRobots]#		#[ochunksizRobots]##[ochunkmaxRobots]##[ohitsizeRobots]#
#[omisssizeRobots]#		#[ohittmissRobots]#
#[ouniqdoubRobots]#
#[oflushRobots]#		#[nhittmissRobots]#
#[nuniqdoubRobots]#
#[nflushRobots]#		#[usedRobots]##[dfltRobots]##[bestRobots]#The robots.txt DB stores downloaded records from robots.txt files. - Increasing this cache may speed up validation if crawling of the URL is allowed.
Crawl Profiles#[slreqProfiles]##[nodszProfiles]##[slempProfiles]##[slfilProfiles]##[slhittmissProfiles]#
#[sluniqdoubProfiles]#
#[slflushProfiles]#		#[ochunksizProfiles]##[ochunkmaxProfiles]##[ohitsizeProfiles]#
#[omisssizeProfiles]#		#[ohittmissProfiles]#
#[ouniqdoubProfiles]#
#[oflushProfiles]#		#[nhittmissProfiles]#
#[nuniqdoubProfiles]#
#[nflushProfiles]#		#[usedProfiles]##[dfltProfiles]##[bestProfiles]#The profile database stores properties for each crawl that is started on the local peer. - Increasing this cache may speed up crawling, but not much space is needed, so the effect may be low.
Totals#[usedTotal]# MB#[currTotal]# MB#[dfltTotal]# MB#[bestTotal]# MBSum of memory amounts
Re-Configuration:
these
custom
values
 
all
default
values
 
all
optimum
values
 		Changes take effect after re-start of YaCy
-
+

FlexTable RAM Index:

+ + + + + + + + #{TableList}# + + + + + + + #{/TableList}# + + + +
TableChunk SizeCountUsed Memory
#[tableIndexPath]##[tableIndexChunkSize]##[tableIndexCount]##[tableIndexMem]# MB
Total Mem = #[TableIndexTotalMem]# MB

- +

+ -

Other caching structures:

+

Other Caching Structures:

@@ -424,7 +204,7 @@ -
TypeHashBlacklistedCache #[blacklistcache.size]#
+

#%env/templates/footer.template%# -- cgit v1.2.3