summaryrefslogtreecommitdiff
path: root/htroot/CrawlCheck_p.java
AgeCommit message (Expand)Author
2016-11-23Added control over Robots.txt active threads maximum number.luccioman
2014-03-14support for multiple sitemaps in robots.txtMichael Peter Christen
2013-10-23added concurrency for mass crawl checkMichael Peter Christen
2013-09-15- the webgraph shall store all links which appear on a web page and notMichael Peter Christen
2013-08-22Redesign of crawler identification and robots steering. A non-p2p userMichael Peter Christen
2013-07-17Added 'final' for all exception blocks as this helps the Java compilerRoland Haeder
2013-07-12refactoring of load_delay: this is a matter of client identificationMichael Peter Christen
2013-07-09- refactoring of log to ConcurrentLog:Michael Peter Christen
2013-05-20reduced locking situation in crawler: shifted synchronized location andMichael Peter Christen
2012-10-28replaced the custom robots.txt loader by the standard http loaderMichael Peter Christen
2012-10-10removed the option to prevent removal of & parts inside of theMichael Peter Christen
2012-10-10added a crawl start checker which makes a simple analysis on the list ofMichael Peter Christen
generated by cgit v1.2.3 (git 2.46.0) at 2026-08-08 11:26:30 +0000