summaryrefslogtreecommitdiff
path: root/test/java
AgeCommit message (Collapse)Author
2018-02-02Added HTML microdata typed items parsing capability.luccioman
This adds the possibility for the HTML parser to gather typed items URLs annotated in HTML tags with itemscope and itemtype attributes (see microdata specification https://www.w3.org/TR/microdata/ ), notably Types from the schema.org vocabulary, but also Types/Classes from any other vocabulary, such as the common ones listed in the RDFa core context ( https://www.w3.org/2011/rdfa-context/rdfa-1.1.html ).
2018-01-29Moved dbtest to the test source folder.luccioman
2018-01-28Added a manual performance test for the HostBalancer.luccioman
Consequently to the report in mantis 776 (http://mantis.tokeek.de/view.php?id=776). Running the perfs test with different control parameters seems to reveal that the YaCy's RowHandleMap used in the balancer depthCache is finally more efficient than for example the ConcurrentHashMap from JDK 8.
2018-01-26Removed time condition on HostBalancer initialization in JUnit test.luccioman
Its initialization in main application usage remains asynchronous.
2018-01-02Fixed SegmentTest test case time dependant occasional failuresluccioman
As highlighted by latest automated Travis builds.
2017-12-09Merge branch 'master' of https://github.com/yacy/yacy_search_server.gitMichael Peter Christen
2017-12-09added a crawl filter based on <div> tag class namesMichael Peter Christen
When a crawl is started, a new field to exclude content from scraping is available. The field can be identified with the class name of div tags. All text contained in such a div tag where the configured class name(s) match are not indexed, while the remaining page is indexed.
2017-12-08Removed use of deprecated Jetty IPAccessHandler for client filtering.luccioman
Upgraded to InetAccessHandler. Added InetPathAccessHandler extension to InetAccessHandler to maintain path patterns capability previously available in IPAccessHandler but lost in InetAccessHandler. Filtering on IPv6 addresses is now supported. Support for deprecated pattern formats such as "192.168." and "192.168.1.1/path" has been removed, but startup automated migration should convert such patterns eventually present in serverClient.
2017-12-01Improved accuracy of URLs search filters : protocol, tld, host, file extluccioman
2017-12-01Fixed URL parsing with fragment and empty pathluccioman
2017-11-22Added a basic JUnit test for the Visio parser (vsdParser)luccioman
2017-11-20Restore initial locale at the end of a JUnit test case which modify it.luccioman
2017-11-20Do locale neutral case conversions on domain names.luccioman
Required to properly run on systems with default locale set to Turkish language, as with this locale the 'i' character has different upper and lower case flavors than with other locales.
2017-11-20Do locale neutral case conversions in MultiProtocolURLluccioman
For any relevant URL parts : host name, URL scheme, session ids or technical parts (see https://url.spec.whatwg.org/#url-writing and https://tools.ietf.org/html/rfc3986 for current standard references). Remaining locale sensitive conversion used for detection of URL word components in urlComps() makes sense but using detected language would be preferable than using the default system locale.
2017-11-20Do locale neutral case conversions in Classificationluccioman
Required for people using Turkish language as their default system locale, as with this locale the 'i' character has different upper and lower case flavors than with other locales.
2017-11-14Updated the generic top-level known domains list.luccioman
Using current IANA reference list at https://www.iana.org/domains/root/db The generated URL hashes on these domains stay the same but performance is greatly improved as a DNS resolve request is required on URL hash computation when the TLD part of the host name is unknown. Hash computation mean time measured on 1541 sample URLs (one on each TLD) and a computer with a DSL connection : about 230ms before change, then only 20ms.
2017-11-08Improved some JUnit tests isolation and resources releaseluccioman
The modified tests were successfull when run manually from an IDE such as Eclipse, but failed occasionnally when run with maven as part of the overall test suite.
2017-11-07Remove old hard-coded holiday dates from DateDection class.luccioman
Replaced with rules based relative to current year as already done for a part of the supported dates.
2017-11-06Added a html parser charset detection unit testluccioman
2017-10-18Consistently encode snapshot image with format requested on the API.luccioman
Previously, calling /api/snapshot.png rendered JPEG encoded images.
2017-10-16Fixed pdf2image conversion with imagemagick on PDFs having transparencyluccioman
The target image format (jpeg) doesn't support transparency, so the Html2ImageTest produced unusable black images when ran on a linux machine having imagemagick package installed.
2017-10-16Fixed ProfilingGraph calculation integer overflows and added test class. luccioman
Complementary to fix proposed in PR #128 by @otteresk.
2017-10-09Adjusted ResponseHeaderTest to succeed on slow or highly loaded CPUluccioman
2017-10-04Added partial bzip2 stream parsing support and bzipParser Junit testluccioman
2017-10-04Fixed RWI distance calculation on multi words search queries.luccioman
Distance was lost when storing/retrieving references to intermediate result container. Now all JUnit tests are again successfully passing!
2017-08-22Added unit tests on the gzip parser.luccioman
2017-08-22Finer control on max links to parse in the html parser.luccioman
2017-08-22Added some unit tests on FileUtils.luccioman
2017-08-21Improved parsing support for OOXML spreadsheets (.xlsx)luccioman
As reported edycop in mantis 765 ( http://mantis.tokeek.de/view.php?id=765 ), parsing of xlsx files was quite incomplete. Now properly support "Shared String Table" entry in Office Open XML spreadsheets, an also detect embedded URLs. Integrating the Apache poi-ooxml library could be an option for finer OOXML formats support, but their SAX style parsing example ( http://poi.apache.org/spreadsheet/how-to.html#xssf_sax_api ) tends to show that a custom SAX handler is still efficient for lightweight and low memory footprint processing.
2017-08-14Implemented partial stream parsing of tar archives.luccioman
Also added JUnit tests for the tar parser and fixed unwanted use of the tar parser as a fallback on files included in a tar archive.
2017-08-14Also handle text content when parsing XML within limits.luccioman
2017-08-14Add junit test for AbstractOperations.addOperand()reger
2017-08-12Updated xml parser limited parsing test for use latest jdk.luccioman
2017-07-11Fixed read/copy on input streams reading sometimes less than expected.luccioman
2017-07-08Started support of partial parsing on large streamed resources.luccioman
Thus enable getpageinfo_p API to return something in a reasonable amount of time on resources over MegaBytes size range. Support added first with the generic XML parser, for other formats regular crawler limits apply as usual.
2017-07-03Removed temporary html parser test codeluccioman
2017-07-03HTML parser : removed unnecessary remaining recursive processingluccioman
Recursive processing was removed in commit 67beef657f82e92f48dd8425073ad81896a2ff4b, but one remained for anchors content(likely omitted from refactoring). It is no more necessary : other links such as images embedded in anchors are currently correctly detected by the parser. More annoying : that remaining recursive processing could lead to almost endless processing when encountering some (invalid) HTML structures involving nested anchors, as detected and reported by lucipher on YaCy forum ( http://forum.yacy-websuche.de/viewtopic.php?f=23&t=6005 ).
2017-06-27Refactored plain-text URLs detection implementation.luccioman
For faster processing (measured about 2 times faster on many real-world examples) and more advanced detection (previous algorithm detected only URLs separated from the rest of the text by a space character).
2017-06-27Ensure lower case conversion consistency with any default locale.luccioman
Especially for Turkish speaking users using "tr" as their system default locale : strings for technical stuff (URLs, tag names, constants...) must not be lower cased with the default locale, as 'I' doesn't becomes 'i' like in other locales such as "en", but becomes 'ı'.
2017-06-26Made mime type and extension normalization locale independent.luccioman
Previously, upper cased mime type was incorrectly normalized when the default locale is Turkish.
2017-06-26Added a generic XML parser, able to parse elements text and URLs.luccioman
This parser adds support for any XML based format other than already supported XML vocabularies such XHTML, RSS/Atom feeds... It will eventually be used as a fallback if one of these specific parsers fail, before falling back to the existing genericParser which extracts not that much useful information except URL tokens.
2017-06-22Improved character encoding detection from Content-Type headerluccioman
Also updated some related JavaDocs
2017-06-21Added a basic JUnit test with test gz files for the gzip parserluccioman
2017-06-21Properly close test files in htmlParser unit testluccioman
2017-06-15Added HT Cache basic statistics (hit rate)luccioman
2017-06-14Made Cache compression level and lock timeout user configurableluccioman
2017-06-14Limit the synchronization blocking time on some Cache operations.luccioman
Using a Reentrant lock instead of the intrinsic synchronization lock permits limiting the blocking time to acquire a lock. Useful on a very busy Cache concurrently accessed by many threads : when the time to acquire a lock is too high, getting/storing content on the cache becomes inefficient, and it is then better to fall back to loading remote resources. Illustrated by the CacheTest stress test and some traces reported in mantis 751 ( http://mantis.tokeek.de/view.php?id=751 )
2017-06-09migrated Solr 5.5 -> Solr 6.6 and from Java 1.7 -> 1.8Michael Peter Christen
Also: now Version 1.921
2017-06-03Ensure file input streams proper closing in both success and failuresluccioman
Also add when possible a warning level log message on input stream closing error instead of failing silently. This could help understanding some IO exceptions such as "too many files open".
2017-06-02Ensure proper closing of file input streams.luccioman
generated by cgit v1.2.3 (git 2.46.0) at 2026-08-08 11:14:43 +0000