summaryrefslogtreecommitdiff
path: root/source/org
AgeCommit message (Collapse)Author
2024-11-25code maintenance - removed warnings and replaced deprecated functionsMichael Peter Christen
2023-11-05bugfixes to zim reader:Michael Peter Christen
2023-11-05Added a zim parser to the surrogate import option.Michael Peter Christen
You can now import zim files into YaCy by simply moving them to the DATA/SURROGATE/IN folder. They will be fetched and after parsing moved to DATA/SURROGATE/OUT. There are exceptions where the parser is not able to identify the original URL of the documents in the zim file. In that case the file is simply ignored. This commit also carries an important fix to the pdf parser and an increase of the maximum parsing speed to 60000 PPM which should make it possible to index up to 1000 files in one second.
2023-11-03modified cache strategy for zim clustersMichael Peter Christen
2023-11-02added parsing rules for wikibooks and wikinews in zim readerMichael Peter Christen
2023-11-01fixed blob iterator in zim cluster definitionMichael Peter Christen
2023-11-01added a cluster cache but it requires more testingMichael Peter Christen
2023-11-01Merge branch 'master' of https://github.com/yacy/yacy_search_server.gitMichael Peter Christen
2023-11-01added an optimized zim file entry iteratorMichael Peter Christen
2023-11-01added a zim importer that can be used for surrogate imports.Michael Peter Christen
Can not be used yet because it requires some security additions to verify that the given urls actually work.
2023-10-29added a ZIM cluster pointer cacheMichael Peter Christen
2023-10-29introduced url list and title list caching and enhanced input streamMichael Peter Christen
performance in ZIM reader
2023-10-28added Zstandard compressed data decompression for ZIM files type 5Michael Peter Christen
also: more generalization and performance enhancements
2023-10-28fixed typo in class nameMichael Peter Christen
2023-10-28refactoring of ZIM reader, simplification, removed unnecessary codeMichael Peter Christen
2023-10-27Fixed a large number of problems in the ZIM reader.Michael Peter Christen
This library was not prepared for large data because it was missing long data types for pointers. I had to modify the code-base in a fundamental way: - Proof-Reading, - unclustering, - refactoring, - naming adoption to https://wiki.openzim.org/wiki/ZIM_file_format, - change of Exception handling, - extension to more attributes as defined in spec (bugfix for mime type loading) - bugfix to long parsing (prevented reading of large files) The code is furthermore very inefficient and requires more attention. However the format is very useful for YaCy as there are numerous data sources for ZIM-Files.
2023-10-27integrated the source code of a openzim file format reader. These areMichael Peter Christen
the raw format reader files with no integration in YaCy yet, which will maybe follow as a next step. The zim file format is documented in https://openzim.org and the reader code was taken from the archived, non-maintained repository at https://github.com/openzim/zimreader-java
2020-04-24modified org.json.* library to fit into the YaCy environmentMichael Peter Christen
as drop-in replacement. Also made some fixes and enhancements to the library.
2020-04-24added org.json.* libraryMichael Peter Christen
from https://android.googlesource.com/platform/libcore/+/refs/heads/master/json/src/main/java/org/json as a preparation step for https://github.com/yacy/yacy_search_server/issues/347
2015-11-04use current tar library for untar filesreger
- remove old source copy
2012-07-05- added @SuppressWarnings to unused servlet method parametersMichael Peter Christen
- removed unnecessary casts - removed unnecessary throw statements
2012-05-14Now used synchronization in all close() methods to make sure all objectsRoland 'Quix0r' Haeder
are 'closed' in an ordered way Conflicts: source/de/anomic/http/server/ChunkedInputStream.java source/de/anomic/http/server/ChunkedOutputStream.java source/de/anomic/http/server/ContentLengthInputStream.java source/net/yacy/cora/protocol/Domains.java source/net/yacy/cora/services/federated/solr/SolrShardingConnector.java source/net/yacy/cora/services/federated/solr/SolrSingleConnector.java source/net/yacy/document/content/dao/PhpBB3Dao.java source/net/yacy/document/parser/html/AbstractTransformer.java source/net/yacy/kelondro/blob/BEncodedHeap.java source/net/yacy/kelondro/blob/HeapReader.java source/net/yacy/kelondro/index/RAMIndexCluster.java source/net/yacy/kelondro/io/ByteCountInputStream.java source/net/yacy/kelondro/logging/ConsoleOutErrHandler.java source/net/yacy/kelondro/table/SQLTable.java
2011-11-25some last-minute performance hacksorbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@8101 6c8d7289-2bf4-0310-a012-ef5d649a1542
2011-03-08*) set SVN propertieslow012
*) minor changes git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@7567 6c8d7289-2bf4-0310-a012-ef5d649a1542
2010-01-10applied code changes that are recommended by PMDorbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6563 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-11-20- fixes for missing or bad hashCode computationorbiter
- fixes for bad equals() methods that had not been used by hash maps and therefore some classes did not work as objects in hash maps. - this may also affect some cases where double-checks should have been, but did not work. git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6495 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-07-15removed compiler warnings in imported classesorbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6220 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-07-14replaced StringBuffer with StringBuilder in tar liborbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6213 6c8d7289-2bf4-0310-a012-ef5d649a1542
2009-07-14added tar classes from apache ant toolsorbiter
git-svn-id: https://svn.berlios.de/svnroot/repos/yacy/trunk@6211 6c8d7289-2bf4-0310-a012-ef5d649a1542
generated by cgit v1.2.3 (git 2.46.0) at 2026-08-08 11:24:46 +0000