| Age | Commit message (Collapse) | Author |
|---|
|
|
|
This is a major step because solr removed support for embedded solr
instances in 9.0 and we want to keep it because we want to ship
YaCy with an embedded solr. It was necessary to add parts of solr
code into YaCy to make this migration possible. Further on with
Solr 9.1 they removed even more parts which are required for embedded
operation, therefore we cannot migrate yet further without big
changes.
If you are running a YaCy instance with Solr 8.x, the migration should
be done automatically. If not you require to first migrate to a YaCy
version 1.93 with Solr 8.x to migrate to Solr 8 data.
|
|
index with solr 8.9.0 will automatically be migrated to 8.11.2. This is a preparation step to migrate to 9.0.0 soon.
|
|
|
|
+ some setting to reduce caches
|
|
- replace new guava 30 with older 25 because that is the correct
dependency for solr 8.8.1. The newer one did actually not work!
- index will be crated in a DATA/INDEX/freeworld/SEGMENTS/solr_8_8_1
subfolder. The older solr_6_6 index is not touched but also not
migrated. The index starts with fresh (empty) content.
- Older indexes must be migrated by hand (export/import) so far until a
better solution is found.
- Large schema adoptions for lucene 8.8.1
|
|
Fixes #292
|
|
Since upgrade from Solr 5.5 to Solr 6.6 (commit 6fe7359), hard
autocommits were still enabled to regularly persist the Solr index to
the file system, but new index entries were no more automatically made
available for use by the application (soft autocommit).
Therefore, YaCy features such as index statistics, that do not perform
an explicit commit (as recommended by Solr documentation) were no more
accurate.
Soft autocommit is now restored as a default, with a time period
expected to be sufficient for accuracy while adding only a reasonable
system load overhead.
Fixes issue #251
|
|
This overrides Solr default to use managed schema. As we don't use
programatic schema changes this directs Solr to use schema.xml, eliminating
the warning.
|
|
it is not longer part of solr-core api
http://lucene.apache.org/solr/6_6_0/index.html
|
|
resulting in startup stop exception after fresh swithch to 1.921
|
|
Reported by Collision on mantis 758 (
http://mantis.tokeek.de/view.php?id=758 ).
Introduced by the new YaCy Solr configuration for Solr 6.6.0 (see commit
6fe735945da97abcbb91ac545fb11cff9d48effc), including now Suggester
configuration.
|
|
Also: now Version 1.921
|
|
there should be no need for reindex by the update
|
|
|
|
|
|
|
|
(coordinate_p is not a mandatory field as such doesn't need to be declared as schema.field)
|
|
|
|
moved and was not cleared anymore. This results in an huge fieldcache.
(http://lucene.apache.org/#highlights-of-the-lucene-release-include
https://issues.apache.org/jira/browse/LUCENE-5666)
Here I try to use DovValues where it is possible.
For this I used the Api-Scheme as new basis für the Solr-Schema.
This needs at least a complete optimization of the Solr-Index to get a
smaller FieldCache.
Everything that is indexed with these setting will not use the
Fieldcache at all.
|
|
avoid warning log
W org.apache.solr.handler.admin.AdminHandlers <requestHandler name="/admin/" class="solr.admin.AdminHandlers" /> is deprecated . It is not required anymore
|
|
This is a very complex migration: many classes had been renamed or
removed, dependencies changed and the solr index type is now aligned to
be a solr cloud repository.
Together with the Solr 5.2 library update, one other dependent library
had been updated as well: httpclient 4.4->4.4.1
Older indexes are migrated from 4_10 to 5_2. However, the new index
structure is more efficient and we recommend to re-index everything.
Please use the index export before you do the update to a large
surrogate xml file. After the update, start with an empty index and then
initialize this with your dump.
|
|
enhancements when using large facets
|
|
|
|
|
|
|
|
This includes also an update to zookeeper 3.4.6 and a new library that
Solr initializes by default: org.restlet from
http://restlet.com/download/current#release=stable&edition=jse&distribution=zip
which is included in version 2.2.1 from may 6th 2014
|
|
lucene_47
|
|
https://cwiki.apache.org/confluence/display/solr/IndexConfig+in+SolrConfig
This appears to be necessary as many times a
java.io.FileNotFoundException: (Too many open files) appears.
See also: https://issues.apache.org/jira/browse/SOLR-4 and desperate
users at
http://stackoverflow.com/questions/3828343/too-many-open-file-exception-while-indexin-using-solr
We cannot force users to do a "ulimit -n 1000000", so this action seems
to be required.
|
|
an index on fields make search facets on that field possible
|
|
|
|
|
|
not-flushed Solr cache is now handled in this way:
- it is smaller by default
- an Solr-internal process is started to flush the cache periodically
(this does NOT clean the cache, just removes old objects)
- a Solr-external process (the standard YaCy cleanup-process) now has
direct access to the solr internal cache and flushes them completely.
The time frame for such a flush is defined by the cleanup-process
frequency, by default 10 minutes.
|
|
|
|
|
|
lower mergeFactor again (5) for less segments
|
|
|
|
string
|
|
noisy loggers quiet
|
|
|
|
frequent index receiving and re-indexing after clickdepth/cr
calculation.
|
|
- move setting of system property solr.directoryFactory=solr.MMapDirectoryFactory to solrcore.properties
- add check of os.arch for 64bit system, if it fails use default/solrcore.x86.properties (if exists) as solrcore.properties
reason: on 32bit MMapDirectoryFactory may fail with.....
Caused by: java.io.IOException: Map failed
at sun.nio.ch.FileChannelImpl.map(FileChannelImpl.java:849)
at org.apache.lucene.store.MMapDirectory.map(MMapDirectory.java:283)
|
|
field for solr
|
|
timeout, less documents from remote searches (too many results had
easily blocked a peer)
|
|
soft commits, reduced caching size of search events, ensured that solr
results are processed before connection is closed to keep that stuff not
too long in RAM
|
|
|
|
- an existing ranking servlet for solr was extended. It is now possible
to set boost values for fields, boost functions and boost queries.
- The ranking can have different instances, but currently only the first
one is used
- added an abstraction layer for fields which can be used for search and
those fields can be edited in the solr ranking configruation
- the ranking value from solr within the field score is used to combine
remote search requests, which all are created using the same locally
defined boost values
- reduced the number of fields which are used for search (makes it
faster)
- replaced some text fields by string fields (makes indexing faster)
- removed classes which had no use
- made a large number of experiments for a better ranking and created a
temporary setting which prefers hits inside titles
- adjusted also the RWI-based ranking computation to 'prefer title'
- made special cases like for portal search where no post-processing and
post-ranking is wanted: this keeps the original ranking order as done by
Solr
- fixed many bugs with old settings for ranking
|
|
structure, but is not filled yet. To have the opportunity of a second
core, multi-core functionality had to be implemented to the
deep-embedded solr:
- migrated the solr_40 directory content to a subdirectory
'collection1'; the previously used default core is now called
collection1
- added solr_40/webgraph subdirectory as second core
- added a servlet configuration for the second core 'webgraph' in
/IndexSchema_p.html
- added instance handling as addition to solr connections: all solr
connectors are now instances of an solr 'instance' object; this required
a complete re-design of the solr embedding
- migrated also caching and sharding ontop of new instance handling
- migrated the search apis to handle now the access to a specific core,
the default core named 'collection1'
- migrated the remote solr search interface to access shards of cores;
for the yacy remote search the default core is now called 'solr'; using
the peer address as solr address
- migrated the solr backup and restore process: old backups cannot be
used after this migration!
- redesign of solr instance handling in all methods which access the
instances: they cannot hold copies of these instances any more; the must
retrieve the actuall connection object every time they want to write to
it (this solves also some bugs when switching the index/network)
- added another schema 'solr.webgraph.schema', the old solr.keys.list is
replaced by solr.collection.schema
|
|
|
|
http://forum.yacy-websuche.de/viewtopic.php?p=27509#p27509
Instead, in the start script is checked if the host is a 64 host and
-Dsolr.directoryFactory=solr.MMapDirectoryFactory is set as java option
Reverted the ramBufferSizeMB setting (this was not enabled anyway)
because that may be too much memory for small peers and embedded
systems.
Activated the mergeFactor 4; this was commented out by mistake
|
