diff options
43 files changed, 2877 insertions, 3068 deletions
diff --git a/JETTY12_MIGRATION.md b/JETTY12_MIGRATION.md deleted file mode 100644 index 22122cfa4..000000000 --- a/JETTY12_MIGRATION.md +++ /dev/null @@ -1,333 +0,0 @@ -# Jetty 12 Migration Contract - -## Scope - -YaCy will migrate its embedded HTTP server from Jetty 9 to Jetty 12 while -remaining on Java 17 and keeping the existing Servlet 4.0 `javax.servlet` -source API. Jetty 12's EE8 environment provides this compatibility layer; the -migration does not require converting YaCy to `jakarta.servlet`. - -This contract covers the dependency boundary only. It does not yet migrate the -Jetty-specific adapters under `source/net/yacy/http`. - -## Solr Boundary - -YaCy embeds Solr 9 but does not directly use Solr's embedded `JettySolrRunner` -or its Jetty-based HTTP/2 clients. Remote Solr access is implemented with -`HttpSolrClient` and `ConcurrentUpdateSolrClient`, both configured with Apache -HttpClient in `RemoteInstance`. - -The three Solr artifacts in `ivy.xml` therefore belong to the private -`solr9-bridge` configuration and explicitly exclude the `org.eclipse.jetty` -and `org.eclipse.jetty.http2` families. They are retrieved below -`build/solr9-bridge/input`, not directly into the runtime `lib` directory. - -Solr 9.0 nevertheless has an eager internal dependency that cannot be removed: -`CoreContainer` constructs both `HttpShardHandlerFactory` and -`UpdateShardHandler`, which create `Http2SolrClient` instances during startup -even in YaCy's standalone embedded configuration. The build therefore rewrites -Solr's Jetty references and the minimal Jetty 9 client/HTTP2 implementation to -the private `net.yacy.solr9.jetty` package. The generated artifacts have the -`solr9-bridge-` prefix under `lib/`; unrelocated Solr and HTTP2 jars are removed -from that directory before compilation. - -Run the boundary guard after resolving dependencies: - -```sh -ant clean compile -test/jetty-solr-dependency-guard.sh -``` - -## Current Jetty 9 Roots - -The direct Jetty dependencies reflect APIs imported by YaCy source code. -Dependencies needed only by another Jetty module remain transitive. - -| Responsibility | Direct Jetty 9 artifact | -| --- | --- | -| HTTP primitives | `jetty-http` | -| Connection and output APIs | `jetty-io` | -| Server and handlers | `jetty-server` | -| Utility, resource, and TLS APIs | `jetty-util` | -| CONNECT and proxy handlers | `jetty-proxy` | -| Authentication and constraints | `jetty-security` | -| Servlet container | `jetty-servlet` | -| QoS and servlet helpers | `jetty-servlets` | -| Web application context | `jetty-webapp` | - -The public `jetty-client` is owned transitively by `jetty-proxy`. YaCy has no -direct client API imports. `jetty-deploy` and `jetty-jmx` are not part of the -current root graph. HTTP/2 exists only inside the relocated Solr island, not as -an embedded-server feature. - -## Jetty 12 EE8 Target Graph - -The minimal target keeps Jetty Core separate from the EE8 Servlet layer. - -| Current responsibility | Jetty 12 target family | -| --- | --- | -| HTTP, IO, server, utilities | Jetty Core `jetty-http`, `jetty-io`, `jetty-server`, `jetty-util` | -| Proxy handlers | Jetty Core `jetty-proxy` and its client transitive | -| Servlet container | `org.eclipse.jetty.ee8:jetty-ee8-servlet` | -| Servlet helpers and QoS | `org.eclipse.jetty.ee8:jetty-ee8-servlets` | -| Web application context | `org.eclipse.jetty.ee8:jetty-ee8-webapp` | -| Servlet authentication | `org.eclipse.jetty.ee8:jetty-ee8-security` plus Jetty Core security transitives | -| Servlet API | Servlet 4.0 in the `javax.servlet` namespace, aligned with the EE8 environment | - -Do not add Jetty deploy, JMX, server-side HTTP/2, or Jakarta EE modules unless -a concrete YaCy feature requires them and its contract is verified separately. -The private Solr 9 HTTP/2 island above is the only current exception. - -### Solr 9 Isolation Gate (P1) - -Jetty 9 client classes and Jetty 12 server classes cannot safely share the -same application classloader because they use overlapping -`org.eclipse.jetty.*` packages with incompatible APIs. - -The P1 investigation rules out the two initially attractive shortcuts: - -1. **A Solr upgrade is not part of this migration.** YaCy remains on Solr - 9.0.0. A later Solr upgrade may provide another migration option, but it is - deliberately not a P1 implementation path. A comparison with Solr 9.10.1 - also showed that upgrading within the Solr 9 line would not remove the - boundary: it selects Jetty 10.0.26 and still eagerly constructs - `Http2SolrClient` instances. -2. **Whole-Solr classloader isolation is not a minimal boundary for YaCy.** - Solr API objects are part of the application boundary: 88 files under - `source/` and `test/` currently import `org.apache.solr` types. A child - classloader would either create incompatible class identities or require a - broad new facade and data conversion layer. - -Solr 10 is also outside this migration contract: although it moves to Jetty -12, the Solr 10 server requires Java 21 and uses the Jakarta Servlet namespace. - -P1 implements a **relocated Solr 9.0.0 Jetty client bridge**. The bridge keeps -the public `org.apache.solr.*` classes visible to YaCy while -rewriting Solr 9.0.0's internal `org.eclipse.jetty.*` references and the -required Jetty 9 client/HTTP2 implementation into a private package. It is -built reproducibly by Ant; no edited jar is stored in the repository. P1 does -not change the Solr or Lucene versions. - -P1 is complete only when a bridge proof passes all of these checks: - -1. only the selected embedded-server Jetty line uses the public - `org.eclipse.jetty` package; -2. no unrelocated Jetty 9/10 class is packaged by the bridge; -3. `EmbeddedSolrConnectorTest` starts and closes a `CoreContainer`; -4. an embedded update followed by a query succeeds; -5. the bridge dependency set and relocation rules are generated by the build; -6. the bridge can be removed without changing YaCy's Solr-facing source API. - -The integrated bridge passes `EmbeddedSolrConnectorTest` (`OK (4 tests)`) with -the original Solr and HTTP2 jars removed. It also passes when -Jetty 12.1.11 Core client, HTTP, IO, proxy, security, server, and utility jars -are present in the same application classpath. The reduced private island is: - -- Solr Core, SolrJ, and Solr Scripting 9.0.0 with only their Jetty references - rewritten; -- Jetty 9.4 client, HTTP, IO, and utility; -- Jetty 9.4 HTTP/2 client, common, and HTTP-client transport. - -Jetty server, servlet, security, proxy, webapp, and XML are not included in -the Solr island. The proof can be repeated with: - -```sh -ant clean compile -test/solr9-jetty-bridge-spike.sh -``` - -Set `JETTY12_CLASSPATH` to a colon-separated set of resolved Jetty 12 jars to -repeat the coexistence variant. The script compiles only its focused test into -a temporary directory; the bridge itself is already produced by `ant compile`. - -All six P1 checks pass. The Solr isolation gate is therefore closed for the -Jetty 12 server migration. Reconsidering Solr remains a separate future -decision, not an automatic part of this migration. - -The optional version comparison that established the limitation of the Solr 9 -line can be repeated without changing YaCy's production dependencies with: - -```sh -javap -classpath ~/.ivy2/cache/org.apache.solr/solr-core/jars/solr-core-9.10.1.jar \ - -private -c org.apache.solr.update.UpdateShardHandler -javap -classpath ~/.ivy2/cache/org.apache.solr/solr-core/jars/solr-core-9.10.1.jar \ - -private -c org.apache.solr.handler.component.HttpShardHandlerFactory -``` - -## Completion Gates - -The dependency phase is complete when all of these checks pass with one public -Jetty version on the resolved classpath and the private Solr island: - -1. `ant clean compile` -2. `test/jetty-solr-dependency-guard.sh` -3. Startup with embedded Solr enabled -4. A proven resolution for the Solr 9 isolation gate -5. A remote Solr request through the Apache-based client -6. Proxy traffic including CONNECT -7. `test/jetty-smoke-test.sh` - -The live gates are split by the environment they require: - -- `test/jetty-smoke-test.sh` checks HTTP methods, ranges, conditional requests, - error dispatch, and gzip request/response handling against a running peer; -- `test/jetty-auth-smoke-test.sh` checks localhost, `bin/apicall.sh`, optional - credentials, and an optional real non-loopback path; -- `test/jetty-peer-start-smoke-test.sh` starts and stops an isolated peer through - explicit harness commands and queries its embedded Solr core; -- `test/remote-solr-smoke-test.sh` queries an explicitly configured external - Solr instance through YaCy's Apache-HttpClient-backed `RemoteInstance`; -- `test/proxy-smoke-test.sh` checks HTTP proxy traffic and an HTTPS CONNECT - tunnel against explicitly configured controlled targets. - -The environment-dependent gates exit with status 2 when their required target -or isolated-peer harness has not been supplied. This is a reported skip, not a -successful verification. - -The final migration acceptance must run the authentication gate with no skips: - -```sh -YACY_SMOKE_REQUIRE_COMPLETE=true \ -YACY_SMOKE_ADMIN_USER=admin \ -YACY_SMOKE_ADMIN_PASSWORD='the configured password' \ -YACY_SMOKE_REMOTE_BASE_URL='http://a-real-non-loopback-peer-address:8090' \ -test/jetty-auth-smoke-test.sh -``` - -The HTTP range contract includes a single satisfiable range (`206`), multiple -satisfiable ranges as `multipart/byteranges`, and an unsatisfiable range -(`416`). - -### Switch-time logging tests - -`Slf4jJulBridgeTest` is version-neutral and must pass both before and after the -server switch. It proves that the public SLF4J 2 provider routes the -`org.eclipse.jetty` logger namespace into `java.util.logging` and therefore the -YaCy logging configuration. - -`Jetty9LoggingFacadeTest` is deliberately a Jetty 9 baseline test. It imports -Jetty 9's removed `org.eclipse.jetty.util.log.Log` API and asserts the old -`Slf4jLog` facade. Remove it together with `Jetty9HttpServerImpl` during the -switch and replace it with a Jetty 12 integration test that starts and stops a -real server while capturing an `org.eclipse.jetty` record through JUL. The -Jetty 12 test must not assert an internal logger implementation class. - -The following implementation phase may then replace `Jetty9HttpServerImpl` and -the remaining Jetty adapter APIs without changing the Solr dependency graph. - -## P2.1 Helper Removal - -`YaCyDefaultServlet` no longer imports Jetty HTTP header/method constants, -MIME lookup, URI joining, writer adaptation, inclusive byte ranges, multipart -output, or `Resource`. The small operations use Servlet/JDK APIs or local -implementations. Static resources are exposed through the container-neutral -`ServletResource` interface; `Jetty9ServletResource` is the only Jetty 9 -adapter for the existing resource behavior. - -`YaCyQoSFilter` and `YaCyDigestCredential` remain explicit container adapters -rather than being replaced by simplified local implementations that could -change request priority or authentication behavior. - -## P2.2 Authentication And Access Rules - -The request-level administrator decision is represented by -`AdminAccessPolicy`: public access, the configured localhost bypass, or the -administrator role. It combines the existing pure `AdminSecurity` checks -without depending on Servlet or Jetty APIs. In particular, the localhost -without account option and the localhost-only stored-hash authentication used -by `bin/apicall.sh` remain supported. - -`AdminAuthenticationContext` carries the true socket peer IP only for the -duration of the current authentication call. `YaCySecurityHandler` publishes -and clears that context, and `YaCyDigestCredential` only adapts Jetty's BASIC -and DIGEST credential objects to the container-neutral password check. - -The portable address/path syntax of `serverClient` is represented by -`InetPathAccessRule`. `InetPathAccessHandler` remains the Jetty 9 matcher -adapter; Jetty 12 can consume the normalized `address|path` rules with its -native path-aware access handler. - -## P2.3 Handler Boundaries - -Proxy request processing and cache processing no longer receive Jetty's -`Request`. `RequestCompletion` is the container-neutral signal that processing -is complete; `AbstractRemoteHandler` adapts it to Jetty 9's -`Request.setHandled(true)`. Consequently `ProxyHandler` and -`ProxyCacheHandler` have no Jetty imports. - -The `proxyClient` regular-expression list is evaluated by the pure -`ProxyAccessPolicy`. The local virtual-host cache used by proxy detection is a -concurrent set because it is populated by both the discovery thread and -request threads. - -The remaining Jetty handler classes now have explicit migration roles: - -| Jetty 9 adapter | Responsibility to reproduce with Jetty 12 | -| --- | --- | -| `AbstractRemoteHandler` | detect proxy traffic and delegate CONNECT tunnelling | -| `CrashProtectionHandler` | outer exception barrier around proxy and servlet handlers | -| `YacyDomainHandler` | rewrite `.yacy` destinations and redispatch into the proxy chain | -| `YaCyErrorHandler` | render the container error page | -| `YaCyQoSFilter` | optional request prioritization when enabled in `web.xml` | - -These classes intentionally remain container adapters. They must be ported -against the corresponding Jetty 12 APIs rather than replaced with servlet-only -approximations that would change CONNECT, error dispatch, or prioritization. - -## P2.4 Embedded Server Bootstrap Contract - -`HttpServerBootstrapConfig` is the common immutable input for Jetty 9 and the -future Jetty 12 implementation. It fixes the following startup values: - -| Concern | Contract | -| --- | --- | -| HTTP binding | constructor host and port | -| Acceptor threads | half the available processors, clamped to 1 through 4 | -| Request header limit | 16,384 bytes | -| Connector idle timeout | 9,000 ms | -| HTTP accept queue | 128 | -| HTTPS | `server.https`, configured SSL port, initialized SSL context only | -| Web root | configured `htRootPath` below the application directory | -| Descriptors | `defaults/web.xml`, optionally `DATA/SETTINGS/web.xml` | -| Request decompression | Gzip inflate buffer of 4,096 bytes | -| Response compression | controlled by `server.response.compress.gzip` | -| Form limit | unlimited at the proxy-handler context boundary | -| Proxy handlers | present only when transparent proxy is enabled | -| Network access | configured `serverClient` address/path rules plus loopback | -| Authentication realm | configured administrator realm, unchanged for DIGEST hashes | - -TLS preparation remains a YaCy bootstrap responsibility because it may import -a configured PKCS#12 file, create/update the JKS file, clear the one-shot -import settings, and construct the JDK `SSLContext`. The container adapter only -attaches that context to its HTTPS connector. - -The request pipeline order is a behavioral requirement: - -1. optional server-client address/path gate; -2. outer crash-protection barrier; -3. `.yacy` domain rewrite; -4. cached proxy response, when transparent proxy is enabled; -5. live HTTP proxy and CONNECT tunnel, when enabled; -6. root web application with monitor filter, admin security, gzip/inflate, and - `YaCyDefaultServlet`; -7. container default handler for requests left unhandled. - -The connection-close listener must remove the matching `ConnectionInfo` entry -created by `MonitorFilter`. The default servlet and monitor filter remain -hard-coded mandatory components; additional servlet mappings come from the -merged web descriptors. - -`YaCyHttpServer` defines the runtime contract used outside the adapter: - -- synchronous start; -- synchronous stop followed by join; -- asynchronous delayed port reconnect without rebuilding the handler graph; -- HTTPS availability and bound-port reporting; -- administrator identity eviction/reload after credential changes; -- container version reporting; -- current non-idle worker-thread count. - -A Jetty 12 implementation must first be added beside `Jetty9HttpServerImpl` -and satisfy this complete contract before the construction site in `yacy.java` -is switched. No caller outside the HTTP package should need a Jetty type or a -Jetty-version condition. @@ -39,6 +39,7 @@ <property name="libt" location="libt"/> <property name="build" location="build/classes/java/main"/> <!-- reuse Gradle build path --> <property name="ivy.compile.stage" location="build/ivy-retrieve/compile"/> + <property name="jetty12.test.classes" location="build/jetty12-server-tests"/> <property name="solr9.bridge.build" location="build/solr9-bridge"/> <property name="solr9.bridge.input" location="${solr9.bridge.build}/input"/> <property name="solr9.bridge.tool" location="${solr9.bridge.build}/tool"/> @@ -104,16 +105,16 @@ --> <delete dir="${ivy.compile.stage}" failonerror="false" /> <mkdir dir="${ivy.compile.stage}" /> - <ivy:retrieve conf="compile" pathid="compile.path" pattern="${ivy.compile.stage}/[artifact]-[revision].[ext]" /> + <ivy:retrieve conf="compile" type="jar,bundle" pathid="compile.path" pattern="${ivy.compile.stage}/[artifact]-[revision].[ext]" /> <delete failonerror="false"> <fileset dir="${lib}" includes="javax.servlet-api-*.jar,jetty-*.jar" /> </delete> <copy todir="${ivy.lib.dir}" overwrite="true"> <fileset dir="${ivy.compile.stage}" includes="**/*" /> </copy> - <ivy:retrieve conf="test" pathid="test.path" pattern="${libt}/[artifact]-[revision].[ext]" /> - <ivy:retrieve conf="solr9-bridge" pathid="solr9.bridge.input.path" pattern="${solr9.bridge.input}/[artifact]-[revision].[ext]" /> - <ivy:retrieve conf="solr9-bridge-tool" pathid="solr9.bridge.tool.path" pattern="${solr9.bridge.tool.libs}/[artifact]-[revision].[ext]" /> + <ivy:retrieve conf="test" type="jar,bundle" pathid="test.path" pattern="${libt}/[artifact]-[revision].[ext]" /> + <ivy:retrieve conf="solr9-bridge" type="jar,bundle" pathid="solr9.bridge.input.path" pattern="${solr9.bridge.input}/[artifact]-[revision].[ext]" /> + <ivy:retrieve conf="solr9-bridge-tool" type="jar,bundle" pathid="solr9.bridge.tool.path" pattern="${solr9.bridge.tool.libs}/[artifact]-[revision].[ext]" /> <property name="target-resolve-already-run" value="true" /> </target> @@ -257,7 +258,37 @@ <target name="compile" depends="compile-core" description="compile YaCy core and YaCy servlets" /> - <target name="all" depends="compile"> + <target name="jetty12-server-test" depends="compile" description="run focused Jetty 12 server tests"> + <delete dir="${jetty12.test.classes}" failonerror="false" /> + <mkdir dir="${jetty12.test.classes}" /> + <javac srcdir="${test}" destdir="${jetty12.test.classes}" + debug="true" debuglevel="lines,vars,source" includeantruntime="false" + release="${javacRelease}" encoding="UTF-8"> + <include name="net/yacy/http/Jetty12*Test.java" /> + <include name="net/yacy/http/servlets/ServletResourceTest.java" /> + <include name="net/yacy/http/servlets/Jetty12QoSFilterTest.java" /> + <classpath> + <pathelement location="${build}" /> + <path refid="compile.path" /> + <fileset dir="${libt}" includes="**/*.jar" /> + </classpath> + <compilerarg value="-Xlint:deprecation" /> + </javac> + <java classname="org.junit.runner.JUnitCore" fork="true" failonerror="true"> + <arg value="net.yacy.http.Jetty12HttpServerTest" /> + <arg value="net.yacy.http.Jetty12ProxyChainTest" /> + <arg value="net.yacy.http.servlets.ServletResourceTest" /> + <arg value="net.yacy.http.servlets.Jetty12QoSFilterTest" /> + <classpath> + <pathelement location="${jetty12.test.classes}" /> + <pathelement location="${build}" /> + <path refid="compile.path" /> + <fileset dir="${libt}" includes="**/*.jar" /> + </classpath> + </java> + </target> + + <target name="all" depends="jetty12-server-test"> </target> <target name="copyMain4Dist" depends="compile"> @@ -468,6 +499,7 @@ </delete> <delete dir="test/DATA" failonerror="false"/> <delete dir="${ivy.compile.stage}" failonerror="false"/> + <delete dir="${jetty12.test.classes}" failonerror="false"/> <delete dir="${solr9.bridge.build}" failonerror="false"/> </target> @@ -51,8 +51,8 @@ <dependency org="org.apache.commons" name="commons-lang3" rev="3.20.0" /> <dependency org="org.apache.httpcomponents" name="httpclient" rev="4.5.14"/> <dependency org="org.apache.httpcomponents" name="httpmime" rev="4.5.14"/> - <dependency org="org.apache.james" name="apache-mime4j-core" rev="0.8.14"/> - <dependency org="org.apache.james" name="apache-mime4j-dom" rev="0.8.14"/> + <dependency org="org.apache.james" name="apache-mime4j-core" rev="0.8.14" conf="compile->master"/> + <dependency org="org.apache.james" name="apache-mime4j-dom" rev="0.8.14" conf="compile->master"/> <dependency org="org.apache.lucene" name="lucene-analysis-common" rev="9.0.0"/> <dependency org="org.apache.lucene" name="lucene-backward-codecs" rev="9.0.0" /> <dependency org="org.apache.lucene" name="lucene-classification" rev="9.0.0" /> @@ -93,18 +93,34 @@ <dependency org="org.bitlet" name="weupnp" rev="0.1.4" /> <dependency org="org.bouncycastle" name="bcmail-jdk18on" rev="1.84" /> <dependency org="com.fasterxml.woodstox" name="woodstox-core" rev="7.2.1" /> - <dependency org="org.eclipse.jetty" name="jetty-http" rev="9.4.58.v20250814" conf="compile->default;solr9-bridge->master"/> - <dependency org="org.eclipse.jetty" name="jetty-io" rev="9.4.58.v20250814" conf="compile->default;solr9-bridge->master"/> + <!-- Jetty 12 Core and EE8 form the public embedded-server runtime. --> + <dependency org="org.eclipse.jetty" name="jetty-http" rev="12.1.11" conf="compile->default"/> + <dependency org="org.eclipse.jetty" name="jetty-io" rev="12.1.11" conf="compile->default"/> <dependency org="org.eclipse.jetty" name="jetty-client" rev="9.4.58.v20250814" conf="solr9-bridge->master"/> - <dependency org="org.eclipse.jetty" name="jetty-proxy" rev="9.4.58.v20250814"/> - <dependency org="org.eclipse.jetty" name="jetty-security" rev="9.4.58.v20250814"/> - <dependency org="org.eclipse.jetty" name="jetty-server" rev="9.4.58.v20250814"/> - <dependency org="org.eclipse.jetty" name="jetty-servlets" rev="9.4.58.v20250814"/> - <dependency org="org.eclipse.jetty" name="jetty-servlet" rev="9.4.58.v20250814"> - <exclude module="jetty-util-ajax" /> + <dependency org="org.eclipse.jetty" name="jetty-proxy" rev="12.1.11" conf="compile->default"/> + <dependency org="org.eclipse.jetty" name="jetty-security" rev="12.1.11" conf="compile->default"/> + <dependency org="org.eclipse.jetty" name="jetty-server" rev="12.1.11" conf="compile->default"/> + <dependency org="org.eclipse.jetty" name="jetty-util" rev="12.1.11" conf="compile->default" /> + <dependency org="org.eclipse.jetty.compression" name="jetty-compression-server" rev="12.1.11" conf="compile->default"/> + <dependency org="org.eclipse.jetty.ee8" name="jetty-ee8-nested" rev="12.1.11" conf="compile->default"> + <exclude org="org.eclipse.jetty.toolchain" module="jetty-servlet-api"/> </dependency> - <dependency org="org.eclipse.jetty" name="jetty-util" rev="9.4.58.v20250814" conf="compile->default;solr9-bridge->master" /> - <dependency org="org.eclipse.jetty" name="jetty-webapp" rev="9.4.58.v20250814" /> + <dependency org="org.eclipse.jetty.ee8" name="jetty-ee8-security" rev="12.1.11" conf="compile->default"> + <exclude org="org.eclipse.jetty.toolchain" module="jetty-servlet-api"/> + </dependency> + <dependency org="org.eclipse.jetty.ee8" name="jetty-ee8-servlet" rev="12.1.11" conf="compile->default"> + <exclude org="org.eclipse.jetty.toolchain" module="jetty-servlet-api"/> + </dependency> + <dependency org="org.eclipse.jetty.ee8" name="jetty-ee8-servlets" rev="12.1.11" conf="compile->default"> + <exclude org="org.eclipse.jetty.toolchain" module="jetty-servlet-api"/> + </dependency> + <dependency org="org.eclipse.jetty.ee8" name="jetty-ee8-webapp" rev="12.1.11" conf="compile->default"> + <exclude org="org.eclipse.jetty.toolchain" module="jetty-servlet-api"/> + </dependency> + <!-- Private Solr bridge inputs remain on Jetty 9. --> + <dependency org="org.eclipse.jetty" name="jetty-http" rev="9.4.58.v20250814" conf="solr9-bridge->master"/> + <dependency org="org.eclipse.jetty" name="jetty-io" rev="9.4.58.v20250814" conf="solr9-bridge->master"/> + <dependency org="org.eclipse.jetty" name="jetty-util" rev="9.4.58.v20250814" conf="solr9-bridge->master" /> <!-- Temporary Solr 9.0 runtime bridge: CoreContainer eagerly constructs an Http2SolrClient for its shard/update handlers even in standalone mode. diff --git a/source/net/yacy/htroot/SettingsAck_p.java b/source/net/yacy/htroot/SettingsAck_p.java index e53d833d5..9836c7435 100644 --- a/source/net/yacy/htroot/SettingsAck_p.java +++ b/source/net/yacy/htroot/SettingsAck_p.java @@ -37,7 +37,7 @@ import java.util.regex.PatternSyntaxException; import net.yacy.cora.protocol.RequestHeader; import net.yacy.data.TransactionManager; -import net.yacy.http.InetPathAccessHandler; +import net.yacy.http.Jetty12HttpServer; import net.yacy.kelondro.util.Formatter; import net.yacy.peers.Network; import net.yacy.peers.Seed; @@ -243,7 +243,7 @@ public class SettingsAck_p { while (st.hasMoreTokens()) { patternCount++; patternStr = st.nextToken(); - InetPathAccessHandler.checkPattern(patternStr); + Jetty12HttpServer.AccessRules.checkPattern(patternStr); } } catch (final IllegalArgumentException e) { prop.put("info", "27"); diff --git a/source/net/yacy/http/AbstractRemoteHandler.java b/source/net/yacy/http/AbstractRemoteHandler.java deleted file mode 100644 index 8f791c2a7..000000000 --- a/source/net/yacy/http/AbstractRemoteHandler.java +++ /dev/null @@ -1,163 +0,0 @@ -// -// AbstractRemoteHandler -// Copyright 2011 by Florian Richter -// First released 2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import java.io.IOException; -import java.net.InetAddress; -import java.util.Locale; -import java.util.Set; -import java.util.concurrent.ConcurrentHashMap; - -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import net.yacy.cora.protocol.Domains; -import net.yacy.cora.protocol.HeaderFramework; -import net.yacy.repository.Blacklist.BlacklistType; -import net.yacy.search.Switchboard; -import net.yacy.search.SwitchboardConstants; - -import org.eclipse.jetty.proxy.ConnectHandler; -import org.eclipse.jetty.server.Request; - -/** - * abstract jetty http handler - * only request to remote hosts (proxy requests) are processed by derived classes - */ -abstract public class AbstractRemoteHandler extends ConnectHandler { - - protected Switchboard sb = null; - private final Set<String> localVirtualHostNames = ConcurrentHashMap.newKeySet(); // updated by discovery thread and request threads - - @Override - protected void doStart() throws Exception { - super.doStart(); - this.sb = Switchboard.getSwitchboard(); - this.localVirtualHostNames.add("localhost"); - this.localVirtualHostNames.add(sb.getConfig("fileHost", "localpeer")); - - // Add some other known local host names - // The remote DNS sometimes takes very long when it is waiting for timeout, therefore we do this concurrently - new Thread(AbstractRemoteHandler.class.getSimpleName() + ".doStart") { - @Override - public void run() { - for (InetAddress localInetAddress : Domains.myPublicIPv4()) { - if (localInetAddress != null) { - if (!localVirtualHostNames.contains(localInetAddress.getHostName())) { - localVirtualHostNames.add(localInetAddress.getHostName()); - localVirtualHostNames.add(localInetAddress.getHostAddress()); // same as getServer().getURI().getHost() - } - - if (!localVirtualHostNames.contains(localInetAddress.getCanonicalHostName())) { - localVirtualHostNames.add(localInetAddress.getCanonicalHostName()); - } - } - } - for (InetAddress localInetAddress : Domains.myPublicIPv6()) { - if (localInetAddress != null) { - if (!localVirtualHostNames.contains(localInetAddress.getHostName())) { - localVirtualHostNames.add(localInetAddress.getHostName()); - localVirtualHostNames.add(localInetAddress.getHostAddress()); // same as getServer().getURI().getHost() - } - - if (!localVirtualHostNames.contains(localInetAddress.getCanonicalHostName())) { - localVirtualHostNames.add(localInetAddress.getCanonicalHostName()); - } - } - } - if (sb.peers != null) { - localVirtualHostNames.addAll(sb.peers.mySeed().getIPs()); - localVirtualHostNames.add(sb.peers.myAlternativeAddress()); // add the "peername.yacy" address - localVirtualHostNames.add(sb.peers.mySeed().getHexHash() + ".yacyh"); // bugfix by P. Dahl - } - } - }.start(); - } - - abstract public void handleRemote(String target, RequestCompletion completion, HttpServletRequest request, - HttpServletResponse response) throws IOException, ServletException; - - @Override - public void handle(String target, Request baseRequest, HttpServletRequest request, - HttpServletResponse response) throws IOException, ServletException { - - String host = request.getHeader("Host"); - if (host == null) return; // no proxy request, continue processing by handlers - String hostOnly = Domains.stripToHostName(host); - - if (localVirtualHostNames.contains(hostOnly)) return; // no proxy request (quick check), continue processing by handlers - if (Domains.isLocal(hostOnly, null)) return; // no proxy, continue processing by handlers - if (sb.peers.myIPs().contains(hostOnly)) { // remote access to my external IP, continue processing by handlers - localVirtualHostNames.addAll(sb.peers.myIPs()); // not available on init, add it now for quickcheck - return; - } - - InetAddress resolvedIP = Domains.dnsResolve(hostOnly); // during testing isLocal() failed to resolve domain against publicIP - if (resolvedIP != null && sb.myPublicIPs().contains(resolvedIP.getHostAddress())) { - localVirtualHostNames.add(resolvedIP.getHostName()); // remember resolved hostname - //localVirtualHostNames.add(resolved.getHostAddress()); // might change ? - return; - } - - // from here we can assume it is a proxy request - // should check proxy use permission - - if (!Switchboard.getSwitchboard().getConfigBool(SwitchboardConstants.PROXY_TRANSPARENT_PROXY, false)) { - // transparent proxy not swiched on - response.sendError(HttpServletResponse.SC_FORBIDDEN,"proxy use not allowed (see System Administration -> Advanced Settings -> Proxy Access Settings -> Transparent Proxy; switched off)."); - baseRequest.setHandled(true); - return; - } - - final String remoteHost = request.getRemoteHost(); - if (!ProxyAccessPolicy.isClientAllowed( - Switchboard.getSwitchboard().getConfig("proxyClient", "*"), remoteHost)) { - // TODO: handle proxy account - response.sendError(HttpServletResponse.SC_FORBIDDEN, - "proxy use not granted for IP " + remoteHost + " (see Advanced Settings -> Proxy Access Settings -> IP-Number filter)."); - baseRequest.setHandled(true); - return; - } - - // check the blacklist - if (Switchboard.urlBlacklist.isListed(BlacklistType.PROXY, hostOnly.toLowerCase(Locale.ROOT), request.getPathInfo())) { - response.sendError(HttpServletResponse.SC_FORBIDDEN, - "URL '" + hostOnly + "' blocked by yacy proxy (blacklisted)"); - baseRequest.setHandled(true); - return; - } - - if (request.getMethod().equalsIgnoreCase(HeaderFramework.METHOD_CONNECT)) { - // will be done by the ConnectHandler - super.handle(target, baseRequest, request, response); - return; - } - - handleRemote(target, () -> baseRequest.setHandled(true), request, response); - - } - -} diff --git a/source/net/yacy/http/AdminAccessPolicy.java b/source/net/yacy/http/AdminAccessPolicy.java deleted file mode 100644 index 5560200ec..000000000 --- a/source/net/yacy/http/AdminAccessPolicy.java +++ /dev/null @@ -1,80 +0,0 @@ -/** - * AdminAccessPolicy - * Copyright 2026 by Michael Peter Christen - * First released 12.07.2026 at https://yacy.net - * - * This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program in the file lgpl21.txt - * If not, see <http://www.gnu.org/licenses/>. - */ - -package net.yacy.http; - -import java.net.MalformedURLException; - -import net.yacy.cora.document.id.MultiProtocolURL; - -/** Container-neutral policy for administrator access to a request path. */ -public final class AdminAccessPolicy { - - public enum Decision { - PUBLIC, - LOCAL_BYPASS, - ADMIN_REQUIRED - } - - private final boolean protectAllPages; - private final boolean privateRobinsonMode; - private final boolean publicSearchPage; - private final boolean allowLocalhostWithoutLogin; - private final String adminUser; - private final String adminHash; - - public AdminAccessPolicy(final boolean protectAllPages, final boolean privateRobinsonMode, - final boolean publicSearchPage, final boolean allowLocalhostWithoutLogin, - final String adminUser, final String adminHash) { - this.protectAllPages = protectAllPages; - this.privateRobinsonMode = privateRobinsonMode; - this.publicSearchPage = publicSearchPage; - this.allowLocalhostWithoutLogin = allowLocalhostWithoutLogin; - this.adminUser = adminUser; - this.adminHash = adminHash; - } - - public Decision decide(final String path, final String socketPeerIp, final String referer, - final String authorizationHeader) { - if (!AdminSecurity.isProtectedPath(path, this.protectAllPages, - this.privateRobinsonMode, this.publicSearchPage)) { - return Decision.PUBLIC; - } - - if (AdminSecurity.isLocalhostAccess(socketPeerIp, refererHost(referer))) { - if (this.allowLocalhostWithoutLogin || AdminSecurity.checkLocalhostLazyAuth( - authorizationHeader, this.adminUser, this.adminHash)) { - return Decision.LOCAL_BYPASS; - } - } - return Decision.ADMIN_REQUIRED; - } - - private static String refererHost(final String referer) { - if (referer == null || referer.isEmpty()) { - return null; - } - try { - return new MultiProtocolURL(referer).getHost(); - } catch (final MalformedURLException e) { - return null; - } - } -} diff --git a/source/net/yacy/http/AdminSecurity.java b/source/net/yacy/http/AdminSecurity.java index 20cffa094..f213ef9de 100644 --- a/source/net/yacy/http/AdminSecurity.java +++ b/source/net/yacy/http/AdminSecurity.java @@ -20,6 +20,9 @@ package net.yacy.http; +import java.net.MalformedURLException; + +import net.yacy.cora.document.id.MultiProtocolURL; import net.yacy.cora.order.Base64Order; import net.yacy.cora.order.Digest; import net.yacy.cora.protocol.Domains; @@ -31,9 +34,14 @@ import net.yacy.cora.protocol.Domains; * * All methods are pure functions on their parameters, free of servlet * container (Jetty) and Switchboard dependencies: the container specific - * classes YaCySecurityHandler, YaCyLoginService and YaCyDigestCredential are - * thin adapters delegating here (extracted from them to ease servlet - * container migration). + * nested classes Jetty12HttpServer.AdminSecurityHandler, .AdminLoginService and + * .AdminCredential are thin adapters delegating here (extracted from them to + * ease servlet container migration). + * + * The complete container-neutral administrator security surface lives in this + * file: the pure check functions, the request-level {@link AccessPolicy} built + * on them, and the {@link AuthenticationContext} that carries the socket peer + * IP through the container's credential verification. */ public final class AdminSecurity { @@ -136,4 +144,81 @@ public final class AdminSecurity { public static String calcHash(final String pw) { return Digest.encodeMD5Hex(Base64Order.standardCoder.encodeString(pw)); } + + /** Container-neutral policy for administrator access to a request path. */ + public static final class AccessPolicy { + + public enum Decision { + PUBLIC, + LOCAL_BYPASS, + ADMIN_REQUIRED + } + + private final boolean protectAllPages; + private final boolean privateRobinsonMode; + private final boolean publicSearchPage; + private final boolean allowLocalhostWithoutLogin; + private final String adminUser; + private final String adminHash; + + public AccessPolicy(final boolean protectAllPages, final boolean privateRobinsonMode, + final boolean publicSearchPage, final boolean allowLocalhostWithoutLogin, + final String adminUser, final String adminHash) { + this.protectAllPages = protectAllPages; + this.privateRobinsonMode = privateRobinsonMode; + this.publicSearchPage = publicSearchPage; + this.allowLocalhostWithoutLogin = allowLocalhostWithoutLogin; + this.adminUser = adminUser; + this.adminHash = adminHash; + } + + public Decision decide(final String path, final String socketPeerIp, final String referer, + final String authorizationHeader) { + if (!AdminSecurity.isProtectedPath(path, this.protectAllPages, + this.privateRobinsonMode, this.publicSearchPage)) { + return Decision.PUBLIC; + } + + if (AdminSecurity.isLocalhostAccess(socketPeerIp, refererHost(referer))) { + if (this.allowLocalhostWithoutLogin || AdminSecurity.checkLocalhostLazyAuth( + authorizationHeader, this.adminUser, this.adminHash)) { + return Decision.LOCAL_BYPASS; + } + } + return Decision.ADMIN_REQUIRED; + } + + private static String refererHost(final String referer) { + if (referer == null || referer.isEmpty()) { + return null; + } + try { + return new MultiProtocolURL(referer).getHost(); + } catch (final MalformedURLException e) { + return null; + } + } + } + + /** Request-bound facts needed while the container verifies admin credentials. */ + public static final class AuthenticationContext { + + private static final ThreadLocal<String> SOCKET_PEER_IP = new ThreadLocal<>(); + + private AuthenticationContext() { + } + + public static void setSocketPeerIp(final String ip) { + SOCKET_PEER_IP.set(ip); + } + + public static void clear() { + SOCKET_PEER_IP.remove(); + } + + public static boolean isLocalhostRequest() { + final String ip = SOCKET_PEER_IP.get(); + return ip != null && Domains.isLocalhost(ip); + } + } } diff --git a/source/net/yacy/http/CrashProtectionHandler.java b/source/net/yacy/http/CrashProtectionHandler.java deleted file mode 100644 index 1456ba778..000000000 --- a/source/net/yacy/http/CrashProtectionHandler.java +++ /dev/null @@ -1,88 +0,0 @@ -/** - * CrashProtectionHandler - * Copyright 2026 by Michael Peter Christen - * First released 12.07.2026 at https://yacy.net - * - * This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program in the file lgpl21.txt - * If not, see <http://www.gnu.org/licenses/>. - */ - -package net.yacy.http; - -import java.io.IOException; -import java.io.PrintWriter; - -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import org.eclipse.jetty.server.Handler; -import org.eclipse.jetty.server.Request; -import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.server.handler.HandlerWrapper; - -import net.yacy.cora.util.ConcurrentLog; - -/** - * Last-resort exception barrier wrapped around the complete handler chain. - * - * Note for servlet container migration: this must stay a container level - * handler (it can not become a servlet filter): inside the servlet context a - * filter would catch servlet exceptions before the containers error dispatch - * and thereby replace the YaCyErrorHandler error page with a plain text stack - * trace. Its purpose is to catch failures outside the servlet context, e.g. - * in the transparent proxy handlers. - */ -public class CrashProtectionHandler extends HandlerWrapper implements Handler { - - public CrashProtectionHandler() { - super(); - } - - public CrashProtectionHandler(Server s, Handler h) { - super(); - this.setServer(s); - this.setHandler(h); - } - - - @Override - public void handle(String target, Request baseRequest, HttpServletRequest request, - HttpServletResponse response) throws IOException, ServletException { - try { - super.handle(target, baseRequest, request, response); - } catch (Exception e) { - ConcurrentLog.severe("HTTP", "event=http.request subsystem=http result=exception method=" + request.getMethod() + - " target=" + target + " status=500 reason=" + e.getMessage()); - // handle all we can - writeResponse(request, response, e); - baseRequest.setHandled(true); - } - } - - private void writeResponse(@SuppressWarnings("unused") HttpServletRequest request, HttpServletResponse response, Exception exc) throws IOException { - PrintWriter out; - try { // prevent exception after partial response (only getWriter not allowed if getOutputStream called before; Servlet API 3.0 ) - out = response.getWriter(); - } catch (IllegalStateException e) { - out = new PrintWriter(response.getOutputStream()); - } - out.println("Ops!"); - out.println(); - out.println("Message: " + exc.getMessage()); - exc.printStackTrace(out); - response.setContentType("text/plain"); - response.setStatus(HttpServletResponse.SC_INTERNAL_SERVER_ERROR); - } -} diff --git a/source/net/yacy/http/HttpServerBootstrapConfig.java b/source/net/yacy/http/HttpServerBootstrapConfig.java index f8ee311ef..0bcaade86 100644 --- a/source/net/yacy/http/HttpServerBootstrapConfig.java +++ b/source/net/yacy/http/HttpServerBootstrapConfig.java @@ -20,8 +20,18 @@ package net.yacy.http; +import java.io.FileInputStream; +import java.io.FileOutputStream; +import java.io.IOException; +import java.security.KeyStore; + +import javax.net.ssl.KeyManagerFactory; +import javax.net.ssl.SSLContext; + +import net.yacy.cora.util.ConcurrentLog; import net.yacy.search.Switchboard; import net.yacy.search.SwitchboardConstants; +import net.yacy.utils.PKCS12Tool; /** Immutable, servlet-container-neutral input for the embedded HTTP server. */ public final class HttpServerBootstrapConfig { @@ -30,7 +40,7 @@ public final class HttpServerBootstrapConfig { public static final long CONNECTOR_IDLE_TIMEOUT_MILLIS = 9_000L; public static final int ACCEPT_QUEUE_SIZE = 128; public static final int REQUEST_INFLATE_BUFFER_SIZE = 4_096; - public static final int MAX_FORM_CONTENT_SIZE = -1; + public static final int MAX_FORM_CONTENT_SIZE = 200_000; private final int httpPort; private final String bindHost; @@ -64,6 +74,19 @@ public final class HttpServerBootstrapConfig { this.adminRealm = adminRealm; } + public int httpPort() { return this.httpPort; } + public String bindHost() { return this.bindHost; } + public int acceptorCount() { return this.acceptorCount; } + public boolean httpsEnabled() { return this.httpsEnabled; } + public int httpsPort() { return this.httpsPort; } + public String htrootPath() { return this.htrootPath; } + public String defaultsWebXml() { return this.defaultsWebXml; } + public String overrideWebXml() { return this.overrideWebXml; } + public boolean gzipResponsesEnabled() { return this.gzipResponsesEnabled; } + public boolean transparentProxyEnabled() { return this.transparentProxyEnabled; } + public String serverClientRules() { return this.serverClientRules; } + public String adminRealm() { return this.adminRealm; } + public static HttpServerBootstrapConfig from(final Switchboard switchboard, final int httpPort, final String bindHost) { final int cores = Runtime.getRuntime().availableProcessors(); @@ -88,16 +111,113 @@ public final class HttpServerBootstrapConfig { return Math.max(1, Math.min(4, availableProcessors / 2)); } - public int httpPort() { return this.httpPort; } - public String bindHost() { return this.bindHost; } - public int acceptorCount() { return this.acceptorCount; } - public boolean httpsEnabled() { return this.httpsEnabled; } - public int httpsPort() { return this.httpsPort; } - public String htrootPath() { return this.htrootPath; } - public String defaultsWebXml() { return this.defaultsWebXml; } - public String overrideWebXml() { return this.overrideWebXml; } - public boolean gzipResponsesEnabled() { return this.gzipResponsesEnabled; } - public boolean transparentProxyEnabled() { return this.transparentProxyEnabled; } - public String serverClientRules() { return this.serverClientRules; } - public String adminRealm() { return this.adminRealm; } + /** Container-neutral preparation of the configured server TLS context. */ + final static class ServerTlsContextFactory { + + private ServerTlsContextFactory() { + } + + static SSLContext create(final Switchboard switchboard) { + String keyStoreFileName = switchboard.getConfig("keyStore", "").trim(); + String keyStorePassword = switchboard.getConfig("keyStorePassword", "").trim(); + final String pkcs12ImportFile = switchboard.getConfig("pkcs12ImportFile", "").trim(); + + if (keyStoreFileName.isEmpty() && keyStorePassword.isEmpty() && pkcs12ImportFile.isEmpty()) { + keyStoreFileName = "defaults/freeworldKeystore"; + keyStorePassword = "freeworld"; + switchboard.setConfig("keyStore", keyStoreFileName); + switchboard.setConfig("keyStorePassword", keyStorePassword); + } + + if (!pkcs12ImportFile.isEmpty()) { + ConcurrentLog.info("SERVER", "Import certificates from import file '" + pkcs12ImportFile + "'."); + try { + final String pkcs12ImportPassword = switchboard.getConfig("pkcs12ImportPwd", "").trim(); + final PKCS12Tool pkcsTool = new PKCS12Tool(pkcs12ImportFile, pkcs12ImportPassword); + if (keyStoreFileName.isEmpty()) { + keyStoreFileName = "DATA/SETTINGS/myPeerKeystore"; + final KeyStore keyStore = KeyStore.getInstance("JKS"); + keyStore.load(null, keyStorePassword.toCharArray()); + try (FileOutputStream output = new FileOutputStream(keyStoreFileName)) { + keyStore.store(output, keyStorePassword.toCharArray()); + } + switchboard.setConfig("keyStore", keyStoreFileName); + } + pkcsTool.importToJKS(keyStoreFileName, keyStorePassword); + switchboard.setConfig("pkcs12ImportFile", ""); + switchboard.setConfig("pkcs12ImportPwd", ""); + } catch (final Exception error) { + ConcurrentLog.severe("SERVER", + "Unable to import certificate from import file '" + pkcs12ImportFile + "'.", error); + } + } else if (keyStoreFileName.isEmpty()) { + return null; + } + + try { + ConcurrentLog.info("SERVER", "Initializing SSL support ..."); + final KeyStore keyStore = KeyStore.getInstance("JKS"); + try (FileInputStream input = new FileInputStream(keyStoreFileName)) { + keyStore.load(input, keyStorePassword.toCharArray()); + } catch (final IOException error) { + ConcurrentLog.warn("SERVER", "Could not read keystore file " + keyStoreFileName); + throw error; + } + final KeyManagerFactory keyManagers = KeyManagerFactory.getInstance( + KeyManagerFactory.getDefaultAlgorithm()); + keyManagers.init(keyStore, keyStorePassword.toCharArray()); + final SSLContext sslContext = SSLContext.getInstance("TLS"); + sslContext.init(keyManagers.getKeyManagers(), null, null); + return sslContext; + } catch (final Exception error) { + final String message = "FATAL ERROR: Unable to initialize the SSL Socket factory. " + + error.getMessage(); + ConcurrentLog.severe("SERVER", message); + System.out.println(message); + return null; + } + } + } + + + /** Container-neutral representation of a server-client address/path rule. */ + public static final class InetPathAccessRule { + + private static final String DEFAULT_PATH = "/*"; + + private final String addressPattern; + private final String pathPattern; + + private InetPathAccessRule(final String addressPattern, final String pathPattern) { + this.addressPattern = addressPattern; + this.pathPattern = pathPattern; + } + + public static InetPathAccessRule parse(final String pattern) { + if (pattern == null || pattern.isEmpty()) { + throw new IllegalArgumentException("Access rule must not be empty"); + } + final int separator = pattern.indexOf('|'); + final String address = separator > 0 ? pattern.substring(0, separator) : pattern; + final String path = separator > 0 && pattern.length() > separator + 1 + ? pattern.substring(separator + 1) + : DEFAULT_PATH; + if (address.isEmpty()) { + throw new IllegalArgumentException("Access rule has no address: " + pattern); + } + return new InetPathAccessRule(address, path); + } + + public String addressPattern() { + return this.addressPattern; + } + + public String pathPattern() { + return this.pathPattern; + } + + public String asJettyPattern() { + return this.addressPattern + '|' + this.pathPattern; + } + } } diff --git a/source/net/yacy/http/InetPathAccessHandler.java b/source/net/yacy/http/InetPathAccessHandler.java deleted file mode 100644 index 6d3651b83..000000000 --- a/source/net/yacy/http/InetPathAccessHandler.java +++ /dev/null @@ -1,162 +0,0 @@ -// InetPathAccessHandler.java -// Copyright 2017 by luccioman; https://github.com/luccioman -// -// This is a part of YaCy, a peer-to-peer based web search engine -// -// LICENSE -// -// This program is free software; you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation; either version 2 of the License, or -// (at your option) any later version. -// -// This program is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with this program; if not, write to the Free Software -// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -package net.yacy.http; - -import java.io.IOException; -import java.net.InetAddress; - -import org.eclipse.jetty.http.pathmap.MappedResource; -import org.eclipse.jetty.http.pathmap.PathMappings; -import org.eclipse.jetty.http.pathmap.PathSpec; -import org.eclipse.jetty.server.handler.InetAccessHandler; -import org.eclipse.jetty.util.InetAddressSet; -import org.eclipse.jetty.util.component.DumpableCollection; - -/** - * InetPathAccessHandler Access Handler - * <p> - * Extends {@link InetAccessHandler} by adding path patterns capabilities as - * previously available in the deprecated IPAccessHandler. - * </p> - * - * Note for servlet container migration: the InetAccessHandler of Jetty 10 and - * later supports path patterns natively ("addr|path" syntax), this class can - * then be removed. - */ -public class InetPathAccessHandler extends InetAccessHandler { - - /** List of white listed paths mapped to adresses sets */ - private final PathMappings<InetAddressSet> white = new PathMappings<>(); - - /** List of black listed paths mapped to adresses sets */ - private final PathMappings<InetAddressSet> black = new PathMappings<>(); - - /** - * @throws IllegalArgumentException when the pattern is malformed - */ - @Override - public void include(final String pattern) throws IllegalArgumentException { - addPattern(pattern, this.white); - } - - /** - * @throws IllegalArgumentException when a pattern is malformed - */ - @Override - public void include(final String... patterns) throws IllegalArgumentException { - for (final String pattern : patterns) { - include(pattern); - } - } - - /** - * @throws IllegalArgumentException when the pattern is malformed - */ - @Override - public void exclude(final String pattern) throws IllegalArgumentException { - addPattern(pattern, this.black); - } - - /** - * @throws IllegalArgumentException when a pattern is malformed - */ - @Override - public void exclude(final String... patterns) throws IllegalArgumentException { - for (final String pattern : patterns) { - exclude(pattern); - } - } - - /** - * Helper method to parse the new pattern and add it to the specified mapping. - * - * @param pattern - * a new pattern to process - * @param pathMappings - * target mapping from paths to addresses sets. Must not be null. - * @throws IllegalArgumentException - * when the pattern is malformed - */ - protected void addPattern(final String pattern, final PathMappings<InetAddressSet> pathMappings) - throws IllegalArgumentException { - final InetPathAccessRule rule = InetPathAccessRule.parse(pattern); - final PathSpec pathSpec = PathSpec.from(rule.pathPattern()); - InetAddressSet addresses = pathMappings.get(pathSpec); - if (addresses == null) { - addresses = new InetAddressSet(); - pathMappings.put(pathSpec, addresses); - } - addresses.add(rule.addressPattern()); - } - - /** - * Helper method to check pattern syntax. - * - * @param pattern pattern to check for syntax errors - * @throws IllegalArgumentException - * when the pattern is malformed - */ - public static void checkPattern(final String pattern) throws IllegalArgumentException { - new InetPathAccessHandler().include(pattern); - } - /** - * Check whether the given address and path are allowed by current rules. - * - * @param address - * the address to check - * @param path - * an eventual path string starting with "/" - * @return true when allowed - */ - protected boolean isAllowed(final InetAddress address, final String path) { - boolean allowed = true; - final String nonNullPath = path != null ? path : "/"; - if (this.white.size() > 0) { - /* Non empty white list patterns : MUST match at least one of it */ - allowed = false; - for (final MappedResource<InetAddressSet> mapping : this.white.getMatches(nonNullPath)) { - if (mapping.getResource().test(address)) { - allowed = true; - break; - } - } - } - if (allowed) { - /* Finally check against black list patterns even when the first step passed */ - for (final MappedResource<InetAddressSet> mapping : this.black.getMatches(nonNullPath)) { - if (mapping.getResource().test(address)) { - allowed = false; - break; - } - } - } - return allowed; - } - - @Override - public void dump(final Appendable out, final String indent) throws IOException { - dumpObjects(out, indent, - DumpableCollection.from("white", this.white.getMappings()), - DumpableCollection.from("black", this.black.getMappings())); - } - -} diff --git a/source/net/yacy/http/InetPathAccessRule.java b/source/net/yacy/http/InetPathAccessRule.java deleted file mode 100644 index c93200829..000000000 --- a/source/net/yacy/http/InetPathAccessRule.java +++ /dev/null @@ -1,62 +0,0 @@ -/** - * InetPathAccessRule - * Copyright 2026 by Michael Peter Christen - * First released 12.07.2026 at https://yacy.net - * - * This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program in the file lgpl21.txt - * If not, see <http://www.gnu.org/licenses/>. - */ - -package net.yacy.http; - -/** Container-neutral representation of a server-client address/path rule. */ -public final class InetPathAccessRule { - - private static final String DEFAULT_PATH = "/*"; - - private final String addressPattern; - private final String pathPattern; - - private InetPathAccessRule(final String addressPattern, final String pathPattern) { - this.addressPattern = addressPattern; - this.pathPattern = pathPattern; - } - - public static InetPathAccessRule parse(final String pattern) { - if (pattern == null || pattern.isEmpty()) { - throw new IllegalArgumentException("Access rule must not be empty"); - } - final int separator = pattern.indexOf('|'); - final String address = separator > 0 ? pattern.substring(0, separator) : pattern; - final String path = separator > 0 && pattern.length() > separator + 1 - ? pattern.substring(separator + 1) - : DEFAULT_PATH; - if (address.isEmpty()) { - throw new IllegalArgumentException("Access rule has no address: " + pattern); - } - return new InetPathAccessRule(address, path); - } - - public String addressPattern() { - return this.addressPattern; - } - - public String pathPattern() { - return this.pathPattern; - } - - public String asJettyPattern() { - return this.addressPattern + '|' + this.pathPattern; - } -} diff --git a/source/net/yacy/http/Jetty12HttpServer.java b/source/net/yacy/http/Jetty12HttpServer.java new file mode 100644 index 000000000..16590266c --- /dev/null +++ b/source/net/yacy/http/Jetty12HttpServer.java @@ -0,0 +1,657 @@ +/** + * Jetty12HttpServer + * Copyright 2026 by Michael Peter Christen + * First released 12.07.2026 at https://yacy.net + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program in the file lgpl21.txt + * If not, see <http://www.gnu.org/licenses/>. + */ + +package net.yacy.http; + +import java.io.IOException; +import java.io.Writer; +import java.net.InetAddress; +import java.net.InetSocketAddress; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.EnumSet; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.function.BooleanSupplier; +import java.util.function.Supplier; + +import javax.net.ssl.SSLContext; +import javax.servlet.DispatcherType; +import javax.servlet.ServletException; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.eclipse.jetty.compression.gzip.GzipCompression; +import org.eclipse.jetty.compression.gzip.GzipDecoderConfig; +import org.eclipse.jetty.compression.server.CompressionConfig; +import org.eclipse.jetty.compression.server.CompressionHandler; +import org.eclipse.jetty.ee8.security.ConstraintSecurityHandler; +import org.eclipse.jetty.ee8.security.RoleInfo; +import org.eclipse.jetty.ee8.security.authentication.DigestAuthenticator; +import org.eclipse.jetty.ee8.servlet.FilterHolder; +import org.eclipse.jetty.ee8.servlet.ServletHolder; +import org.eclipse.jetty.ee8.webapp.WebAppContext; +import org.eclipse.jetty.http.HttpMethod; +import org.eclipse.jetty.http.HttpVersion; +import org.eclipse.jetty.http.MimeTypes; +import org.eclipse.jetty.io.Connection; +import org.eclipse.jetty.security.HashLoginService; +import org.eclipse.jetty.security.UserPrincipal; +import org.eclipse.jetty.security.UserStore; +import org.eclipse.jetty.server.Connector; +import org.eclipse.jetty.server.Handler; +import org.eclipse.jetty.server.HttpConfiguration; +import org.eclipse.jetty.server.HttpConnectionFactory; +import org.eclipse.jetty.server.Request; +import org.eclipse.jetty.server.Response; +import org.eclipse.jetty.server.SecureRequestCustomizer; +import org.eclipse.jetty.server.Server; +import org.eclipse.jetty.server.ServerConnector; +import org.eclipse.jetty.server.SslConnectionFactory; +import org.eclipse.jetty.server.handler.InetAccessHandler; +import org.eclipse.jetty.util.Callback; +import org.eclipse.jetty.util.security.Credential; +import org.eclipse.jetty.util.ssl.SslContextFactory; + +import net.yacy.cora.protocol.ConnectionInfo; +import net.yacy.cora.protocol.RequestHeader; +import net.yacy.cora.util.ConcurrentLog; +import net.yacy.http.servlets.MonitorFilter; +import net.yacy.http.servlets.YaCyDefaultServlet; +import net.yacy.peers.operation.yacyBuildProperties; +import net.yacy.search.Switchboard; +import net.yacy.search.SwitchboardConstants; +import net.yacy.server.serverAccessTracker; + +/** + * The complete Jetty 12 server adapter behind {@link YaCyHttpServer}. + * + * <p>Every Jetty-12-specific class of the default (non-proxy) runtime is a + * static nested class of this file: bootstrap and connectors, administrator + * security, access rules, crash protection and the error page. This is the + * single surface to port at the next servlet-container migration; the optional + * transparent-proxy handlers live in {@link Jetty12ProxyChain}.</p> + */ +public class Jetty12HttpServer implements YaCyHttpServer { + + private final Server server; + private final AdminLoginService loginService; + + public Jetty12HttpServer(final int port, final String host) { + final Switchboard switchboard = Switchboard.getSwitchboard(); + final HttpServerBootstrapConfig bootstrap = HttpServerBootstrapConfig.from(switchboard, port, host); + final SSLContext sslContext = bootstrap.httpsEnabled() + ? HttpServerBootstrapConfig.ServerTlsContextFactory.create(switchboard) + : null; + this.loginService = new AdminLoginService(); + this.loginService.setName(bootstrap.adminRealm()); + this.server = createServer(bootstrap.httpPort(), bootstrap.bindHost(), bootstrap.acceptorCount(), + sslContext, bootstrap.httpsPort()); + Handler requestPipeline = createWebAppHandler(this.server, bootstrap, this.loginService); + if (bootstrap.transparentProxyEnabled()) { + requestPipeline = Jetty12ProxyChain.wrap(requestPipeline, switchboard); + } else { + requestPipeline = new DisabledProxyHandler(requestPipeline); + } + final Handler crashProtected = new CrashProtectionHandler(requestPipeline); + this.server.setHandler(AccessRules.wrap(crashProtected, bootstrap.serverClientRules())); + } + + Jetty12HttpServer(final int port, final String host, final int acceptorCount, + final SSLContext sslContext, final int sslPort) { + this.loginService = null; + this.server = createServer(port, host, acceptorCount, sslContext, sslPort); + } + + private static Server createServer(final int port, final String host, final int acceptorCount, + final SSLContext sslContext, final int sslPort) { + final Server server = new Server(); + final Connection.Listener connectionCloseMonitor = new Connection.Listener() { + @Override + public void onClosed(final Connection connection) { + if (connection.getEndPoint().getRemoteSocketAddress() instanceof InetSocketAddress) { + final InetSocketAddress remote = + (InetSocketAddress) connection.getEndPoint().getRemoteSocketAddress(); + ConnectionInfo.removeServerConnection(MonitorFilter.connectionId( + remote.getAddress().getHostAddress(), remote.getPort())); + } + } + }; + + final HttpConfiguration httpConfiguration = new HttpConfiguration(); + httpConfiguration.setRequestHeaderSize(HttpServerBootstrapConfig.REQUEST_HEADER_SIZE); + final ServerConnector httpConnector = new ServerConnector(server, acceptorCount, -1, + new HttpConnectionFactory(httpConfiguration)); + configureConnector(httpConnector, host, port, "httpd-" + host + ":" + port, + connectionCloseMonitor); + httpConnector.setAcceptQueueSize(HttpServerBootstrapConfig.ACCEPT_QUEUE_SIZE); + server.addConnector(httpConnector); + + if (sslContext != null) { + final SslContextFactory.Server sslContextFactory = new SslContextFactory.Server(); + sslContextFactory.setSslContext(sslContext); + final HttpConfiguration httpsConfiguration = new HttpConfiguration(httpConfiguration); + httpsConfiguration.addCustomizer(createSecureRequestCustomizer()); + final ServerConnector httpsConnector = new ServerConnector(server, acceptorCount, -1, + new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString()), + new HttpConnectionFactory(httpsConfiguration)); + configureConnector(httpsConnector, host, sslPort, "ssld:" + sslPort, + connectionCloseMonitor); + httpsConnector.setAcceptQueueSize(HttpServerBootstrapConfig.ACCEPT_QUEUE_SIZE); + server.addConnector(httpsConnector); + } + return server; + } + + static SecureRequestCustomizer createSecureRequestCustomizer() { + final SecureRequestCustomizer secureRequests = new SecureRequestCustomizer(); + // Preserve Jetty 9 behavior and support YaCy's configurable/self-signed certificates, + // whose subject does not necessarily match the requested peer hostname. + secureRequests.setSniHostCheck(false); + return secureRequests; + } + + private static void configureConnector(final ServerConnector connector, final String host, + final int port, final String name, final Connection.Listener listener) { + connector.setHost(host); + connector.setPort(port); + connector.setName(name); + connector.setIdleTimeout(HttpServerBootstrapConfig.CONNECTOR_IDLE_TIMEOUT_MILLIS); + connector.addBean(listener); + } + + private static Handler createWebAppHandler(final Server server, + final HttpServerBootstrapConfig bootstrap, + final AdminLoginService loginService) { + final WebAppContext webApp = new WebAppContext(); + webApp.setServer(server); + webApp.setContextPath("/"); + webApp.setMaxFormContentSize(HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE); + webApp.setErrorHandler(new ErrorPageHandler()); + webApp.setBaseResource(webApp.getResourceFactory().newResource(Path.of(bootstrap.htrootPath()))); + webApp.setDefaultsDescriptor(bootstrap.defaultsWebXml()); + if (Files.exists(Path.of(bootstrap.overrideWebXml()))) { + webApp.setDescriptor(bootstrap.overrideWebXml()); + } + + final ServletHolder defaultServlet = new ServletHolder(YaCyDefaultServlet.class); + defaultServlet.setInitParameter("resourceBase", bootstrap.htrootPath()); + defaultServlet.setAsyncSupported(true); + webApp.addServlet(defaultServlet, "/*"); + + final FilterHolder monitorFilter = new FilterHolder(MonitorFilter.class); + monitorFilter.setAsyncSupported(true); + webApp.addFilter(monitorFilter, "/*", EnumSet.of(DispatcherType.REQUEST)); + + final AdminSecurityHandler security = new AdminSecurityHandler(); + security.setLoginService(loginService); + webApp.setSecurityHandler(security); + + final CompressionHandler compressionHandler = new CompressionHandler(webApp.get()); + final GzipCompression gzip = new GzipCompression(); + final GzipDecoderConfig decoder = new GzipDecoderConfig(); + decoder.setBufferSize(HttpServerBootstrapConfig.REQUEST_INFLATE_BUFFER_SIZE); + gzip.setDefaultDecoderConfig(decoder); + compressionHandler.putCompression(gzip); + compressionHandler.putConfiguration("/*", + createCompressionConfig(bootstrap.gzipResponsesEnabled())); + return compressionHandler; + } + + static CompressionConfig createCompressionConfig(final boolean gzipResponsesEnabled) { + final CompressionConfig.Builder compression = CompressionConfig.builder() + .compressIncludeMethod("GET") + .decompressIncludeMethod("POST"); + for (final String type : MimeTypes.DEFAULTS.getMimeMap().values()) { + if ("image/svg+xml".equals(type)) { + compression.compressExcludePath("*.svgz").decompressExcludePath("*.svgz"); + } else if (type.startsWith("image/") || type.startsWith("audio/") + || type.startsWith("video/")) { + compression.compressExcludeMimeType(type).decompressExcludeMimeType(type); + } + } + for (final String type : new String[] {"application/compress", "application/zip", + "application/gzip", "application/bzip2", "application/brotli", + "application/x-xz", "application/x-rar-compressed"}) { + compression.compressExcludeMimeType(type).decompressExcludeMimeType(type); + } + if (!gzipResponsesEnabled) { + compression.compressExcludeMethod("GET").compressExcludeMethod("POST"); + } + return compression.build(); + } + + @Override + public void startupServer() throws Exception { + this.server.setStopAtShutdown(true); + this.server.start(); + } + + @Override + public void stop() throws Exception { + this.server.stop(); + this.server.join(); + } + + @Override + public void reconnect(final int milliseconds) { + new Thread(() -> { + if (milliseconds > 0) { + try { + Thread.sleep(milliseconds); + } catch (final InterruptedException interrupted) { + Thread.currentThread().interrupt(); + return; + } + } + try { + if (!this.server.isRunning() || this.server.isStopped()) { + this.server.start(); + } + final int httpPort = Switchboard.getSwitchboard().getLocalPort(); + final int httpsPort = Switchboard.getSwitchboard().getConfigInt( + SwitchboardConstants.SERVER_SSLPORT, 8443); + for (final Connector connector : this.server.getConnectors()) { + final ServerConnector networkConnector = (ServerConnector) connector; + final int desiredPort = connector.getName().startsWith("ssl") ? httpsPort : httpPort; + if (networkConnector.getPort() != desiredPort) { + networkConnector.close(); + networkConnector.stop(); + networkConnector.setPort(desiredPort); + networkConnector.start(); + ConcurrentLog.info("SERVER", "set new port for Jetty connector " + connector.getName()); + } + } + } catch (final Exception error) { + ConcurrentLog.logException(error); + } + }, "Jetty12HttpServer.reconnect").start(); + } + + @Override + public boolean withSSL() { + for (final Connector connector : this.server.getConnectors()) { + if (connector.getName().startsWith("ssl")) { + return true; + } + } + return false; + } + + @Override + public int getSslPort() { + for (final Connector connector : this.server.getConnectors()) { + if (connector.getName().startsWith("ssl")) { + return ((ServerConnector) connector).getLocalPort(); + } + } + return -1; + } + + int getHttpPort() { + for (final Connector connector : this.server.getConnectors()) { + if (connector.getName().startsWith("httpd")) { + return ((ServerConnector) connector).getLocalPort(); + } + } + return -1; + } + + @Override + public void resetUser(final String username) { + if (this.loginService != null) { + this.loginService.reloadUser(username); + } + } + + @Override + public void removeUser(final String username) { + if (this.loginService != null) { + this.loginService.removeCachedUser(username); + } + } + + @Override + public String getVersion() { + return "Jetty " + Server.getVersion(); + } + + @Override + public int getServerThreads() { + return this.server.getThreadPool().getThreads() - this.server.getThreadPool().getIdleThreads(); + } + + @Override + public String toString() { + return this.server.dump() + "\n\n" + this.server.getState(); + } + + /** Adapts YaCy's address/path rules to Jetty 12's native access handler. */ + public static final class AccessRules { + + private AccessRules() { + } + + /** Validate one configured address/path expression with Jetty's active parser. */ + public static void checkPattern(final String pattern) { + final InetAccessHandler validator = new InetAccessHandler(); + validator.include(HttpServerBootstrapConfig.InetPathAccessRule.parse(pattern).asJettyPattern()); + } + + static Handler wrap(final Handler next, final String configuredRules) { + if (configuredRules == null || "*".equals(configuredRules.trim())) { + return next; + } + final InetAccessHandler access = new InetAccessHandler(next); + int accepted = 0; + for (final String configuredRule : configuredRules.split(",")) { + final String pattern = configuredRule.trim(); + if (pattern.isEmpty()) { + continue; + } + try { + final HttpServerBootstrapConfig.InetPathAccessRule rule = HttpServerBootstrapConfig.InetPathAccessRule.parse(pattern); + access.include(rule.asJettyPattern()); + accepted++; + } catch (final IllegalArgumentException error) { + ConcurrentLog.severe("SERVER", "Server Access Settings - IP filter: " + error.getMessage()); + } + } + if (accepted == 0) { + return next; + } + final String loopbackAddress = InetAddress.getLoopbackAddress().getHostAddress(); + access.include(loopbackAddress); + ConcurrentLog.info("SERVER", "activated IP access restriction to: [" + + loopbackAddress + "," + configuredRules + "]"); + return access; + } + } + + /** Last-resort exception barrier around Jetty 12's complete request pipeline. */ + static final class CrashProtectionHandler extends Handler.Wrapper { + + CrashProtectionHandler(final Handler next) { + super(next); + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) + throws Exception { + final AtomicBoolean completed = new AtomicBoolean(); + final Callback protectedCallback = new Callback() { + @Override + public void succeeded() { + if (completed.compareAndSet(false, true)) { + callback.succeeded(); + } + } + + @Override + public void failed(final Throwable failure) { + handleFailure(request, response, callback, completed, failure); + } + }; + try { + return super.handle(request, response, protectedCallback); + } catch (final Throwable failure) { + handleFailure(request, response, callback, completed, failure); + return true; + } + } + + private static void handleFailure(final Request request, final Response response, + final Callback callback, final AtomicBoolean completed, final Throwable failure) { + if (!completed.compareAndSet(false, true)) { + return; + } + ConcurrentLog.severe("HTTP", "event=http.request subsystem=http result=exception method=" + + request.getMethod() + " target=" + request.getHttpURI().getPath() + + " status=500 reason=" + failure.getMessage()); + if (response.isCommitted()) { + callback.failed(failure); + return; + } + Response.writeError(request, response, callback, 500, "Internal Server Error"); + } + } + + /** Rejects CONNECT explicitly when transparent proxy support is disabled. */ + static final class DisabledProxyHandler extends Handler.Wrapper { + + private static final String REJECTION_MESSAGE = "Transparent proxy is disabled"; + private static final String REJECTION_HEADER = "X-YaCy-Proxy-Error"; + + DisabledProxyHandler(final Handler next) { + super(next); + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) + throws Exception { + if (HttpMethod.CONNECT.is(request.getMethod())) { + response.getHeaders().put(REJECTION_HEADER, REJECTION_MESSAGE); + Response.writeError(request, response, callback, 403, REJECTION_MESSAGE); + return true; + } + return super.handle(request, response, callback); + } + } + + /** YaCy-branded EE8 error page for Jetty 12. */ + static final class ErrorPageHandler extends org.eclipse.jetty.ee8.nested.ErrorHandler { + + @Override + protected void writeErrorPageBody(final HttpServletRequest request, final Writer writer, + final int code, final String message, final boolean showStacks) throws IOException { + final String uri = request.getRequestURI(); + this.writeErrorPageMessage(request, writer, code, message, uri); + if (showStacks) { + this.writeErrorPageStacks(request, writer); + } + writer.write("<br/><hr /><small>YaCy " + yacyBuildProperties.getVersion() + + " - <i> powered by Jetty </i> - </small>"); + for (int i = 0; i < 20; i++) { + writer.write("<br/> \n"); + } + } + } + + /** Jetty 12 EE8 adapter for YaCy's container-neutral administrator policy. */ + static final class AdminSecurityHandler extends ConstraintSecurityHandler { + + @Override + protected void doStart() throws Exception { + if (getAuthenticator() == null && "DIGEST".equalsIgnoreCase(getAuthMethod())) { + setAuthenticator(createDigestAuthenticator()); + } + super.doStart(); + } + + static DigestAuthenticator createDigestAuthenticator() { + final DigestAuthenticator authenticator = new DigestAuthenticator(); + // YaCy stores the RFC 2617 MD5 HA1 value, not the clear-text password. + authenticator.setAlgorithm("MD5"); + return authenticator; + } + + @Override + public void handle(final String pathInContext, + final org.eclipse.jetty.ee8.nested.Request baseRequest, + final HttpServletRequest request, final HttpServletResponse response) + throws IOException, ServletException { + AdminSecurity.AuthenticationContext.setSocketPeerIp(baseRequest.getRemoteAddr()); + try { + super.handle(pathInContext, baseRequest, request, response); + } finally { + AdminSecurity.AuthenticationContext.clear(); + } + } + + @Override + protected RoleInfo prepareConstraintInfo(final String pathInContext, + final org.eclipse.jetty.ee8.nested.Request request) { + final Switchboard switchboard = Switchboard.getSwitchboard(); + final String remoteIp = request.getRemoteAddr(); + serverAccessTracker.track(remoteIp, pathInContext); + final AdminSecurity.AccessPolicy policy = new AdminSecurity.AccessPolicy( + switchboard.getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_All_PAGES, false), + switchboard.isRobinsonMode() && !switchboard.isPublicRobinson(), + switchboard.getConfigBool(SwitchboardConstants.PUBLIC_SEARCHPAGE, true), + switchboard.getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_FOR_LOCALHOST, false), + switchboard.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin"), + switchboard.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "")); + final AdminSecurity.AccessPolicy.Decision decision = policy.decide(pathInContext, remoteIp, + request.getHeader(RequestHeader.REFERER), + request.getHeader(RequestHeader.AUTHORIZATION)); + if (decision == AdminSecurity.AccessPolicy.Decision.PUBLIC) { + return super.prepareConstraintInfo(pathInContext, request); + } + if (decision == AdminSecurity.AccessPolicy.Decision.LOCAL_BYPASS) { + return null; + } + final RoleInfo roleInfo = new RoleInfo(); + roleInfo.setChecked(true); + roleInfo.addRole(SwitchboardConstants.ADMIN_ACCOUNT_ROLE); + return roleInfo; + } + } + + /** Jetty 12 login-service adapter for YaCy's single built-in administrator. */ + static final class AdminLoginService extends HashLoginService { + + record AdminCredentialConfig(String username, String hash, String realm) { + } + + private final Supplier<AdminCredentialConfig> credentials; + private UserStore userStore; + + AdminLoginService() { + this(AdminLoginService::configuredCredentials); + } + + AdminLoginService(final Supplier<AdminCredentialConfig> credentials) { + this.credentials = credentials; + } + + private static AdminCredentialConfig configuredCredentials() { + final Switchboard switchboard = Switchboard.getSwitchboard(); + return new AdminCredentialConfig( + switchboard.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin"), + switchboard.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, ""), + switchboard.getConfig(SwitchboardConstants.ADMIN_REALM, "")); + } + + @Override + protected void doStart() throws Exception { + this.userStore = new UserStore(); + this.setUserStore(this.userStore); + super.doStart(); + } + + @Override + protected void doStop() throws Exception { + super.doStop(); + if (this.userStore != null) { + this.userStore.stop(); + this.userStore = null; + } + } + + @Override + protected UserPrincipal loadUserInfo(final String username) { + if (username == null || username.isEmpty()) { + return null; + } + UserPrincipal user = super.loadUserInfo(username); + if (user != null || this.userStore == null) { + return user; + } + final AdminCredentialConfig configured = this.credentials.get(); + if (!username.equals(configured.username())) { + return null; + } + final AdminCredential credential = new AdminCredential( + username, configured.hash(), configured.realm(), configured.username()); + this.userStore.addUser(username, credential, + new String[] {SwitchboardConstants.ADMIN_ACCOUNT_ROLE}); + user = this.userStore.getUserPrincipal(username); + return user; + } + + synchronized boolean removeCachedUser(final String username) { + if (this.userStore == null || this.userStore.getUserPrincipal(username) == null) { + return false; + } + this.userStore.removeUser(username); + return true; + } + + synchronized void reloadUser(final String username) { + this.removeCachedUser(username); + this.loadUserInfo(username); + } + } + + /** Jetty 12 credential facade over YaCy's container-neutral password verifier. */ + static final class AdminCredential extends Credential { + + private static final long serialVersionUID = 1L; + + private final String username; + private final String configuredHash; + private final String realm; + private final String configuredAdminUser; + private final Credential digestCredential; + private final BooleanSupplier localhostRequest; + + AdminCredential(final String username, final String configuredHash, + final String realm, final String configuredAdminUser) { + this(username, configuredHash, realm, configuredAdminUser, + AdminSecurity.AuthenticationContext::isLocalhostRequest); + } + + AdminCredential(final String username, final String configuredHash, + final String realm, final String configuredAdminUser, + final BooleanSupplier localhostRequest) { + this.username = username; + this.configuredHash = configuredHash; + this.realm = realm; + this.configuredAdminUser = configuredAdminUser; + this.localhostRequest = localhostRequest; + this.digestCredential = configuredHash.startsWith("MD5:") + ? Credential.getCredential(configuredHash) + : null; + } + + @Override + public boolean check(final Object credentials) { + if (credentials instanceof Credential) { + return this.digestCredential != null + && ((Credential) credentials).check(this.digestCredential); + } + if (credentials instanceof String) { + return AdminSecurity.checkAdminPassword(this.username, this.configuredHash, + this.realm, this.configuredAdminUser, + this.localhostRequest.getAsBoolean(), (String) credentials); + } + throw new UnsupportedOperationException("Unsupported administrator credential type"); + } + } +} diff --git a/source/net/yacy/http/Jetty12ProxyChain.java b/source/net/yacy/http/Jetty12ProxyChain.java new file mode 100644 index 000000000..136918ad2 --- /dev/null +++ b/source/net/yacy/http/Jetty12ProxyChain.java @@ -0,0 +1,614 @@ +/** + * Jetty12ProxyChain + * Copyright 2026 by Michael Peter Christen + * First released 12.07.2026 at https://yacy.net + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program in the file lgpl21.txt + * If not, see <http://www.gnu.org/licenses/>. + */ + +package net.yacy.http; + +import java.io.ByteArrayOutputStream; +import java.io.IOException; +import java.nio.ByteBuffer; +import java.util.Date; +import java.util.Locale; +import java.util.function.Predicate; + +import org.eclipse.jetty.client.HttpClient; +import org.eclipse.jetty.client.Result; +import org.eclipse.jetty.http.HttpField; +import org.eclipse.jetty.http.HttpFields; +import org.eclipse.jetty.http.HttpHeader; +import org.eclipse.jetty.http.HttpMethod; +import org.eclipse.jetty.http.HttpURI; +import org.eclipse.jetty.proxy.ProxyHandler; +import org.eclipse.jetty.server.Handler; +import org.eclipse.jetty.server.Request; +import org.eclipse.jetty.server.Response; +import org.eclipse.jetty.util.Callback; +import org.eclipse.jetty.util.HostPort; + +import net.yacy.cora.date.GenericFormatter; +import net.yacy.cora.document.id.DigestURL; +import net.yacy.cora.protocol.Domains; +import net.yacy.cora.protocol.RequestHeader; +import net.yacy.cora.protocol.ResponseHeader; +import net.yacy.crawler.data.Cache; +import net.yacy.document.TextParser; +import net.yacy.repository.Blacklist.BlacklistType; +import net.yacy.search.Switchboard; +import net.yacy.server.http.AlternativeDomainNames; +import net.yacy.server.http.HTTPDProxyHandler; + +/** + * The complete Jetty 12 transparent-proxy handler chain. + * + * <p>Every handler of the optional transparent-proxy feature is a static + * nested class of this file; {@link #wrap(Handler, Switchboard)} composes them + * in the contract order: {@code .yacy} domain rewrite, CONNECT tunnel, then + * the policy-gated cache/forward sequence in front of the web application. + * The default (non-proxy) runtime lives in {@link Jetty12HttpServer}.</p> + */ +final class Jetty12ProxyChain { + + private Jetty12ProxyChain() { + } + + /** Compose the transparent-proxy pipeline around the web application handler. */ + static Handler wrap(final Handler next, final Switchboard switchboard) { + final Handler forwardProxy = new ForwardProxyHandler(switchboard); + final Handler proxyCache = new ProxyCacheHandler(switchboard); + final Handler proxyPipeline = new Handler.Sequence(proxyCache, forwardProxy); + final Handler proxyPolicy = new ProxyPolicyHandler(proxyPipeline, switchboard); + Handler pipeline = new Handler.Sequence(proxyPolicy, next); + pipeline = new ConnectTunnelHandler(pipeline, switchboard); + return new DomainHandler(pipeline, switchboard.peers); + } + + /** Applies proxy classification and authorization once before cache or forwarding. */ + static final class ProxyPolicyHandler extends Handler.Wrapper { + + @FunctionalInterface + interface Permission { + String rejectionReason(Request request); + } + + private final Predicate<Request> proxyRequest; + private final Permission permission; + + ProxyPolicyHandler(final Handler proxyPipeline, final Switchboard switchboard) { + this(proxyPipeline, request -> isProxyRequest(switchboard, request), + request -> rejectionReason(switchboard, request)); + } + + ProxyPolicyHandler(final Handler proxyPipeline, + final Predicate<Request> proxyRequest, final Permission permission) { + super(proxyPipeline); + this.proxyRequest = proxyRequest; + this.permission = permission; + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) + throws Exception { + if (!this.proxyRequest.test(request)) { + return false; + } + final String rejection = this.permission.rejectionReason(request); + if (rejection != null) { + Response.writeError(request, response, callback, 403, rejection); + return true; + } + return super.handle(request, response, callback); + } + + static boolean isProxyRequest(final Switchboard switchboard, final Request request) { + if (isDirectPeerRequest(request, switchboard.getConfig("fileHost", "localpeer"))) { + return false; + } + final String host = Request.getServerName(request); + if (switchboard.peers != null) { + if (switchboard.peers.myIPs().contains(host) + || host.equalsIgnoreCase(switchboard.peers.myAlternativeAddress())) { + return false; + } + if (switchboard.peers.mySeed() != null + && (switchboard.peers.mySeed().getIPs().contains(host) + || host.equalsIgnoreCase( + switchboard.peers.mySeed().getHexHash() + ".yacyh"))) { + return false; + } + } + final java.net.InetAddress resolved = Domains.dnsResolve(host); + return resolved == null || !switchboard.myPublicIPs().contains(resolved.getHostAddress()); + } + + static boolean isDirectPeerRequest(final Request request, final String fileHost) { + final String host = Request.getServerName(request); + final int targetPort = request.getHttpURI().getPort(); + if (targetPort >= 0 && targetPort != Request.getLocalPort(request)) { + return false; + } + if (host == null || "localhost".equalsIgnoreCase(host) + || fileHost.equalsIgnoreCase(host) + || Domains.isThisHostIP(host)) { + return true; + } + final java.net.InetAddress resolved = Domains.dnsResolve(host); + return resolved != null && Domains.isLocal(host, resolved); + } + + static String rejectionReason(final Switchboard switchboard, final Request request) { + final String remoteAddress = Request.getRemoteAddr(request); + if (!ProxyAccessPolicy.isClientAllowed( + switchboard.getConfig("proxyClient", "*"), remoteAddress)) { + return "proxy use not granted for IP " + remoteAddress; + } + final String requestHost = Request.getServerName(request); + if (requestHost == null) { + return "proxy target has no host"; + } + final String host = requestHost.toLowerCase(Locale.ROOT); + if (Switchboard.urlBlacklist.isListed(BlacklistType.PROXY, host, + request.getHttpURI().getCanonicalPath())) { + return "URL '" + host + "' blocked by yacy proxy"; + } + return null; + } + } + + /** Rewrites peer-domain authorities before requests enter the proxy chain. */ + static final class DomainHandler extends Handler.Wrapper { + + private final AlternativeDomainNames resolver; + private final Predicate<String> localAddress; + + DomainHandler(final Handler next, final AlternativeDomainNames resolver) { + this(next, resolver, host -> Domains.isLocal(host, null)); + } + + DomainHandler(final Handler next, final AlternativeDomainNames resolver, + final Predicate<String> localAddress) { + super(next); + this.resolver = resolver; + this.localAddress = localAddress; + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) + throws Exception { + if (this.resolver == null) { + return super.handle(request, response, callback); + } + final String host = Request.getServerName(request); + if (host == null) { + return super.handle(request, response, callback); + } + final String resolved = this.resolver.resolve(host); + if (resolved == null) { + return super.handle(request, response, callback); + } + final HostPort destination = new HostPort(resolved); + final String destinationHost = destination.getHost(); + if (this.resolver.myIPs().contains(destinationHost) || this.localAddress.test(destinationHost)) { + return super.handle(request, response, callback); + } + final int destinationPort = destination.getPort(80); + final HttpURI.Mutable rewritten = HttpURI.build(request.getHttpURI()) + .authority(destinationHost, destinationPort); + if (rewritten.getScheme() == null) { + rewritten.scheme(request.isSecure() ? "https" : "http"); + } + final HttpURI rewrittenUri = rewritten.asImmutable(); + final HttpFields rewrittenHeaders = HttpFields.build(request.getHeaders()) + .put(HttpHeader.HOST, destinationHost + (destinationPort == 80 ? "" : ":" + destinationPort)) + .asImmutable(); + final Request rewrittenRequest = new Request.Wrapper(request) { + @Override + public HttpURI getHttpURI() { + return rewrittenUri; + } + + @Override + public HttpFields getHeaders() { + return rewrittenHeaders; + } + }; + return this.getHandler().handle(rewrittenRequest, response, callback); + } + } + + /** Jetty 12 CONNECT tunnel with YaCy's existing proxy authorization rules. */ + static final class ConnectTunnelHandler extends org.eclipse.jetty.server.handler.ConnectHandler { + + @FunctionalInterface + interface Permission { + String rejectionReason(Request request, HostPort destination); + } + + private final Permission permission; + + ConnectTunnelHandler(final Handler next, final Switchboard switchboard) { + this(next, (request, destination) -> rejectionReason(switchboard, request, destination)); + } + + ConnectTunnelHandler(final Handler next, final Permission permission) { + super(next); + this.permission = permission; + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) + throws Exception { + if (!HttpMethod.CONNECT.is(request.getMethod())) { + return super.handle(request, response, callback); + } + final HostPort destination; + try { + final String target = request.getHttpURI().getAuthority() != null + ? request.getHttpURI().getAuthority() : request.getHttpURI().getPath(); + if (target == null) { + Response.writeError(request, response, callback, 400, "Invalid CONNECT destination"); + return true; + } + destination = new HostPort(target); + } catch (final IllegalArgumentException error) { + Response.writeError(request, response, callback, 400, "Invalid CONNECT destination"); + return true; + } + final String rejection = this.permission.rejectionReason(request, destination); + if (rejection != null) { + Response.writeError(request, response, callback, 403, rejection); + return true; + } + return super.handle(request, response, callback); + } + + private static String rejectionReason(final Switchboard switchboard, final Request request, + final HostPort destination) { + final String remoteAddress = Request.getRemoteAddr(request); + if (!ProxyAccessPolicy.isClientAllowed(switchboard.getConfig("proxyClient", "*"), remoteAddress)) { + return "proxy use not granted for IP " + remoteAddress; + } + final String destinationHost = destination.getHost().toLowerCase(Locale.ROOT); + if (Switchboard.urlBlacklist.isListed(BlacklistType.PROXY, destinationHost, "/")) { + return "URL '" + destinationHost + "' blocked by yacy proxy"; + } + return null; + } + } + + /** Serves fresh YaCy proxy-cache entries before a request reaches the network. */ + static final class ProxyCacheHandler extends Handler.Abstract { + + record CachedResponse(HttpFields headers, byte[] content) { + } + + @FunctionalInterface + interface Lookup { + CachedResponse find(Request request); + } + + private final Lookup lookup; + + ProxyCacheHandler(final Switchboard switchboard) { + this(request -> findCached(switchboard, request)); + } + + ProxyCacheHandler(final Lookup lookup) { + this.lookup = lookup; + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) { + if (!"GET".equals(request.getMethod())) { + return false; + } + final CachedResponse cached = this.lookup.find(request); + if (cached == null) { + return false; + } + response.setStatus(203); + response.getHeaders().add(cached.headers()); + response.write(true, ByteBuffer.wrap(cached.content()), callback); + return true; + } + + private static CachedResponse findCached(final Switchboard switchboard, final Request request) { + try { + if (switchboard.crawler == null || switchboard.crawler.defaultProxyProfile == null) { + return null; + } + final DigestURL url = new DigestURL(request.getHttpURI().toString()); + final ResponseHeader responseHeader = Cache.getResponseHeader(url.hash()); + if (responseHeader == null) { + return null; + } + final RequestHeader requestHeader = new RequestHeader(); + request.getHeaders().forEach(field -> requestHeader.add(field.getName(), field.getValue())); + final net.yacy.crawler.retrieval.Request crawlerRequest = + new net.yacy.crawler.retrieval.Request(null, url, + requestHeader.referer() == null ? null + : new DigestURL(requestHeader.referer().toNormalform(true)).hash(), + "", responseHeader.lastModified(), + switchboard.crawler.defaultProxyProfile.handle(), 0, + switchboard.crawler.defaultProxyProfile.timezoneOffset()); + final net.yacy.crawler.retrieval.Response cachedResponse = + new net.yacy.crawler.retrieval.Response(crawlerRequest, requestHeader, + responseHeader, switchboard.crawler.defaultProxyProfile, false, null); + final byte[] content = Cache.getContent(url.hash()); + if (content == null || !cachedResponse.isFreshForProxy()) { + return null; + } + final HttpFields.Mutable headers = HttpFields.build(); + responseHeader.forEach((name, value) -> headers.add(name, value)); + return new CachedResponse(headers.asImmutable(), content); + } catch (final Exception invalidCacheEntry) { + return null; + } + } + } + + /** Streams an already authorized Jetty 12 HTTP proxy request. */ + static final class ForwardProxyHandler extends ProxyHandler.Forward { + + interface Capture { + void append(ByteBuffer content); + + void complete(); + } + + @FunctionalInterface + interface CaptureFactory { + Capture begin(Request request, org.eclipse.jetty.client.Response upstream); + } + + private final CaptureFactory captureFactory; + private final Switchboard switchboard; + private final int timeout; + + ForwardProxyHandler(final Switchboard switchboard) { + this(new ProxyResponseStore(switchboard), switchboard, + switchboard.getConfigInt("proxy.clientTimeout", 10000)); + } + + ForwardProxyHandler(final CaptureFactory captureFactory) { + this(captureFactory, null, 10000); + } + + private ForwardProxyHandler(final CaptureFactory captureFactory, + final Switchboard switchboard, final int timeout) { + this.captureFactory = captureFactory; + this.switchboard = switchboard; + this.timeout = timeout; + } + + @Override + public boolean handle(final Request request, final Response response, final Callback callback) { + if (!HttpMethod.GET.is(request.getMethod()) && !HttpMethod.POST.is(request.getMethod()) + && !HttpMethod.HEAD.is(request.getMethod())) { + Response.writeError(request, response, callback, 501, "Unsupported proxy request method"); + return true; + } + if (this.switchboard != null) { + this.switchboard.proxyLastAccess = System.currentTimeMillis(); + } + return super.handle(request, response, callback); + } + + @Override + protected void configureHttpClient(final HttpClient httpClient) { + super.configureHttpClient(httpClient); + httpClient.setConnectTimeout(this.timeout); + httpClient.setIdleTimeout(this.timeout); + } + + @Override + protected void addProxyHeaders(final Request clientRequest, + final org.eclipse.jetty.client.Request upstreamRequest) { + addViaHeader(clientRequest, upstreamRequest); + if (this.switchboard != null + && this.switchboard.getConfigBool("proxy.sendXForwardedForHeader", true)) { + final String remoteAddress = Request.getRemoteAddr(clientRequest); + if (!Domains.isThisHostIP(remoteAddress)) { + upstreamRequest.headers(headers -> headers.put(HttpHeader.X_FORWARDED_FOR, + remoteAddress)); + } + } + } + + @Override + protected HttpURI rewriteHttpURI(final Request request) { + if (request.getHttpURI().isAbsolute()) { + return request.getHttpURI(); + } + return HttpURI.build(request.getHttpURI()) + .scheme(request.isSecure() ? "https" : "http") + .authority(Request.getServerName(request), Request.getServerPort(request)) + .asImmutable(); + } + + @Override + protected org.eclipse.jetty.client.Response.CompleteListener newServerToProxyResponseListener( + final Request clientRequest, final org.eclipse.jetty.client.Request upstreamRequest, + final Response clientResponse, final Callback clientCallback) { + return new ProxyResponseListener(clientRequest, upstreamRequest, clientResponse, clientCallback) { + private Capture capture; + + @Override + public void onHeaders(final org.eclipse.jetty.client.Response upstreamResponse) { + super.onHeaders(upstreamResponse); + this.capture = ForwardProxyHandler.this.captureFactory.begin( + clientRequest, upstreamResponse); + } + + @Override + public void onContent(final org.eclipse.jetty.client.Response upstreamResponse, + final org.eclipse.jetty.io.Content.Chunk chunk, final Runnable demander) { + if (this.capture != null) { + this.capture.append(chunk.getByteBuffer().asReadOnlyBuffer()); + } + super.onContent(upstreamResponse, chunk, demander); + } + + @Override + public void onComplete(final Result result) { + super.onComplete(result); + if (result.isSucceeded() && this.capture != null) { + whenComplete((ignored, failure) -> { + if (failure == null) { + this.capture.complete(); + } + }); + } + } + }; + } + + @Override + protected void onProxyToClientResponseComplete(final Request clientRequest, + final org.eclipse.jetty.client.Request upstreamRequest, + final org.eclipse.jetty.client.Response upstreamResponse, + final Response clientResponse, final Callback clientCallback) { + final StringBuilder message = new StringBuilder(96); + message.append(GenericFormatter.SHORT_SECOND_FORMATTER.format(new Date())).append(' ') + .append(Request.getRemoteAddr(clientRequest)).append(' ') + .append(clientRequest.getMethod()).append(' ') + .append(clientRequest.getHttpURI()); + HTTPDProxyHandler.proxyLog.fine(message.toString()); + super.onProxyToClientResponseComplete(clientRequest, upstreamRequest, upstreamResponse, + clientResponse, clientCallback); + } + } + + /** Bridges a completed Jetty 12 proxy response into YaCy's cache/index path. */ + static final class ProxyResponseStore implements ForwardProxyHandler.CaptureFactory { + + private final Switchboard switchboard; + + ProxyResponseStore(final Switchboard switchboard) { + this.switchboard = switchboard; + } + + @Override + public ForwardProxyHandler.Capture begin(final Request request, + final org.eclipse.jetty.client.Response upstream) { + try { + if (this.switchboard.crawler == null + || this.switchboard.crawler.defaultProxyProfile == null) { + return null; + } + final DigestURL url = new DigestURL(request.getHttpURI().toString()); + final ResponseHeader responseHeader = new ResponseHeader(upstream.getStatus()); + for (final HttpField field : upstream.getHeaders()) { + responseHeader.add(field.getName(), field.getValue()); + } + final net.yacy.crawler.retrieval.Request crawlerRequest = + new net.yacy.crawler.retrieval.Request(null, url, null, "", + responseHeader.lastModified(), + this.switchboard.crawler.defaultProxyProfile.handle(), 0, + this.switchboard.crawler.defaultProxyProfile.timezoneOffset()); + final net.yacy.crawler.retrieval.Response yacyResponse = + new net.yacy.crawler.retrieval.Response(crawlerRequest, null, responseHeader, + this.switchboard.crawler.defaultProxyProfile, false, null); + final String storeError = yacyResponse.shallStoreCacheForProxy(); + final boolean storeHTCache = yacyResponse.profile().storeHTCache(); + final String supportError = TextParser.supports(url, yacyResponse.getMimeType()); + if (storeError != null || (!storeHTCache && supportError == null)) { + return null; + } + return new CacheCapture(this.switchboard, yacyResponse); + } catch (final IOException invalidUrl) { + return null; + } + } + + static final class CacheCapture implements ForwardProxyHandler.Capture { + + private final Switchboard switchboard; + private final net.yacy.crawler.retrieval.Response response; + private final int maxContentSize; + private final ByteArrayOutputStream content = new ByteArrayOutputStream(); + private boolean discarded; + + private CacheCapture(final Switchboard switchboard, + final net.yacy.crawler.retrieval.Response response) { + this(switchboard, response, + (int) net.yacy.crawler.retrieval.Response.CRAWLER_MAX_SIZE_TO_CACHE); + } + + CacheCapture(final Switchboard switchboard, + final net.yacy.crawler.retrieval.Response response, + final int maxContentSize) { + this.switchboard = switchboard; + this.response = response; + this.maxContentSize = maxContentSize; + } + + @Override + public synchronized void append(final ByteBuffer buffer) { + if (this.discarded) { + return; + } + final ByteBuffer copy = buffer.slice(); + if (copy.remaining() > this.maxContentSize - this.content.size()) { + this.content.reset(); + this.discarded = true; + return; + } + final byte[] bytes = new byte[copy.remaining()]; + copy.get(bytes); + this.content.writeBytes(bytes); + } + + @Override + public void complete() { + final byte[] bytes; + synchronized (this) { + if (this.discarded) { + return; + } + bytes = this.content.toByteArray(); + } + if (bytes.length == 0) { + return; + } + final Thread writer = new Thread(() -> { + try { + if (Cache.getResponseHeader(this.response.url().hash()) != null) { + Cache.delete(this.response.url().hash()); + } + this.response.setContent(bytes); + Cache.store(this.response.url(), this.response.getResponseHeader(), bytes); + this.switchboard.toIndexer(this.response); + } catch (final IOException ignored) { + // A proxy response must still reach its client when cache storage fails. + } + }, "Jetty12ProxyChain.ResponseStore(" + this.response.url().toNormalform(true) + ")"); + writer.setPriority(Thread.MIN_PRIORITY); + writer.start(); + } + + synchronized int bufferedSize() { + return this.content.size(); + } + + synchronized boolean isDiscarded() { + return this.discarded; + } + } + } +} diff --git a/source/net/yacy/http/Jetty9HttpServerImpl.java b/source/net/yacy/http/Jetty9HttpServerImpl.java deleted file mode 100644 index 137330acc..000000000 --- a/source/net/yacy/http/Jetty9HttpServerImpl.java +++ /dev/null @@ -1,549 +0,0 @@ -// -// Jetty9HttpServerImpl -// Copyright 2011 by Florian Richter -// First released 13.04.2011 at https://yacy.net -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import java.io.FileInputStream; -import java.io.FileOutputStream; -import java.io.IOException; -import java.net.InetAddress; -import java.net.InetSocketAddress; -import java.security.KeyStore; -import java.util.EnumSet; -import java.util.StringTokenizer; - -import javax.net.ssl.KeyManagerFactory; -import javax.net.ssl.SSLContext; -import javax.servlet.DispatcherType; - -import org.eclipse.jetty.http.HttpMethod; -import org.eclipse.jetty.http.HttpVersion; -import org.eclipse.jetty.io.Connection; -import org.eclipse.jetty.server.Connector; -import org.eclipse.jetty.server.Handler; -import org.eclipse.jetty.server.HttpConfiguration; -import org.eclipse.jetty.server.HttpConnectionFactory; -import org.eclipse.jetty.server.SecureRequestCustomizer; -import org.eclipse.jetty.server.Server; -import org.eclipse.jetty.server.ServerConnector; -import org.eclipse.jetty.server.SslConnectionFactory; -import org.eclipse.jetty.server.handler.ContextHandler; -import org.eclipse.jetty.server.handler.ContextHandlerCollection; -import org.eclipse.jetty.server.handler.DefaultHandler; -import org.eclipse.jetty.server.handler.HandlerList; -import org.eclipse.jetty.server.handler.InetAccessHandler; -import org.eclipse.jetty.server.handler.gzip.GzipHandler; -import org.eclipse.jetty.servlet.FilterHolder; -import org.eclipse.jetty.servlet.ServletHolder; -import org.eclipse.jetty.util.resource.Resource; -import org.eclipse.jetty.util.ssl.SslContextFactory; -import org.eclipse.jetty.webapp.WebAppContext; - -import net.yacy.cora.protocol.ConnectionInfo; -import net.yacy.cora.util.ConcurrentLog; -import net.yacy.http.servlets.MonitorFilter; -import net.yacy.http.servlets.YaCyDefaultServlet; -import net.yacy.search.Switchboard; -import net.yacy.search.SwitchboardConstants; -import net.yacy.utils.PKCS12Tool; - -/** - * class to embedded Jetty 9 http server into YaCy - */ -public class Jetty9HttpServerImpl implements YaCyHttpServer { - - private final Server server; - - /** - * @param port TCP Port to listen for http requests - * @param host The network interface this connector binds to as an IP address or a hostname. - */ - public Jetty9HttpServerImpl(final int port, final String host) { - final Switchboard sb = Switchboard.getSwitchboard(); - final HttpServerBootstrapConfig bootstrap = HttpServerBootstrapConfig.from(sb, port, host); - - this.server = new Server(); - - // remove the ConnectionInfo tracking entry (added per request by the MonitorFilter) - // when the tcp connection closes; added as bean to each connector below - final Connection.Listener connectionCloseMonitor = new Connection.Listener() { - @Override - public void onOpened(final Connection connection) { - } - @Override - public void onClosed(final Connection connection) { - final InetSocketAddress remote = connection.getEndPoint().getRemoteAddress(); - if (remote != null) { - ConnectionInfo.removeServerConnection(MonitorFilter.connectionId(remote.getAddress().getHostAddress(), remote.getPort())); - } - } - }; - - final HttpConfiguration httpConfig = new HttpConfiguration(); - httpConfig.setRequestHeaderSize(HttpServerBootstrapConfig.REQUEST_HEADER_SIZE); - final HttpConnectionFactory hcf = new HttpConnectionFactory(httpConfig); - final ServerConnector connector = new ServerConnector(this.server, null, null, null, bootstrap.acceptorCount(), -1, hcf); - connector.setPort(bootstrap.httpPort()); - connector.setHost(bootstrap.bindHost()); - connector.setName("httpd-" + bootstrap.bindHost() + ":" + Integer.toString(bootstrap.httpPort())); - connector.setIdleTimeout(HttpServerBootstrapConfig.CONNECTOR_IDLE_TIMEOUT_MILLIS); - connector.setAcceptQueueSize(HttpServerBootstrapConfig.ACCEPT_QUEUE_SIZE); - connector.addBean(connectionCloseMonitor); - - this.server.addConnector(connector); - - - // add ssl/https connector - final boolean useSSL = bootstrap.httpsEnabled(); - - if (useSSL) { - final SslContextFactory sslContextFactory = new SslContextFactory.Server(); - final SSLContext sslContext = this.initSslContext(sb); - if (sslContext != null) { - - final int sslport = bootstrap.httpsPort(); - sslContextFactory.setSslContext(sslContext); - - // SSL HTTP Configuration - final HttpConfiguration https_config = new HttpConfiguration(); - https_config.addCustomizer(new SecureRequestCustomizer()); - - // SSL Connector - final ServerConnector sslConnector = new ServerConnector(this.server, - new SslConnectionFactory(sslContextFactory, HttpVersion.HTTP_1_1.asString()), - new HttpConnectionFactory(https_config)); - sslConnector.setPort(sslport); - sslConnector.setName("ssld:" + Integer.toString(sslport)); // name must start with ssl (for withSSL() to work correctly) - sslConnector.setIdleTimeout(HttpServerBootstrapConfig.CONNECTOR_IDLE_TIMEOUT_MILLIS); - sslConnector.addBean(connectionCloseMonitor); - - this.server.addConnector(sslConnector); - ConcurrentLog.info("SERVER", "SSL support initialized successfully on port " + sslport); - } - } - - final YacyDomainHandler domainHandler = new YacyDomainHandler(); - domainHandler.setAlternativeResolver(sb.peers); - - // configure root context - final WebAppContext htrootContext = new WebAppContext(); - htrootContext.setContextPath("/"); - final String htrootpath = bootstrap.htrootPath(); - ConcurrentLog.info("Jetty9HttpServerImpl", "htrootpath = " + htrootpath); - htrootContext.setErrorHandler(new YaCyErrorHandler()); // handler for custom error page - try { - htrootContext.setBaseResource(Resource.newResource(htrootpath)); - - // set web.xml to use - // make use of Jetty feature to define web.xml other as default WEB-INF/web.xml - // and to use a DefaultsDescriptor merged with a individual web.xml - // use defaults/web.xml as default and look in DATA/SETTINGS for local addition/changes - htrootContext.setDefaultsDescriptor(bootstrap.defaultsWebXml()); - final Resource webxml = Resource.newResource(bootstrap.overrideWebXml()); - if (webxml.exists()) { - htrootContext.setDescriptor(webxml.getName()); - } - - } catch (final IOException ex) { - if (htrootContext.getBaseResource() == null) { - ConcurrentLog.severe("SERVER", "could not find directory: htroot "); - } else { - ConcurrentLog.warn("SERVER", "could not find: defaults/web.xml or DATA/SETTINGS/web.xml"); - } - } - - // as fundamental component leave this hardcoded, other servlets may be defined in web.xml only - final ServletHolder sholder = new ServletHolder(YaCyDefaultServlet.class); - sholder.setInitParameter("resourceBase", htrootpath); - sholder.setAsyncSupported(true); // needed for YaCyQoSFilter - //sholder.setInitParameter("welcomeFile", "index.html"); // default is index.html, welcome.html - htrootContext.addServlet(sholder, "/*"); - - // as fundamental component this filter is hardcoded too: it feeds the - // Connections_p.html monitoring and rejects requests above the connection limit - final FilterHolder monitorFilter = new FilterHolder(MonitorFilter.class); - monitorFilter.setAsyncSupported(true); - htrootContext.addFilter(monitorFilter, "/*", EnumSet.of(DispatcherType.REQUEST)); - - final GzipHandler gzipHandler = new GzipHandler(); - /* - * Decompression of incoming requests body is required for index distribution - * APIs /yacy/transferRWI.html and /yacy/transferURL.html This was previously - * handled by a GZIPRequestWrapper in the YaCyDefaultServlet. - */ - gzipHandler.setInflateBufferSize(HttpServerBootstrapConfig.REQUEST_INFLATE_BUFFER_SIZE); - - if (!bootstrap.gzipResponsesEnabled()) { - /* Gzip compression of responses can be disabled by user configuration */ - gzipHandler.setExcludedMethods(HttpMethod.GET.asString(), HttpMethod.POST.asString()); - } - htrootContext.setGzipHandler(gzipHandler); - - // ----------------------------------------------------------------------------- - // here we set and map the mandatory servlets, needed for typical YaCy operation - // to make sure they are available even if removed in individual web.xml - // additional, optional or individual servlets or servlet mappings can be set in web.xml - - // in Jetty 9 servlet should be set only once - // therefore only the settings in web.xml is used - //add SolrSelectServlet - //htrootContext.addServlet(SolrSelectServlet.class, "/solr/select"); // uses the default core, collection1 - //htrootContext.addServlet(SolrSelectServlet.class, "/solr/collection1/select"); // the same servlet, identifies the collection1 core using the path - //htrootContext.addServlet(SolrSelectServlet.class, "/solr/webgraph/select"); // the same servlet, identifies the webgraph core using the path - - //htrootContext.addServlet(SolrServlet.class, "/solr/collection1/admin/luke"); - //htrootContext.addServlet(SolrServlet.class, "/solr/webgraph/admin/luke"); - - // add proxy?url= servlet - //htrootContext.addServlet(YaCyProxyServlet.class,"/proxy.html"); - - // add GSA servlet - //htrootContext.addServlet(GSAsearchServlet.class,"/gsa/search"); - // --- eof default servlet mappings -------------------------------------------- - - // define list of YaCy specific general handlers - final HandlerList handlers = new HandlerList(); - if (bootstrap.transparentProxyEnabled()) { - // Proxyhandlers are only needed if feature activated (save resources if not used) - ConcurrentLog.info("SERVER", "load Jetty handler for transparent proxy"); - handlers.setHandlers(new Handler[]{domainHandler, new ProxyCacheHandler(), new ProxyHandler()}); - } else { - handlers.setHandlers(new Handler[]{domainHandler}); - } - // context handler for dispatcher and security (hint: dispatcher requires a context) - final ContextHandler context = new ContextHandler(); - context.setServer(this.server); - context.setContextPath("/"); - context.setHandler(handlers); - context.setMaxFormContentSize(HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE); - // make YaCy handlers (in context) and servlet context handlers available (both contain root context "/") - // logic: 1. YaCy handlers are called if request not handled (e.g. proxy) then servlets handle it - final ContextHandlerCollection allrequesthandlers = new ContextHandlerCollection(); - allrequesthandlers.setServer(this.server); - allrequesthandlers.addHandler(context); - allrequesthandlers.addHandler(htrootContext); - allrequesthandlers.addHandler(new DefaultHandler()); // if not handled by other handler - - final YaCyLoginService loginService = new YaCyLoginService(); - // This is part of the built-in administrator's DIGEST password hash. - // Changing it invalidates the configured administrator password hash. - loginService.setName(bootstrap.adminRealm()); - - final YaCySecurityHandler securityHandler = new YaCySecurityHandler(); - securityHandler.setLoginService(loginService); - - htrootContext.setSecurityHandler(securityHandler); - - // wrap all handlers - final Handler crashHandler = new CrashProtectionHandler(this.server, allrequesthandlers); - // check server access restriction and add InetAccessHandler if restrictions are needed - // otherwise don't (to save performance) - final String white = bootstrap.serverClientRules(); - if (!white.equals("*")) { // full ip (allowed ranges 0-255 or prefix 10.0-255,0,0-100 or CIDR notation 192.168.1.0/24) - final StringTokenizer st = new StringTokenizer(white, ","); - final InetAccessHandler whiteListHandler; - if (white.contains("|")) { - /* - * At least one pattern includes a path definition : we must use the - * InetPathAccessHandler as InetAccessHandler doesn't support path patterns - */ - whiteListHandler = new InetPathAccessHandler(); - } else { - whiteListHandler = new InetAccessHandler(); - } - int i = 0; - while (st.hasMoreTokens()) { - final String pattern = st.nextToken(); - try { - whiteListHandler.include(pattern); - } catch (final IllegalArgumentException nex) { // catch format exception on wrong ip address pattern - ConcurrentLog.severe("SERVER", "Server Access Settings - IP filter: " + nex.getMessage()); - continue; - } - i++; - } - if (i > 0) { - final String loopbackAddress = InetAddress.getLoopbackAddress().getHostAddress(); - whiteListHandler.include(loopbackAddress); - whiteListHandler.setHandler(crashHandler); - this.server.setHandler(whiteListHandler); - - ConcurrentLog.info("SERVER","activated IP access restriction to: [" + loopbackAddress + "," + white +"]"); - } else { - this.server.setHandler(crashHandler); // InetAccessHandler not needed - } - } else { - this.server.setHandler(crashHandler); // InetAccessHandler not needed - } - } - - /** - * start http server - */ - public void startupServer() throws Exception { - // option to finish running requests on shutdown -// server.setGracefulShutdown(3000); - this.server.setStopAtShutdown(true); - this.server.start(); - } - - /** - * stop http server and wait for it - */ - public void stop() throws Exception { - this.server.stop(); - this.server.join(); - } - - /** - * @return true if ssl/https connector is available - */ - public boolean withSSL() { - final Connector[] clist = this.server.getConnectors(); - for (final Connector c:clist) { - if (c.getName().startsWith("ssl")) return true; - } - return false; - } - - /** - * The port of actual running ssl connector - * @return the ssl/https port or -1 if not active - */ - public int getSslPort() { - final Connector[] clist = this.server.getConnectors(); - for (final Connector c:clist) { - if (c.getName().startsWith("ssl")) { - final int port =((ServerConnector)c).getLocalPort(); - return port; - } - } - return -1; - } - - /** - * reconnect with new port settings (after waiting milsec) - routine returns - * immediately - * checks http and ssl connector for new port settings - * @param milsec wait time - */ - public void reconnect(final int milsec) { - - new Thread("Jetty8HttpServer.reconnect") { - - @Override - public void run() { - if (milsec > 0) try { - Thread.sleep(milsec); - } catch (final Exception e) { - ConcurrentLog.logException(e); - } - try { - if (!Jetty9HttpServerImpl.this.server.isRunning() || Jetty9HttpServerImpl.this.server.isStopped()) { - Jetty9HttpServerImpl.this.server.start(); - } - - // reconnect with new settings (instead to stop/start server, just manipulate connectors - final Connector[] cons = Jetty9HttpServerImpl.this.server.getConnectors(); - final int port = Switchboard.getSwitchboard().getLocalPort(); - final int sslport = Switchboard.getSwitchboard().getConfigInt(SwitchboardConstants.SERVER_SSLPORT, 8443); - for (final Connector con : cons) { - // check http connector - if (con.getName().startsWith("httpd") && ((ServerConnector)con).getPort() != port) { - ((ServerConnector)con).close(); - con.stop(); - if (!con.isStopped()) { - ConcurrentLog.warn("SERVER", "Reconnect: Jetty Connector failed to stop"); - } - ((ServerConnector)con).setPort(port); - con.start(); - ConcurrentLog.info("SERVER", "set new port for Jetty connector " + con.getName()); - continue; - } - // check https connector - if (con.getName().startsWith("ssl") && ((ServerConnector)con).getPort() != sslport) { - ((ServerConnector)con).close(); - con.stop(); - if (!con.isStopped()) { - ConcurrentLog.warn("SERVER", "Reconnect: Jetty Connector failed to stop"); - } - ((ServerConnector)con).setPort(sslport); - con.start(); - ConcurrentLog.info("SERVER", "set new port for Jetty connector " + con.getName()); - } - } - } catch (final Exception ex) { - ConcurrentLog.logException(ex); - } - } - }.start(); - } - - /** - * Forces the login service to reload the built-in administrator credentials - * after they were changed in the configuration. - * @param username - */ - public void resetUser(final String username) { - final YaCySecurityHandler hx = this.server.getChildHandlerByClass(YaCySecurityHandler.class); - if (hx != null) { - final YaCyLoginService loginservice = (YaCyLoginService) hx.getLoginService(); - if (loginservice.removeUser(username)) { // remove old credential from cache - loginservice.loadUserInfo(username); - } - } - } - - /** - * Removes the built-in administrator from the login service cache. - * @param username - */ - public void removeUser(final String username) { - final YaCySecurityHandler hx = this.server.getChildHandlerByClass(YaCySecurityHandler.class); - if (hx != null) { - final YaCyLoginService loginservice = (YaCyLoginService) hx.getLoginService(); - loginservice.removeUser(username); - } - } - - /** - * get Jetty version - * @return version_string - */ - public String getVersion() { - return "Jetty " + Server.getVersion(); - } - - /** - * Init SSL Context from config settings - * @param sb Switchboard - * @return default or sslcontext according to config - */ - private SSLContext initSslContext(final Switchboard sb) { - - // getting the keystore file name - String keyStoreFileName = sb.getConfig("keyStore", "").trim(); - - // getting the keystore pwd - String keyStorePwd = sb.getConfig("keyStorePassword", "").trim(); - - // take a look if we have something to import - final String pkcs12ImportFile = sb.getConfig("pkcs12ImportFile", "").trim(); - - // if no keyStore and no import is defined, then set the default key - if (keyStoreFileName.isEmpty() && keyStorePwd.isEmpty() && pkcs12ImportFile.isEmpty()) { - keyStoreFileName = "defaults/freeworldKeystore"; - keyStorePwd = "freeworld"; - sb.setConfig("keyStore", keyStoreFileName); - sb.setConfig("keyStorePassword", keyStorePwd); - } - - if (pkcs12ImportFile.length() > 0) { - ConcurrentLog.info("SERVER", "Import certificates from import file '" + pkcs12ImportFile + "'."); - - try { - // getting the password - final String pkcs12ImportPwd = sb.getConfig("pkcs12ImportPwd", "").trim(); - - // creating tool to import cert - final PKCS12Tool pkcsTool = new PKCS12Tool(pkcs12ImportFile,pkcs12ImportPwd); - - // creating a new keystore file - if (keyStoreFileName.isEmpty()) { - // using the default keystore name - keyStoreFileName = "DATA/SETTINGS/myPeerKeystore"; - - // creating an empty java keystore - final KeyStore ks = KeyStore.getInstance("JKS"); - ks.load(null,keyStorePwd.toCharArray()); - try ( - /* Automatically closed by this try-with-resources statement */ - final FileOutputStream ksOut = new FileOutputStream(keyStoreFileName); - ) { - ks.store(ksOut, keyStorePwd.toCharArray()); - } - - // storing path to keystore into config file - sb.setConfig("keyStore", keyStoreFileName); - } - - // importing certificate - pkcsTool.importToJKS(keyStoreFileName, keyStorePwd); - - // removing entries from config file - sb.setConfig("pkcs12ImportFile", ""); - sb.setConfig("pkcs12ImportPwd", ""); - - // deleting original import file - // TODO: should we do this - } catch (final Exception e) { - ConcurrentLog.severe("SERVER", "Unable to import certificate from import file '" + pkcs12ImportFile + "'.",e); - } - } else if (keyStoreFileName.isEmpty()) return null; - - // get the ssl context - try { - ConcurrentLog.info("SERVER","Initializing SSL support ..."); - - // creating a new keystore instance of type (java key store) - if (ConcurrentLog.isFine("SERVER")) ConcurrentLog.fine("SERVER", "Initializing keystore ..."); - final KeyStore ks = KeyStore.getInstance("JKS"); - - // loading keystore data from file - if (ConcurrentLog.isFine("SERVER")) ConcurrentLog.fine("SERVER","Loading keystore file " + keyStoreFileName); - final FileInputStream stream = new FileInputStream(keyStoreFileName); - try { - ks.load(stream, keyStorePwd.toCharArray()); - } finally { - try { - stream.close(); - } catch(final IOException ioe) { - ConcurrentLog.warn("SERVER", "Could not close input stream on file " + keyStoreFileName); - } - } - - // creating a keystore factory - if (ConcurrentLog.isFine("SERVER")) ConcurrentLog.fine("SERVER","Initializing key manager factory ..."); - final KeyManagerFactory kmf = KeyManagerFactory.getInstance(KeyManagerFactory.getDefaultAlgorithm()); - kmf.init(ks,keyStorePwd.toCharArray()); - - // initializing the ssl context - if (ConcurrentLog.isFine("SERVER")) ConcurrentLog.fine("SERVER","Initializing SSL context ..."); - final SSLContext sslcontext = SSLContext.getInstance("TLS"); - sslcontext.init(kmf.getKeyManagers(), null, null); - - return sslcontext; - } catch (final Exception e) { - final String errorMsg = "FATAL ERROR: Unable to initialize the SSL Socket factory. " + e.getMessage(); - ConcurrentLog.severe("SERVER",errorMsg); - System.out.println(errorMsg); - return null; - } - } - - public int getServerThreads() { - return this.server == null ? 0 : this.server.getThreadPool().getThreads() - this.server.getThreadPool().getIdleThreads(); - } - - @Override - public String toString() { - return this.server.dump() + "\n\n" + this.server.getState(); - } -} diff --git a/source/net/yacy/http/ProxyCacheHandler.java b/source/net/yacy/http/ProxyCacheHandler.java deleted file mode 100644 index 27e043191..000000000 --- a/source/net/yacy/http/ProxyCacheHandler.java +++ /dev/null @@ -1,94 +0,0 @@ -// -// ProxyCacheHandler -// Copyright 2004 by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany -// Copyright 2011 by Florian Richter -// First released 2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// -package net.yacy.http; - -import java.io.IOException; -import java.util.Map.Entry; - -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - - -import net.yacy.cora.document.id.DigestURL; -import net.yacy.cora.protocol.RequestHeader; -import net.yacy.cora.protocol.ResponseHeader; -import net.yacy.crawler.data.Cache; -import net.yacy.crawler.retrieval.Response; - -/** - * jetty http handler serves pages from cache if available and valid - */ -public class ProxyCacheHandler extends AbstractRemoteHandler { - - private void handleRequestFromCache(@SuppressWarnings("unused") HttpServletRequest request, HttpServletResponse response, ResponseHeader cachedResponseHeader, byte[] content) throws IOException { - - // TODO: check if-modified - for (Entry<String, String> entry : cachedResponseHeader.entrySet()) { - response.addHeader(entry.getKey(), entry.getValue()); - } - response.setStatus(HttpServletResponse.SC_NON_AUTHORITATIVE_INFORMATION); - response.getOutputStream().write(content); - // we handled this request, break out of handler chain - } - - @Override - public void handleRemote(String target, RequestCompletion completion, HttpServletRequest request, HttpServletResponse response) throws IOException, ServletException { - if (request.getMethod().equals("GET")) { - String queryString = request.getQueryString() != null ? "?" + request.getQueryString() : ""; - DigestURL url = new DigestURL(request.getRequestURL().toString() + queryString); - ResponseHeader cachedResponseHeader = Cache.getResponseHeader(url.hash()); - - if (cachedResponseHeader != null) { - RequestHeader proxyHeaders = ProxyHandler.convertHeaderFromJetty(request); - // TODO: this convertion is only necessary - final net.yacy.crawler.retrieval.Request yacyRequest = new net.yacy.crawler.retrieval.Request( - null, - url, - proxyHeaders.referer() == null ? null : new DigestURL(proxyHeaders.referer().toNormalform(true)).hash(), - "", - cachedResponseHeader.lastModified(), - sb.crawler.defaultProxyProfile.handle(), - 0, - sb.crawler.defaultProxyProfile.timezoneOffset()); - - final Response cachedResponse = new Response( - yacyRequest, - proxyHeaders, - cachedResponseHeader, - sb.crawler.defaultProxyProfile, - false, - null); - byte[] cacheContent = Cache.getContent(url.hash()); - if (cacheContent != null && cachedResponse.isFreshForProxy()) { - handleRequestFromCache(request, response, cachedResponseHeader, cacheContent); - completion.complete(); - } - } - - } - } - -} diff --git a/source/net/yacy/http/ProxyHandler.java b/source/net/yacy/http/ProxyHandler.java deleted file mode 100644 index 9c86a6b09..000000000 --- a/source/net/yacy/http/ProxyHandler.java +++ /dev/null @@ -1,299 +0,0 @@ -// -// ProxyHandler -// Copyright 2004 by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany -// Copyright 2011 by Florian Richter -// First released 2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import java.io.ByteArrayOutputStream; -import java.io.IOException; -import java.io.OutputStream; -import java.net.SocketException; -import java.util.Date; -import java.util.Enumeration; - -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import net.yacy.cora.date.GenericFormatter; -import net.yacy.cora.document.id.DigestURL; -import net.yacy.cora.protocol.ClientIdentification; -import net.yacy.cora.protocol.Domains; -import net.yacy.cora.protocol.HeaderFramework; -import net.yacy.cora.protocol.RequestHeader; -import net.yacy.cora.protocol.ResponseHeader; -import net.yacy.cora.protocol.http.HTTPClient; -import net.yacy.crawler.data.Cache; -import net.yacy.crawler.retrieval.Response; -import net.yacy.document.TextParser; -import net.yacy.server.http.HTTPDProxyHandler; -import net.yacy.server.http.MultiOutputStream; - -import org.apache.http.Header; -import org.apache.http.HttpResponse; - -/** - * jetty http handler - * proxies request, caches responses and adds urls to crawler - */ -public class ProxyHandler extends AbstractRemoteHandler { - - protected int timeout = 10000; - - @Override - protected void doStart() throws Exception { - super.doStart(); - timeout = sb.getConfigInt("proxy.clientTimeout", 10000); - } - - private void convertHeaderToJetty(HttpResponse in, HttpServletResponse out) { - for(Header h: in.getAllHeaders()) { - out.addHeader(h.getName(), h.getValue()); - } - } - - private void cleanResponseHeader(HttpResponse headers) { - headers.removeHeaders(HeaderFramework.CONTENT_ENCODING); - headers.removeHeaders(HeaderFramework.CONTENT_LENGTH); - } - - private void deleteFromCache(final byte[] hash) { - // long size = -1; - ResponseHeader rh = Cache.getResponseHeader(hash); - if (rh != null) { - // delete the cache - // if ((size = rh.getContentLength()) == 0) { - // byte[] b = Cache.getContent(hash); - // if (b != null) size = b.length; - // } - try { - Cache.delete(hash); - } catch (final IOException e) { - // log refresh miss - HTTPDProxyHandler.proxyLog.fine(e.getMessage()); - } - } - } - - private void storeToCache(final Response yacyResponse, final byte[] cacheArray) { - final Thread t = new Thread() { - @Override - public void run() { - if (yacyResponse == null) return; - this.setName("ProxyHandler.storeToCache(" + yacyResponse.url().toNormalform(true) + ")"); - - // the cache does either not exist or is (supposed to be) stale - deleteFromCache(yacyResponse.url().hash()); - - if (cacheArray == null || cacheArray.length <= 0) return; - - yacyResponse.setContent(cacheArray); - try { - Cache.store(yacyResponse.url(), yacyResponse.getResponseHeader(), cacheArray); - sb.toIndexer(yacyResponse); - } catch (IOException e) { - //log.logWarning("cannot write " + response.url() + " to Cache (1): " + e.getMessage(), e); - } - } - }; - t.setPriority(Thread.MIN_PRIORITY); - t.start(); - } - - @Override - public void handleRemote(String target, RequestCompletion completion, HttpServletRequest request, - HttpServletResponse response) throws IOException, ServletException { - - sb.proxyLastAccess = System.currentTimeMillis(); - - RequestHeader proxyHeaders = ProxyHandler.convertHeaderFromJetty(request); - setProxyHeaderForClient(request, proxyHeaders); - - // send request - try (final HTTPClient client = new HTTPClient(ClientIdentification.yacyProxyAgent)) { - client.setTimout(timeout); - client.setHeader(proxyHeaders.entrySet()); - client.setRedirecting(false); - String queryString = request.getQueryString() != null ? "?" + request.getQueryString() : ""; - DigestURL digestURI = new DigestURL(request.getScheme(), request.getServerName(), request.getServerPort(), request.getRequestURI() + queryString); - if (request.getMethod().equals(HeaderFramework.METHOD_GET)) { - client.GET(digestURI, false); - } else if (request.getMethod().equals(HeaderFramework.METHOD_POST)) { - client.POST(digestURI, request.getInputStream(), request.getContentLength(), false); - } else if (request.getMethod().equals(HeaderFramework.METHOD_HEAD)) { - client.HEADResponse(digestURI, false); - } else { - throw new ServletException("Unsupported Request Method"); - } - HttpResponse clientresponse = client.getHttpResponse(); - int statusCode = clientresponse.getStatusLine().getStatusCode(); - final ResponseHeader responseHeaderLegacy = new ResponseHeader(statusCode, clientresponse.getAllHeaders()); - - if (responseHeaderLegacy.isEmpty()) { - throw new SocketException(clientresponse.getStatusLine().toString()); - } - cleanResponseHeader(clientresponse); - - // reserver cache entry - final net.yacy.crawler.retrieval.Request yacyRequest = new net.yacy.crawler.retrieval.Request( - null, - digestURI, - null, //requestHeader.referer() == null ? null : new DigestURI(requestHeader.referer()).hash(), - "", - responseHeaderLegacy.lastModified(), - sb.crawler.defaultProxyProfile.handle(), - 0, - sb.crawler.defaultProxyProfile.timezoneOffset()); //sizeBeforeDelete < 0 ? 0 : sizeBeforeDelete); - final Response yacyResponse = new Response( - yacyRequest, - null, - responseHeaderLegacy, - sb.crawler.defaultProxyProfile, - false, - null - ); - - final String storeError = yacyResponse.shallStoreCacheForProxy(); - final boolean storeHTCache = yacyResponse.profile().storeHTCache(); - final String supportError = TextParser.supports(yacyResponse.url(), yacyResponse.getMimeType()); - - if ( - /* - * Now we store the response into the htcache directory if - * a) the response is cacheable AND - */ - (storeError == null) && - /* - * b) the user has configured to use the htcache OR - * c) the content should be indexed - */ - ((storeHTCache) || (supportError != null)) - ) { - // we don't write actually into a file, only to RAM, and schedule writing the file. - int l = responseHeaderLegacy.size(); - final ByteArrayOutputStream byteStream = new ByteArrayOutputStream((l < 32) ? 32 : l); - final OutputStream toClientAndMemory = new MultiOutputStream(new OutputStream[] {response.getOutputStream(), byteStream}); - convertHeaderToJetty(clientresponse, response); - response.setStatus(statusCode); - client.writeTo(toClientAndMemory); - - // cached bytes - storeToCache(yacyResponse, byteStream.toByteArray()); - } else { - // no caching - /*if (log.isFine()) log.logFine(reqID +" "+ url.toString() + " not cached." + - " StoreError=" + ((storeError==null)?"None":storeError) + - " StoreHTCache=" + storeHTCache + - " SupportError=" + supportError);*/ - convertHeaderToJetty(clientresponse, response); - response.setStatus(statusCode); - - if (statusCode == HttpServletResponse.SC_OK) { // continue to serve header to client e.g. HttpStatus = 302 (while skiping content) - client.writeTo(response.getOutputStream()); // may throw exception on httpStatus=302 while gzip encoded inputstream - } - - } - } catch (final SocketException se) { - throw new ServletException("Socket Exception: " + se.getMessage()); - } - - // we handled this request, break out of handler chain - logProxyAccess(request); - completion.complete(); - } - - /** - * Convert ServletRequest header to modifiable YaCy RequestHeader - * - * @param request ServletRequest - * @return RequestHeader created from ServletRequest - */ - public static RequestHeader convertHeaderFromJetty(HttpServletRequest request) { - RequestHeader result = new RequestHeader(); - Enumeration<String> headerNames = request.getHeaderNames(); - while (headerNames.hasMoreElements()) { - String headerName = headerNames.nextElement(); - Enumeration<String> headers = request.getHeaders(headerName); - while (headers.hasMoreElements()) { - String header = headers.nextElement(); - result.add(headerName, header); - } - } - return result; - } - - /** - * adds specific header elements for the connection of the internal - * httpclient to the remote server according to local config - * - * @param header header for http client (already preset with headers from - * original ServletRequest) - * @param origServletRequest original request/header - */ - private void setProxyHeaderForClient(final HttpServletRequest origServletRequest, final HeaderFramework header) { - - header.remove(RequestHeader.KEEP_ALIVE); - header.remove(HeaderFramework.CONTENT_LENGTH); - - // setting the X-Forwarded-For header - if (sb.getConfigBool("proxy.sendXForwardedForHeader", true)) { - String ip = origServletRequest.getRemoteAddr(); - if (!Domains.isThisHostIP(ip)) { // if originator is local host no user ip to forward (= request from localhost) - header.put(HeaderFramework.X_FORWARDED_FOR, origServletRequest.getRemoteAddr()); - } - } - - String httpVersion = origServletRequest.getProtocol(); - HTTPDProxyHandler.modifyProxyHeaders(header, httpVersion); - } - - public final static synchronized void logProxyAccess(HttpServletRequest request) { - - final StringBuilder logMessage = new StringBuilder(80); - - // Timestamp - logMessage.append(GenericFormatter.SHORT_SECOND_FORMATTER.format(new Date())); - logMessage.append(' '); - - // Remote Host - final String clientIP = request.getRemoteAddr(); - logMessage.append(clientIP); - logMessage.append(' '); - - // Method - final String requestMethod = request.getMethod(); - logMessage.append(requestMethod); - logMessage.append(' '); - - // URL - logMessage.append(request.getRequestURL()); - final String requestArgs = request.getQueryString(); - if (requestArgs != null) { - logMessage.append("?").append(requestArgs); - } - - HTTPDProxyHandler.proxyLog.fine(logMessage.toString()); - - } -} diff --git a/source/net/yacy/http/RequestCompletion.java b/source/net/yacy/http/RequestCompletion.java deleted file mode 100644 index 7dce04d59..000000000 --- a/source/net/yacy/http/RequestCompletion.java +++ /dev/null @@ -1,28 +0,0 @@ -/** - * RequestCompletion - * Copyright 2026 by Michael Peter Christen - * First released 12.07.2026 at https://yacy.net - * - * This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program in the file lgpl21.txt - * If not, see <http://www.gnu.org/licenses/>. - */ - -package net.yacy.http; - -/** Container-neutral callback used when a handler has completed a request. */ -@FunctionalInterface -public interface RequestCompletion { - - void complete(); -} diff --git a/source/net/yacy/http/AdminAuthenticationContext.java b/source/net/yacy/http/ServletRequestHeaderAdapter.java index 5080c9b05..e2d36d796 100644 --- a/source/net/yacy/http/AdminAuthenticationContext.java +++ b/source/net/yacy/http/ServletRequestHeaderAdapter.java @@ -1,5 +1,5 @@ /** - * AdminAuthenticationContext + * ServletRequestHeaderAdapter * Copyright 2026 by Michael Peter Christen * First released 12.07.2026 at https://yacy.net * @@ -20,26 +20,28 @@ package net.yacy.http; -import net.yacy.cora.protocol.Domains; +import java.util.Enumeration; -/** Request-bound facts needed while the container verifies admin credentials. */ -public final class AdminAuthenticationContext { +import javax.servlet.http.HttpServletRequest; - private static final ThreadLocal<String> SOCKET_PEER_IP = new ThreadLocal<>(); +import net.yacy.cora.protocol.RequestHeader; - private AdminAuthenticationContext() { - } - - public static void setSocketPeerIp(final String ip) { - SOCKET_PEER_IP.set(ip); - } +/** Converts servlet request headers into YaCy's mutable header representation. */ +public final class ServletRequestHeaderAdapter { - public static void clear() { - SOCKET_PEER_IP.remove(); + private ServletRequestHeaderAdapter() { } - public static boolean isLocalhostRequest() { - final String ip = SOCKET_PEER_IP.get(); - return ip != null && Domains.isLocalhost(ip); + public static RequestHeader from(final HttpServletRequest request) { + final RequestHeader result = new RequestHeader(); + final Enumeration<String> names = request.getHeaderNames(); + while (names.hasMoreElements()) { + final String name = names.nextElement(); + final Enumeration<String> values = request.getHeaders(name); + while (values.hasMoreElements()) { + result.add(name, values.nextElement()); + } + } + return result; } } diff --git a/source/net/yacy/http/YaCyDigestCredential.java b/source/net/yacy/http/YaCyDigestCredential.java deleted file mode 100644 index f11c676d6..000000000 --- a/source/net/yacy/http/YaCyDigestCredential.java +++ /dev/null @@ -1,99 +0,0 @@ -// -// YaCyDigestCredential -// Copyright 2011 by Florian Richter -// First released 16.04.2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import net.yacy.search.Switchboard; -import net.yacy.search.SwitchboardConstants; - -import org.eclipse.jetty.util.security.Credential; - - - -/** - * implementation of YaCy's admin password as jetty Credential - * supporting BASIC and DIGEST authentication - * and using MD5 digested passwords/credentials. Following RFC recommendation (to use the realm in MD5 hash) - * expecting a MD5 hash in format MD5( username:realm:password ), realm configured in yacy.init adminRealm - * (a credential in format MD5( username:password ) is also accepted with BASIC auth) - * - * This is a thin adapter to the servlet container: the password verification - * logic is in the container neutral {@link AdminSecurity}. - */ -public class YaCyDigestCredential extends Credential { - - private static final long serialVersionUID = -3527894085562480001L; - - private String hash; // remember password hash, either MD5(Base64(user:pwd)) or with encryption prefix "MD5:" + MD5(user:realm:pwd) - private String foruser; // remember the user as YaCy credential is username:pwd (not just pwd) - private Credential c; - - @Override - public boolean check(Object credentials) { - - if (credentials instanceof Credential) { // for DIGEST auth - if (this.c == null) { - /* credential may be null after switching from BASIC to DIGEST authentication without re-encoding the password */ - return false; - } - Credential credential = (Credential) credentials; - return credential.check(this.c); - } - if (credentials instanceof String) { // for BASIC auth - final Switchboard sb = Switchboard.getSwitchboard(); - // The "fromLocalhost" exception (see AdminSecurity.checkAdminPassword) lets a - // localhost caller submit the stored password hash itself as the password. This - // is what the bin/*.sh scripts do via bin/apicall.sh: they read the hash from the - // configuration file to steer a local peer, because the cleartext password is - // never stored anywhere. - // - // We must therefore know whether THIS request comes from localhost. Jetty's - // Credential.check() is not given the request, so YaCySecurityHandler publishes - // the true socket peer IP through AdminAuthenticationContext for this request. - return AdminSecurity.checkAdminPassword(this.foruser, this.hash, - sb.getConfig(SwitchboardConstants.ADMIN_REALM, ""), - sb.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin"), - AdminAuthenticationContext.isLocalhostRequest(), - (String) credentials); - } - throw new UnsupportedOperationException(); - } - - /** - * create Credential object from config file hash - * - * @param configHash hash as in config file hash(adminuser:pwd) - * @return - */ - public static Credential getCredentialForAdmin(String username, String configHash) { - YaCyDigestCredential yc = new YaCyDigestCredential(); - if (configHash.startsWith("MD5:")) { - yc.c = Credential.getCredential(configHash); // for DIGEST auth - } - yc.foruser = username; - yc.hash = configHash; - return yc; - } - -} diff --git a/source/net/yacy/http/YaCyErrorHandler.java b/source/net/yacy/http/YaCyErrorHandler.java deleted file mode 100644 index c83938855..000000000 --- a/source/net/yacy/http/YaCyErrorHandler.java +++ /dev/null @@ -1,53 +0,0 @@ -//
-// YaCyErrorHandler
-// ----------------
-// Copyright 2014 by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany
-// First released 2014 at https://yacy.net
-//
-// $LastChangedDate$
-// $LastChangedRevision$
-// $LastChangedBy$
-//
-// This library is free software; you can redistribute it and/or
-// modify it under the terms of the GNU Lesser General Public
-// License as published by the Free Software Foundation; either
-// version 2.1 of the License, or (at your option) any later version.
-//
-// This library is distributed in the hope that it will be useful,
-// but WITHOUT ANY WARRANTY; without even the implied warranty of
-// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
-// Lesser General Public License for more details.
-//
-// You should have received a copy of the GNU Lesser General Public License
-// along with this program in the file lgpl21.txt
-// If not, see <http://www.gnu.org/licenses/>.
-//
-
-package net.yacy.http;
-
-import java.io.IOException;
-import java.io.Writer;
-import javax.servlet.http.HttpServletRequest;
-import net.yacy.peers.operation.yacyBuildProperties;
-import org.eclipse.jetty.server.handler.ErrorHandler;
-
-/**
- * Custom Handler to serve error pages called by the HttpResponse.sendError method
- */
-public class YaCyErrorHandler extends ErrorHandler {
-
- @Override
- protected void writeErrorPageBody(HttpServletRequest request, Writer writer, int code, String message, boolean showStacks)
- throws IOException {
- String uri = request.getRequestURI();
-
- writeErrorPageMessage(request, writer, code, message, uri);
- if (showStacks) {
- writeErrorPageStacks(request, writer);
- }
- writer.write("<br/><hr /><small>YaCy " + yacyBuildProperties.getVersion() + " - <i> powered by Jetty </i> - </small>");
- for (int i = 0; i < 20; i++) {
- writer.write("<br/> \n");
- }
- }
-}
diff --git a/source/net/yacy/http/YaCyHttpServer.java b/source/net/yacy/http/YaCyHttpServer.java index 60cd9c219..0279d8f2f 100644 --- a/source/net/yacy/http/YaCyHttpServer.java +++ b/source/net/yacy/http/YaCyHttpServer.java @@ -23,8 +23,8 @@ package net.yacy.http; /** * Servlet-container neutral interface to YaCy's embedded http server. * This is the only view the rest of the code base has on the server; - * all container specific code (currently Jetty 9, see {@link Jetty9HttpServerImpl}) - * stays behind this interface to ease migration to newer container versions. + * all container-specific code in {@link Jetty12HttpServer} and + * {@link Jetty12ProxyChain} stays behind this interface. */ public interface YaCyHttpServer { diff --git a/source/net/yacy/http/YaCyLoginService.java b/source/net/yacy/http/YaCyLoginService.java deleted file mode 100644 index ef9086cf3..000000000 --- a/source/net/yacy/http/YaCyLoginService.java +++ /dev/null @@ -1,123 +0,0 @@ -// -// YaCyLoginService -// Copyright 2011 by Florian Richter -// First released 16.04.2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import net.yacy.search.Switchboard; -import net.yacy.search.SwitchboardConstants; -import org.eclipse.jetty.security.AbstractLoginService; -import org.eclipse.jetty.security.HashLoginService; -import org.eclipse.jetty.security.LoginService; -import org.eclipse.jetty.security.UserStore; -import org.eclipse.jetty.server.UserIdentity; -import org.eclipse.jetty.util.security.Credential; - -/** - * Jetty login service for YaCy's built-in administrator account. - * With DIGEST auth Jetty uses the name of the login service - * as realmname (which is part of all password hashes) - */ -public class YaCyLoginService extends HashLoginService implements LoginService { - - private UserStore _userStore; // user cache for known/authenticated users - - /** - * Initialize a user cache - * @throws Exception - */ - @Override - protected void doStart() throws Exception { - _userStore = new UserStore(); - this.setUserStore(_userStore); - super.doStart(); - } - - /** - * Free space used by user cache - * @throws Exception - */ - @Override - protected void doStop() throws Exception { - super.doStop(); - if (_userStore != null) { - _userStore.stop(); - _userStore = null; - } - } - - /** - * Load the built-in administrator from the authenticated-user cache or configuration. - * @param username - * @return known user or null - */ - @Override - protected AbstractLoginService.UserPrincipal loadUserInfo(String username) { - if (username == null || username.isEmpty()) { - return null; // quick exit - } - - AbstractLoginService.UserPrincipal theUser = super.loadUserInfo(username); // load from cache (the internal _userStore) - if (theUser == null) { - final Switchboard sb = Switchboard.getSwitchboard(); - final String adminuser = sb.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin"); - Credential credential = null; - String[] roles = null; - if (username.equals(adminuser)) { - final String adminAccountBase64MD5 = sb.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, ""); - // in YaCy the credential hash is composed of username:pwd so the username is needed to create valid credential - // not just the password (as usually in Jetty). As the accountname for the std. adminuser is not stored a useridentity - // is created for current user (and the pwd checked against the stored username:pwd setting) - credential = YaCyDigestCredential.getCredentialForAdmin(username, adminAccountBase64MD5); - // TODO: YaCy user:pwd hashes should longterm likely be switched to separable username + pwd-hash entries - // and/or the standard admin account username should be fix = "admin" - roles = new String[]{SwitchboardConstants.ADMIN_ACCOUNT_ROLE}; - } - - if (credential != null) { // if credential exist, user is known, create or get info - theUser = new AbstractLoginService.UserPrincipal(username, credential); - _userStore.addUser(username, credential, roles); // add to jetty user cache - _userStore.getUserIdentity(username).getUserPrincipal(); - theUser.authenticate(credential); - } - } - return theUser; - } - - /** - * Delete the administrator identity from the internal cache. When present, - * the identity is logged out before removal. - * @param username - * @return true if user deleted, if not found in user cache false - */ - public boolean removeUser(String username) { - UserIdentity uid = _userStore.getUserIdentity(username); - if (uid != null) { - logout(uid); - _userStore.removeUser(username); - return true; - } - return false; - } - -} diff --git a/source/net/yacy/http/YaCySecurityHandler.java b/source/net/yacy/http/YaCySecurityHandler.java deleted file mode 100644 index 9e6923d07..000000000 --- a/source/net/yacy/http/YaCySecurityHandler.java +++ /dev/null @@ -1,130 +0,0 @@ -// -// YaCySecurityHandler -// Copyright 2011 by Florian Richter -// First released 16.04.2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import java.io.IOException; - -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletResponse; - -import net.yacy.cora.protocol.RequestHeader; -import net.yacy.search.Switchboard; -import net.yacy.search.SwitchboardConstants; -import net.yacy.server.serverAccessTracker; - -import org.eclipse.jetty.security.ConstraintSecurityHandler; -import org.eclipse.jetty.security.RoleInfo; -import org.eclipse.jetty.server.Request; - -/** - * jetty security handler - * demands authentication for pages with _p. inside - * and updates AccessTracker - * - * This is a thin adapter to the servlet container: the decision logic is in - * the container-neutral {@link AdminAccessPolicy} and {@link AdminSecurity}. - */ -public class YaCySecurityHandler extends ConstraintSecurityHandler { - - /** - * Request-scoped hand-off of the client IP to the admin password check. - * <p> - * <b>What happens here and why:</b> YaCy accepts the stored admin password hash itself - * as a password, but only for requests originating from localhost. This is used by the - * {@code bin/*.sh} scripts (via {@code bin/apicall.sh}), which read the hash from the - * configuration file to steer a local peer - the cleartext password is never stored - * anywhere. That exception is evaluated in {@link YaCyDigestCredential#check(Object)}, - * which the servlet container invokes through the {@link YaCyLoginService}. - * <p> - * The problem: Jetty's {@link org.eclipse.jetty.util.security.Credential} API only - * receives the submitted password, not the request, so the credential can not tell on - * its own whether the request comes from localhost. This handler is the single place - * that both (a) sees the request and (b) spans the whole authentication of that request: - * {@code prepareConstraintInfo()}, the authenticator, the login service and finally - * {@code Credential.check()} all run synchronously inside {@code super.handle()} on this - * same thread. We therefore publish the request's true socket peer IP here for the - * duration of the request and clear it in a {@code finally} block (so it can not leak - * across pooled request threads). The credential check then reads a value that is bound - * to the exact request being authenticated. - * <p> - * This replaced a former process-global "last localhost access" timestamp with a 100ms - * window: that value was not tied to the request being checked and could be opened by - * unrelated concurrent localhost traffic. The IP used here is the real socket peer - * ({@link Request#getRemoteAddr()}); the client-controlled and spoofable X-Real-IP - * header is never consulted for this authentication decision. - */ - @Override - public void handle(final String pathInContext, final Request baseRequest, - final HttpServletRequest request, final HttpServletResponse response) - throws IOException, ServletException { - AdminAuthenticationContext.setSocketPeerIp(baseRequest.getRemoteAddr()); - try { - super.handle(pathInContext, baseRequest, request, response); - } finally { - AdminAuthenticationContext.clear(); - } - } - - /** - * create the constraint for the given path - * for urls containing *_p. (like info_p.html) admin access is required, - * on localhost = admin setting no constraint is set - * @param pathInContext - * @param request - * @return RoleInfo with - * isChecked=true if any security contraint applies (compare reference implementation org.eclipse.jetty.security.ConstraintSecurityHandler) - * role = "admin" for resource name containint _p. - */ - @Override - protected RoleInfo prepareConstraintInfo(String pathInContext, Request request) { - final Switchboard sb = Switchboard.getSwitchboard(); - - // Use the true socket peer IP for the access-control decision below, never the - // client-controlled (spoofable) X-Real-IP header that RequestHeader.client() would apply. - final String remoteip = request.getRemoteAddr(); - serverAccessTracker.track(remoteip, pathInContext); - - final AdminAccessPolicy policy = new AdminAccessPolicy( - sb.getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_All_PAGES, false), - sb.isRobinsonMode() && !sb.isPublicRobinson(), - sb.getConfigBool(SwitchboardConstants.PUBLIC_SEARCHPAGE, true), - sb.getConfigBool(SwitchboardConstants.ADMIN_ACCOUNT_FOR_LOCALHOST, false), - sb.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_USER_NAME, "admin"), - sb.getConfig(SwitchboardConstants.ADMIN_ACCOUNT_B64MD5, "")); - final AdminAccessPolicy.Decision decision = policy.decide(pathInContext, remoteip, - request.getHeader(RequestHeader.REFERER), request.getHeader(RequestHeader.AUTHORIZATION)); - if (decision == AdminAccessPolicy.Decision.PUBLIC) { - return super.prepareConstraintInfo(pathInContext, request); - } - if (decision == AdminAccessPolicy.Decision.LOCAL_BYPASS) { - return null; - } - RoleInfo roleinfo = new RoleInfo(); - roleinfo.setChecked(true); // RoleInfo.setChecked() : in Jetty this means - marked to have any security constraint - roleinfo.addRole(SwitchboardConstants.ADMIN_ACCOUNT_ROLE); - return roleinfo; - } -} diff --git a/source/net/yacy/http/YacyDomainHandler.java b/source/net/yacy/http/YacyDomainHandler.java deleted file mode 100644 index a61a72fac..000000000 --- a/source/net/yacy/http/YacyDomainHandler.java +++ /dev/null @@ -1,125 +0,0 @@ -// -// YacyDomainHandler -// Copyright 2005 by Michael Peter Christen; mc@yacy.net, Frankfurt a. M., Germany -// Copyright 2011 by Florian Richter -// First released 13.04.2011 at https://yacy.net -// -// $LastChangedDate$ -// $LastChangedRevision$ -// $LastChangedBy$ -// -// This library is free software; you can redistribute it and/or -// modify it under the terms of the GNU Lesser General Public -// License as published by the Free Software Foundation; either -// version 2.1 of the License, or (at your option) any later version. -// -// This library is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU -// Lesser General Public License for more details. -// -// You should have received a copy of the GNU Lesser General Public License -// along with this program in the file lgpl21.txt -// If not, see <http://www.gnu.org/licenses/>. -// - -package net.yacy.http; - -import java.io.IOException; -import java.util.Enumeration; -import java.util.Vector; - -import javax.servlet.RequestDispatcher; -import javax.servlet.ServletException; -import javax.servlet.http.HttpServletRequest; -import javax.servlet.http.HttpServletRequestWrapper; -import javax.servlet.http.HttpServletResponse; - -import net.yacy.cora.protocol.Domains; -import net.yacy.server.http.AlternativeDomainNames; - -import org.eclipse.jetty.server.Request; -import org.eclipse.jetty.server.handler.AbstractHandler; - - -/** - * handling of request to virtual ".yacy" domain determines public adress from - * seedlist and forwards modified/wrapped request to it - * - * Note for servlet container migration: this must stay a container level - * handler (it can not become a servlet filter), because it cooperates with the - * ProxyHandler chain: the re-dispatched request with the rewritten (remote) - * host is picked up and forwarded to the peer by the transparent proxy - * handlers, before the local servlet context would handle it. - */ -public class YacyDomainHandler extends AbstractHandler { - - private AlternativeDomainNames alternativeResolvers; - - public void setAlternativeResolver(AlternativeDomainNames resolver) { - this.alternativeResolvers = resolver; - } - - @Override - public void handle(String target, Request baseRequest, HttpServletRequest request, - HttpServletResponse response) throws IOException, ServletException { - String host = request.getServerName(); // is hostname (without port) - String resolved = alternativeResolvers.resolve(host); // is a host|ip with port - if (resolved != null) { - int newPort = Domains.stripToPort(resolved); - String newHost = Domains.stripToHostName(resolved); - if (alternativeResolvers.myIPs().contains(newHost)) return; - if (Domains.isLocal(newHost, null)) return; - RequestDispatcher dispatcher = request.getRequestDispatcher(target); - dispatcher.forward(new DomainRequestWrapper(request, newHost, newPort), response); - baseRequest.setHandled(true); - } - } - - private class DomainRequestWrapper extends HttpServletRequestWrapper { - - final private String newServerName; - final private int newServerPort; - - public DomainRequestWrapper(HttpServletRequest request, String serverName, int serverPort) { - super(request); - this.newServerName = serverName; - this.newServerPort = serverPort; - } - - @Override - public String getServerName() { - return newServerName; - } - - @Override - public int getServerPort() { - return newServerPort; - } - - @Override - public StringBuffer getRequestURL() { - StringBuffer buf = new StringBuffer(this.getScheme() + "://" + newServerName + ":" + newServerPort + this.getPathInfo()); - return buf; - } - - @Override - public String getHeader(String name) { - if (name.equals("Host")) { - return newServerName + (newServerPort != 80 ? ":" + newServerPort : ""); - } - return super.getHeader(name); - } - - @Override - public Enumeration<String> getHeaders(String name) { - if (name.equals("Host")) { - Vector<String> header = new Vector<String>(); - header.add(newServerName + (newServerPort != 80 ? ":" + newServerPort : "")); - return header.elements(); - } - return super.getHeaders(name); - } - } - -} diff --git a/source/net/yacy/http/servlets/Jetty9ServletResource.java b/source/net/yacy/http/servlets/Jetty9ServletResource.java deleted file mode 100644 index 0a4054e45..000000000 --- a/source/net/yacy/http/servlets/Jetty9ServletResource.java +++ /dev/null @@ -1,119 +0,0 @@ -/** - * Jetty9ServletResource - * Copyright 2026 by Michael Peter Christen - * First released 12.07.2026 at https://yacy.net - * - * This library is free software; you can redistribute it and/or - * modify it under the terms of the GNU Lesser General Public - * License as published by the Free Software Foundation; either - * version 2.1 of the License, or (at your option) any later version. - * - * This library is distributed in the hope that it will be useful, - * but WITHOUT ANY WARRANTY; without even the implied warranty of - * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU - * Lesser General Public License for more details. - * - * You should have received a copy of the GNU Lesser General Public License - * along with this program in the file lgpl21.txt - * If not, see <http://www.gnu.org/licenses/>. - */ - -package net.yacy.http.servlets; - -import java.io.File; -import java.io.IOException; -import java.io.InputStream; -import java.io.OutputStream; -import java.net.URL; - -import org.eclipse.jetty.util.resource.Resource; - -/** Jetty 9 adapter for the static-resource operations used by YaCy. */ -final class Jetty9ServletResource implements ServletResource { - - private final Resource delegate; - - private Jetty9ServletResource(final Resource delegate) { - this.delegate = delegate; - } - - static void disableDefaultCaches() { - Resource.setDefaultUseCaches(false); - } - - static ServletResource from(final String location) throws IOException { - return wrap(Resource.newResource(location)); - } - - static ServletResource from(final File file) throws IOException { - return wrap(Resource.newResource(file)); - } - - static ServletResource from(final URL url) throws IOException { - return wrap(Resource.newResource(url)); - } - - private static ServletResource wrap(final Resource resource) { - return resource == null ? null : new Jetty9ServletResource(resource); - } - - @Override - public ServletResource addPath(final String path) throws IOException { - return wrap(this.delegate.addPath(path)); - } - - @Override - public boolean exists() { - return this.delegate.exists(); - } - - @Override - public boolean isDirectory() { - return this.delegate.isDirectory(); - } - - @Override - public long lastModified() { - return this.delegate.lastModified(); - } - - @Override - public long length() { - return this.delegate.length(); - } - - @Override - public String getName() { - return this.delegate.getName(); - } - - @Override - public File getFile() throws IOException { - return this.delegate.getFile(); - } - - @Override - public InputStream getInputStream() throws IOException { - return this.delegate.getInputStream(); - } - - @Override - public String getListHTML(final String base, final boolean parent, final String query) throws IOException { - return this.delegate.getListHTML(base, parent, query); - } - - @Override - public void writeTo(final OutputStream output, final long start, final long count) throws IOException { - this.delegate.writeTo(output, start, count); - } - - @Override - public void close() { - this.delegate.close(); - } - - @Override - public String toString() { - return this.delegate.toString(); - } -} diff --git a/source/net/yacy/http/servlets/LLMAdminProxyServlet.java b/source/net/yacy/http/servlets/LLMAdminProxyServlet.java index 1f8ffd810..37e89d2d0 100644 --- a/source/net/yacy/http/servlets/LLMAdminProxyServlet.java +++ b/source/net/yacy/http/servlets/LLMAdminProxyServlet.java @@ -219,7 +219,7 @@ public class LLMAdminProxyServlet extends HttpServlet { } /** - * Check for administrator access, mirroring the rules of YaCySecurityHandler: + * Check for administrator access, mirroring the active server security handler: * localhost access with the localhost-admin setting is granted without credentials, * everything else requires an authenticated user with the admin right. Sends the * authentication challenge (401) when credentials are missing. diff --git a/source/net/yacy/http/servlets/MonitorFilter.java b/source/net/yacy/http/servlets/MonitorFilter.java index 23b5affb4..5f9b495f5 100644 --- a/source/net/yacy/http/servlets/MonitorFilter.java +++ b/source/net/yacy/http/servlets/MonitorFilter.java @@ -46,7 +46,7 @@ import net.yacy.cora.protocol.RequestHeader; * * This is a plain servlet filter (former Jetty handler MonitorHandler); the * tracking entry of a connection is removed on connection close by a - * servlet-container specific listener, see Jetty9HttpServerImpl. + * servlet-container specific listener, see Jetty12HttpServer. */ public class MonitorFilter implements Filter { diff --git a/source/net/yacy/http/servlets/ServletResource.java b/source/net/yacy/http/servlets/ServletResource.java index e5ee160fa..ea306b8d3 100644 --- a/source/net/yacy/http/servlets/ServletResource.java +++ b/source/net/yacy/http/servlets/ServletResource.java @@ -20,34 +20,213 @@ package net.yacy.http.servlets; +import java.io.File; import java.io.IOException; import java.io.InputStream; import java.io.OutputStream; -import java.io.File; +import java.net.URI; +import java.net.URISyntaxException; +import java.net.URL; +import java.net.URLConnection; +import java.nio.file.Files; +import java.nio.file.Path; +import java.util.Comparator; +import java.util.stream.Stream; + +/** JDK-only static resource implementation shared by all servlet containers. */ +final class ServletResource { + + private static final int COPY_BUFFER_SIZE = 8192; + + private final Path path; + private final URL url; + + private ServletResource(final Path path, final URL url) { + this.path = path; + this.url = url; + } + + static ServletResource from(final String location) throws IOException { + if (location == null) { + return null; + } + if (location.contains("://") || location.startsWith("file:") || location.startsWith("jar:")) { + return from(new URL(location)); + } + return from(Path.of(location)); + } + + static ServletResource from(final File file) { + return file == null ? null : from(file.toPath()); + } + + static ServletResource from(final URL url) throws IOException { + if (url == null) { + return null; + } + if ("file".equalsIgnoreCase(url.getProtocol())) { + try { + return from(Path.of(url.toURI())); + } catch (final URISyntaxException | IllegalArgumentException error) { + throw new IOException("Invalid file resource URL: " + url, error); + } + } + return new ServletResource(null, url); + } + + private static ServletResource from(final Path path) { + return new ServletResource(path.toAbsolutePath().normalize(), null); + } + + public ServletResource addPath(final String child) throws IOException { + if (child == null) { + throw new IllegalArgumentException("Resource path must not be null"); + } + final String relative = child.replace('\\', '/').replaceFirst("^/+", ""); + final Path normalizedRelative = Path.of(relative).normalize(); + if (normalizedRelative.isAbsolute() || normalizedRelative.startsWith("..")) { + throw new IllegalArgumentException("Resource path escapes its base: " + child); + } + if (this.path != null) { + final Path resolved = this.path.resolve(normalizedRelative).normalize(); + if (!resolved.startsWith(this.path)) { + throw new IllegalArgumentException("Resource path escapes its base: " + child); + } + return from(resolved); + } + try { + final URI base = this.url.toURI(); + final String baseText = base.toString().endsWith("/") ? base.toString() : base + "/"; + return from(URI.create(baseText).resolve(relative).normalize().toURL()); + } catch (final URISyntaxException error) { + throw new IOException("Invalid resource URL: " + this.url, error); + } + } + + public boolean exists() { + if (this.path != null) { + return Files.exists(this.path); + } + try (InputStream ignored = this.openConnection().getInputStream()) { + return true; + } catch (final IOException error) { + return false; + } + } + + public boolean isDirectory() { + return this.path != null && Files.isDirectory(this.path); + } -/** Container-neutral view of a static resource served by YaCy. */ -public interface ServletResource extends AutoCloseable { + public long lastModified() { + try { + return this.path != null + ? Files.getLastModifiedTime(this.path).toMillis() + : this.openConnection().getLastModified(); + } catch (final IOException error) { + return 0L; + } + } - ServletResource addPath(String path) throws IOException; + public long length() { + try { + return this.path != null ? Files.size(this.path) : this.openConnection().getContentLengthLong(); + } catch (final IOException error) { + return -1L; + } + } - boolean exists(); + public String getName() { + return this.path != null ? this.path.toString() : this.url.toExternalForm(); + } - boolean isDirectory(); + public File getFile() throws IOException { + if (this.path == null) { + throw new IOException("Resource is not backed by a file: " + this.url); + } + return this.path.toFile(); + } - long lastModified(); + public InputStream getInputStream() throws IOException { + return this.path != null ? Files.newInputStream(this.path) : this.openConnection().getInputStream(); + } - long length(); + public String getListHTML(final String base, final boolean parent, final String query) throws IOException { + if (this.path == null || !Files.isDirectory(this.path)) { + return null; + } + final StringBuilder html = new StringBuilder(512); + html.append("<!DOCTYPE html><html><head><meta charset=\"utf-8\"><title>Directory: ") + .append(escapeHtml(base)).append("</title></head><body><h1>Directory: ") + .append(escapeHtml(base)).append("</h1><ul>"); + if (parent) { + html.append("<li><a href=\"../\">../</a></li>"); + } + try (Stream<Path> children = Files.list(this.path)) { + children.sorted(Comparator.comparing(path -> path.getFileName().toString(), String.CASE_INSENSITIVE_ORDER)) + .forEach(child -> appendDirectoryEntry(html, base, child)); + } + return html.append("</ul></body></html>").toString(); + } - String getName(); + private static void appendDirectoryEntry(final StringBuilder html, final String base, final Path child) { + final String name = child.getFileName().toString(); + final boolean directory = Files.isDirectory(child); + try { + final String encoded = new URI(null, null, name, null).toASCIIString(); + html.append("<li><a href=\"").append(escapeHtml(base)).append(encoded); + if (directory) { + html.append('/'); + } + html.append("\">").append(escapeHtml(name)); + if (directory) { + html.append('/'); + } + html.append("</a></li>"); + } catch (final URISyntaxException error) { + throw new IllegalArgumentException("Invalid directory entry: " + name, error); + } + } - File getFile() throws IOException; + private static String escapeHtml(final String value) { + return value.replace("&", "&").replace("\"", """) + .replace("<", "<").replace(">", ">"); + } - InputStream getInputStream() throws IOException; + public void writeTo(final OutputStream output, final long start, final long count) throws IOException { + if (start < 0 || count < -1) { + throw new IllegalArgumentException("Invalid resource range: " + start + "+" + count); + } + try (InputStream input = this.getInputStream()) { + input.skipNBytes(start); + final byte[] buffer = new byte[COPY_BUFFER_SIZE]; + long remaining = count; + while (remaining != 0) { + final int requested = remaining < 0 ? buffer.length : (int) Math.min(buffer.length, remaining); + final int read = input.read(buffer, 0, requested); + if (read < 0) { + break; + } + output.write(buffer, 0, read); + if (remaining > 0) { + remaining -= read; + } + } + } + } - String getListHTML(String base, boolean parent, String query) throws IOException; + private URLConnection openConnection() throws IOException { + final URLConnection connection = this.url.openConnection(); + connection.setUseCaches(false); + return connection; + } - void writeTo(OutputStream output, long start, long count) throws IOException; + public void close() { + // Streams are opened per operation and closed by their caller. + } @Override - void close(); + public String toString() { + return this.getName(); + } } diff --git a/source/net/yacy/http/servlets/UrlProxyServlet.java b/source/net/yacy/http/servlets/UrlProxyServlet.java index 378664448..32101d132 100644 --- a/source/net/yacy/http/servlets/UrlProxyServlet.java +++ b/source/net/yacy/http/servlets/UrlProxyServlet.java @@ -45,7 +45,7 @@ import net.yacy.cora.protocol.HeaderFramework; import net.yacy.cora.protocol.RequestHeader; import net.yacy.cora.protocol.ResponseHeader; import net.yacy.cora.util.ConcurrentLog; -import net.yacy.http.ProxyHandler; +import net.yacy.http.ServletRequestHeaderAdapter; import net.yacy.kelondro.util.FileUtils; import net.yacy.search.Switchboard; import net.yacy.server.http.ChunkedInputStream; @@ -152,7 +152,7 @@ public class UrlProxyServlet extends HttpServlet implements Servlet { hostwithport += ":" + proxyurl.getPort(); } // 4 - get target url - RequestHeader yacyRequestHeader = ProxyHandler.convertHeaderFromJetty(request); + RequestHeader yacyRequestHeader = ServletRequestHeaderAdapter.from(request); yacyRequestHeader.remove(RequestHeader.KEEP_ALIVE); yacyRequestHeader.remove(HeaderFramework.CONTENT_LENGTH); diff --git a/source/net/yacy/http/servlets/YaCyDefaultServlet.java b/source/net/yacy/http/servlets/YaCyDefaultServlet.java index e90a7619f..b1234b989 100644 --- a/source/net/yacy/http/servlets/YaCyDefaultServlet.java +++ b/source/net/yacy/http/servlets/YaCyDefaultServlet.java @@ -141,7 +141,8 @@ public class YaCyDefaultServlet extends HttpServlet { protected static final File TMPDIR = new File(System.getProperty("java.io.tmpdir"));
protected static final int SIZE_FILE_THRESHOLD = 1024 * 1024 * 1024; // 1GB is a lot but appropriate for multi-document pushed using the push_p.json servlet
protected static final FileItemFactory DISK_FILE_ITEM_FACTORY = new DiskFileItemFactory(SIZE_FILE_THRESHOLD, TMPDIR);
- /* ------------------------------------------------------------ */
+
+
@Override
public void init() throws UnavailableException {
final Switchboard sb = Switchboard.getSwitchboard();
@@ -159,14 +160,12 @@ public class YaCyDefaultServlet extends HttpServlet { this._acceptRanges = this.getInitBoolean("acceptRanges", this._acceptRanges);
this._dirAllowed = this.getInitBoolean("dirAllowed", this._dirAllowed);
- Jetty9ServletResource.disableDefaultCaches(); // caching is handled internally (prevent double caching) -
- final String rb = this.getInitParameter("resourceBase");
- try {
- if (rb != null) {
- this._resourceBase = Jetty9ServletResource.from(rb); - } else {
- this._resourceBase = Jetty9ServletResource.from(sb.getConfig(SwitchboardConstants.HTROOT_PATH, SwitchboardConstants.HTROOT_PATH_DEFAULT)); //default + final String rb = this.getInitParameter("resourceBase"); + try { + if (rb != null) { + this._resourceBase = ServletResource.from(rb); + } else { + this._resourceBase = ServletResource.from(sb.getConfig(SwitchboardConstants.HTROOT_PATH, SwitchboardConstants.HTROOT_PATH_DEFAULT)); //default }
} catch (final IOException e) { ConcurrentLog.severe("FILEHANDLER", "event=http.resource subsystem=http result=missing-resource-base reason=" + e.getMessage()); @@ -179,7 +178,6 @@ public class YaCyDefaultServlet extends HttpServlet { this.templateMethodCache = new ConcurrentHashMap<>();
}
- /* ------------------------------------------------------------ */
protected boolean getInitBoolean(final String name, final boolean dft) {
final String value = this.getInitParameter(name);
if (value == null || value.length() == 0) {
@@ -192,7 +190,6 @@ public class YaCyDefaultServlet extends HttpServlet { || value.startsWith("1"));
}
- /* ------------------------------------------------------------ */
/**
* get Resource to serve. Map a path to a resource. The default
* implementation calls HttpContext.getResource but derived servlets may
@@ -208,7 +205,7 @@ public class YaCyDefaultServlet extends HttpServlet { r = this._resourceBase.addPath(pathInContext);
} else {
final URL u = this._servletContext.getResource(pathInContext);
- r = Jetty9ServletResource.from(u); + r = ServletResource.from(u); }
if (ConcurrentLog.isFine("FILEHANDLER")) {
@@ -220,13 +217,11 @@ public class YaCyDefaultServlet extends HttpServlet { return r;
}
-
- /* ------------------------------------------------------------ */
+
protected boolean hasDefinedRange(final Enumeration<String> reqRanges) {
return (reqRanges != null && reqRanges.hasMoreElements());
}
- /* ------------------------------------------------------------ */
@Override
protected void doGet(final HttpServletRequest request, final HttpServletResponse response)
throws ServletException, IOException {
@@ -281,7 +276,7 @@ public class YaCyDefaultServlet extends HttpServlet { if (!hasClass && (resource == null || !resource.exists()) && !pathInContext.contains("..")) {
// try to get this in the alternative htDocsPath
if (resource != null) resource.close();
- resource = Jetty9ServletResource.from(new File(this._htDocsPath, pathInContext)); + resource = ServletResource.from(new File(this._htDocsPath, pathInContext)); }
if (ConcurrentLog.isFine("FILEHANDLER")) {
@@ -363,7 +358,6 @@ public class YaCyDefaultServlet extends HttpServlet { }
}
- /* ------------------------------------------------------------ */
@Override
protected void doPost(final HttpServletRequest request, final HttpServletResponse response) throws ServletException, IOException { @@ -469,7 +463,6 @@ public class YaCyDefaultServlet extends HttpServlet { }
} - /* ------------------------------------------------------------ */
/* (non-Javadoc)
* @see javax.servlet.http.HttpServlet#doTrace(javax.servlet.http.HttpServletRequest, javax.servlet.http.HttpServletResponse)
*/
@@ -478,14 +471,12 @@ public class YaCyDefaultServlet extends HttpServlet { resp.sendError(HttpServletResponse.SC_METHOD_NOT_ALLOWED);
}
- /* ------------------------------------------------------------ */
@Override
protected void doOptions(final HttpServletRequest req, final HttpServletResponse resp)
throws ServletException, IOException {
resp.setHeader("Allow", "GET,HEAD,POST,OPTIONS");
}
- /* ------------------------------------------------------------ */
/**
* Finds a matching welcome file for the supplied path.
* The filename to look is set as servlet context init parameter
@@ -506,7 +497,7 @@ public class YaCyDefaultServlet extends HttpServlet { }
return null;
}
- /* ------------------------------------------------------------ */
+
/* Check modification date headers.
* send a 304 response instead of content if not modified since
*/
@@ -549,7 +540,6 @@ public class YaCyDefaultServlet extends HttpServlet { return true;
}
- /* ------------------------------------------------------------------- */
protected void sendDirectory(final HttpServletRequest request,
final HttpServletResponse response,
final ServletResource resource, @@ -577,7 +567,6 @@ public class YaCyDefaultServlet extends HttpServlet { response.getOutputStream().write(data);
}
- /* ------------------------------------------------------------ */
/**
* send static content
*
@@ -730,7 +719,6 @@ public class YaCyDefaultServlet extends HttpServlet { }
}
- /* ------------------------------------------------------------ */
protected void writeHeaders(final HttpServletResponse response, final ServletResource resource, final long count) { if (response.getContentType() == null) {
final String extensionmime;
diff --git a/source/net/yacy/http/servlets/YaCyQoSFilter.java b/source/net/yacy/http/servlets/YaCyQoSFilter.java index b6f9746bf..d129890a1 100644 --- a/source/net/yacy/http/servlets/YaCyQoSFilter.java +++ b/source/net/yacy/http/servlets/YaCyQoSFilter.java @@ -1,49 +1,43 @@ -/**
- * YaCyQoSFilter
- * Copyright 2015 by Burkhard Buelte
- * First released 26.04.2015 at https://yacy.net
- *
- * This library is free software; you can redistribute it and/or
- * modify it under the terms of the GNU Lesser General Public
- * License as published by the Free Software Foundation; either
- * version 2.1 of the License, or (at your option) any later version.
- *
- * This library is distributed in the hope that it will be useful,
- * but WITHOUT ANY WARRANTY; without even the implied warranty of
- * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU
- * Lesser General Public License for more details.
- *
- * You should have received a copy of the GNU Lesser General Public License
- * along with this program in the file lgpl21.txt
- * If not, see <http://www.gnu.org/licenses/>.
- */
-package net.yacy.http.servlets;
-
-import javax.servlet.ServletRequest;
-import net.yacy.cora.protocol.Domains;
-import org.eclipse.jetty.servlets.QoSFilter;
-
-/**
- * Quality of Service Filter based on Jetty QosFilter
- * to prioritize requests from localhost
- * The intention is to improve the responsivness of web/user interface for the local admin
- * To activate this filter uncomment the predefined filter setting in web.xml
- */
-public class YaCyQoSFilter extends QoSFilter {
-
- /**
- * set priority for localhost to max
- * @param request
- * @return priority
- */
- @Override
- protected int getPriority(ServletRequest request) {
- if (request.getServerName().equalsIgnoreCase(Domains.LOCALHOST)) {
- return 10; // highest priority for "localhost"
- } else if (Domains.isLocalhost(request.getRemoteAddr())) {
- return 9;
- } else {
- return super.getPriority(request); // standard: authenticated = 2, other = 1 or 0
- }
- }
-}
+/** + * YaCyQoSFilter + * Copyright 2026 by Michael Peter Christen + * First released 12.07.2026 at https://yacy.net + * + * This library is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * This library is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public License + * along with this program in the file lgpl21.txt + * If not, see <http://www.gnu.org/licenses/>. + */ + +package net.yacy.http.servlets; + +import javax.servlet.ServletRequest; + +import org.eclipse.jetty.ee8.servlets.QoSFilter; + +import net.yacy.cora.protocol.Domains; + +/** Preserves YaCy's localhost request priority on Jetty 12 EE8. */ +@SuppressWarnings("deprecation") +public class YaCyQoSFilter extends QoSFilter { + + @Override + protected int getPriority(final ServletRequest request) { + if (request.getServerName().equalsIgnoreCase(Domains.LOCALHOST)) { + return 10; + } + if (Domains.isLocalhost(request.getRemoteAddr())) { + return 9; + } + return super.getPriority(request); + } +} diff --git a/source/net/yacy/yacy.java b/source/net/yacy/yacy.java index 251df47c5..a2b2f62ba 100644 --- a/source/net/yacy/yacy.java +++ b/source/net/yacy/yacy.java @@ -68,7 +68,7 @@ import net.yacy.data.TransactionManager; import net.yacy.data.Translator;
import net.yacy.gui.YaCyApp;
import net.yacy.gui.framework.Browser;
-import net.yacy.http.Jetty9HttpServerImpl;
+import net.yacy.http.Jetty12HttpServer;
import net.yacy.http.YaCyHttpServer;
import net.yacy.kelondro.util.FileUtils;
import net.yacy.kelondro.util.Formatter;
@@ -300,7 +300,7 @@ public final class yacy { try {
// start http server
YaCyHttpServer httpServer;
- httpServer = new Jetty9HttpServerImpl(port, host);
+ httpServer = new Jetty12HttpServer(port, host);
httpServer.startupServer();
sb.setHttpServer(httpServer);
// TODO: this has no effect on Jetty (but needed to reflect configured value and limit is still used)
diff --git a/test/java/net/yacy/http/AdminSecurityTest.java b/test/java/net/yacy/http/AdminSecurityTest.java index 196eae3ea..02725d72f 100644 --- a/test/java/net/yacy/http/AdminSecurityTest.java +++ b/test/java/net/yacy/http/AdminSecurityTest.java @@ -112,35 +112,35 @@ public class AdminSecurityTest { public void testAdminAccessPolicy() { final String user = "admin"; final String hash = AdminSecurity.calcHash(user + ":secret"); - final AdminAccessPolicy localAllowed = new AdminAccessPolicy( + final AdminSecurity.AccessPolicy localAllowed = new AdminSecurity.AccessPolicy( false, false, true, true, user, hash); - Assert.assertEquals(AdminAccessPolicy.Decision.PUBLIC, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.PUBLIC, localAllowed.decide("/index.html", "192.0.2.1", null, null)); - Assert.assertEquals(AdminAccessPolicy.Decision.ADMIN_REQUIRED, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.ADMIN_REQUIRED, localAllowed.decide("/Settings_p.html", "192.0.2.1", null, null)); - Assert.assertEquals(AdminAccessPolicy.Decision.LOCAL_BYPASS, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.LOCAL_BYPASS, localAllowed.decide("/Settings_p.html", "127.0.0.1", null, null)); - Assert.assertEquals(AdminAccessPolicy.Decision.ADMIN_REQUIRED, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.ADMIN_REQUIRED, localAllowed.decide("/Settings_p.html", "127.0.0.1", "https://example.org/", null)); - final AdminAccessPolicy loginRequired = new AdminAccessPolicy( + final AdminSecurity.AccessPolicy loginRequired = new AdminSecurity.AccessPolicy( false, false, true, false, user, hash); - Assert.assertEquals(AdminAccessPolicy.Decision.ADMIN_REQUIRED, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.ADMIN_REQUIRED, loginRequired.decide("/Settings_p.html", "127.0.0.1", null, null)); final String lazyAuth = "Basic " + Base64Order.standardCoder.encodeString(user + ":" + hash); - Assert.assertEquals(AdminAccessPolicy.Decision.LOCAL_BYPASS, + Assert.assertEquals(AdminSecurity.AccessPolicy.Decision.LOCAL_BYPASS, loginRequired.decide("/Settings_p.html", "127.0.0.1", null, lazyAuth)); } /** The credential context is request-bound and fails closed after cleanup. */ @Test public void testAdminAuthenticationContext() { - AdminAuthenticationContext.clear(); - Assert.assertFalse(AdminAuthenticationContext.isLocalhostRequest()); - AdminAuthenticationContext.setSocketPeerIp("127.0.0.1"); - Assert.assertTrue(AdminAuthenticationContext.isLocalhostRequest()); - AdminAuthenticationContext.clear(); - Assert.assertFalse(AdminAuthenticationContext.isLocalhostRequest()); + AdminSecurity.AuthenticationContext.clear(); + Assert.assertFalse(AdminSecurity.AuthenticationContext.isLocalhostRequest()); + AdminSecurity.AuthenticationContext.setSocketPeerIp("127.0.0.1"); + Assert.assertTrue(AdminSecurity.AuthenticationContext.isLocalhostRequest()); + AdminSecurity.AuthenticationContext.clear(); + Assert.assertFalse(AdminSecurity.AuthenticationContext.isLocalhostRequest()); } } diff --git a/test/java/net/yacy/http/HttpServerBootstrapConfigTest.java b/test/java/net/yacy/http/HttpServerBootstrapConfigTest.java index 7f2f71120..9faecb2fd 100644 --- a/test/java/net/yacy/http/HttpServerBootstrapConfigTest.java +++ b/test/java/net/yacy/http/HttpServerBootstrapConfigTest.java @@ -20,6 +20,6 @@ public class HttpServerBootstrapConfigTest { Assert.assertEquals(9_000L, HttpServerBootstrapConfig.CONNECTOR_IDLE_TIMEOUT_MILLIS); Assert.assertEquals(128, HttpServerBootstrapConfig.ACCEPT_QUEUE_SIZE); Assert.assertEquals(4_096, HttpServerBootstrapConfig.REQUEST_INFLATE_BUFFER_SIZE); - Assert.assertEquals(-1, HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE); + Assert.assertEquals(200_000, HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE); } } diff --git a/test/java/net/yacy/http/InetPathAccessHandlerTest.java b/test/java/net/yacy/http/InetPathAccessHandlerTest.java deleted file mode 100644 index 641ad8bf1..000000000 --- a/test/java/net/yacy/http/InetPathAccessHandlerTest.java +++ /dev/null @@ -1,355 +0,0 @@ -// InetPathAccessHandlerTest.java -// Copyright 2017 by luccioman; https://github.com/luccioman -// -// This is a part of YaCy, a peer-to-peer based web search engine -// -// LICENSE -// -// This program is free software; you can redistribute it and/or modify -// it under the terms of the GNU General Public License as published by -// the Free Software Foundation; either version 2 of the License, or -// (at your option) any later version. -// -// This program is distributed in the hope that it will be useful, -// but WITHOUT ANY WARRANTY; without even the implied warranty of -// MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the -// GNU General Public License for more details. -// -// You should have received a copy of the GNU General Public License -// along with this program; if not, write to the Free Software -// Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA - -package net.yacy.http; - -import java.net.InetAddress; -import java.net.UnknownHostException; - -import org.junit.Assert; -import org.junit.Test; - -/** - * Unit tests for the {@link InetPathAccessHandler} class. - */ -public class InetPathAccessHandlerTest { - - @Test - public void testPortableRuleParsing() { - final InetPathAccessRule addressOnly = InetPathAccessRule.parse("192.168.1.0/24"); - Assert.assertEquals("192.168.1.0/24", addressOnly.addressPattern()); - Assert.assertEquals("/*", addressOnly.pathPattern()); - - final InetPathAccessRule addressAndPath = InetPathAccessRule.parse("127.0.0.1|/api/*"); - Assert.assertEquals("127.0.0.1", addressAndPath.addressPattern()); - Assert.assertEquals("/api/*", addressAndPath.pathPattern()); - Assert.assertEquals("127.0.0.1|/api/*", addressAndPath.asJettyPattern()); - } - - /** - * Check the handler allow the given ip/path pairs. - * - * @param handler - * the handler to test. Must not be null. - * @param ipAndPaths - * array of ip address and path pairs. Must not be null. - * @throws UnknownHostException - * when a test address is incorrect. - */ - private void assertAllowed(final InetPathAccessHandler handler, final String[][] ipAndPaths) - throws UnknownHostException { - for (final String[] ipAndPath : ipAndPaths) { - final String ip = ipAndPath[0]; - final String path = ipAndPath[1]; - Assert.assertTrue("Should allow " + ip + path, handler.isAllowed(InetAddress.getByName(ip), path)); - } - } - - /** - * Check the handler dos not allow the given ip/path pairs. - * - * @param handler - * the handler to test. Must not be null. - * @param ipAndPaths - * array of ip address and path pairs. Must not be null. - * @throws UnknownHostException - * when a test address is incorrect. - */ - private void assertRejected(final InetPathAccessHandler handler, final String[][] ipAndPaths) - throws UnknownHostException { - for (final String[] ipAndPath : ipAndPaths) { - final String ip = ipAndPath[0]; - final String path = ipAndPath[1]; - Assert.assertFalse("Should not allow " + ip + path, handler.isAllowed(InetAddress.getByName(ip), path)); - } - } - - /** - * Test inclusion with a single white listed IPv4 address. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeSingleIPv4() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.2"); - - final String[][] allowed = { { "10.10.1.2", "/" }, // matching address, root path - { "10.10.1.2", "/foo/bar" }, // matching address, non root path - { "10.10.1.2", null } // matching address, no path information provided - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.3", "/" }, // non matching address, root path - { null, null } // no address nor path information provided - }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with a single white listed IPv6 address. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeSingleIPv6() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("2001:db8::ff00:42:8329"); - - final String[][] allowed = { { "2001:db8::ff00:42:8329", "/" }, // matching address, root path - { "2001:0db8:0000:0000:0000:ff00:0042:8329", "/" }, // matching address in long representation, root - // path - { "2001:db8::ff00:42:8329", "/foo/bar" }, // matching address, non root path - { "2001:db8::ff00:42:8329", null } // matching address, no path information provided - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "2001:db8::ff00:42:8539", "/" }, // non matching address, root path - { null, null } // no address nor path information provided - }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with a single white listed IPV4 address and path. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeSingleAddressAndPath() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.2|/foo/bar"); - - final String[][] allowed = { { "10.10.1.2", "/foo/bar" } // matching address, matching path - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.3", "/" }, // non matching address, non matching path - { "10.10.1.3", "/foo/bar" }, // non matching address, even if matching path - { "10.10.1.2", "/" }, // matching address, but non matching root path - { "10.10.1.2", "/foo" }, // matching address, but non matching parent path - { "10.10.1.2", "/foo/" }, // matching address, but non matching parent path - { "10.10.1.2", "/foo/wrong" }, // matching address, but non matching sub path - { "10.10.1.2", "/foo/bar/file.txt" } // matching address, but non matching sub path with file - }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with a single white listed IPV4 address and wildcard path. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeSingleAddressAndWildcardPath() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.2|/foo/*"); - - final String[][] allowed = { { "10.10.1.2", "/foo/bar" }, // matching address, matching sub path - { "10.10.1.2", "/foo/bar/sub" }, // matching address, matching sub path - { "10.10.1.2", "/foo/file.txt" }, // matching address, matching sub path with file - { "10.10.1.2", "/foo" }, // matching address, matching path - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.3", "/" }, // non matching address, non matching path - { "10.10.1.3", "/foo/bar" }, // non matching address, event if matching path - { "10.10.1.2", "/" }, // matching address, but non matching root path - { "10.10.1.2", null }, // matching address, but no path information provided - { null, "/foo/bar" } // no address provided, event if matching path - }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with a single white listed IPV4 address and wildcard path - * suffix. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeSingleAddressAndWildcardSuffix() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.2|*.html"); - - final String[][] allowed = { { "10.10.1.2", "/index.html" }, // matching address, matching file path - { "10.10.1.2", "/foo/bar/index.html" }, // matching address, matching file with parent path - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.3", "/" }, // non matching address, non matching path - { "10.10.1.3", "/index.html" }, // non matching address, event if matching file path - { "10.10.1.2", "/" }, // matching address, but non matching root path - { "10.10.1.2", "/index.txt" }, // matching address, but non matching file path - { "10.10.1.2", null }, // matching address, but no path information provided - { null, "/index.html" } // no address provided, event if matching path - }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with ranges of white listed addresses. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeRanges() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.1-255"); // legacy IPv4 range format used by IPAddressMap - handler.include("192.168.128.0-192.168.128.255"); // inclusive range of IPv4 addresses - handler.include("2001:db8::ff00:42:8329-2001:db8::ff00:42:ffff"); // inclusive range of IPv6 addresses - handler.include("192.168.1.0/24"); // CIDR notation on IPv4 - handler.include("2001:db8::aaaa:0:0/96"); // CIDR notation on IPv6 - - final String[][] allowed = { { "10.10.1.1", "/" }, // matching legacy IPv4 range - { "10.10.1.255", "/" }, // matching legacy IPv4 range - { "192.168.128.0", "/" }, // matching second range of IPv4 addresses - { "192.168.128.255", "/" }, // matching second range of IPv4 addresses - { "2001:db8::ff00:42:8329", "/" }, // matching IPv6 range - { "2001:db8::ff00:42:99ff", "/" }, // matching IPv6 range - { "192.168.1.0", "/" }, // matching IPv4 CIDR notation range - { "192.168.1.255", "/" }, // matching IPv4 CIDR notation range - { "2001:db8::aaaa:1:1", "/" }, // matching IPv6 CIDR notation range - { "2001:db8::aaaa:ffff:ffff", "/" } // matching IPv6 CIDR notation range - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.9.1.1", "/" }, { "10.10.2.1", "/" }, { "192.168.127.1", "/" }, - { "2001:db8::ff00:43:1234", "/" }, { "192.168.2.1", "/" }, { "2001:db8::aabb:ffff:ffff", "/" } }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with ranges of white listed addresses associated with wildcard - * paths. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeRangesAndWildcardPaths() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.1-255|/foo/*"); // legacy IPv4 range format used by IPAddressMap - handler.include("192.168.128.0-192.168.128.255|/path/*"); // inclusive range of IPv4 adresses - handler.include("2001:db8::ff00:42:8329-2001:db8::ff00:42:ffff|/root/*"); // inclusive range of IPv6 adresses - handler.include("192.168.1.0/24|/www/*"); // CIDR notation - - final String[][] allowed = { { "10.10.1.1", "/foo/bar" }, // matching legacy IPv4 range and path - { "10.10.1.255", "/foo/bar" }, // matching legacy IPv4 range and path - { "192.168.128.0", "/path/index.html" }, // matching second range of IPv4 addresses and path - { "192.168.128.255", "/path/file.txt" }, // matching second range of IPv4 addresses and path - { "2001:db8::ff00:42:8329", "/root/index.txt" }, // matching IPv6 range and path - { "2001:db8::ff00:42:99ff", "/root/image.jpg" }, // matching IPv6 range and path - { "192.168.1.0", "/www/resource" }, // matching IPv4 CIDR notation range and path - { "192.168.1.255", "/www/home" } }; // matching IPv4 CIDR notation range and path - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.9.1.1", "/" }, { "10.9.1.1", "/foo/bar" }, { "10.10.2.1", "/" }, - { "10.10.2.1", "/foo/bar" }, { "192.168.127.1", "/" }, { "192.168.127.1", "/path/index.html" }, - { "2001:db8::ff00:43:1234", "/" }, { "2001:db8::ff00:43:1234", "/root/index.txt" }, - { "192.168.2.1", "/" }, { "192.168.2.1", "/www/content" } }; - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion with multiple patterns using the same path - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeMultiplePatternsOnSamePath() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.1|/foo/bar"); // a single address pattern - handler.include("192.168.128.0-192.168.128.255|/foo/bar"); // inclusive range of IPv4 adresses - - final String[][] allowed = { { "10.10.1.1", "/foo/bar" }, // matching single address pattern - { "192.168.128.0", "/foo/bar" }, { "192.168.128.255", "/foo/bar" } // matching range pattern - }; - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.1", "/" }, // matching single address pattern bu root path - { "127.0.0.1", "/" }, // non matching address - }; - this.assertRejected(handler, rejected); - } - - /** - * Test exclusion with a single white listed IPV4 address and path. - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testExcludeSingleAddressAndPath() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.exclude("10.10.1.2|/foo/bar"); - - final String[][] allowed = { { "10.10.1.3", "/" }, // non matching address, non matching path - { "10.10.1.3", "/foo/bar" }, // non matching address, even if matching path - { "10.10.1.2", "/" }, // matching address, but non matching root path - { "10.10.1.2", "/foo" }, // matching address, but non matching parent path - { "10.10.1.2", "/foo/" }, // matching address, but non matching parent path - { "10.10.1.2", "/foo/wrong" }, // matching address, but non matching sub path - { "10.10.1.2", "/foo/bar/file.txt" } // matching address, but non matching sub path with file - }; - - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.2", "/foo/bar" } // matching address, matching path - }; - - this.assertRejected(handler, rejected); - } - - /** - * Test inclusion and exclusion rules applied on the same address - * - * @throws UnknownHostException - * when a test address is incorrect. Should not happen. - */ - @Test - public void testIncludeExcludeOnSameAddress() throws UnknownHostException { - final InetPathAccessHandler handler = new InetPathAccessHandler(); - handler.include("10.10.1.1-10.10.1.255"); // include a range of addresses without path restrictions - handler.exclude("10.10.1.2|/foo/bar"); // exclude a specific address and path - - final String[][] allowed = { { "10.10.1.3", "/" }, // matching included addresses range - { "10.10.1.2", "/" }, // matching excluded address, but non matching root path - { "10.10.1.2", "/foo" }, // matching excluded address, but non matching parent path - { "10.10.1.2", "/foo/wrong" }, // matching excluded address, but non matching sub path - { "10.10.1.2", "/foo/bar/file.txt" } // matching excluded address, but non matching sub path with file - }; - - this.assertAllowed(handler, allowed); - - final String[][] rejected = { { "10.10.1.2", "/foo/bar" } // matching excluded address and path - }; - - this.assertRejected(handler, rejected); - } -} diff --git a/test/java/net/yacy/http/Jetty12HttpServerTest.java b/test/java/net/yacy/http/Jetty12HttpServerTest.java new file mode 100644 index 000000000..160188358 --- /dev/null +++ b/test/java/net/yacy/http/Jetty12HttpServerTest.java @@ -0,0 +1,427 @@ +package net.yacy.http; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertNotNull; +import static org.junit.Assert.assertNull; +import static org.junit.Assert.assertSame; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.Socket; +import java.nio.charset.StandardCharsets; +import java.util.concurrent.atomic.AtomicBoolean; +import java.util.concurrent.atomic.AtomicReference; +import java.util.logging.Level; +import java.util.logging.LogRecord; +import java.util.logging.Logger; + +import javax.net.ssl.SSLContext; +import javax.servlet.http.HttpServlet; +import javax.servlet.http.HttpServletRequest; +import javax.servlet.http.HttpServletResponse; + +import org.eclipse.jetty.compression.server.CompressionConfig; +import org.eclipse.jetty.ee8.security.authentication.DigestAuthenticator; +import org.eclipse.jetty.ee8.servlet.ServletContextHandler; +import org.eclipse.jetty.ee8.servlet.ServletHolder; +import org.eclipse.jetty.security.UserIdentity; +import org.eclipse.jetty.server.Handler; +import org.eclipse.jetty.server.Request; +import org.eclipse.jetty.server.Response; +import org.eclipse.jetty.server.SecureRequestCustomizer; +import org.eclipse.jetty.server.Server; +import org.eclipse.jetty.server.ServerConnector; +import org.eclipse.jetty.server.handler.InetAccessHandler; +import org.eclipse.jetty.util.Callback; +import org.eclipse.jetty.util.security.Credential; +import org.junit.Test; +import org.junit.experimental.runners.Enclosed; +import org.junit.runner.RunWith; + +import net.yacy.cora.order.Digest; +import net.yacy.search.SwitchboardConstants; + +/** + * Focused tests for the consolidated {@link Jetty12HttpServer} adapter and its + * nested default-runtime handlers. The transparent-proxy chain is covered by + * {@link Jetty12ProxyChainTest}. + */ +@RunWith(Enclosed.class) +public class Jetty12HttpServerTest { + + /** Lifecycle, connector and compression contract of the bootstrap. */ + public static class ServerLifecycleTest { + + @Test + public void startsAndStopsHttpConnectorOnEphemeralPort() throws Exception { + final Logger jettyLogger = Logger.getLogger("org.eclipse.jetty.server.Server"); + final AtomicBoolean receivedJettyLog = new AtomicBoolean(); + final java.util.logging.Handler capture = new java.util.logging.Handler() { + @Override + public void publish(final LogRecord record) { + receivedJettyLog.set(true); + } + + @Override + public void flush() { + } + + @Override + public void close() { + } + }; + capture.setLevel(Level.ALL); + jettyLogger.addHandler(capture); + final Jetty12HttpServer server = new Jetty12HttpServer( + 0, "127.0.0.1", 1, null, -1); + assertFalse(server.withSSL()); + assertEquals(-1, server.getSslPort()); + assertTrue(server.getVersion().startsWith("Jetty 12.1.11")); + + try { + server.startupServer(); + assertTrue(server.getHttpPort() > 0); + assertTrue(server.getServerThreads() >= 0); + } finally { + server.stop(); + jettyLogger.removeHandler(capture); + } + assertTrue("Jetty 12 lifecycle log did not reach JUL", receivedJettyLog.get()); + } + + @Test + public void configuresHttpsConnectorWhenSslContextIsAvailable() throws Exception { + final Jetty12HttpServer server = new Jetty12HttpServer( + 0, "127.0.0.1", 1, SSLContext.getDefault(), 0); + assertTrue(server.withSSL()); + try { + server.startupServer(); + assertTrue(server.getHttpPort() > 0); + assertTrue(server.getSslPort() > 0); + } finally { + server.stop(); + } + } + + @Test + public void acceptsConfiguredCertificatesWithoutJetty12SniHostRejection() { + final SecureRequestCustomizer secureRequests = + Jetty12HttpServer.createSecureRequestCustomizer(); + assertFalse(secureRequests.isSniHostCheck()); + assertFalse(secureRequests.isSniRequired()); + } + + @Test + public void preservesJetty9SvgCompressionContract() { + final CompressionConfig compression = Jetty12HttpServer.createCompressionConfig(true); + assertTrue(compression.isCompressMimeTypeSupported("image/svg+xml")); + assertFalse(compression.isCompressMimeTypeSupported("image/png")); + assertTrue(compression.isCompressMethodSupported("GET")); + } + + @Test + public void preservesJetty9WebApplicationFormContentLimit() throws Exception { + final Server server = new Server(); + final ServerConnector connector = new ServerConnector(server, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + server.addConnector(connector); + + final ServletContextHandler webApp = new ServletContextHandler(); + webApp.setContextPath("/"); + webApp.setMaxFormContentSize(HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE); + webApp.addServlet(new ServletHolder(new HttpServlet() { + private static final long serialVersionUID = 1L; + + @Override + protected void doPost(final HttpServletRequest request, + final HttpServletResponse response) throws java.io.IOException { + response.setStatus(HttpServletResponse.SC_OK); + response.getWriter().print(request.getParameter("value").length()); + } + }), "/*"); + server.setHandler(webApp); + + try { + server.start(); + final String accepted = postForm(connector.getLocalPort(), "value=" + "a".repeat(1024)); + assertTrue(accepted, accepted.startsWith("HTTP/1.1 200")); + assertTrue(accepted, accepted.contains("1024")); + + final String rejected = postForm(connector.getLocalPort(), + "value=" + "a".repeat(HttpServerBootstrapConfig.MAX_FORM_CONTENT_SIZE)); + assertTrue(rejected, rejected.startsWith("HTTP/1.1 400")); + } finally { + server.stop(); + server.join(); + } + } + + private static String postForm(final int port, final String body) throws Exception { + final byte[] content = body.getBytes(StandardCharsets.US_ASCII); + try (Socket client = new Socket("127.0.0.1", port)) { + final OutputStream output = client.getOutputStream(); + output.write(("POST / HTTP/1.1\r\nHost: 127.0.0.1:" + port + + "\r\nContent-Type: application/x-www-form-urlencoded" + + "\r\nContent-Length: " + content.length + + "\r\nConnection: close\r\n\r\n").getBytes(StandardCharsets.US_ASCII)); + output.write(content); + output.flush(); + return readAll(client.getInputStream()); + } + } + } + + /** Address/path rule adaptation to Jetty 12's native access handler. */ + public static class AccessRulesTest { + + private static final Handler NEXT = new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, final Callback callback) { + return false; + } + }; + + @Test + public void leavesUnrestrictedPipelineUntouched() { + assertSame(NEXT, Jetty12HttpServer.AccessRules.wrap(NEXT, "*")); + } + + @Test + public void usesNativeJetty12HandlerForAddressAndPathRule() { + final Handler wrapped = Jetty12HttpServer.AccessRules.wrap(NEXT, "192.0.2.0/24|/api/*"); + assertTrue(wrapped instanceof InetAccessHandler); + assertSame(NEXT, ((InetAccessHandler) wrapped).getHandler()); + } + + @Test + public void ignoresEmptyRuleList() { + assertSame(NEXT, Jetty12HttpServer.AccessRules.wrap(NEXT, " , ")); + } + + @Test + public void validatesConfiguredAddressAndPathPattern() { + Jetty12HttpServer.AccessRules.checkPattern("192.0.2.0/24|/api/*"); + } + + @Test(expected = IllegalArgumentException.class) + public void rejectsMalformedConfiguredAddressPattern() { + Jetty12HttpServer.AccessRules.checkPattern("not an address|/api/*"); + } + } + + /** Administrator credential verification for BASIC and DIGEST. */ + public static class AdminCredentialTest { + + @Test + public void verifiesBasicPasswordAgainstRealmHash() { + final String hash = "MD5:" + Digest.encodeMD5Hex("admin:YaCy:test-password"); + final Jetty12HttpServer.AdminCredential credential = new Jetty12HttpServer.AdminCredential( + "admin", hash, "YaCy", "admin"); + assertTrue(credential.check("test-password")); + assertFalse(credential.check("wrong-password")); + } + + @Test + public void acceptsConfiguredHashAsPasswordOnlyFromLocalhost() { + final String hash = "MD5:" + Digest.encodeMD5Hex("admin:YaCy:test-password"); + final Jetty12HttpServer.AdminCredential remoteCredential = new Jetty12HttpServer.AdminCredential( + "admin", hash, "YaCy", "admin", () -> false); + final Jetty12HttpServer.AdminCredential localCredential = new Jetty12HttpServer.AdminCredential( + "admin", hash, "YaCy", "admin", () -> true); + assertFalse(remoteCredential.check(hash)); + assertTrue(localCredential.check(hash)); + } + + @Test + public void acceptsJettyDigestCredentialForConfiguredDigest() { + final String hash = "MD5:" + Digest.encodeMD5Hex("admin:YaCy:test-password"); + final Jetty12HttpServer.AdminCredential credential = new Jetty12HttpServer.AdminCredential( + "admin", hash, "YaCy", "admin"); + assertTrue(credential.check(Credential.getCredential(hash))); + } + + @Test + public void digestChallengeUsesTheAlgorithmOfYacysStoredHa1() { + final DigestAuthenticator authenticator = + Jetty12HttpServer.AdminSecurityHandler.createDigestAuthenticator(); + assertEquals("MD5", authenticator.getAlgorithm()); + } + } + + /** Administrator login-service cache and credential reload contract. */ + public static class AdminLoginServiceTest { + + @Test + public void reloadsChangedAdministratorCredentialAndRole() throws Exception { + final AtomicReference<Jetty12HttpServer.AdminLoginService.AdminCredentialConfig> configured = + new AtomicReference<>(credentials("first-password")); + final Jetty12HttpServer.AdminLoginService service = + new Jetty12HttpServer.AdminLoginService(configured::get); + service.setName("YaCy"); + service.start(); + try { + final UserIdentity first = service.login("admin", "first-password", null, ignored -> null); + assertNotNull(first); + assertTrue(first.isUserInRole(SwitchboardConstants.ADMIN_ACCOUNT_ROLE)); + assertTrue(service.removeCachedUser("admin")); + assertFalse(service.removeCachedUser("admin")); + + configured.set(credentials("second-password")); + service.reloadUser("admin"); + assertNull(service.login("admin", "first-password", null, ignored -> null)); + assertNotNull(service.login("admin", "second-password", null, ignored -> null)); + } finally { + service.stop(); + } + } + + private static Jetty12HttpServer.AdminLoginService.AdminCredentialConfig credentials( + final String password) { + return new Jetty12HttpServer.AdminLoginService.AdminCredentialConfig("admin", + "MD5:" + Digest.encodeMD5Hex("admin:YaCy:" + password), "YaCy"); + } + } + + /** Exception barrier around the complete request pipeline. */ + public static class CrashProtectionHandlerTest { + + @Test + public void convertsSynchronousHandlerFailureTo500Response() throws Exception { + final Server server = new Server(); + final ServerConnector connector = new ServerConnector(server, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + server.addConnector(connector); + final Handler failing = new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, + final Callback callback) { + throw new IllegalStateException("expected test failure"); + } + }; + server.setHandler(new Jetty12HttpServer.CrashProtectionHandler(failing)); + try { + server.start(); + final java.net.http.HttpResponse<String> response = + java.net.http.HttpClient.newHttpClient().send( + java.net.http.HttpRequest.newBuilder(java.net.URI.create( + "http://127.0.0.1:" + connector.getLocalPort() + "/failure")) + .GET().build(), + java.net.http.HttpResponse.BodyHandlers.ofString()); + assertEquals(500, response.statusCode()); + } finally { + server.stop(); + server.join(); + } + } + } + + /** CONNECT rejection while normal traffic passes when the proxy is disabled. */ + public static class DisabledProxyHandlerTest { + + @Test + public void passesReverseProxyHostToWebApplication() throws Exception { + final Handler handler = new Jetty12HttpServer.DisabledProxyHandler(fallback()); + final String response = request(handler, + "GET /resource HTTP/1.1\r\n" + + "Host: search.example.org\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 204")); + } + + @Test + public void rejectsConnectWith403() throws Exception { + final Handler handler = new Jetty12HttpServer.DisabledProxyHandler(fallback()); + final String response = requestHeaders(handler, + "CONNECT example.invalid:443 HTTP/1.1\r\n" + + "Host: example.invalid:443\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 403")); + assertTrue(response, response.contains( + "X-YaCy-Proxy-Error: Transparent proxy is disabled")); + } + + @Test + public void passesDirectPeerRequestToWebApplication() throws Exception { + final Handler handler = new Jetty12HttpServer.DisabledProxyHandler(fallback()); + final String response = request(handler, + "GET /ConfigBasic.html HTTP/1.1\r\n" + + "Host: 127.0.0.1:%d\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 204")); + } + + private static Handler fallback() { + return new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, + final Callback callback) { + response.setStatus(204); + callback.succeeded(); + return true; + } + }; + } + + private static String request(final Handler handler, final String requestTemplate) + throws Exception { + return request(handler, requestTemplate, false); + } + + private static String requestHeaders(final Handler handler, final String requestTemplate) + throws Exception { + return request(handler, requestTemplate, true); + } + + private static String request(final Handler handler, final String requestTemplate, + final boolean headersOnly) throws Exception { + final Server server = new Server(); + final ServerConnector connector = new ServerConnector(server, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + server.addConnector(connector); + server.setHandler(handler); + try { + server.start(); + try (Socket client = new Socket("127.0.0.1", connector.getLocalPort())) { + final OutputStream output = client.getOutputStream(); + output.write(String.format(requestTemplate, connector.getLocalPort()) + .getBytes(StandardCharsets.US_ASCII)); + output.flush(); + return headersOnly ? readHeaders(client.getInputStream()) + : readAll(client.getInputStream()); + } + } finally { + server.stop(); + server.join(); + } + } + + private static String readHeaders(final InputStream input) throws Exception { + final ByteArrayOutputStream result = new ByteArrayOutputStream(); + int state = 0; + int value; + while ((value = input.read()) >= 0) { + result.write(value); + state = switch (state) { + case 0 -> value == '\r' ? 1 : 0; + case 1 -> value == '\n' ? 2 : 0; + case 2 -> value == '\r' ? 3 : 0; + case 3 -> value == '\n' ? 4 : 0; + default -> state; + }; + if (state == 4) { + break; + } + } + return result.toString(StandardCharsets.ISO_8859_1); + } + } + + static String readAll(final InputStream input) throws Exception { + final ByteArrayOutputStream result = new ByteArrayOutputStream(); + input.transferTo(result); + return result.toString(StandardCharsets.ISO_8859_1); + } +} diff --git a/test/java/net/yacy/http/Jetty12ProxyChainTest.java b/test/java/net/yacy/http/Jetty12ProxyChainTest.java new file mode 100644 index 000000000..8cf811d5c --- /dev/null +++ b/test/java/net/yacy/http/Jetty12ProxyChainTest.java @@ -0,0 +1,457 @@ +package net.yacy.http; + +import static org.junit.Assert.assertArrayEquals; +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; + +import java.io.ByteArrayOutputStream; +import java.io.InputStream; +import java.io.OutputStream; +import java.net.InetSocketAddress; +import java.net.ServerSocket; +import java.net.Socket; +import java.nio.ByteBuffer; +import java.nio.charset.StandardCharsets; +import java.util.Collections; +import java.util.Set; +import java.util.concurrent.CountDownLatch; +import java.util.concurrent.TimeUnit; +import java.util.concurrent.atomic.AtomicInteger; +import java.util.concurrent.atomic.AtomicReference; + +import org.eclipse.jetty.http.HttpFields; +import org.eclipse.jetty.http.HttpHeader; +import org.eclipse.jetty.server.Handler; +import org.eclipse.jetty.server.Request; +import org.eclipse.jetty.server.Response; +import org.eclipse.jetty.server.Server; +import org.eclipse.jetty.server.ServerConnector; +import org.eclipse.jetty.util.Callback; +import org.junit.Test; +import org.junit.experimental.runners.Enclosed; +import org.junit.runner.RunWith; + +import com.sun.net.httpserver.HttpServer; + +import net.yacy.server.http.AlternativeDomainNames; + +/** + * Focused tests for the consolidated {@link Jetty12ProxyChain} transparent-proxy + * handlers. The default-runtime adapter is covered by {@link Jetty12HttpServerTest}. + */ +@RunWith(Enclosed.class) +public class Jetty12ProxyChainTest { + + /** Single policy gate in front of the cache/forward sequence. */ + public static class ProxyPolicyHandlerTest { + + @Test + public void numericLoopbackIsRecognizedAsThisHost() throws Exception { + assertDirectPeerRequest("/api/version.xml", true); + } + + @Test + public void numericLoopbackOnAnotherPortRemainsAProxyTarget() throws Exception { + assertDirectPeerRequest("http://127.0.0.1:9/resource", false); + } + + @Test + public void unresolvedAbsoluteHostRemainsAProxyTarget() throws Exception { + assertDirectPeerRequest("http://example.invalid/resource", false); + } + + @Test + public void directRequestFallsThroughWithoutPermissionOrProxyPipeline() throws Exception { + final AtomicInteger classification = new AtomicInteger(); + final AtomicInteger permission = new AtomicInteger(); + final AtomicInteger proxyCalls = new AtomicInteger(); + final Handler proxy = responseHandler(202, proxyCalls); + final Handler gate = new Jetty12ProxyChain.ProxyPolicyHandler(proxy, request -> { + classification.incrementAndGet(); + return false; + }, request -> { + permission.incrementAndGet(); + return null; + }); + final String response = request(new Handler.Sequence(gate, responseHandler(204, null)), + "GET /ConfigBasic.html HTTP/1.1\r\nHost: 127.0.0.1:%d\r\n" + + "Connection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 204")); + assertEquals(1, classification.get()); + assertEquals(0, permission.get()); + assertEquals(0, proxyCalls.get()); + } + + @Test + public void rejectedRequestNeverReachesProxyPipeline() throws Exception { + final AtomicInteger permission = new AtomicInteger(); + final AtomicInteger proxyCalls = new AtomicInteger(); + final Handler gate = new Jetty12ProxyChain.ProxyPolicyHandler( + responseHandler(202, proxyCalls), + request -> true, request -> { + permission.incrementAndGet(); + return "proxy use not granted"; + }); + final String response = request(gate, + "GET http://example.invalid/resource HTTP/1.1\r\n" + + "Host: example.invalid\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 403")); + assertEquals(1, permission.get()); + assertEquals(0, proxyCalls.get()); + } + + @Test + public void cacheHitEvaluatesPolicyOnceAndSkipsForwarding() throws Exception { + final AtomicInteger classification = new AtomicInteger(); + final AtomicInteger permission = new AtomicInteger(); + final AtomicInteger forwarding = new AtomicInteger(); + final HttpFields headers = HttpFields.build().put("Content-Type", "text/plain").asImmutable(); + final Handler cache = new Jetty12ProxyChain.ProxyCacheHandler(request -> + new Jetty12ProxyChain.ProxyCacheHandler.CachedResponse(headers, + "cached".getBytes(StandardCharsets.UTF_8))); + final Handler proxyPipeline = new Handler.Sequence(cache, responseHandler(202, forwarding)); + final Handler gate = countingGate(proxyPipeline, classification, permission); + + final String response = proxyRequest(gate); + assertTrue(response, response.startsWith("HTTP/1.1 203")); + assertEquals(1, classification.get()); + assertEquals(1, permission.get()); + assertEquals(0, forwarding.get()); + } + + @Test + public void cacheMissEvaluatesPolicyOnceBeforeForwarding() throws Exception { + final AtomicInteger classification = new AtomicInteger(); + final AtomicInteger permission = new AtomicInteger(); + final AtomicInteger lookups = new AtomicInteger(); + final AtomicInteger forwarding = new AtomicInteger(); + final Handler cache = new Jetty12ProxyChain.ProxyCacheHandler(request -> { + lookups.incrementAndGet(); + return null; + }); + final Handler proxyPipeline = new Handler.Sequence(cache, responseHandler(202, forwarding)); + final Handler gate = countingGate(proxyPipeline, classification, permission); + + final String response = proxyRequest(gate); + assertTrue(response, response.startsWith("HTTP/1.1 202")); + assertEquals(1, classification.get()); + assertEquals(1, permission.get()); + assertEquals(1, lookups.get()); + assertEquals(1, forwarding.get()); + } + + private static Handler responseHandler(final int status, final AtomicInteger calls) { + return new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, + final Callback callback) { + if (calls != null) { + calls.incrementAndGet(); + } + response.setStatus(status); + callback.succeeded(); + return true; + } + }; + } + + private static Handler countingGate(final Handler proxyPipeline, + final AtomicInteger classification, final AtomicInteger permission) { + return new Jetty12ProxyChain.ProxyPolicyHandler(proxyPipeline, request -> { + classification.incrementAndGet(); + return true; + }, request -> { + permission.incrementAndGet(); + return null; + }); + } + + private static String proxyRequest(final Handler handler) throws Exception { + return request(handler, "GET http://example.invalid/resource HTTP/1.1\r\n" + + "Host: example.invalid\r\nConnection: close\r\n\r\n"); + } + + private static void assertDirectPeerRequest(final String target, final boolean expected) + throws Exception { + final Handler check = new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, + final Callback callback) { + final boolean actual = Jetty12ProxyChain.ProxyPolicyHandler.isDirectPeerRequest( + request, "localpeer"); + response.setStatus(actual == expected ? 204 : 500); + callback.succeeded(); + return true; + } + }; + final String host = target.startsWith("http://") + ? java.net.URI.create(target).getRawAuthority() : "127.0.0.1:%d"; + final String response = request(check, "GET " + target + " HTTP/1.1\r\nHost: " + + host + "\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 204")); + } + } + + /** {@code .yacy} authority rewrite in front of the proxy selection. */ + public static class DomainHandlerTest { + + @Test + public void rewritesPeerAuthorityBeforeCallingNextHandler() throws Exception { + final AtomicReference<String> authority = new AtomicReference<>(); + final AtomicReference<String> hostHeader = new AtomicReference<>(); + final Handler capture = new Handler.Abstract() { + @Override + public boolean handle(final Request request, final Response response, + final Callback callback) { + authority.set(request.getHttpURI().getAuthority()); + hostHeader.set(request.getHeaders().get(HttpHeader.HOST)); + response.setStatus(204); + callback.succeeded(); + return true; + } + }; + final Server server = new Server(); + final ServerConnector connector = new ServerConnector(server, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + server.addConnector(connector); + server.setHandler(new Jetty12ProxyChain.DomainHandler(capture, resolver(), ignored -> false)); + try { + server.start(); + try (Socket client = new Socket("127.0.0.1", connector.getLocalPort())) { + final OutputStream output = client.getOutputStream(); + output.write(("GET /status?q=1 HTTP/1.1\r\nHost: peer.yacy\r\nConnection: close\r\n\r\n") + .getBytes(StandardCharsets.US_ASCII)); + output.flush(); + try (InputStream input = client.getInputStream()) { + while (input.read() >= 0) { + // Drain the response so the exchange completes before assertions. + } + } + } + assertEquals("198.51.100.7:8090", authority.get()); + assertEquals("198.51.100.7:8090", hostHeader.get()); + } finally { + server.stop(); + server.join(); + } + } + + private static AlternativeDomainNames resolver() { + return new AlternativeDomainNames() { + @Override + public String resolve(final String name) { + return "peer.yacy".equals(name) ? "198.51.100.7:8090" : null; + } + + @Override public String myAlternativeAddress() { return "local.yacy"; } + @Override public Set<String> myIPs() { return Collections.singleton("127.0.0.1"); } + @Override public int myPort() { return 8090; } + @Override public String myName() { return "local"; } + @Override public String myID() { return "local-id"; } + }; + } + } + + /** Raw byte tunnelling for permitted CONNECT destinations. */ + public static class ConnectTunnelHandlerTest { + + @Test + public void tunnelsBytesToPermittedDestination() throws Exception { + try (ServerSocket origin = new ServerSocket(0, 1)) { + final Thread echo = new Thread(() -> echoOnce(origin), "connect-origin-echo"); + echo.start(); + final Server proxy = new Server(); + final ServerConnector connector = new ServerConnector(proxy, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + proxy.addConnector(connector); + proxy.setHandler(new Jetty12ProxyChain.ConnectTunnelHandler( + (Handler) null, (request, destination) -> null)); + try { + proxy.start(); + try (Socket client = new Socket("127.0.0.1", connector.getLocalPort())) { + final OutputStream output = client.getOutputStream(); + output.write(("CONNECT 127.0.0.1:" + origin.getLocalPort() + + " HTTP/1.1\r\nHost: 127.0.0.1:" + origin.getLocalPort() + + "\r\n\r\n").getBytes(StandardCharsets.US_ASCII)); + output.flush(); + final InputStream input = client.getInputStream(); + final String headers = readHeaders(input); + assertEquals(true, headers.startsWith("HTTP/1.1 200")); + output.write("tunnel-data".getBytes(StandardCharsets.US_ASCII)); + output.flush(); + assertEquals("tunnel-data", new String(input.readNBytes(11), StandardCharsets.US_ASCII)); + } + } finally { + proxy.stop(); + proxy.join(); + echo.join(5000L); + } + } + } + + private static void echoOnce(final ServerSocket origin) { + try (Socket socket = origin.accept()) { + final byte[] data = socket.getInputStream().readNBytes(11); + socket.getOutputStream().write(data); + socket.getOutputStream().flush(); + } catch (final Exception error) { + throw new AssertionError(error); + } + } + + private static String readHeaders(final InputStream input) throws Exception { + final ByteArrayOutputStream bytes = new ByteArrayOutputStream(); + int matched = 0; + while (matched < 4) { + final int next = input.read(); + if (next < 0) { + break; + } + bytes.write(next); + final byte expected = new byte[] {'\r', '\n', '\r', '\n'}[matched]; + matched = next == expected ? matched + 1 : (next == '\r' ? 1 : 0); + } + return bytes.toString(StandardCharsets.US_ASCII); + } + } + + /** Streaming forward proxy for absolute and transparent origin-form requests. */ + public static class ForwardProxyHandlerTest { + + @Test + public void forwardsAbsoluteHttpRequest() throws Exception { + assertForwarded(true); + } + + @Test + public void forwardsTransparentOriginFormRequest() throws Exception { + assertForwarded(false); + } + + private static void assertForwarded(final boolean absoluteTarget) throws Exception { + final ByteArrayOutputStream captured = new ByteArrayOutputStream(); + final CountDownLatch captureComplete = new CountDownLatch(1); + final HttpServer origin = HttpServer.create(new InetSocketAddress("127.0.0.1", 0), 0); + origin.createContext("/resource", exchange -> { + final byte[] body = "proxied-content".getBytes(StandardCharsets.UTF_8); + exchange.sendResponseHeaders(200, body.length); + exchange.getResponseBody().write(body); + exchange.close(); + }); + origin.start(); + final Server proxy = new Server(); + final ServerConnector connector = new ServerConnector(proxy, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + proxy.addConnector(connector); + proxy.setHandler(new Jetty12ProxyChain.ForwardProxyHandler( + (request, upstream) -> new Jetty12ProxyChain.ForwardProxyHandler.Capture() { + @Override + public void append(final ByteBuffer content) { + final byte[] bytes = new byte[content.remaining()]; + content.get(bytes); + captured.writeBytes(bytes); + } + + @Override + public void complete() { + captureComplete.countDown(); + } + })); + try { + proxy.start(); + try (Socket client = new Socket("127.0.0.1", connector.getLocalPort())) { + final OutputStream output = client.getOutputStream(); + final String target = absoluteTarget + ? "http://127.0.0.1:" + origin.getAddress().getPort() + "/resource" + : "/resource"; + output.write(("GET " + target + " HTTP/1.1\r\nHost: 127.0.0.1:" + + origin.getAddress().getPort() + "\r\nConnection: close\r\n\r\n") + .getBytes(StandardCharsets.US_ASCII)); + output.flush(); + final String response = readAll(client.getInputStream()); + assertTrue(response.startsWith("HTTP/1.1 200")); + assertTrue(response.contains("proxied-content")); + } + assertTrue(captureComplete.await(2, TimeUnit.SECONDS)); + assertArrayEquals("proxied-content".getBytes(StandardCharsets.UTF_8), + captured.toByteArray()); + } finally { + proxy.stop(); + proxy.join(); + origin.stop(0); + } + } + } + + /** Cache hits are served with 203 without a network round trip. */ + public static class ProxyCacheHandlerTest { + + @Test + public void servesFreshCachedResponseWithoutNetworkHandler() throws Exception { + final HttpFields headers = HttpFields.build().put("Content-Type", "text/plain").asImmutable(); + final Handler cache = new Jetty12ProxyChain.ProxyCacheHandler(request -> + new Jetty12ProxyChain.ProxyCacheHandler.CachedResponse(headers, + "cached-content".getBytes(StandardCharsets.UTF_8))); + final String response = request(cache, + "GET http://example.invalid/resource HTTP/1.1\r\n" + + "Host: example.invalid\r\nConnection: close\r\n\r\n"); + assertTrue(response, response.startsWith("HTTP/1.1 203")); + assertTrue(response, response.contains("cached-content")); + } + } + + /** Size-capped copy of streamed proxy responses for the cache/index path. */ + public static class ProxyResponseStoreTest { + + @Test + public void discardsCacheCaptureWhenStreamingResponseExceedsLimit() { + final Jetty12ProxyChain.ProxyResponseStore.CacheCapture capture = + new Jetty12ProxyChain.ProxyResponseStore.CacheCapture(null, null, 4); + + capture.append(ByteBuffer.wrap(new byte[] {1, 2})); + capture.append(ByteBuffer.wrap(new byte[] {3, 4})); + assertFalse(capture.isDiscarded()); + assertEquals(4, capture.bufferedSize()); + + capture.append(ByteBuffer.wrap(new byte[] {5})); + assertTrue(capture.isDiscarded()); + assertEquals(0, capture.bufferedSize()); + + capture.append(ByteBuffer.wrap(new byte[] {6, 7})); + assertEquals(0, capture.bufferedSize()); + capture.complete(); + } + } + + static String request(final Handler handler, final String requestTemplate) throws Exception { + final Server server = new Server(); + final ServerConnector connector = new ServerConnector(server, 1, 1); + connector.setHost("127.0.0.1"); + connector.setPort(0); + server.addConnector(connector); + server.setHandler(handler); + try { + server.start(); + try (Socket client = new Socket("127.0.0.1", connector.getLocalPort())) { + final OutputStream output = client.getOutputStream(); + output.write(String.format(requestTemplate, connector.getLocalPort()) + .getBytes(StandardCharsets.US_ASCII)); + output.flush(); + return readAll(client.getInputStream()); + } + } finally { + server.stop(); + server.join(); + } + } + + static String readAll(final InputStream input) throws Exception { + final ByteArrayOutputStream result = new ByteArrayOutputStream(); + input.transferTo(result); + return result.toString(StandardCharsets.ISO_8859_1); + } +} diff --git a/test/java/net/yacy/http/Jetty9LoggingFacadeTest.java b/test/java/net/yacy/http/Jetty9LoggingFacadeTest.java deleted file mode 100644 index bba2b6924..000000000 --- a/test/java/net/yacy/http/Jetty9LoggingFacadeTest.java +++ /dev/null @@ -1,16 +0,0 @@ -package net.yacy.http; - -import static org.junit.Assert.assertEquals; - -import org.eclipse.jetty.util.log.Log; -import org.junit.Test; - -/** Jetty 9 baseline only; replace this test when Jetty9HttpServerImpl is removed. */ -public class Jetty9LoggingFacadeTest { - - @Test - public void jetty9UsesItsSlf4jFacade() { - assertEquals("org.eclipse.jetty.util.log.Slf4jLog", - Log.getLogger("org.eclipse.jetty.yacy.logging.test").getClass().getName()); - } -} diff --git a/test/java/net/yacy/http/servlets/Jetty12QoSFilterTest.java b/test/java/net/yacy/http/servlets/Jetty12QoSFilterTest.java new file mode 100644 index 000000000..2920e44e2 --- /dev/null +++ b/test/java/net/yacy/http/servlets/Jetty12QoSFilterTest.java @@ -0,0 +1,34 @@ +package net.yacy.http.servlets; + +import static org.junit.Assert.assertEquals; + +import java.lang.reflect.Proxy; + +import javax.servlet.ServletRequest; + +import org.junit.Test; + +public class Jetty12QoSFilterTest { + + @Test + public void givesLocalhostServerNameHighestPriority() { + final ServletRequest request = (ServletRequest) Proxy.newProxyInstance( + ServletRequest.class.getClassLoader(), new Class<?>[] {ServletRequest.class}, + (proxy, method, arguments) -> "getServerName".equals(method.getName()) + ? "localhost" : defaultValue(method.getReturnType())); + assertEquals(10, new YaCyQoSFilter().getPriority(request)); + } + + private static Object defaultValue(final Class<?> type) { + if (!type.isPrimitive()) { + return null; + } + if (type == boolean.class) { + return false; + } + if (type == char.class) { + return '\0'; + } + return 0; + } +} diff --git a/test/java/net/yacy/http/servlets/ServletResourceTest.java b/test/java/net/yacy/http/servlets/ServletResourceTest.java new file mode 100644 index 000000000..bac0cd4ec --- /dev/null +++ b/test/java/net/yacy/http/servlets/ServletResourceTest.java @@ -0,0 +1,65 @@ +package net.yacy.http.servlets; + +import static org.junit.Assert.assertEquals; +import static org.junit.Assert.assertFalse; +import static org.junit.Assert.assertTrue; +import static org.junit.Assert.fail; + +import java.io.ByteArrayOutputStream; +import java.nio.charset.StandardCharsets; +import java.nio.file.Files; +import java.nio.file.Path; + +import org.junit.Rule; +import org.junit.Test; +import org.junit.rules.TemporaryFolder; + +public class ServletResourceTest { + + @Rule + public TemporaryFolder temporaryFolder = new TemporaryFolder(); + + @Test + public void servesFileMetadataAndRangesWithoutJetty() throws Exception { + final Path root = this.temporaryFolder.newFolder("root").toPath(); + final Path file = Files.writeString(root.resolve("sample.txt"), "0123456789", StandardCharsets.UTF_8); + final ServletResource base = ServletResource.from(root.toString()); + final ServletResource resource = base.addPath("/sample.txt"); + + assertTrue(resource.exists()); + assertFalse(resource.isDirectory()); + assertEquals(10L, resource.length()); + assertEquals(file.toFile(), resource.getFile()); + assertTrue(resource.lastModified() > 0L); + + final ByteArrayOutputStream range = new ByteArrayOutputStream(); + resource.writeTo(range, 2L, 4L); + assertEquals("2345", range.toString(StandardCharsets.UTF_8)); + } + + @Test + public void rejectsPathsOutsideResourceBase() throws Exception { + final ServletResource base = ServletResource.from( + this.temporaryFolder.newFolder("root").toPath().toString()); + try { + base.addPath("../outside.txt"); + fail("Path traversal must be rejected"); + } catch (final IllegalArgumentException expected) { + assertTrue(expected.getMessage().contains("escapes its base")); + } + } + + @Test + public void createsEscapedDirectoryListing() throws Exception { + final Path root = this.temporaryFolder.newFolder("root").toPath(); + Files.writeString(root.resolve("<entry>.txt"), "content", StandardCharsets.UTF_8); + Files.createDirectory(root.resolve("child")); + final ServletResource directory = ServletResource.from(root.toUri().toURL()); + + final String listing = directory.getListHTML("/files/", true, null); + assertTrue(listing.contains("href=\"../\"")); + assertTrue(listing.contains("%3Centry%3E.txt")); + assertTrue(listing.contains("<entry>.txt")); + assertTrue(listing.contains("href=\"/files/child/\"")); + } +} diff --git a/test/jetty-auth-smoke-test.sh b/test/jetty-auth-smoke-test.sh index 4970a7e47..573350120 100755 --- a/test/jetty-auth-smoke-test.sh +++ b/test/jetty-auth-smoke-test.sh @@ -39,7 +39,8 @@ bin/apicall.sh 'ConfigAccounts_p.html' >/dev/null echo "ok 2 - bin/apicall.sh localhost authentication" if [ -n "${YACY_SMOKE_ADMIN_USER:-}" ] && [ -n "${YACY_SMOKE_ADMIN_PASSWORD:-}" ]; then - authenticated_status=$(status --user "$YACY_SMOKE_ADMIN_USER:$YACY_SMOKE_ADMIN_PASSWORD" \ + authenticated_status=$(status --anyauth \ + --user "$YACY_SMOKE_ADMIN_USER:$YACY_SMOKE_ADMIN_PASSWORD" \ "$base_url$protected_path") [ "$authenticated_status" = 200 ] || { echo "FAIL: administrator login returned $authenticated_status" >&2; exit 1; } echo "ok 3 - administrator credentials" diff --git a/test/jetty-smoke-test.sh b/test/jetty-smoke-test.sh index 0ff61ccc2..c8e4722e2 100755 --- a/test/jetty-smoke-test.sh +++ b/test/jetty-smoke-test.sh @@ -203,8 +203,7 @@ assert_body_contains 'YaCy ' pass "YaCy 404 error page" request GET /env/grafics/YaCyLogo2012.svg 200 \ - --header 'Accept-Encoding: gzip' \ - --raw + --header 'Accept-Encoding: gzip' assert_header_contains Content-Encoding gzip gzip -dc "$response_body" > "$work_dir/gzip-decoded" || fail "gzip response cannot be decompressed" cmp "$work_dir/static-full" "$work_dir/gzip-decoded" >/dev/null 2>&1 || \ diff --git a/test/jetty-solr-dependency-guard.sh b/test/jetty-solr-dependency-guard.sh index 21e208194..f90302a4f 100755 --- a/test/jetty-solr-dependency-guard.sh +++ b/test/jetty-solr-dependency-guard.sh @@ -1,8 +1,7 @@ #!/usr/bin/env sh -# Guard the classpath boundary needed to migrate YaCy's embedded server from -# Jetty 9 to Jetty 12 while Solr 9's Jetty client is relocated into a private -# package and kept out of YaCy source code. +# Guard the production Jetty 12 classpath while Solr 9's Jetty 9 client remains +# relocated into a private package and out of YaCy source code. set -eu @@ -25,6 +24,14 @@ if grep -R -n -E \ fail "YaCy source must not use Solr's Jetty-backed clients or runner" fi +if grep -R -n -E \ + --include='YaCyDefaultServlet.java' \ + --include='*ServletResource.java' \ + 'org\.eclipse\.jetty|Jetty9ServletResource' \ + source/net/yacy/http/servlets >/dev/null 2>&1; then + fail "YaCyDefaultServlet resources must remain servlet-container neutral" +fi + if grep -E \ 'name="(jetty-deploy|jetty-jmx)"' \ ivy.xml >/dev/null 2>&1; then @@ -34,8 +41,29 @@ fi grep -E 'name="jetty-client".*conf="solr9-bridge->master"' ivy.xml >/dev/null 2>&1 || \ fail "jetty-client must only be a direct input of the Solr 9 bridge" -grep -E 'org="org.eclipse.jetty" name="jetty-io"' ivy.xml >/dev/null 2>&1 || \ - fail "jetty-io must be an explicit dependency because YaCy imports its API" +server_jetty_version=12.1.11 + +for artifact in jetty-http jetty-io jetty-proxy jetty-security jetty-server jetty-util; do + grep -E "org=\"org.eclipse.jetty\" name=\"$artifact\" rev=\"$server_jetty_version\" conf=\"compile->default\"" ivy.xml >/dev/null 2>&1 || \ + fail "$artifact $server_jetty_version must be on the production compile classpath" +done + +for artifact in jetty-ee8-nested jetty-ee8-security jetty-ee8-servlet jetty-ee8-servlets jetty-ee8-webapp; do + grep -E "org=\"org.eclipse.jetty.ee8\" name=\"$artifact\" rev=\"$server_jetty_version\" conf=\"compile->default\"" ivy.xml >/dev/null 2>&1 || \ + fail "$artifact $server_jetty_version must be an explicit production EE8 dependency" +done +grep -E "org=\"org.eclipse.jetty.compression\" name=\"jetty-compression-server\" rev=\"$server_jetty_version\" conf=\"compile->default\"" ivy.xml >/dev/null 2>&1 || \ + fail "Jetty 12 compression server support must be on the production classpath" + +if grep -E 'conf="jetty12-migration|rev="9\.4\.58\.v20250814" conf="compile' ivy.xml >/dev/null 2>&1; then + fail "ivy.xml still contains an isolated migration configuration or public Jetty 9 dependency" +fi + +if grep -R -n -E --include='*.java' \ + '(Jetty9HttpServerImpl|AbstractRemoteHandler|YaCyLoginService|YaCySecurityHandler|YaCyDigestCredential|YacyDomainHandler|InetPathAccessHandler)' \ + source >/dev/null 2>&1; then + fail "obsolete Jetty 9 adapter references remain in production source" +fi for artifact in http2-client http2-common http2-http-client-transport; do grep -E "org=\"org.eclipse.jetty.http2\" name=\"$artifact\".*conf=\"solr9-bridge->master\"" ivy.xml >/dev/null 2>&1 || \ @@ -52,14 +80,6 @@ grep -E 'org="org.eclipse.jetty.toolchain" name="jetty-servlet-api" rev="4.0.9"' grep -E 'exclude org="javax.servlet" module="javax.servlet-api"' ivy.xml >/dev/null 2>&1 || \ fail "transitive javax.servlet-api artifacts must be excluded" -expected_jetty_version=$(sed -n \ - 's/.*org="org.eclipse.jetty" name="jetty-server" rev="\([^"]*\)".*/\1/p' \ - ivy.xml) -[ -n "$expected_jetty_version" ] || \ - fail "could not determine the public Jetty version from jetty-server in ivy.xml" -[ "$(printf '%s\n' "$expected_jetty_version" | wc -l | tr -d ' ')" -eq 1 ] || \ - fail "jetty-server must declare exactly one public Jetty version" - if [ -d lib ]; then public_jetty_count=0 for artifact in lib/jetty-*.jar; do @@ -72,21 +92,12 @@ if [ -d lib ]; then esac public_jetty_count=$((public_jetty_count + 1)) case $(basename "$artifact") in - *-"$expected_jetty_version".jar) ;; - *) fail "public Jetty artifact is not on version $expected_jetty_version: $artifact" ;; + *-"$server_jetty_version".jar) ;; + *) fail "production Jetty artifact is not on version $server_jetty_version: $artifact" ;; esac done [ "$public_jetty_count" -gt 0 ] || \ - fail "no public Jetty $expected_jetty_version artifacts found" - - case "$expected_jetty_version" in - 12.*) - for artifact in lib/jetty-continuation-*.jar; do - [ -e "$artifact" ] || continue - fail "Jetty 9-only artifact remains on the Jetty 12 classpath: $artifact" - done - ;; - esac + fail "no production Jetty $server_jetty_version artifacts found" servlet_api_count=0 for artifact in lib/*servlet-api-*.jar; do @@ -103,6 +114,12 @@ if [ -d lib ]; then for pattern in \ 'jetty-deploy-*.jar' \ 'jetty-jmx-*.jar' \ + 'jetty-slf4j-impl-*.jar' \ + 'jetty-jakarta-servlet-api-*.jar' \ + 'jakarta.servlet-api-*.jar' \ + 'jetty-ee9-*.jar' \ + 'jetty-ee10-*.jar' \ + 'jetty-ee11-*.jar' \ 'solr-core-*.jar' \ 'solr-solrj-*.jar' \ 'solr-scripting-*.jar' \ @@ -144,4 +161,4 @@ if [ -d lib ]; then done fi -echo "PASS: Solr 9 uses only the relocated Jetty client and SLF4J 1.7 island." +echo "PASS: production Jetty 12 and the relocated Solr 9 Jetty 9 dependencies are separated." |