Moves FilesEntitlements validation to a separate class. This is the final PR to make EntitlementsInitialization a simpler "orchestrator" of the various steps in the initialization phase.
Our path comparison for file access is string based, due to the fact that we need to support Paths created for different file systems/platforms.
However, Windows files and paths are (sort of) case insensitive.
This PR fixes the problem by abstracting String comparison operations and making them case sensitive or not based on the host OS.
Now that entitlements are always used, there is no need to run tests
with security manager (a future enhancement will run tests with
entitlements). This commit removes setting up security manager from
tests.
We already had logging in FileAccessTree as result of debugging the \\pipe\ failures a while ago; this PR slightly improves the logs to provide more information.
Design and implementation of Entitlement with this level of detail needs to stay close to the code, and take advantage of javadoc features like linking and class-references to help us with refactorings and future code changes.
The bulk of the information went into the package-info file for the main library, but I split up some parts and referenced them from the main doc, where I thought it made sense (mainly: the bridge sub-project for some implementation details, PolicyManager, EntitlementInitialization and FileAccessTree); this way they still can be reached from the "overview" while being closer to where the information really belongs.
Relates to ES-11284
This change refactors the known directory resolution such as modules,
plugins, lib, etc. into a PathLookup. This is one of the steps towards
allowing unit tests to provide their own PathLookup for resolution so
we can enable entitlements there.
ES-11584
With this PR we restrict the paths we allow access to, forbidding plugins to specify/request entitlements for reading or writing to specific protected directories.
I added this validation to EntitlementInitialization, as I wanted to fail fast and this is the earliest occurrence where we have all we need: PathLookup to resolve relative paths, policies (for plugins, server, agents) and the Paths for the specific directories we want to protect.
Relates to ES-10918
On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID.
However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc).
In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given.
This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level.
Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case.
Fixes#126809
In Jackson 2.15 a maximum string length of 50k characters was
introduced. We worked around that by override the length to max int on
all parsers created by xcontent. Jackson 2.17 introduced a similar limit
on field names. This commit mimics the workaround for string length by
overriding the max name length to be unlimited.
relates #58952
These entitlements should not be necessary. Server does not directly
create any outbound connections, and netty is not a dependency of server
so it should not exist within the server policy.
This commit cleans up how entitlement test methods are discovered. It
also adds another robustness check to ensure an annotation doesn't exist
on a private method.
In the unexpected case that Elasticsearch dies due to a segfault or
other similar native issue, a core dump is useful in diagnosing the
problem. Yet core dumps are written to the working directory, which is
read-only for most installations of Elasticsearch. This commit changes
the working directory to the logs dir which should always be writeable.
This PR expands the README for the entitlements package; it adds information for Elasticsearch developers on how the system is structured, how to build policies, what to do when new code "does not work" because it's bumping into a NotEntitledException, and how to apply a policy patch in case of an emergency.
This PR fixes 2 issues discovered around PolicyUtils (and PolicyUtilsTests) when running CI on Windows:
- in tests, absolute paths like always are different, this fix updates the tests to account for the difference.
- on Windows, Files.move was failing because we were parsing the Entitlement policy but not closing the stream in a timely manner. This causes plugin installation (and related CI tests) to fail 70% of the time. Fixed by closing the stream properly
Fixes#126176
This PR replaces the parsing and formatting of SecurityManager policies with the parsing and formatting of Entitlements policy during plugin installation.
Relates to ES-10923
The transform API for instrumentation should return null when no
transformation occurs. This commit fixes our entitlement transformer to
return null instead of the original buffer.
closes#125920
* Use ALL-UNNAMED for the unnamed module in warnings
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Using ASM CheckClassAdapter was key to diagnose the issue we had with incorrect signatures for some check methods.
In this PR I polished up the code I used to pinpoint the issue, and made it available via a system property so it can be turned on if we need it (and it's always on for Entitlements IT tests too).
It is also turned on in case we get VerifyErrors during retransformClasses early in the Entitlement agent bootstrap: retransformClasses runs in the native part of the JVM, so the VerifyError it produces is not so readable (e.g. it lacks a full stack trace and a description); in case this happens, we re-apply the transformation with verification turned on to get a meaningful error before dying.
With #124904 we introduced server policy patching via system properties; now that we have the possibility to merge policies and scopes, it's worth applying the same mechanism to plugin policies too.
This PR changes the behaviour of plugin policy overrides; now they are not replacing the policy entirely, but patch it by adding the entitlements and scopes specified in the "patch policy"
* Use child loggers broadly in PolicyManager
* Pass ModuleEntitlements to notEntitled
* Store logger name instead of object.
Some of our unit tests check for equality of `ModuleEntitlements` objects,
and they are entitled to do so (no pun intended).
* Alright, let's cache 'em
* Memoize at point of creation, not point of use
* Explanatory comments
The ingest-attachment module uses Tika to parse some content; Tika in turn uses some libraries from java.desktop to perform its tasks.
In turn, the JDK loads one (or more) native libraries for its implementation as part of class initialization. This means we need to grant load_native_libraries to java.desktop so that because AWT can load libraries for itself.
This PR adds the ability to override entitlement policies for Elasticsearch plugins and modules via a system property.
The system property is in the form es.entitlements.policy.<plugin name>, and accepts a versioned policy:
versions:
- version1
- versionN
policy:
<a standard entitlement policy>
For example:
versions:
- 9.1.0
policy:
ALL-UNNAMED:
- set_https_connection_properties
- outbound_network
- files:
- relative_path: ".config/gcloud"
relative_to: home
mode: read
The versioned policy needs to be base64 encoded. For example, to pass the above policy to a test cluster via gradle run:
./gradlew run --debug-jvm -Dtests.jvm.argline="-Des.entitlements.policy.repository-gcs=dmVyc2lvbnM6CiAgLSA5LjEuMApwb2xpY3k6CiAgQUxMLVVOTkFNRUQ6CiAgICAtIHNldF9odHRwc19jb25uZWN0aW9uX3Byb3BlcnRpZXMKICAgIC0gb3V0Ym91bmRfbmV0d29yawogICAgLSBmaWxlczoKICAgICAgLSByZWxhdGl2ZV9wYXRoOiAiLmNvbmZpZy9nY2xvdWQiCiAgICAgICAgcmVsYXRpdmVfdG86IGhvbWUKICAgICAgICBtb2RlOiByZWFkCg=="
The versions listed in the policy are string-matched against Build.version().current(); it is possible to specify any number of versions. If the list is empty/there is no versions field, the policy is assumed to match any Elasticsearch versions.
The override policy specified for any given plugin replaces the embedded policy for that plugin. See how EntitlementsAllowedViaOverrideIT replaces an empty policy for the entitlement-test-plugin with a policy that allows load_native_libraries and access to files in the test read_dir.
Also tested manually with an override with a different version, with an override with an invalid policy and with a valid override (see command above).
Relates to ES-11009
When enabling FIPS `javax.net.ssl.trustStore` is not necessarily set.
This change adds FIPS entitlements based on
`org.bouncycastle.fips.approved_only=true`, which enforces usage of FIPS
approved functionality only.
Additionally, this PR grants read access to a custom trust store if
provided via `javax.net.ssl.trustStore`, otherwise read access to the
default JDK trust store is granted.
Relates to ES-11025.
Writing tests for #123861, turns out that #124195 is not enough.
We really need new IT test cases for "always allowed" actions: in order to be sure they are allowed, we need to setup the plugin with no policy.
This PR adds test cases for that, plus the support for writing test functions that accept one Environment parameter: many test paths we test and allow/deny are relative to paths in Environment, so it's useful to have access to it (see readAccessConfigDirectory as an example)
* Refactor: findRequestingFrame
* INFO instead of WARN for NotEntitledException.
Some of these are expected, so an INFO seems more appropriate.
The stack trace tends to attract attention even when entitlements are not the
cause of a problem, so let's avoid the stack trace, but still include stack
frame info from the frame of interest.
* Use child loggers for Not Entitled logs
* Use warn, and include compoenent name
* Fix ALL_UNNAMED
* Mute entitlement warnings from repositories
* PR feedback
* Common out the Not Entitled prefix.
We're alerting on this, so let's not rely on every caller of notEntitled to remember it.
This will rethrow the `NoSuchFileException` when encountering an invalid
symbolic link when following links during file (read) entitlement
checks.
Relates to https://github.com/elastic/elasticsearch/pull/124133
(ES-11019)
Even if the contract for JDK methods using the SecurityManager states that the exception throw is of type SecurityException, many libraries (including our own, apparently!) violates that and use the type actually thrown by SecurityManager, AccessControlException. A prime example is the GCS/CSP libraries.
In order to maintain compatibility for them, we need to inherit from the more specific AccessControlException; this is less desirable, as AccessControlException is marked as deprecated for removal alongside the other SecurityManager classes, but we discussed and found this is the best short term solution.
More work will be needed -- again, this is a short term solution.
Replaces #123984
Our check methods injected by the instrumenter receive "this" as the second parameter.
For internal classes like AbstractDelegateHttpsURLConnection we generally use a base type; in this case we were using javax.net.ssl.HttpsURLConnection, which is incorrect as AbstractDelegateHttpsURLConnection derives from java.net.HttpURLConnection.
This was not failing in our tests because we don't actually use that parameter in that check function.
Also, it was not failing on transform, just on retransformClasses, and only in JDK 24. Apparently, JDK 24 introduced new validation there (to be confirmed).
And it was failing just on cloud as the APM agent there (which is loaded before our agent) connects to a https endpoint - our IT tests, and ./gradlew run --with-apm-server, use a http endpoint. Using https makes the JVM load AbstractDelegateHttpsURLConnection, making it one of the classes we need to retransform, triggering the VerifyError.
Makes the execution and use of enrich policies project-aware.
Note: this does not make the enrich cache project-aware. That is to be
handled in a follow-up PR.
This PR adjusts the list of supported ciphers to reflect ciphers
available in JDK 24.
JDK 24 [drops](https://bugs.openjdk.org/browse/JDK-8245545) support for
`TLS_RSA` suites. These ciphers will no longer be supported in
Elasticsearch with a bundled JDK with version >= 24. JDK's of lower
versions will continue to support to dropped ciphers.
I will follow up this PR with a separate docs PR.
* Fix Gradle Deprecation warning as declaring an is- property with a Boolean type has been deprecated.
* Make use of new layout.settingsFolder api to address some cross project references
* Fix buildParams snapshot check for multiprojet projects
* Remove @UpdateForV9 annotation in RuntimeVersionFeature
Re ES-10338 the Core/Infra team agreed that this is not blocked on security manager removal. We should go ahead and remove these 2 instances of the @UpdateForV9 annotation, the code will be cleaned up soon with the removal of Security Manager.
* Remove @UpdateForV9 annotation from SystemJvmOptions
re ES-10338 the Core/Infra team agreed this is not blocked on security manager removal. We should go ahead and remove this instance of the @UpdateForV9 annotation.
All CLIs in elasticsearch support command line flags for controlling the
output level. When --silent is used, the expectation is that normal
logging is omitted. Yet the log4j logger is still configured to output
error level logs. This commit sets the appropriate log level for log4j
depending on the Terminal log level.
This commit reworks the settings globs to be more useable. Primarily it
expands the values so that the settings may be lists, iterating over
each value. Additionally it simplifies the function to also allow
non-glob settings so that this single method may be used to lookup all
values for a given setting pattern, whether it contains a glob or not.
The setting based paths could be either absolute or relative, and they
are always relative to the config dir. This commit renames the
path_setting to make it clear it is related to config, and removes the
relative variant.
SSL file utils currently only handle security manager access control
exceptions around file read checks. This PR extends these to support
entitlement checks as well.
There is no easy way to unit test this since we can't run unit tests
with entitlements enabled (for now). The PR includes a REST test
instead.
Relates: https://github.com/elastic/elasticsearch/issues/121960
Bit of a random find. Looks like we're effectively spending something
like ~0.5% of the total CPU time on resolving virtual calls for these
methods. Inlining `IOUtils` reduces their size quite a bit, removes
dead-code for rethrowing `IOException` that is never actually thrown
and hopefully as a result improve compilation a little here.
This adds an exclusive parameter for FilesEntitlement where a path can be made exclusive for a certain
module. Should two modules attempt to both specify the same path as exclusive an exception is
thrown.
Ryan Ernst