Using a static `diff` or epsilon just doesn't work for this test as the
scores can be very large, but relatively close.
Maybe there is a simpler way, but my mind wasn't wanting to "math" very
much.
For example, the seed that this previously failed on had scores like
`1.726524E9` and `1.7265239E9`, which, given their size, are really
close together (within 128). But a static epsilon wouldn't capture that.
closes: https://github.com/elastic/elasticsearch/issues/128485
Follow-up to #126492 to apply the json parsing optimization to strings
containing unicode characters and some backslash-escaped characters.
Supporting backslash-escaped strings is tricky as it requires modifying the
string. There are two types of modification: some just remove the backslash
(e.g. \", \\), and some replace the whole escape sequence with a new
character (e.g. \n, \r, \u00e5). In this implementation, the optimization
only supports the first case--removing the backslash. This is done by
making a copy of the data, skipping the backslash. It should still be more
optimized than full String decoding, but it won't be as fast as
non-backslashed strings where we can directly reference the input bytes.
Relates to #129072.
* Support multiple plugin source paths
* Refactor: remove unncessary PathLookup method.
It's only called in one place, and there's no need to override it for testing.
Removing it just makes things simpler.
* Refactor: local var for pathLookup
* Fix bugs in test build info parsing
* Fix representative_class in test
* Move BridgeUtilTests.
Tests in org.elasticsearch.entitlement.bridge are going to be uniquely hard to
test once we patch the bridge into java.base, due to Java's prohibition on
split packages.
Let's just move this guy to another package.
* Upcast (?!) Java23EntitlementChecker to EntitlementChecker
* Empty TestPathLookup
* Create PolicyManager during bootstrap, allowing us to share initialization
* Use empty component path list instead of null
* Downcast to the class of the check method.
In our unit test, we have a mock checker that doesn't extend
EntitlementChecker, so downcasting to that would require us to needlessly
rework the unit test.
* Fix javadoc typos
This ensures we package an aggregation zip with all artifacts we want to publish to maven central as part of a release.
Running zipAggregation will produce a zip file in the build/nmcp/zip folder. The content of this zip is meant to match the maven artifacts we have currently declared as dra maven artifacts.
When parsing documents, we receive the document as UTF-8 encoded data which
we then parse and convert the fields to java-native UTF-16 encoded Strings.
We then convert these strings back to UTF-8 for storage in lucene.
This patch skips the redundant conversion, instead passing lucene a
direct reference to the received UTF-8 bytes when possible.
This PR is a precursor to #126492.
It does three things:
1. Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
2. Refactor the Text class to use a new EncodedBytes record instead of
the elasticsearch BytesReference.
3. Add the XContentString interface, with the Text class implementing
that interface.
These changes were originally implemented in #127666 and #128316,
however they were reverted in #128484 due to problems caused by the
mutable nature of java ByteBuffers. This is resolved by instead using a
new immutable EncodedBytes record.
Instead of waiting for the next run of the `ClusterStateObserver` (which
might be arbitrarily far in the future, but bound by the timeout if one
is set), we notify the listener immediately that the task has been
cancelled. While doing so, we ensure we invoke the listener only once.
Fixes#117971
Entitlement instrumentation works by reflectively calling back into the
entitlements lib to grab the checker. It must be fully in place before
any classes are instrumented. This commit fixes a bug that was
introduced by refactoring which caused the checker to not be set until
after all classes were instrumented. In some situations this could lead
the checker to being null when it is grab (and statically cached) by the
entitlement bridge.
* Rename and encapsulate InitializeArgs
* Move ElasticsearchEntitlementChecker out of api package.
It's an implementation detail that doesn't need to be exposed to the rest of
the system.
* Stub TestPathLookup (not yet implemented)
* Revert "Fix the Text class package change in example plugins (#128316)"
This reverts commit cc486480e3.
* Revert "Update Text class to use native java ByteBuffer (#127666)"
This reverts commit db0c3c7a28.
Co-authored-by: Lorenzo Dematté <lorenzo.dematte@elastic.co>
* Use package to suppress warning for entitlement self-test
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Entitlements do a stack walk to find the calling class. When method
refences are used in a lambda, the frame ends up hidden in the stack
walk. In the case of using a method reference with
AccessController.doPrivileged, the call looks like it is the jdk itself,
so the call is trivially allowed. This commit adds hidden frames to the
stack walk so that the lambda frame created for the method reference is
included. Several internal packages are then necessary to filter out of
the stack.
This PR is a precursor to #126492.
It does three things:
- Move org.elasticsearch.common.text.Text from :server to
org.elasticsearch.xcontent.Text in :libs:x-content.
- Refactor the Text class to use a java-native ByteBuffer instead
of the elasticsearch BytesReference.
- Add the XContentString interface, with the Text class implementing
that interface.
While backporting entitlement initialization refactorings, I realized there is a mismatch in getVersionSpecificCheckerClass signature, and also that this function in the backports is used in more places (DynamicInstrumentation), making it "strange" to have this in EntitlementInitialization. This PR extracts the function to a separate static class (package-private) and makes the signature uniform with backports.
This will need to be backported manually to the 8.x branches, and will make the backported version of DynamicInstrumentation cleaner.
Moves FilesEntitlements validation to a separate class. This is the final PR to make EntitlementsInitialization a simpler "orchestrator" of the various steps in the initialization phase.
Our path comparison for file access is string based, due to the fact that we need to support Paths created for different file systems/platforms.
However, Windows files and paths are (sort of) case insensitive.
This PR fixes the problem by abstracting String comparison operations and making them case sensitive or not based on the host OS.
Now that entitlements are always used, there is no need to run tests
with security manager (a future enhancement will run tests with
entitlements). This commit removes setting up security manager from
tests.
We already had logging in FileAccessTree as result of debugging the \\pipe\ failures a while ago; this PR slightly improves the logs to provide more information.
Design and implementation of Entitlement with this level of detail needs to stay close to the code, and take advantage of javadoc features like linking and class-references to help us with refactorings and future code changes.
The bulk of the information went into the package-info file for the main library, but I split up some parts and referenced them from the main doc, where I thought it made sense (mainly: the bridge sub-project for some implementation details, PolicyManager, EntitlementInitialization and FileAccessTree); this way they still can be reached from the "overview" while being closer to where the information really belongs.
Relates to ES-11284
This change refactors the known directory resolution such as modules,
plugins, lib, etc. into a PathLookup. This is one of the steps towards
allowing unit tests to provide their own PathLookup for resolution so
we can enable entitlements there.
ES-11584
With this PR we restrict the paths we allow access to, forbidding plugins to specify/request entitlements for reading or writing to specific protected directories.
I added this validation to EntitlementInitialization, as I wanted to fail fast and this is the earliest occurrence where we have all we need: PathLookup to resolve relative paths, policies (for plugins, server, agents) and the Paths for the specific directories we want to protect.
Relates to ES-10918
On x64, we are testing if we support vector capabilities (1 = "basic" = AVX2, 2 = "advanced" = AVX-512) in order to enable and choose a native implementation for some vector functions, using CPUID.
However, under some circumstances, this is not sufficient: the OS on which we are running also needs to support AVX/AVX2 etc; basically, it needs to acknowledge it knows about the additional register and that it is able to handle them e.g. in context switches. To do that we need to a) test if the CPU has xsave feature and b) use the xgetbv to test if the OS set it (declaring it supports AVX/AVX2/etc).
In most cases this is not needed, as all modern OSes do that, but for some virtualized situations (hypervisors, emulators, etc.) all the component along the chain must support it, and in some cases this is not a given.
This PR introduces a change to the x64 version of vec_caps to check for OS support too, and a warning on the Java side in case the CPU supports vector capabilities but those are not enabled at OS level.
Tested by passing noxsave to my linux box kernel boot options, and ensuring that the avx flags "disappear" from /proc/cpuinfo, and we fall back to the "no native vector" case.
Fixes#126809
In Jackson 2.15 a maximum string length of 50k characters was
introduced. We worked around that by override the length to max int on
all parsers created by xcontent. Jackson 2.17 introduced a similar limit
on field names. This commit mimics the workaround for string length by
overriding the max name length to be unlimited.
relates #58952
Ignacio Vera