File entitlements which read a setting may actually contain urls. This
commit adds an optional `ignore_url` property for the entitlement to
skip any values which are urls.
The NamedComponentReader reads a file created upon plugin installation
for stable plugins from the plugin installation dir. This commit passes
the plugins directory through to entitlements and grants server access.
* Consider entitlement lib as system module
Entitlements sometimes needs to perform sensitive operations,
particularly within the FileAccessTree. This commit expands the
trivially allowed check to include entitlements as one of the system
modules alongside the jdk. One consequence is that the self test must be
moved outside entitlements.
* [CI] Auto commit changes from spotless
* remove old method call
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Some file paths are OS specific. This commit adds a `platform` property
to each file in a files entitlement that can be used to limit that file
to a specific platform.
Co-authored-by: Moritz Mack <mmack@apache.org>
Co-authored-by: Lorenzo Dematte <lorenzo.dematte@elastic.co>
* java.base entitlements
* SuppressForbidden, and add a missing test
* Revert logging back to commented-out printlns
* Merge FileCheckActions and rename for overloads
* Remove stray logger
* Remove more traces of logging change
* Remove more traces of logging
With Security Manager we have SecuredConfigFileSettingAccessPermission.
This commit adds an entitlement equivalent. With each entry in files
entitlement, a `path_setting` can now be used. The value may be an
explicit setting, or a setting glob with a single `*`.
relates ES-10844
This commit switches to using records in place of classes for FileData
impls. It moves them outside the interface so that they can be private.
This does not change the fact the interface is sealed, nor the factory
methods for creating them.
While testing https://github.com/elastic/elasticsearch/pull/122591, I
realized we need to grand read/write permission to the logs dir to
server.
This PR adds the `logsDir` to the bootstrap parameters, and uses it in
the `server` policy.
Currently the SLM health indicator in health report turns YELLOW when snapshots fail for a number of times. However, the SLM health indicator stays GREEN if snapshot is not completed (no success or failure) for a long time. This change adds a new optional setting unhealthy_if_no_snapshot_within to SLM policy, that sets a time threshold. If the SLM policy has not had a successful snapshot for longer than the threshold, the SLM health indicator will turn YELLOW.
Temp dir access is necessary all over Elasticsearch, and in general is
not sensitive. With Security Manager all code in ES, including plugins,
are given read/write access to the temp dir. This commit mimicks that
behavior with entitlements.
This commit adds instrumentation for File methods that require write
permission. No server or plugins use these methods, so no policy changes
were necessary. Note that since we are not planning to restrict temp
file creation, the bootstrap self test on file writing was removed,
which failed with these changes.
This commit removes "TLSv1.1" from the list of default protocols in
Elasticsearch (starting with ES9.0)
TLSv1.1 has been deprecated by the IETF since March 2021
This affects a variety of TLS contexts, include
- The HTTP Server (Rest API)
- Transport protocol (including CCS and CCR)
- Outgoing connections for features that have configurable SSL
settings. This includes
- reindex
- watcher
- security realms (SAML, OIDC, LDAP, etc)
- monitoring exporters
- inference services
In practice, however, TLSv1.1 has been disabled in most Elasticsearch
deployments since around 7.12 because most JDK releases have disabled
TLSv1.1 (by default) starting in April 2021
That is, if you run a default installation of Elasticsearch (for any
currently supported version of ES) that uses the bundled JVM then
TLSv1.1 is already disabled.
And, since ES9+ requires JDK21+, all supported JDKs ship with TLSv1.1
disabled by default.
In addition, incoming HTTP connections to Elastic Cloud deployments
have required TLSv1.2 or higher since April 2020
This change simply makes it clear that Elasticsearch does not
attempt to enable TLSv1.1 and administrators who wish to use that
protocol will need to explicitly enable it in both the JVM and in
Elasticsearch.
Resolves: #108057
Like the plugin being testing, the entitled test plugin needs access to
dynamic elements (namely, file paths). This commit dynamically generates
the entitlement policy for the entitlted test plugin when it is
installed. It also adds using the file entitltlement as an example.
The only real path separators are either forward or back slash. Trying
to use something else like newline fails to even parse as a path on
windows. This commit removes testing of other separators.
closes#121872
This change replaces FileEntitlement with FilesEntitlement so that we can have exactly one entitlement
class per module (or possibly future scope). This cleans up our policy files so that all files are located
together to allow access, and this opens up the design for future optimizations.
* Report componentName in ModuleEntitlements.
This lets us produce error messages that guide the user to add the right
entitlement to the right plugin/server/etc.
* Include component names in errors and logs
* Name APM agent specifically.
Avoids confusion with the entitlements agent.
* Entitlement component names package private
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Fix FileAccessTree for prefixes that aren't parents
* Support backslashes
* Whoops, nio
* Move normalization responsibility to FileEntitlement
* Normalize to native separators
* Avoid forbidden API
This updates the PolicyParser to allow static methods to have an ExternalEntitlement annotation. This
removes a limitation where constructors cannot properly support type-erasure with different types of
data structures for internal entitlement generation and external entitlement generation (for example
List<Object> from the parser and List<SomeData> from an internal builder). We continue to enforce
that only one constructor/method may be annotated with ExternalEntitlement per Entitlement class.
Instrumented methods in EntitlementChecker are loosely grouped by the
associated entitlement. This commit sorts the methods within groups to
allow more clear placement of additional instrumented methods.
* Refactor: remove duplicate canWrite methods.
This serves as a good example of how Path and File handling could be
specialized in the future, but as long as they are identical, the duplication
causes more harm than good.
* Refactor: just one neverEntitled.
The original motivation was to avoid allocating a lambda object on each call,
but since that's a highly optimized operation in the JVM, it's unlikely to make
a difference in practice, and this smacks of premature optimization.
We're pretty liberal about lambdas elsewhere, so let's not sweat it here until
we have some evidence that it matters.
* Remove dead code
This commit adds an EntitlementTest annotation that can be used on
classes containing test actions for entitlements. The annotation mirrors
the parameters of CheckAction. Only file check actions are currently
converted, the rest can be moved and annotated as followups. Note that
the check action name is simply the method name, no fancy name
manipulation is done.
This commit adds the data dirs, config dir and temp dir into
entitlement bootstrapping. It doesn't yet use them in entitlement
policies, but makes them available to use within initialization.
This adds basic flag entitlement validation when creating PolicyManager. If a module has the same flag
entitlement as part of it's policy multiple times we will throw an IllegalArgumentException. With this
validation we can safely assume FileEntitlement is the only one we currently have that allows multiple entitlements in a policy.
This commit adds FileEntitlement to entitlements. It does not add checks
for all file access methods yet, instead opting for example read and
write methods.
Each module contains a sorted array of paths with read and write
permissions. Binary search is used to quickly identify the closest path
to determine whether a target path can be read or written.
Some important things about FileEntitlement are the path can either be a
file or a directory. All directories grant recursive permission. The
mode is either read or read_write. All operations like create or delete
are considered write.
relates ES-10354
Co-authored-by: Jack Conradson <osjdconrad@gmail.com>
This change closes a hole where we assumed any check against an unnamed-module from any
classloader was for one of our apm agent. This was not the case and made it so scripts could in theory
have the same entitlements as apm agent. Instead we now check to see if a class is part of the apm
package in an unnamed module to ensure it's actually for the apm agent.
Relates to ES-10192
A new query parameter `?include_source_on_error` was added for create / index, update and bulk REST APIs to control
if to include the document source in the error response in case of parsing errors. The default value is `true`.
This commit adds an AbstractEntitlementsIT and moves the entitlement
cluster setup into a bespoke EntitlementTestRule. That allows most of
the common code to be deduplicated. This change also automatically
creates a temp dir which the test passes along into the test cluster.
The mrjar plugin adds support for sourcesets named in the form mainNN,
which adds the appropriate compiler and other settings for that version
of Java, and produces a multi-release jar. Having multi-release jars
only makes sense for versions of java newer than the minimum compile
version. This commit adds validation that the version is not too old.
Note that the check is slightly relaxed; it allows mainNN where NN is
equal to the min java version. This is due to the desire to keep
code using incubating modules separate because warnings must be
disabled.
When an entitlement is available to policy files, it should have one
constructor that the parser uses. This commit adjusts the policy parser
to scan the constructor to find that one annotated constructor, and
errors if more than one is found.
This commit combines the different entitlement test plugins used for
integration tests into a single plugin. When installing the plugin in
the integ test cluster, the entitlement policy and plugin properties are
adjusted to test the desired combination of settings. This allows the
plugin policy to be dynamically generated by each test, rather than rely
on a static policy.
Instrument write access to System properties by means of the `WriteSystemPropertiesEntitlement`.
`System.setProperties(Properties)` is always denied.
Part of #ES-10359
This adds a new exception called PolicyValdidationException. This exception is intended to be used during construction of entitlements that require parsing additional arguments. This allows PolicyParser to give improved error messaging about a specific entitlement that may have failed during the parsing process.
Support for `ST_EXTENT_AGG` was added in https://github.com/elastic/elasticsearch/pull/118829, and then partially optimized in https://github.com/elastic/elasticsearch/pull/118829. This optimization worked only for cartesian_shape fields, and worked by extracting the Extent from the doc-values and re-encoding it as a WKB `BBOX` geometry. This does not work for geo_shape, where we need to retain all 6 integers stored in the doc-values, in order to perform the datelline choice only at reduce time during the final phase of the aggregation.
Since both geo_shape and cartesian_shape perform the aggregations using integers, and the original Extent values in the doc-values are integers, this PR expands the previous optimization by:
* Saving all Extent values into a multi-valued field in an IntBlock for both cartesian_shape and geo_shape
* Simplifying the logic around merging intermediate states for all cases (geo/cartesian and grouped and non-grouped aggs)
* Widening test cases for testing more combinations of aggregations and types, and fixing a few bugs found
* Enhancing cartesian extent to convert from 6 ints to 4 ints at block loading time (for efficiency)
* Fixing bugs in both cartesian and geo extents for generating intermediate state with missing groups (flaky tests in serverless)
* Moved the int order to always match Rectangle for 4-int and Extent for 6-int cases (improved internal consistency)
Since the PR already changed the meaning of the invalid/infinite values of the intermediate state integers, it was already not compatible with the previous cluster versions. We disabled mixed-cluster testing to prevent errors as a result of that. This leaves us the opportunity to make further changes that are mixed-cluster incompatible, hence the decision to perform this consistency update now.
* Refactor: separate check method name vs signature parsing
* Cosmetic: change checker comment format
* Entitlements for JDK-wide global state
* [CI] Auto commit changes from spotless
* Comment explaining entitlement add-exports
* @SuppressForbidden
* Refactor: rename dummy subclases
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
* Use $$ for static check methods
* Always use class name from checker method name
* Spotless
* Varargs assertParseCheckerMethodSignatureThrows
* Remove hasReceiver again
* Spotless
* Cosmetic changes
* Change to new check method naming convention
* Grant all entitlements to system modules
* [CI] Auto commit changes from spotless
* Make NO_ENTITLEMENTS_MODULE non-null
* Initialize NO_ENTITLEMENTS_MODULE with @BeforeClass.
Looks like @WithoutSecurityManager doesn't work with static initializers.
* Move check to public method
* Logging adjustments
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This updates the gradle wrapper to 8.12
We addressed deprecation warnings due to the update that includes:
- Fix change in TestOutputEvent api
- Fix deprecation in groovy syntax
- Use latest ospackage plugin containing our fix
- Remove project usages at execution time
- Fix deprecated project references in repository-old-versions
* Reapply "[Build] Do not invalidate configuration cache when branch is switched (#118894)" (#119300)
The original PR (#118894) has broken serverless.
* Fix gitinfo plugin for serverless usage
* Update buildscan git revision reference
During entitlements initialization the transformer is added to
instrumenation after some classes are already loaded. Currently we end
up force loading (though not initializing) all classes that want to
transform. This commit simplifies the retransform to only apply to
classes which we know are already loaded by the jdk, which
Instrumentation provides.
* More robust frame skipping
* Cosmetic improvements for clarity
* Explicit set of runtime classes
* Pass entitlements runtime module to PolicyManager ctor
* Use the term "entitlements module" and filter instead of dropWhile
* [CI] Auto commit changes from spotless
---------
Co-authored-by: elasticsearchmachine <infra-root+elasticsearchmachine@elastic.co>
This commit simplifies the entitlements instrumentation service and
instrumenter a bit. It especially removes some repetition in the
instrumenter tests.
* WIP: Tool to find all public caller from a starting list of (JDK) methods.
* Add public-callers-finder tool, extract common stuff to common module
* Adjustments to visibility/functions and classes and modules to print out
* Spotless
* Missing gradle configuration
* Add details in README as requested in PR
* Update ASM version
* Including protected methods
This PR adds support for ST_EXTENT_AGG aggregation, i.e., computing a bounding box over a set of points/shapes (Cartesian or geo). Note the difference between this aggregation and the already implemented scalar function ST_EXTENT.
This isn't a very efficient implementation, and future PRs will attempt to read these extents directly from the doc values.
We currently always use longitude wrapping, i.e., we may wrap around the dateline for a smaller bounding box. Future PRs will let the user control this behavior.
Fixes#104659.
The "sythetic" tests are the only unit tests for the instrumenter. This
commit renames the test suite to be more clear it is the place to put
instrumenter tests.
This change introduces optional source filtering directly within source loaders (both synthetic and stored).
The main benefit is seen in synthetic source loaders, as synthetic fields are stored independently.
By filtering while loading the synthetic source, generating the source becomes linear in the number of fields that match the filter.
This update also modifies the get document API to apply source filters earlier—directly through the source loader.
The search API, however, is not affected in this change, since the loaded source is still used by other features (e.g., highlighting, fields, nested hits),
and source filtering is always applied as the final step.
A follow-up will be required to ensure careful handling of all search-related scenarios.
This PR expands `PolicyManager` to actually use `Policy` and
`Entitlement` classes for checks, instead of hardcoding them.
It also introduces a separate `PluginsResolver`, with a dedicated
function to map a Class to a Plugin (name). `PluginsResolver` is
initialized with data from `PluginsLoader`, and then its resolve
function is used internally in `PolicyManager` to find a plugin policy
(and then test against the entitlements declared in the policy).
Instead of doing an "if" statement, which doesn't lend itself to
vectorization, I switched to expand to the bits and multiply the 1s and
0s.
This led to a marginal speed improvement on ARM.
I expect that Panama vector could be used here to be even faster, but I
didn't want to spend anymore time on this for the time being.
```
Benchmark (dims) Mode Cnt Score Error Units
IpBitVectorScorerBenchmark.dotProductByteIfStatement 768 thrpt 5 2.952 ± 0.026 ops/us
IpBitVectorScorerBenchmark.dotProductByteUnwrap 768 thrpt 5 4.017 ± 0.068 ops/us
IpBitVectorScorerBenchmark.dotProductFloatIfStatement 768 thrpt 5 2.987 ± 0.124 ops/us
IpBitVectorScorerBenchmark.dotProductFloatUnwrap 768 thrpt 5 4.726 ± 0.136 ops/us
```
Benchmark I used.
https://gist.github.com/benwtrent/b0edb3975d2f03356c1a5ea84c72abc9
There are two clear code paths depending if a h3 bin belongs to even resolutions (class II) or
uneven resolutions (class III). especializing the code paths for each type leads to an improvement in performance.
Following up
https://github.com/elastic/elasticsearch/pull/117332#discussion_r1856803255,
I refactored `InstrumenterImpl` tests, splitting them into 2 suites: -
`SyntheticInstrumenterImplTests`, which tests the mechanics of
instrumentation using ad-hoc test cases. This should see little change
now that we have our Instrumenter working as intended -
`InstrumenterImplTests`, which is back to its original intent to make
sure (1) the right arguments make it all the way to the check methods,
and (2) if the check method throws, that exception correctly bubbles up
through the instrumented method.
The PR also includes a little change to `InstrumenterImpl` construction
to clean it up a bit and make it more testable.
I goofed on the bit * byte and bit * float comparisons. Naturally, these
should be bigendian and compare the dimensions with the binary ones
appropriately.
Additionally, I added a test to ensure that this is handled correctly.
This commit adjusts the common lib of entitlement tools to use
elasticsearch.build so that it gets java version configuration
automatically. Additionally the mrjar plugin is removed from the core
lib since it is not used there.
As each version of Java is released, there may be additional methods we
want to instrument for entitlements. Since new methods won't exist in
the base version of Java that Elasticsearch is compiled with, we need to
hava different classes and compilation for each version.
This commit adds a scaffolding for adding the classes for new versions
of Java. Unfortunately it requires several classes in different
locations. But hopefully these are infrequent enough that the
boilerplate is ok. We could consider adding a helper Gradle task to
templatize the new classes in the future if it is too cumbersome. Note
that the example for Java23 does not have anything meaningful in it yet,
it's only meant as an example until we find go through classes and
methods that were added after Java 21.
When a parallel stream is opened, the jdk uses an internal fork join
pool to do work on processing the stream. This pool is internal to the
jdk, and so it should always be allowed to create threads. This commit
modifies SecureSM to account for this innocuous thread group and
threads.
Static fields dont do well in Gradle with configuration cache enabled.
- Use buildParams extension in build scripts
- Keep BuildParams.ci for now for easy serverless migration
- Tweak testing doc
Tim Vernum