This PR adds four new templates that are automatically installed from the Monitoring plugin.
In 8.x, Metricbeat will be writing its data in ECS compliant format, even when used with xpack
mode enabled (stack monitoring). In order to continue to support the legacy data format, new
mappings have been created with the new ECS fields for indexing data, and alias fields for the
legacy format which point to the corresponding ECS fields.
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Co-authored-by: Mat Schaffer <mat@schaffer.me>
* [DOCS] Enroll additional nodes on Docker
* Remove -p option for second node
Co-authored-by: Fabio Busatto <52658645+bytebilly@users.noreply.github.com>
* Rename nodes to align with other Docker docs
* Add elastic network to first node docker run command
* Remove hyphen from node names
Co-authored-by: Fabio Busatto <52658645+bytebilly@users.noreply.github.com>
Updates the remote clusters version compatibility table to include 7.17 and 8.x versions.
Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>
If a node reaches the flood stage watermark then we automatically apply
the `read_only_allow_delete` block to all its indices to prevent any
further growth in data. Users are expected to fix the disk space issue
by adding more space or deleting indices. However some users may prefer
to fix the disk space issues by modifying some of the index settings,
perhaps removing replicas or adjusting an allocation filter to move
shards onto nodes with more space. Today this isn't possible since the
`read_only_allow_delete` block also applies to metadata writes. Blocking
metadata writes isn't necessary to protect against further increases in
disk usage, and makes it harder for users to resolve the disk space
issue, so this commit removes the `METADATA_WRITE` level from the block
definition.
per issue 60780, decision from team to remove experimental language from HDR Histogram percentiles and ranks. Feature has been in production for quite some time.
closes#60780
* [DOCS] Add docs for verifying CA fingerprint
* Update openssl command and explanatory text
* Explain copying CA cert if fingerprint validation isn't possible
* Incorporate new section into the main security config page
* Clarify how cert is used
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
* Split into two, separate sections
* Rename file and update text based on feedback
* Update ref to use new filename
* Remove extra word
Co-authored-by: Ioannis Kakavas <ikakavas@protonmail.com>
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
The ES code base is quite JSON heavy. It uses a lot of multi-line JSON requests in tests which need to be escaped and concatenated which in turn makes them hard to read. Let's try to leverage Java 15 text blocks for representing them.
A recent change [1] to how we load our allow list changed the resolution for how Painless looks up
methods of super classes. However, functional interface loading was not changed which caused a
bug where a functional interface would not look at its super interfaces for the functional interface
method [2].
This fixes the issue by going through each super interface until the functional interface method is
found when the target interface doesn't have the functional interface method.
[1] #76955
[2] #81696
Also a big thanks to @megglos and @TheFireCookie for their help with this issue.
* [DOCS] Remove sentence about security being disabled by default
* Updating introduction
* Remove minimal security page
* Clarify configuring security before starting ES
* Clarifications
* Remove old file
* Add set passwords page
* Update change passwords page, clarify TLS adjustments, and other edits
* Update test
* Minor clarification to intro text
Co-authored-by: Elastic Machine <elasticmachine@users.noreply.github.com>
Previously the ML model snapshot upgrade endpoint did not
provide a way to reliably monitor progress. This could lead
to the upgrade assistant UI thinking that a model snapshot
upgrade had finished when it actually hadn't.
This change adds a new "stats" API that allows external
interested parties to find out the status of each model
snapshot upgrade and which node (if any) each is running on.
Fixes#81519
Apply suggestions from Docker Inc about how to update the `cacerts` in
our Ubuntu-based Docker image. Instead of copying around files and
symlinking, instead install `ca-certificates` and `p11-kit`, and use the
latter to regenerate Java's `cacerts`, as well as ensuring it is
regenerated if the system ca certs are updated.
We say to mark repos as readonly to prevent corruption, but there's
other ways to prevent corruption that people sometimes use instead (e.g.
denying writes at the filesystem/bucket level). It's reasonable to think
that the readonly flag is redundant in that situation but it's not: they
should still mark the repo as readonly tho to bypass the cache and
re-read its contents on each access. This commit adds docs to that
effect.
Co-authored-by: James Rodewig <james.rodewig@elastic.co>
Reverts an anchor change from #46711.
Previous versions of the docs use the `_shrinking_an_index` anchor for this
section. Preserving that anchor will prevent doc build breaks in future releases.
* Expose the index age in ILM explain output.
ILM already exposes the `age` that ILM will use to transition to the next phase, based on that phase's `min_age`. The `index_age` is based only on the index creation date and it's used to trigger a rollover.
Resolves#64429
Force merge action is a very costly action. It may take several hours to run for big indices. But current force merge rest api do not support wait_for_completion parameter.
This adds support for the wait_for_completion parameter.
`GET _nodes/stats` returns statistics about indexing pressure for each node.
With this commit `GET _cluster/stats` now returns stats about indexing pressure
computed by aggregating the indexing pressure stats of each node in the
cluster.
Closes#79788
Today the _Size your shards_ docs focus on shard size and count, but in
fact index count and field count are also important. This commit expands
these docs a bit to cover this observation too.
Today the same-shard allocation decider falls back to checking the
hostname if the node has no host address. In practice nodes will always
have an address so the fallback is dead code. This commit removes that
dead code.
Relates #80702 which will add the ability to distinguish nodes by
hostname regardless of whether they have an address or not, and #80767
which optimizes this area of code - this refactoring should make the
optimization simpler.
In order to perform a kNN search on a `dense_vector` field, it must have
`index: true` in its mapping. This commit clarifies the error message. Before
the message was confusing, because the user likely didn't touch the `index`
parameter and might not even be aware of it.
It adds a note to the docs clarifying that when coming from 7.x, you must
explicitly update `index: true` and reindex the vectors.
Relates to #78473.
The script context cache is deprecated.
In 7.16 the default value of `script.max_compilation_rate` was switched from `"use-context"`,
to `"150/5m"`.
That means uses would have to explicitly set `"use-context"` to use any of the context cache families of settings:
`script.context.*.max_compilations_rate`
`script.context.*.cache_max_size`
`script.context.*.cache_expire`
On upgrades to 7.16, if the customer was using the default and had set any of those settings without setting
`script.max_compilation_rate: "use-context"`, the upgrade would be fail.
To avoid an unintentional breaking change, the script service will now **implicitly** use the script context cache if
`script.max_compilation_rate` is **unset** and any of the context cache family of settings is set.
The context cache will also be used if `script.max_compilation_rate: "use-context"`, as before.
Fixes: #81486
Today, a search request with PIT would fail immediately if any
associated indices or nodes are gone, which is inconsistent when
allow_partial_search_results is true.
Relates #81256
The hidden index docs did not mention that dot-prefixed patterns default
to matching hidden indices. This PR adds a note explaining the behavior
and why it's like that.
The current `multi_match` docs contain an erroneous reference to the `combined_fields` query. This updates the reference to reference the correct query.
Relates to https://github.com/elastic/elasticsearch/pull/76893
This adds a geo point(s) field for the scripting fields api. This field only supports get(default),
get(index, default), and iterator right now. This also adds the ability to create new a GeoPoint through
the allow list, so users can specify a default that makes sense. It does not include reset as this field
currently does not wrap the data to be read-only.
Allow users to pass an ENROLLMENT_TOKEN environment variable that
would be passed as the `--enrollment-token` parameter to the
elasticsearch executable running in the container. This allows an
elasticsearch node running in docker to enroll itself to an
existing secured cluster.
Resolves: #81068
* Script: fields API for IP mapped type
Adds support in the scripting fields API for the `ip` mapped type,
including the runtime script type.
Adds a new value object, `IPAddress`, to avoid exposing Java's
`InetAddress`. `InetAddress` may cause name resolution if whitelisted
improperly.
`field('ip')`, implemented by `IpDocValuesField` exposes:
`IPAddress get(IPAddress)`
`IPAddress get(int, IPAddress)`
`Iterator<IPAddress> iterator()`
`List asStrings()`
`String asString(String)`
`String asString(int, String)`
`IPAddress` exposes:
`boolean isV4()`
`boolean isV6()`
`String toString()`
Refs: #79105
The searchable snapshot action mounts snapshots as indices
with a different prefix depending of the phase. This commit
tries to mention them in the docs.
System indices should be hidden from users. Since they are already restricted
indices, a users that can't view restricted indices already can't see or access
them, but they should also be hidden for superusers or users that are otherwise
granted advanced privileges.
To the greatest degree possible, we apply hidden settings in the transport
layer, so that the system can create an index or alias that is set to visible,
for example, when operating in a mixed cluster mode. However, in the case of
aliases created by templates, we hide the alias in the service layer.
This change has broken a number of tests that were relaying unnecessarily on
wildcard searches. In general, the fix for these issues was to apply
expand_wildcards=open,hidden to the request.
* Force system indices to be hidden in IndexMetadata
* Hide system data streams
* Update feature migration tests
* ML datafeed config defaults to searching hidden indices
* Prevent unmanaged system indices from becoming visible
* Change validation in TransportUpdateSettingsAction
* Validate index creation settings in transport action
* Make sure system data stream backing indices are hidden
* Make sure transport request adds hidden index setting if missing
* Validate and set default for autocreated system indices
* Add some code to hide system aliases
* Hide system aliases in create index service
* Hide system aliases when adding them via alias endpoints
* Check system indices when simulating and validating templates
* Add known issue for reenabling tests
* Update docs/changelog/79512.yaml
Manipulating the contents of a snapshot repository is a very bad idea,
but it turns out we don't call this out in the docs anywhere. This
commit adds a warning about this.
The documentations states that if the `weight` field is missing, and no
explicit missing configuration is provided, a default value of 1 is used.
This is incorrect and does not match the implementation of the weighted
average aggregator. In this specific case the document is skipped, instead.
Adam Locke