Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Ability to Read-Only a Data Stream #100439

Open
BenB196 opened this issue Oct 6, 2023 · 3 comments
Open

Ability to Read-Only a Data Stream #100439

BenB196 opened this issue Oct 6, 2023 · 3 comments
Labels
:Data Management/Data streams Data streams and their lifecycles >enhancement Team:Data Management Meta label for data/management team

Comments

@BenB196
Copy link

BenB196 commented Oct 6, 2023

Description

Enhancement

It would be nice that a data stream could exist in a strictly historical/archival form, similar to how an index can be made read-only, but at a data stream level (data stream mappings can't be updated, data stream can't be rolled over, even if "forced").

Use Case

Because data streams mainly target timeseries data, there are cases where a data stream is no longer actively needed, but needs to be kept around because it contains historical data. An issue that I've observed is that there is no way to mark a data stream as archived and that it shouldn't receive things like mapping updates or rollovers.

An example of where this is an issue is with Elastic Fleet. If I have a data stream with a specific namespace value, that I eventually stop using, Elasticsearch knows not to roll it over because the leading index has 0 docs, but Fleet doesn't seem to know this, and if the Fleet integration gets an update that changes mappings, Fleet will see this data stream and force a rollover, causing 0 doc indices to exist.

It would be nice to have the ability to mark a data stream in a way that things like Fleet (or other similar products), no-op changes on archived data streams.

@BenB196 BenB196 added >enhancement needs:triage Requires assignment of a team area label labels Oct 6, 2023
@BenB196 BenB196 changed the title Ability to Close/Lock/Read-Only a Data Stream Ability to Read-Only a Data Stream Oct 6, 2023
@stu-elastic stu-elastic added :Data Management/Data streams Data streams and their lifecycles and removed needs:triage Requires assignment of a team area label labels Oct 6, 2023
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-data-management (Team:Data Management)

@joegallo
Copy link
Contributor

Related to #89283

@dakrone
Copy link
Member

dakrone commented Mar 4, 2025

Fleet will see this data stream and force a rollover, causing 0 doc indices to exist.

Fleet should now be using lazy rollovers which should avoid the 0-document indices.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Data Management/Data streams Data streams and their lifecycles >enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Development

No branches or pull requests

5 participants