Support object auto-flattening in the presence of nested and configured objects

[salvatore-campagna]

Description

In LogsDB it's desirable to avoid mapping errors through object auto-flattening, similar to subobjects: false, as described in #106812.

However, if sub-objects are disabled by setting subobjects: false it means it is not possible to correctly handled a sub-object field of type nested. This issue arises also because NestedObjectMapper extends ObjectMapper. More so, objects with params like enabled: false are not allowed in the mapping.

A use case where the parent is an object and one or more of its sub-objects are a nested field is quite common also considering that:

• subobjects: true has always been the default value for fields of type object.
• a nested field as a sub-object field is typically used to model a one-to-many relationship between the parent field and the sub-object (imagine an exception as the parent object and the stack trace log lines as the sub-object, or an alert firing and the list of rules as sub-obejcts).

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[salvatore-campagna]

We need to explore the idea of determining the default value to use for subobjects dynamically by setting it to false based on:

• whether nested field types being used
• object enabled property being used
• object dynamic property being used
  We need to to this to guarantee backward compatibility and to make sure we do not break integrations expecting subobjects: true.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[kkrik-es]

We followed a different approach, introducing a new mode subobjects: auto. This mode applies auto-flattening, similar to subobjects: false, but makes use of any pre-defined objects that can't be flattened, like nested objects and objects with incompatible params.

[flash1293]

@kkrik-es What's the current status of this? It seems like the original PR got reverted due to performance issues and now there is a new PR implementing this which hasn't been merged yet, is that right?

[kkrik-es]

Addressing this is not trivial, so further effort will have to be prioritized.