- Release Signoff Checklist
- Summary
- Motivation
- Proposal
- Design Details
- Production Readiness Review Questionnaire
- Implementation History
- Drawbacks
- Alternatives
Items marked with (R) are required prior to targeting to a milestone / release.
- (R) Enhancement issue in release milestone, which links to KEP dir in kubernetes/enhancements (not the initial KEP PR)
- (R) KEP approvers have approved the KEP status as
implementable - (R) Design details are appropriately documented
- (R) Test plan is in place, giving consideration to SIG Architecture and SIG Testing input (including test refactors)
- e2e Tests for all Beta API Operations (endpoints)
- (R) Ensure GA e2e tests for meet requirements for Conformance Tests
- (R) Minimum Two Week Window for GA e2e tests to prove flake free
- (R) Graduation criteria is in place
- (R) all GA Endpoints must be hit by Conformance Tests
- (R) Production readiness review completed
- (R) Production readiness review approved
- "Implementation History" section is up-to-date for milestone
- User-facing documentation has been created in kubernetes/website, for publication to kubernetes.io
- Supporting documentation—e.g., additional design documents, links to mailing list discussions/SIG meetings, relevant PRs/issues, release notes
This KEP defines Enum Types support for OpenAPI. Currently, types in our API have fields
that are actually enums but represented as plain string. This KEP proposes a +enum
marker to type aliases that represent enums. The OpenAPI generator will have
the capability to recognize the marker and auto-detect possible values for an enum.
Currently, types that are actually enums are represented as plain strings. For example,
here is the definition of Protocol, which must be one of TCP, UDP, and SCTP.
package core
// ...
// Protocol defines network protocols supported for things like container ports.
type Protocol string
const (
// ProtocolTCP is the TCP protocol.
ProtocolTCP Protocol = "TCP"
// ProtocolUDP is the UDP protocol.
ProtocolUDP Protocol = "UDP"
// ProtocolSCTP is the SCTP protocol.
ProtocolSCTP Protocol = "SCTP"
)
// ContainerPort represents a network port in a single container
type ContainerPort struct {
// ...
// Required: Supports "TCP", "UDP" and "SCTP"
// +optional
Protocol Protocol
// ...
}In order to convey that the Protocol type is an enum, every field of the type
have to comment its values like ContainerPort does. The same goes for validation -
protocol being one of "TCP", "UDP", and "SCTP" is duplicated among the code base.
Tools and UIs can also benefit from the additional piece of information. For example, Protocol can be shown as a dropdown of possible values instead of a textbox. The user no longer needs to see an error of invalid value after submits the request.
- Add a marker that indicates a field as an enum.
- Make the Kubernetes OpenAPI generator recognize the marker.
- Deduce values of an enum type from its Go type definition.
- Detect and annotate enum types in all built-in types.
- Unify the marker with that of kube-builder's enum mechanism.
This KEP does not provide support for validation that is more complicated than an enum type like regex or prefix. Instead, use other validation mechanism that is or will be supported.
This KEP treat any enum type as a closed enum. Any change to the set of possible values should be treated as creating a new type and thus should follow the standard procedure of changing an API.
When generating OpenAPI schema, the de facto enum types should be reflected as enums in the resulting schema.
We define enum types with following properties:
- All enum types are closed, i.e., the type includes a finite number of possible values. However, for compatibility, a client may allow unknown values from responses.
- All enum types should be string. There can be no enum integers, floats, etc
Additionally, This KEP makes the following assumptions on the implementation:
- The canonical definition of a type is written in go and follows the standard Kubernetes convention.
- All enum types have their own type alias. i.e. , no raw string enums.
The Enum Pattern of a type, appearing in a go source file, is defined as follows:
package internal
// FooType defines a type alias, underlying type must be string
type FooType string
// in the same package, define constants with the alias
const (
BarConst FooType = "Bar"
BazConst FooType = "Baz"
)
// in one or more resources or nested objects, one of the fields is of
// the defined alias type.
// ExampleResource should refer FooType
type ExampleResource struct {
FooField FooType
}Enum Type is the type that follows such pattern (FooType in the example above).
Enum Values are the possible values of the Enum Type (Bar and Baz are Enum Values for FooType).
The Enum Tag is the marker // +enum that must be in the closest comment blocks
of an enum type following Enum Pattern.
package internal
// FooType define sa type alias, underlying type must be string
// +enum
type FooType stringAn actual example looks like
package core
// Protocol defines network protocols supported for things like container ports.
// +enum
type Protocol stringNot all types following the Enum Pattern are actually enum types. Here is a counter-example.
package core
// StorageMedium defines ways that storage can be allocated to a volume.
type StorageMedium string
// These are the valid value for StorageMedium
const (
StorageMediumDefault StorageMedium = "" // use whatever the default is for the node
StorageMediumMemory StorageMedium = "Memory" // use memory (tmpfs)
StorageMediumHugePages StorageMedium = "HugePages" // use hugepages
StorageMediumHugePagesPrefix StorageMedium = "HugePages-" // prefix for full medium notation HugePages-<size>
)Here, StorageMedium can have infinite number of possible values, which disqualify it as an enum type.
The parser of Go compiler and, as a result, gengo, does not support type aliases. See kubernetes/gengo#224 for details.
As of Kubernetes 1.23, we do not have any type aliases in kubernetes/kubernetes repo. Any attempt to add aliases would break the code generation.
Example: kubernetes/kubernetes#106300
However, as of 1.23, the enum marker is the only marker to be added to a type declaration, and would be the first marker to be affected.
Until there is a fix to gengo, the enum generator has the following limitations:
- the enum marker must not be added to aliases
- an aliased enum type or value SHOULD NOT have comments. Otherwise, the comments will be squashed with these of the original with undefined ordering.
If the fix to gengo does not arrive in this release cycle, detection and enforcement the rules above will be added to the verify-*.sh scripts.
The syntax proposed in this KEP is different from existing implementation of kube-builder. Namely, kube-builder requires possible values in the marker itself, and is treated as a validation.
package example
// Alias
// +kubebuilder:validation:Enum=Lion;Wolf;Dragon
type Alias stringThe two approaches are compatible, but eventually they will have to be unified. To retain compatibility with existing users of kube-builder, the auto-deduction will be added to kube-builder once the work is done with built-in types.
This feature, when enabled, breaks existing code using swagger to generate client or server stub from the spec for languages with built-in enum support, including Java and C#. Take the following spec as an example.
definitions:
Foo:
type: object
properties:
bar:
type: string
enum:
- A
- B
- CThe generated Java object looks like
public class Foo {
@JsonAdapter(BarEnum.Adapter.class)
public enum BarEnum {
A("A"),
B("B"),
C("C");
// ...
}
private BarEnum bar = null;
}If the enum field was not in the definition, the Bar field would be a String.
Maintainers of the clients should include the change in a new major release.
If, rather than update code generator or use of the generated client, a client would prefer enum fields to be strings as they were previously, a simple preprocessor
will be provided to prune the enum information similar to that of api-server (see below).
If the feature is disabled through the feature gate, api-server will remove all aspects about
enum types from the schema, making the result identical to that of the prior format.
After this feature graduates to GA, the pruning code will be removed from api-server.
Testing should focus on kubernetes/kube-openapi repository. There will be
integration tests against cases with valid and invalid definition of enum types.
In the kubernetes/kubernetes repository, as part of OpenAPI v3 test suites,
there should be integration and e2e tests that validate present of enum fields.
- Feature implemented behind the feature gate
- Initial tests, both in k/k and k/kube-openapi, are completed and enabled
- All enums in Kubernetes built-in types have enum tags properly added.
- Enum syntax of kube-builder updated to match that of this KEP.
- OpenAPI-level validation of enum types in built-in types removed in favor of that provided by the schema.
Enable/disable the OpenAPIEnum feature gate.
The API is still compatible even with enum types enabled. For the conversion from v2 to v3 (see OpenAPI v3 KEP), enum types will be carried over and thus should not change the behavior of v3.
- Feature gate (also fill in values in
kep.yaml)- Feature gate name:
OpenAPIEnum - Components depending on the feature gate:
kube-apiserver
- Feature gate name:
Yes. Any clients generated from the new schema will see enum fields. For languages with enum support, OpenAPI language-level generator (swagger) will create Language-specific enum types instead of plain strings.
Yes. By disabling the feature gate and restart api-server.
This feature needs access only to information immutable at run-time. There should be no problem re-enabling it.
N/A
If the API-server has multiple replicas, with some instances not yet enabling this feature, whether the returned OpenAPI Spec contains enum types will depend on which instance handle the request which could result in inconsistent responses for the same request.
Standard API Server metrics apply to this feature. For example, when
apiserver_request_duration_seconds is too high for /openapi endpoints.
Enabling, disabling, and re-enabling the feature gate yields expected outcomes.
Is the rollout accompanied by any deprecations and/or removals of features, APIs, fields of API types, flags, etc.?
No.
N/A. Workloads that query the OpenAPI specs automatically use this feature if enabled.
- Other (treat as last resort)
- Details: the returned OpenAPI Spec contains enum types.
This feature does not affect the SLO of API Server or any other components.
What are the SLIs (Service Level Indicators) an operator can use to determine the health of the service?
- Metrics
- Metric name: apiserver_request_duration_seconds
- Components exposing the metric: kube-api-server
Are there any missing metrics that would be useful to have to improve observability of this feature?
kube-apiserver- Usage description:
- Impact of its outage on the feature: The
/openapiendpoint is unavailable - Impact of its degraded performance or high-error rates on the feature: The
/openapiendpoint is degraded.
- Impact of its outage on the feature: The
- Usage description:
No. This feature only adds enum fields to enum types.
No.
No.
No. Enum information only appears in /openapi/ endpoints but not standard APIs.
Will enabling / using this feature result in increasing time taken by any operations covered by existing SLIs/SLOs?
No.
Will enabling / using this feature result in non-negligible increase of resource usage (CPU, RAM, disk, IO, ...) in any components?
No. The addition of enum fields take negligible amount of memory.
This feature is part of API server. The feature is unavailable if API server is unavailable.
N/A. This feature fails if and only if the API Server becomes unavailable.
N/A. This feature fails if and only if the API Server becomes unavailable.
- 09-09-2021
SummaryandMotivationsections being merged, signaling SIG acceptance - 09-09-2021
Proposalsection being merged, signaling agreement on a proposed design - 09-14-2021 Enum type generator merged as
kubernetes/kube-openapi#242 - 11-16-2021 Enum type support for OpenAPI v2 merged as #105057
This KEP will break any users that expect enum types to be plain strings. This is the main reason for not implementing this feature in OpenAPI v2 (see Alternative below).
This change can go along with OpenAPI v3 instead of v2, which prevent the breakage mentioned above. However, an affected client can release a new major version to introduce the breaking change itself instead of waiting for new major version of OpenAPI. In this way, we can "make things right" and benefit from the improvement much earlier.