Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[ML] Return empty usage statistics in the event of an failure #124799

Merged

Conversation

jonathan-buttner
Copy link
Contributor

@jonathan-buttner jonathan-buttner commented Mar 13, 2025

This PR refactors the inference xpack usage to return an empty response on a failure instead of a the failure. Returning a failure causes the upstream _xpack/usage to fail as well.

Ideally our get all models action would still return some models even if parsing one of them fails. We don't have that logic in place at the moment though.

Example failure:

[2025-03-13T14:48:47,408][WARN ][o.e.x.i.a.TransportInferenceUsageAction] [runTask-0] Retrieving inference usage failed with error: failed java.lang.IllegalArgumentException: failed
        at org.elasticsearch.inference@9.1.0-SNAPSHOT/org.elasticsearch.xpack.inference.action.TransportInferenceUsageAction.lambda$localClusterStateOperation$1(TransportInferenceUsageAction.java:74)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListener$2.onResponse(ActionListener.java:258)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.support.ContextPreservingActionListener.onResponse(ContextPreservingActionListener.java:33)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListener$3.onResponse(ActionListener.java:413)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:203)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.tasks.TaskManager$1.onResponse(TaskManager.java:197)
        at org.elasticsearch.server@9.1.0-SNAPSHOT/org.elasticsearch.action.ActionListenerImplementations$RunBeforeActionListener.onResponse(ActionListenerImplementations.java:350)

@jonathan-buttner jonathan-buttner added >non-issue :ml Machine learning Team:ML Meta label for the ML team auto-backport Automatically create backport pull requests when merged v8.19.0 v9.1.0 labels Mar 13, 2025
@jonathan-buttner
Copy link
Contributor Author

@elasticmachine test this please

@jonathan-buttner jonathan-buttner marked this pull request as ready for review March 14, 2025 21:13
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

@jonathan-buttner jonathan-buttner merged commit fbf4580 into elastic:main Mar 19, 2025
16 of 17 checks passed
@jonathan-buttner jonathan-buttner deleted the ml-refactor-inference-usage branch March 19, 2025 14:40
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.x

jonathan-buttner added a commit to jonathan-buttner/elasticsearch that referenced this pull request Mar 19, 2025
elasticsearchmachine pushed a commit that referenced this pull request Mar 19, 2025
…124799) (#125227)

* [ML] Return empty usage statistics in the event of an failure (#124799)

* Returning empty usage on failure

* Adding logging

* Fixing tests
smalyshev pushed a commit to smalyshev/elasticsearch that referenced this pull request Mar 21, 2025
omricohenn pushed a commit to omricohenn/elasticsearch that referenced this pull request Mar 28, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged :ml Machine learning >non-issue Team:ML Meta label for the ML team v8.19.0 v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants