How to show Metadata/Sessions/Tags when using Litellm version sdk

[spiritanand]

Question

I have been trying,

1. to add metadata and session and user id so I can view them in the litellm dashboard and
2. also track usage based off the metadata

So far I have been unable to do either since there has been no easy way for me to access the usage or set any metadata to the trace.

[spiritanand]

For a slightly different context and what I have tried so far - BerriAI/litellm#12802

[spiritanand]

from agents import (
    Agent,
    FunctionTool,
    RunContextWrapper,
    Runner,
    set_default_openai_client,
    Usage,
    RunConfig,
    ModelSettings,
)
from openai import AsyncOpenAI
from openai.types.responses.response_usage import InputTokensDetails, OutputTokensDetails

# start of the try catch block
custom_client = AsyncOpenAI(base_url=LITELLM_URL, api_key=LITELLM_KEY)
            set_default_openai_client(custom_client)

            result = await Runner.run(
                agent, self.input_list, max_turns=max_turns, context=context
            )


# in the finally block after the agent result has been streamed back, we use the result from the agent

            # Extract usage information from the final result
            if hasattr(result, 'usage') and result.usage:
                response_usage = result.usage
                usage = (
                    Usage(
                        requests=1,
                        input_tokens=response_usage.prompt_tokens,
                        output_tokens=response_usage.completion_tokens,
                        total_tokens=response_usage.total_tokens,
                        input_tokens_details=InputTokensDetails(
                            cached_tokens=getattr(
                                response_usage.prompt_tokens_details, "cached_tokens", 0
                            )
                            or 0
                        ),
                        output_tokens_details=OutputTokensDetails(
                            reasoning_tokens=getattr(
                                response_usage.completion_tokens_details, "reasoning_tokens", 0
                            )
                            or 0
                        ),
                    )
                    if response_usage
                    else Usage()
                )
                logger.info(f"Agent stream usage statistics - {usage}")
            else:
                usage = Usage()
                logger.warning("Agent stream usage statistics - No usage information returned from Litellm")

this results in the else case being triggered i.e. no usage info

similarly,

        result = Runner.run_streamed(
            agent, self.input_list, max_turns=max_turns, context=context, run_config=RunConfig(model_settings=ModelSettings(metadata={"thread_id": "myThreadId"}))
        )

adding metadata does not appear on the litellm dashboard. am I using this incorrectly?

[spiritanand]

@seratch the problem could be with the litellm python sdk itself, because even passing values (like meteadata) directly to it (by editing it in the .venv - /.venv/lib/python3.12/site-packages/agents/extensions/models/litellm_model.py) resulted in no changes

Example
Line 304 - /agents/extensions/models/litellm_model.py

        ret = await litellm.acompletion(
            model=self.model,
            messages=converted_messages,
            tools=converted_tools or None,
            temperature=model_settings.temperature,
            top_p=model_settings.top_p,
            frequency_penalty=model_settings.frequency_penalty,
            presence_penalty=model_settings.presence_penalty,
            max_tokens=model_settings.max_tokens,
            tool_choice=self._remove_not_given(tool_choice),
            response_format=self._remove_not_given(response_format),
            parallel_tool_calls=parallel_tool_calls,
            stream=stream,
            stream_options=stream_options,
            reasoning_effort=reasoning_effort,
            extra_headers={**HEADERS, **(model_settings.extra_headers or {})},
            api_key=self.api_key,
            base_url=self.base_url,
            # **extra_kwargs,
            metadata={"litellm_session_id": "1234567890"}
        )

there is no metadata reflected on litellm dashboard

what do you think?

[seratch]

We don't have the bandwidth to check the internals of LiteLLM in depth, but as you saw in the code, this SDK does not do anything to prevent you from passing the parameter (you can use model_settings.extra_args too). This is just my guess but there may be something that could be improved on the litellm.acompletion method side. I found litellm.completion internals explicitly passes the additional metadata params to the model call but I couldn't quickly find the same part on the acompletion method side. This might be my misunderstanding but you may want to do some tests with both completion and acompletion directly.

[github-actions]

This issue is stale because it has been open for 7 days with no activity.

[github-actions]

This issue was closed because it has been inactive for 3 days since being marked as stale.