fix(llm-server): llm server timesout issue #173
Conversation
WalkthroughThe pull request updates the chat functionality in the LLMProvider by adding a new Changes
Sequence Diagram(s)sequenceDiagram
participant App
participant LLMProvider
participant Model
App->>LLMProvider: chat(input, timeoutMs)
note right of LLMProvider: Start timeoutPromise & model.chat call concurrently
LLMProvider->>Model: chat(input.messages)
alt Response received before timeout
Model-->>LLMProvider: response
LLMProvider-->>App: response
else Timeout reached
LLMProvider-->>App: timeout error (logged with duration)
end
Possibly related PRs
Suggested reviewers
Poem
Warning There were issues while running some tools. Please review the errors and either fix the tool’s configuration or disable the tool if it’s a critical failure. 🔧 ESLint
llm-server/src/main.tsOops! Something went wrong! :( ESLint: 8.57.1 ESLint couldn't find the plugin "eslint-plugin-prettier". (The package "eslint-plugin-prettier" was not found when loaded as a Node module from the directory "/llm-server".) It's likely that the plugin isn't installed correctly. Try reinstalling by running the following: The plugin "eslint-plugin-prettier" was referenced from the config file in "llm-server/.eslintrc.cjs". If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team. llm-server/src/llm-provider.tsOops! Something went wrong! :( ESLint: 8.57.1 ESLint couldn't find the plugin "eslint-plugin-prettier". (The package "eslint-plugin-prettier" was not found when loaded as a Node module from the directory "/llm-server".) It's likely that the plugin isn't installed correctly. Try reinstalling by running the following: The plugin "eslint-plugin-prettier" was referenced from the config file in "llm-server/.eslintrc.cjs". If you still can't figure out the problem, please stop by https://eslint.org/chat/help to chat with the team. ✨ Finishing Touches
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (3)
llm-server/src/main.ts (1)
140-141: Fix typo in the TODO comment and consider a configurable timeoutThe comment has some grammar and spelling issues. Also, hardcoding the timeout value (200000ms = 200s) might not be flexible enough for different use cases.
-// TODO make it to dynamic Now is 200 second by defult. +// TODO: Make timeout value dynamic. Currently set to 200 seconds by default. -const response = await this.llmProvider.chat(input, 200000); +const defaultTimeoutMs = parseInt(process.env.CHAT_TIMEOUT_MS || '200000', 10); +const response = await this.llmProvider.chat(input, defaultTimeoutMs);Consider making the timeout configurable via environment variables to allow better flexibility without code changes.
llm-server/src/llm-provider.ts (2)
107-126: Improve timeout handling with more context and better typingThe implementation of the timeout mechanism is good, but the error message could be more descriptive and the type handling could be improved.
async chat(input: MessageInput, timeoutMs: number): Promise<string> { try { const model = this.getModelInstance(input.model); // Set a timeout dynamically based on the provided value const timeoutPromise = new Promise<string>((_, reject) => setTimeout( - () => reject(new Error('Chat request timed out')), + () => reject(new Error(`Chat request for model ${input.model} timed out after ${timeoutMs}ms`)), timeoutMs, ), ); // Race between the actual model call and the timeout const completion = await Promise.race([ model.chat(input.messages), timeoutPromise, ]); - return (completion as any).choices[0].message.content || ''; + // Safely handle response structure + if (completion && + typeof completion === 'object' && + 'choices' in completion && + Array.isArray(completion.choices) && + completion.choices.length > 0 && + completion.choices[0].message?.content) { + return completion.choices[0].message.content; + } + return ''; } catch (error) {
127-127: Enhance error logging to distinguish timeout errorsThe current error logging doesn't distinguish between timeout errors and other types of errors.
- this.logger.error(`Error in chat (Timeout: ${timeoutMs}ms):`, error); + if (error instanceof Error && error.message.includes('timed out')) { + this.logger.error(`Timeout occurred in chat (Timeout: ${timeoutMs}ms):`, error); + } else { + this.logger.error(`Error in chat (Timeout: ${timeoutMs}ms):`, error); + }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (3)
llm-server/src/llm-provider.ts(1 hunks)llm-server/src/main.ts(1 hunks)llm-server/src/model/remote-model-instance.ts(0 hunks)
💤 Files with no reviewable changes (1)
- llm-server/src/model/remote-model-instance.ts
Sma1lboy
changed the title
Summary by CodeRabbit