Fixes for OpenAI / Azure OpenAI compatibility with LiteLLM router #1760
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Signed-off-by: dbczumar <[email protected]>
Signed-off-by: dbczumar <[email protected]>
Signed-off-by: dbczumar <[email protected]>
dbczumar
commented
Nov 5, 2024
Comment on lines
+268
to
+275
| if api_config.api_key is not None: | ||
| litellm_params["api_key"] = api_config.api_key | ||
| if api_config.api_base is not None: | ||
| litellm_params["api_base"] = api_config.api_base | ||
| if api_config.api_version is not None: | ||
| litellm_params["api_version"] = api_config.api_version | ||
| if api_config.azure_ad_token is not None: | ||
| litellm_params["azure_ad_token"] = api_config.azure_ad_token |
There was a problem hiding this comment.
@okhat For OpenAI and Azure OpenAI, API keys must be defined in the Router's model_list directly, due to initialization logic in https://github.com/BerriAI/litellm/blob/0fe8cde7c78d6b975b936f7802a4124b58bd253a/litellm/router.py#L3999-L4001
dbczumar
commented
Nov 5, 2024
Comment on lines
-245
to
-247
| # Use the API key and base from the kwargs, or from the environment. | ||
| api_key = kwargs.pop("api_key", None) or os.getenv(f"{provider}_API_KEY") | ||
| api_base = kwargs.pop("api_base", None) or os.getenv(f"{provider}_API_BASE") |
There was a problem hiding this comment.
These are now part of the router definition, so they don't need to be passed to the text_completion() API
Signed-off-by: dbczumar <[email protected]>
dbczumar
commented
Nov 5, 2024
Comment on lines
+187
to
+188
| Note: The API configurations are removed from the specified `llm_kwargs`, if present, mutating | ||
| the input dictionary. |
There was a problem hiding this comment.
This avoids duplicating code in litellm_text_completion, which pops the api key and base from the user-specified kwargs
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Fixes for OpenAI / Azure OpenAI compatibility with LiteLLM router