Skip to content

Fix handling of OpenAI-compatible Gemini req/res #5712

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 1 commit into
base: main
Choose a base branch
from
Draft

Fix handling of OpenAI-compatible Gemini req/res #5712

wants to merge 1 commit into from

Conversation

diksipav
Copy link

@diksipav diksipav commented Jun 25, 2025 •
edited
Loading

NOTE: This is Draft PR because I'm not sure you want to accept this approach, also the async-openai fork should be minimally updated in order for this to work. It would be great if someone from the Meilisearch can test this PR together with the fork update.

Fixes #5684

I used create_stream_byot when Gemini is used. The first difference is that index is not returned inside tool_calls elements. Gemini returns index: None.

This is OpenAI response:

{
  "id": "chatcmpl-BmFNfh09effTOgFZYLMgnXUS6Jk9g",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": null,
        "function_call": null,
        "tool_calls": [
          {
            "index": 0,
            "id": "call_oOWWo0ObgBAsp2bblvOQVnDu",
            "type": "function",
            "function": {
              "name": "_meiliSearchInIndex",
              "arguments": "" // also comes in chunks
            }
          }
        ],
        "role": "assistant",
        "refusal": null
      },
      "finish_reason": null,
      "logprobs": null
    }
  ],
  "created": 1750838567,
  "model": "gpt-3.5-turbo-0125",
  "service_tier": "default",
  "system_fingerprint": null,
  "object": "chat.completion.chunk",
  "usage": null
}

And this is Gemini response:

{
  "id": "SKpbaNfgG_7rkdUPhs3joQg",
  "choices": [
    {
      "index": 0,
      "delta": {
        "content": null,
        "function_call": null,
        "tool_calls": [
          {
            "index": null,
            "id": "",
            "type": "function",
            "function": {
              "name": "_meiliSearchInIndex",
              "arguments": "{\"q\":\"search engine\",\"index_uid\":\"movies\"}"
            }
          }
        ],
        "role": "assistant",
        "refusal": null
      },
      "finish_reason": "tool_calls",
      "logprobs": null
    }
  ],
  "created": 1750837832,
  "model": "gemini-2.5-flash",
  "service_tier": null,
  "system_fingerprint": null,
  "object": "chat.completion.chunk",
  "usage": null
}

So I updated the async_openai type in the Meilisearch fork (I only have this locally, didn't create a PR for that since I don't know if you wanna take this approach):

async_openai::types::chat
pub struct ChatCompletionMessageToolCallChunk {
    pub index: Option<u32>,
    pub id: Option<String>,
    pub r#type: Option<ChatCompletionToolType>,
    pub function: Option<FunctionCallStream>,
}

I only tested with one tool call, not sure how will all this behave in case of multiple tool calls.

Another difference is that Gemini sends in the same chunk tool_calls: Some(..) and finish_reason: Some(tool_calls) while OpenAI after collecting tool call chunks it sends a final chunk with tool_calls: None and finish_reason: Some(tool_calls).

So I updated the logic to accumulate tool calls as they arrive and as soon there is finish_reason: Some(tool_calls), to process accumulated tool calls immediately, regardless of the value of tool_calls.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Chat completions with gemini
1 participant
@diksipav