Request fragments #5596

dureuill · 2025-05-27T16:29:21Z

Usage page: https://www.notion.so/meilisearch/Search-in-images-usage-1c14b06b651f80c1bf9effe56dbeef54

API side

Add searchRequestFragments embedder setting
Add indexingRequestFragments embedder setting
Add multimodal experimental feature
Add media search parameter

Implementation side

github-actions · 2025-06-25T13:52:14Z

Hello, I'm a bot 🤖

You are receiving this message because you declared that this PR make changes to the Meilisearch database.
Depending on the nature of the change, additional actions might be required on your part. The following sections detail the additional actions depending on the nature of the change, please copy the relevant section in the description of your PR, and make sure to perform the required actions.

Thank you for contributing to Meilisearch ❤️

This PR makes forward-compatible changes

Forward-compatible changes are changes to the database such that databases created in an older version of Meilisearch are still valid in the new version of Meilisearch. They usually represent additive changes, like adding a new optional attribute or setting.

Detail the change to the DB format and why they are forward compatible
Forward-compatibility: A database created before this PR and using the features touched by this PR was able to be opened by a Meilisearch produced by the code of this PR.

This PR makes breaking changes

Breaking changes are changes to the database such that databases created in an older version of Meilisearch need changes to remain valid in the new version of Meilisearch. This typically happens when the way to store the data changed (change of database, new required key, etc). This can also happen due to breaking changes in the API of an experimental feature. ⚠️ This kind of changes are more difficult to achieve safely, so proceed with caution and test dumpless upgrade right before merging the PR.

Detail the changes to the DB format,
- which are compatible, and why
- which are not compatible, why, and how they will be fixed up in the upgrade
/!\ Ensure all the read operations still work!
- If the change happened in milli, you may need to check the version of the database before doing any read operation
- If the change happened in the index-scheduler, make sure the new code can immediately read the old database
- If the change happened in the meilisearch-auth database, reach out to the team; we don't know yet how to handle these changes
Write the code to go from the old database to the new one
- If the change happened in milli, the upgrade function should be written and called here
- If the change happened in the index-scheduler, we've never done it yet, but the right place to do it should be here
Write an integration test here ensuring you can read the old database, upgrade to the new database, and read the new database as expected

dureuill · 2025-06-30T08:39:19Z

crates/milli/src/vector/json_template/mod.rs

+        doc_alloc: &'doc Bump,
+    ) -> Result<Value, Error> {
+        let document = ParseableDocument::new(document, doc_alloc);
+        let v: Vec<u32> = vec![];


will create a fields field, but it will be empty!

consider removing the fields field or to populate it

dureuill · 2025-06-30T09:26:47Z

crates/milli/src/vector/extractor.rs

+
+impl<'doc> Extractor<'doc> for RequestFragmentExtractor<'doc> {
+    type DocumentMetadata = ();
+    type Input = Value;


consider renaming Input?

crates/milli/src/vector/extractor.rs

dureuill · 2025-06-30T09:37:15Z

crates/milli/src/vector/session.rs

+        self.embed_chunks(unused_vectors_distribution)
+    }
+
+    pub fn drain(mut self, unused_vectors_distribution: &C::ErrorMetadata) -> Result<C> {


consider adding a panicking, albeit it might be a trade-off we don't want to make 🤔

dureuill · 2025-06-30T09:39:44Z

crates/milli/src/vector/session.rs

+                    error,
+                    self.embedder_name,
+                    unused_vectors_distribution,
+                    &self.metadata,


consider passing metadata as owned here and clearing inputs in this case

crates/milli/src/vector/mod.rs

dureuill · 2025-06-30T09:55:08Z

crates/milli/src/vector/mod.rs

@@ -389,6 +506,9 @@ impl ArroyWrapper {
            let reader = reader?;
            let mut searcher = reader.nns(limit);
            if let Some(filter) = filter {
+                if reader.item_ids().is_disjoint(filter) {


@Kerollmops consider doing this at the arroy level

dureuill

Catch and return an error when searchFragments is empty

crates/milli/src/vector/settings.rs

crates/milli/src/update/new/indexer/write.rs

crates/milli/src/update/new/extract/vectors/mod.rs

Kerollmops marked this pull request as draft May 28, 2025 07:34

dureuill force-pushed the request-fragments branch from 9dfe57c to 1ed4d9d Compare June 17, 2025 09:08

dureuill requested review from Kerollmops and irevoire June 23, 2025 15:24

dureuill force-pushed the request-fragments branch 3 times, most recently from 759bac4 to 15a7679 Compare June 25, 2025 09:07

dureuill added this to the v1.16.0 milestone Jun 25, 2025

dureuill added the db change A database was modified label Jun 25, 2025

dureuill force-pushed the request-fragments branch from d034946 to 1fc6890 Compare June 29, 2025 22:15

dureuill added 4 commits June 30, 2025 09:28

prompt: Publishes some types

4b93e02

Expand json_template module

7be9351

Add vector::db module

0fba187

Index uses the vector::db stuff

31b26c9

dureuill force-pushed the request-fragments branch from 1fc6890 to f5c23f8 Compare June 30, 2025 07:54

dureuill commented Jun 30, 2025

View reviewed changes

crates/milli/src/vector/settings.rs Outdated Show resolved Hide resolved

dureuill commented Jun 30, 2025

View reviewed changes

crates/milli/src/update/new/indexer/write.rs Outdated Show resolved Hide resolved

dureuill commented Jun 30, 2025

View reviewed changes

crates/milli/src/update/new/extract/vectors/mod.rs Outdated Show resolved Hide resolved

dureuill commented Jun 30, 2025

View reviewed changes

crates/milli/src/update/new/extract/vectors/mod.rs Outdated Show resolved Hide resolved

dureuill added 6 commits July 1, 2025 10:08

Add vector::extractor module

077a49c

Add vector::session module

62dafbc

Add new vector errors

583db42

ArroyWrapper changes

da8f207

RuntimeEmbedder and RuntimeFragments

98bc815

EmbeddingConfigs -> RuntimeEmbedders

d303efb

dureuill added 4 commits July 1, 2025 10:08

Change embedder API

44c35e8

REST embedder supports fragments

8ebc1e8

Add new parameters to openai and rest embedders

ccbcb76

New error when too many fragments in settings

8b85674

dureuill force-pushed the request-fragments branch from bd0c135 to 1c07b17 Compare July 1, 2025 08:09

dureuill added 12 commits July 1, 2025 10:20

Vector settings to add indexingFragments and searchFragments

c1985a0

Support indexingFragments and searchFragments in settings

d47787a

Clear DB on clear documents

c415502

Implement in new document indexer

5991c05

Implement in old settings indexer and old dump import indexer

ca30ab1

New search errors

ee045ff

Add media to search

d2464b3

Use media to determine search kind

b3ab389

Analytics

cd5b124

Fix tests

9f72ec0

Fix snaps

ae32bf6

multimodal experimental feature

7db5495

dureuill force-pushed the request-fragments branch from 1c07b17 to 7db5495 Compare July 1, 2025 08:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Request fragments #5596

Request fragments #5596

dureuill commented May 27, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jun 25, 2025

Uh oh!

dureuill Jun 30, 2025

Uh oh!

dureuill Jun 30, 2025

Uh oh!

Uh oh!

dureuill Jun 30, 2025

Uh oh!

dureuill Jun 30, 2025

Uh oh!

Uh oh!

dureuill Jun 30, 2025

Uh oh!

dureuill left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Request fragments #5596

Are you sure you want to change the base?

Request fragments #5596

Conversation

dureuill commented May 27, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

API side

Implementation side

Uh oh!

github-actions bot commented Jun 25, 2025

Hello, I'm a bot 🤖

This PR makes forward-compatible changes

This PR makes breaking changes

Uh oh!

dureuill Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

dureuill Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dureuill Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

dureuill Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dureuill Jun 30, 2025

Choose a reason for hiding this comment

Uh oh!

dureuill left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

dureuill commented May 27, 2025 •

edited

Loading