Introduce a new route to export indexes #5670

Kerollmops · 2025-06-12T15:56:53Z

This PR introduces a new route to export/transfer indexes and documents to another Meilisearch instance. This PR is linked to this one on the Mini-Dashboard meilisearch/mini-dashboard#622.

Fixes #5713

To Do

This PR makes forward-compatible changes

Forward-compatible changes are changes to the database such that databases created in an older version of Meilisearch are still valid in the new version of Meilisearch. They usually represent additive changes, like adding a new optional attribute or setting.

Detail the change to the DB format and why they are forward compatible.
We introduce a new Export task in the task queue. There are no settings, experimental flags, or anything but a new task type.
Forward compatibility: A Meilisearch produced by this PR's code could open a database created before this PR and using the features touched by this PR.
I opened a database generated with Meilisearch from the main branch and opened it with a database generated with Meilisearch from this branch. Once opened, I called the export route to export documents to another Meilisearch.

github-actions · 2025-06-12T15:57:05Z

Hello, I'm a bot 🤖

You are receiving this message because you declared that this PR make changes to the Meilisearch database.
Depending on the nature of the change, additional actions might be required on your part. The following sections detail the additional actions depending on the nature of the change, please copy the relevant section in the description of your PR, and make sure to perform the required actions.

Thank you for contributing to Meilisearch ❤️

This PR makes forward-compatible changes

Forward-compatible changes are changes to the database such that databases created in an older version of Meilisearch are still valid in the new version of Meilisearch. They usually represent additive changes, like adding a new optional attribute or setting.

Detail the change to the DB format and why they are forward compatible
Forward-compatibility: A database created before this PR and using the features touched by this PR was able to be opened by a Meilisearch produced by the code of this PR.

This PR makes breaking changes

Breaking changes are changes to the database such that databases created in an older version of Meilisearch need changes to remain valid in the new version of Meilisearch. This typically happens when the way to store the data changed (change of database, new required key, etc). This can also happen due to breaking changes in the API of an experimental feature. ⚠️ This kind of changes are more difficult to achieve safely, so proceed with caution and test dumpless upgrade right before merging the PR.

Detail the changes to the DB format,
- which are compatible, and why
- which are not compatible, why, and how they will be fixed up in the upgrade
/!\ Ensure all the read operations still work!
- If the change happened in milli, you may need to check the version of the database before doing any read operation
- If the change happened in the index-scheduler, make sure the new code can immediately read the old database
- If the change happened in the meilisearch-auth database, reach out to the team; we don't know yet how to handle these changes
Write the code to go from the old database to the new one
- If the change happened in milli, the upgrade function should be written and called here
- If the change happened in the index-scheduler, we've never done it yet, but the right place to do it should be here
Write an integration test here ensuring you can read the old database, upgrade to the new database, and read the new database as expected

crates/index-scheduler/src/utils.rs

crates/index-scheduler/src/scheduler/create_batch.rs

crates/meilisearch-types/src/task_view.rs

crates/meilisearch-types/src/tasks.rs

crates/meilisearch/src/routes/export.rs

crates/index-scheduler/src/processing.rs

crates/index-scheduler/src/scheduler/process_batch.rs

crates/index-scheduler/src/scheduler/process_export.rs

crates/index-scheduler/Cargo.toml

Mubelotix · 2025-07-01T09:17:54Z

crates/index-scheduler/src/processing.rs

+        ExporingTheSettings,
+        ExporingTheDocuments,


Typo

Suggested change

ExporingTheSettings,

ExporingTheDocuments,

ExportingTheSettings,

ExportingTheDocuments,

Mubelotix · 2025-07-01T09:18:47Z

crates/index-scheduler/src/scheduler/create_batch.rs

+        // 3. we batch the export.
+        let to_export = self.queue.tasks.get_kind(rtxn, Kind::Export)? & enqueued;
+        if !to_export.is_empty() {
+            let task_id = to_export.iter().next().expect("There must be only one export task");


Suggested change

let task_id = to_export.iter().next().expect("There must be only one export task");

let task_id = to_export.iter().next().expect("There must be at least one export task");

Mubelotix · 2025-07-01T09:21:25Z

crates/meilisearch/src/routes/export.rs

+    tag = "Export",
+    security(("Bearer" = ["export", "*"])),
+    responses(
+        (status = OK, description = "Known nodes are returned", body = Export, content_type = "application/json", example = json!(


Suggested change

(status = OK, description = "Known nodes are returned", body = Export, content_type = "application/json", example = json!(

(status = 202, description = "Export successfully enqueued", body = SummarizedTaskView, content_type = "application/json", example = json!(

Mubelotix · 2025-07-01T09:21:56Z

crates/meilisearch/src/routes/export.rs

+    // TODO make it experimental?
+    // index_scheduler.features().check_network("Using the /network route")?;
+


Suggested change

// TODO make it experimental?

// index_scheduler.features().check_network("Using the /network route")?;

Kerollmops · 2025-07-01T09:45:42Z

crates/index-scheduler/src/scheduler/process_export.rs

+                request.send_string("").map_err(into_backoff_error)
            })?;
+            let already_existed = response.status() == 200;


Suggested change

request.send_string("").map_err(into_backoff_error)

})?;

let already_existed = response.status() == 200;

request.send_bytes(Default::default()).map_err(into_backoff_error)

})?;

let index_exists = response.status() == 200;

Kerollmops · 2025-07-01T09:51:32Z

crates/meilisearch-types/src/tasks.rs

-        indexes: BTreeMap<IndexUidPattern, DetailsExportIndexSettings>,
+        indexes: BTreeMap<String, DetailsExportIndexSettings>,


Nope, can we keep the index UID pattern, here, please?

Kerollmops · 2025-07-01T09:51:41Z

crates/meilisearch-types/src/tasks.rs

-                    indexes: indexes.iter().map(|(p, s)| (p.clone(), s.clone().into())).collect(),
+                    indexes: BTreeMap::new(),


Kerollmops · 2025-07-01T09:51:47Z

crates/meilisearch-types/src/tasks.rs

-                    indexes: indexes.iter().map(|(p, s)| (p.clone(), s.clone().into())).collect(),
+                    indexes: BTreeMap::new(),


Kerollmops · 2025-07-01T09:51:54Z

crates/meilisearch-types/src/tasks.rs

-                    indexes: indexes.iter().map(|(p, s)| (p.clone(), s.clone().into())).collect(),
+                    indexes: BTreeMap::new(),


And keep this.

Kerollmops · 2025-07-01T09:52:28Z

crates/index-scheduler/src/scheduler/process_export.rs

@@ -127,7 +157,7 @@ impl IndexScheduler {
            progress.update_progress(progress_step);

            output.insert(
-                (*pattern).clone(),
+                uid.clone(),


We just need to convert the uid into an IndexUidPattern, here.

Kerollmops · 2025-07-01T09:53:03Z

crates/index-scheduler/src/scheduler/process_export.rs

@@ -30,7 +30,7 @@ impl IndexScheduler {
        payload_size: Option<&Byte>,
        indexes: &BTreeMap<IndexUidPattern, ExportIndexSettings>,
        progress: Progress,
-    ) -> Result<BTreeMap<IndexUidPattern, DetailsExportIndexSettings>> {
+    ) -> Result<BTreeMap<String, DetailsExportIndexSettings>> {


Keep this IndexUidPattern, please.

Kerollmops added this to the v1.16.0 milestone Jun 12, 2025

Kerollmops added the db change A database was modified label Jun 12, 2025

Kerollmops changed the title ~~Introduce a new route to export documents~~ Introduce a new route to export indexes Jun 12, 2025

Kerollmops force-pushed the export-and-transfer-route branch 2 times, most recently from a7685fe to 4ce1621 Compare June 16, 2025 12:46

Kerollmops added 12 commits June 25, 2025 15:26

Introduce a new route to export documents and enqueue the export task

e74c3b6

Working first implementation

e023ee4

Export embeddings

e8795d2

Implement a retry strategy

acb7c0a

Make clippy happy

7c448bc

Support task cancelation

3e2f468

Make clippy happy again

bc08cd0

Support no pattern when exporting

3329248

Support JSON value as filters

ee812b3

Make tests happy

2d4f7c6

Parallelize document upload

c621651

Gzip-compress the content

a743da3

Kerollmops force-pushed the export-and-transfer-route branch from 3335525 to a743da3 Compare June 25, 2025 13:27

Add the payload size to the parameters

6303121

Kerollmops force-pushed the export-and-transfer-route branch from d21c1fc to 6303121 Compare June 26, 2025 11:57

Kerollmops added 6 commits June 26, 2025 15:45

Introduce new analytics to the export route

e6e9a03

Remove the skip embeddings boolean in the settings

0bb7866

Better compute aggragates

bf13268

Improve OpenAPI schema

e3003c1

Fix clippy and more utoipa issues

b956918

Update tasks tests

0f1dd36

Kerollmops force-pushed the export-and-transfer-route branch from 1bb7abb to 0f1dd36 Compare June 26, 2025 16:11

Kerollmops added 2 commits June 26, 2025 18:25

Fix some api key errors

7fa1c41

Fix more tests

657bbf5

Better handle task abortion

7219299

Kerollmops force-pushed the export-and-transfer-route branch from 445b8e1 to 7219299 Compare June 27, 2025 10:33

Kerollmops marked this pull request as ready for review June 27, 2025 13:08

Kerollmops marked this pull request as draft June 27, 2025 14:36

Kerollmops requested a review from Mubelotix June 30, 2025 10:11

Kerollmops marked this pull request as ready for review June 30, 2025 11:35

Mubelotix reviewed Jun 30, 2025

View reviewed changes

Kerollmops added 3 commits June 30, 2025 18:31

Fix most of the easy issues

8503735

Display an accurate number of uploaded documents

ad03c86

Better behavior when null indexes

f4bb6cb

Kerollmops requested a review from Mubelotix June 30, 2025 16:59

Mubelotix added 3 commits July 1, 2025 11:02

Add the overrideSettings parameter

efd5fd9

Add override setttings to analytics

9cfbef4

Count exported documents by index name, not pattern

259fc06

Mubelotix reviewed Jul 1, 2025

View reviewed changes

Kerollmops commented Jul 1, 2025

View reviewed changes

	let task_id = to_export.iter().next().expect("There must be only one export task");
	let task_id = to_export.iter().next().expect("There must be at least one export task");

	(status = OK, description = "Known nodes are returned", body = Export, content_type = "application/json", example = json!(
	(status = 202, description = "Export successfully enqueued", body = SummarizedTaskView, content_type = "application/json", example = json!(

		// TODO make it experimental?
		// index_scheduler.features().check_network("Using the /network route")?;

		indexes: BTreeMap<IndexUidPattern, DetailsExportIndexSettings>,
		indexes: BTreeMap<String, DetailsExportIndexSettings>,

		indexes: indexes.iter().map(\|(p, s)\| (p.clone(), s.clone().into())).collect(),
		indexes: BTreeMap::new(),

Introduce a new route to export indexes #5670

Are you sure you want to change the base?

Introduce a new route to export indexes #5670

Conversation

Kerollmops commented Jun 12, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

To Do

This PR makes forward-compatible changes

Uh oh!

github-actions bot commented Jun 12, 2025

Hello, I'm a bot 🤖

This PR makes forward-compatible changes

This PR makes breaking changes

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Kerollmops Jul 1, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Kerollmops commented Jun 12, 2025 •

edited

Loading

Kerollmops Jul 1, 2025 •

edited

Loading