Skip to content

tests: Faster batches:: IT tests #5626

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 6 commits into
base: main
Choose a base branch
from

Conversation

martin-g
Copy link
Contributor

@martin-g martin-g commented Jun 2, 2025

Pull Request

Related issue

#4840

What does this PR do?

  • Use shared server + unique indices where possible

@martin-g
Copy link
Contributor Author

martin-g commented Jun 3, 2025

@irevoire I will need some help with this one.
The tests use index.get_batch(...) passing hardcoded values like 0 and 1.
But as far as I understand it the batches are global, i.e. per server, not per index.
This makes it very hard (impossible) to use shared server!

Am I correct ?

Copy link
Member

@irevoire irevoire left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I yeah I see the issue, these tests were a bit rushed it looks like.

The best solution IMO is to create a new method similar to this one:

#[track_caller]
pub fn uid(&self) -> u64 {
if let Some(uid) = self["uid"].as_u64() {
uid
} else if let Some(uid) = self["taskUid"].as_u64() {
uid
} else {
panic!("Didn't find any task id in: {self}");
}
}

Except its batch_uid and it retrieve the batchUid field or panic.

Then the usage is that you must wait for the task to finish and once it's finished you can retrieve the associated batch with this new method from the task body

let (task, _status_code) = index.create(None).await;
index.wait_task(task.uid()).await.succeeded();
let (_response, code) = index.get_batch(0).await;
let (_response, code) = index.get_batch(task.uid() as u32).await;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
let (_response, code) = index.get_batch(task.uid() as u32).await;
let (_response, code) = index.get_batch(task.batch_uid()).await;

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hm.
It seems the batch_uid property is set only for status=processing tasks:

if processing.contains(task.uid) {
Task {
status: Status::Processing,
batch_uid: Some(batch.uid),

At the moment the new method fails with:

thread 'batches::test_summarized_document_addition_or_update' panicked at crates/meilisearch/tests/batches/mod.rs:276:43:
Didn't find any batch id in: {
  "taskUid": 19,
  "indexUid": "961f61b5-5ac7-4b03-adbd-5839519d886e",
  "status": "enqueued",
  "type": "documentAdditionOrUpdate",
  "enqueuedAt": "[date]"
}

Copy link
Member

@irevoire irevoire Jun 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It should also be set for all the finished tasks.
Basically, it should always be set except when the task is enqueued, iirc and that's why we have to wait for the task to finishes before calling this method

martin-g added 6 commits June 10, 2025 14:12
Use shared server + unique indices where possible

Related-to: meilisearch#4840

Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
Signed-off-by: Martin Tzvetanov Grigorov <mgrigorov@apache.org>
@martin-g martin-g force-pushed the faster-batches-it-tests branch from 4d8e40d to 9e31d6c Compare June 10, 2025 11:12
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants