I also modified one thing in the code (I just did some simple research):

.call((raw_bytes_data.to_vec(),), None)?
to:

.call1((PyBytes::new(py, raw_bytes_data),))?
Reason:
This reduces data copying from 2 operations (Rust Vec creation + Python internal copy) to 1 operation (direct copy to Python memory).

@chandr-andr ,In 8cc74de, I made further optimizations to improve performance. I only conducted simple tests with pytest, no benchmarking. Can you run some tests? If there's a performance regression, feel free to roll back the code.
It is best to increase the memory usage test!

@bymoye Thank you very much for refactoring, it looks awesome.
I cannot run benchmarks right now cuz I have to work (unfortunately).

Will run them in the evening

@bymoye I run benchmarks, I have, I found a difference, not big, but you didn't do anything worse.

The rest of the changes look awesome, thank you a lot!

ok！Thank you for testing

@chandr-andr I checked your code, and I think the current test is not right. Because the engine benchmark should test the performance of the engine, not the performance of the database. So the database query should not use ORDER BY RANDOM, but to reduce the database pressure as much as possible (so that the database is almost no pressure) to complete the query, and then return to Python to get the same result.
The data will be converted and other actions (completed by the engine).
We can return as much data as possible to complete the performance test.

For example, a single query uses SELECT 1;
Multiple queries use id< 101; id < 1001; id < 10001;
and returns the actual data instead of the current "ok";

Time Complexity: The original code has a time complexity of $O(N \cdot D)$, while the optimized code reduces it to $O(N)$, significantly lowering complexity and preventing performance degradation.
Space Complexity: Both versions have a space complexity of $O(N)$.

Optimized: Memory is allocated once and used throughout, ensuring stability and efficiency.
Original: Potential memory jitter (repeated allocation and deallocation) introduces additional overhead.

Memory Copy:

Original: Memory copy operations occur $N \cdot (D - 1)$ times (handled internally by Rust).
Optimized: Reduced to $0$ copies, as all operations are performed via references.

Total Copied Elements:

Original: $N \cdot (D + 1)$ elements copied.
Optimized: Reduced to $2N$ elements.

These optimizations should bring significant performance improvements to querying data values ​​(especially for big data).