@ankane had mentioned wanting to keep dimensionality reduction separate from pgvector at the time we created this issue, although has started looking into product quantization specifically to improve for IVFFLAT performance. pgvector/pgvector#27

This issue is really targeted at general dimensionality reduction algorithms and uses, like PCA as a first algorithm, but ultimately the whole family like TruncatedSVD, TSNE etc.

One related thought is that we've also added support for clustering algorithm's to PostgresML since this issue was created, but limited algorithm's to online implementations so that the API can be the same train/predict similar to regression/classification tasks, which leaves writing all the cluster_id's for input rows as a job handled outside of the API for now. If we can do the same for PCA, (except transform instead of predict), that'd be ideal.

I think #795 is a very similar PR to add a clustering task. I think step 1 is to define a new Task for dimensionality_reduction, and a new set of Algorithms like pca, that can be passed to train, and ultimately applies the scikit fit, which is how I started adding clustering task in #795. The Rust compiler spelled out the rest of the changes that needed to be made until I got into bindings/scikit.rs and bindings/scikit.py, which is just a little bit of glue.

Then it's a matter of making sure transform is analogously exposed on those types of models, in place of predict and takes a row to reduce.

Add dimensionality reduction algorithms for vectors #661

Description

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions