Skip to content

Silas final pipelines #934

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 35 commits into from
Aug 21, 2023
Merged

Silas final pipelines #934

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
35 commits
Select commit Hold shift + click to select a range
6c14956
Remote save
SilasMarvin Jul 14, 2023
3b2e3b5
Working remote embeddings
SilasMarvin Jul 17, 2023
f1d6bf7
Compiling
SilasMarvin Jul 19, 2023
22f280e
Commit before moving everything to lazy
SilasMarvin Jul 19, 2023
ec090ca
Working lazy python
SilasMarvin Jul 21, 2023
58c01a3
Commit before moving adjusting Javascript macros
SilasMarvin Jul 21, 2023
9e7b146
Working javascript sdk
SilasMarvin Jul 22, 2023
abb4f5e
Working javascript sdk
SilasMarvin Jul 26, 2023
76ccf3a
The start of working pipelines
SilasMarvin Jul 28, 2023
cfcc66b
Working pipelines in python
SilasMarvin Aug 3, 2023
a9dcbc9
Uncomment
SilasMarvin Aug 3, 2023
8b48750
Added to_dict function
SilasMarvin Aug 3, 2023
6e3f1e6
Small changes and prep for progress bars
SilasMarvin Aug 4, 2023
5365557
Working progress bars and many other small but exciting things
SilasMarvin Aug 4, 2023
66476ff
Prepping to push to test pypi
SilasMarvin Aug 7, 2023
f2613d7
Prepping for javascript
SilasMarvin Aug 8, 2023
92c9623
Improvments to javascript and updates to the python sdk deploy script
SilasMarvin Aug 8, 2023
8a4e3cf
Prepping for real tests
SilasMarvin Aug 8, 2023
12bb3a8
Updated sql
SilasMarvin Aug 9, 2023
2b5b68b
Python examples translated to use pipelines
SilasMarvin Aug 9, 2023
447fc80
Mostly cleaned up and documented crate, and cleaned up python README …
SilasMarvin Aug 10, 2023
333c5e6
Ready for test deployments
SilasMarvin Aug 10, 2023
11bcce2
Updated manual build file for python
SilasMarvin Aug 10, 2023
845bf02
Build fast
SilasMarvin Aug 11, 2023
4904a1a
Small tweaks
SilasMarvin Aug 11, 2023
64dc7e2
Prepping for another test release
SilasMarvin Aug 11, 2023
c3b274c
Prepping to expand query_builder
SilasMarvin Aug 11, 2023
cb143a5
Massive cleanups to macros
SilasMarvin Aug 11, 2023
c66b07b
Massive cleanups to macros
SilasMarvin Aug 11, 2023
b7d4c2d
Ready to release
SilasMarvin Aug 11, 2023
a2c87b1
Formatting
SilasMarvin Aug 13, 2023
dd9c3ab
Renamed files
SilasMarvin Aug 21, 2023
5568608
Added removed file
SilasMarvin Aug 21, 2023
4a2e98d
Removed unnecessary file
SilasMarvin Aug 21, 2023
e673af4
Updated sdk version to 0.9
SilasMarvin Aug 21, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Python examples translated to use pipelines
  • Loading branch information
@SilasMarvin
SilasMarvin committed Aug 21, 2023
commit 2b5b68b29349c82ef27ef81a685db09969e72f3d
164 changes: 82 additions & 82 deletions pgml-sdks/rust/pgml/javascript/tests/typescript-tests/test.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,88 +32,88 @@ const generate_dummy_documents = (count: number) => {
}
return docs;
}
//
// ///////////////////////////////////////////////////
// // Test the API exposed is correct ////////////////
// ///////////////////////////////////////////////////
//
// it("can create collection", () => {
// let collection = pgml.newCollection("test_j_c_ccc_0");
// expect(collection).toBeTruthy();
// });
//
// it("can create model", () => {
// let model = pgml.newModel("test", "openai", {
// "tester": "test 0123948712394871234987"
// });
// expect(model).toBeTruthy();
// });
//
// it("can create splitter", () => {
// let splitter = pgml.newSplitter();
// expect(splitter).toBeTruthy();
// });
//
// it("can create pipeline", () => {
// let model = pgml.newModel();
// let splitter = pgml.newSplitter();
// let pipeline = pgml.newPipeline("test_j_p_ccc_0", model, splitter);
// expect(pipeline).toBeTruthy();
// });
//
// it("can create builtins", () => {
// let builtins = pgml.newBuiltins();
// expect(builtins).toBeTruthy();
// });
//
// ///////////////////////////////////////////////////
// // Test various vector searches ///////////////////
// ///////////////////////////////////////////////////
//
// it("can vector search with local embeddings", async () => {
// let model = pgml.newModel();
// let splitter = pgml.newSplitter();
// let pipeline = pgml.newPipeline("test_j_p_cvswle_0", model, splitter);
// let collection = pgml.newCollection("test_j_c_cvswle_2");
// await collection.upsert_documents(generate_dummy_documents(3));
// await collection.add_pipeline(pipeline);
// let results = await collection.vector_search("Here is some query", pipeline);
// expect(results).toHaveLength(3);
// await collection.archive();
// });
//
// it("can vector search with remote embeddings", async() => {
// let model = pgml.newModel("text-embedding-ada-002", "openai");
// let splitter = pgml.newSplitter();
// let pipeline = pgml.newPipeline("test_j_p_cvswre_0", model, splitter);
// let collection = pgml.newCollection("test_j_c_cvswre_0");
// await collection.upsert_documents(generate_dummy_documents(3));
// await collection.add_pipeline(pipeline);
// let results = await collection.vector_search("Here is some query", pipeline);
// expect(results).toHaveLength(3);
// });
//
// it("can vector search with query builder", async() => {
// let model = pgml.newModel();
// let splitter = pgml.newSplitter();
// let pipeline = pgml.newPipeline("test_j_p_cvswqb_0", model, splitter);
// let collection = pgml.newCollection("test_j_c_cvswqb_0");
// await collection.upsert_documents(generate_dummy_documents(3));
// await collection.add_pipeline(pipeline);
// let results = await collection.query().vector_recall("Here is some query", pipeline).limit(10).run();
// expect(results).toHaveLength(3);
// });
//
// it("can vector search with query builder with remote embeddings", async() => {
// let model = pgml.newModel("text-embedding-ada-002", "openai");
// let splitter = pgml.newSplitter();
// let pipeline = pgml.newPipeline("test_j_p_cvswqbwre_0", model, splitter);
// let collection = pgml.newCollection("test_j_c_cvswqbwre_0");
// await collection.upsert_documents(generate_dummy_documents(3));
// await collection.add_pipeline(pipeline);
// let results = await collection.query().vector_recall("Here is some query", pipeline).limit(10).run();
// expect(results).toHaveLength(3);
// });

///////////////////////////////////////////////////
// Test the API exposed is correct ////////////////
///////////////////////////////////////////////////

it("can create collection", () => {
let collection = pgml.newCollection("test_j_c_ccc_0");
expect(collection).toBeTruthy();
});

it("can create model", () => {
let model = pgml.newModel("test", "openai", {
"tester": "test 0123948712394871234987"
});
expect(model).toBeTruthy();
});

it("can create splitter", () => {
let splitter = pgml.newSplitter();
expect(splitter).toBeTruthy();
});

it("can create pipeline", () => {
let model = pgml.newModel();
let splitter = pgml.newSplitter();
let pipeline = pgml.newPipeline("test_j_p_ccc_0", model, splitter);
expect(pipeline).toBeTruthy();
});

it("can create builtins", () => {
let builtins = pgml.newBuiltins();
expect(builtins).toBeTruthy();
});

///////////////////////////////////////////////////
// Test various vector searches ///////////////////
///////////////////////////////////////////////////

it("can vector search with local embeddings", async () => {
let model = pgml.newModel();
let splitter = pgml.newSplitter();
let pipeline = pgml.newPipeline("test_j_p_cvswle_0", model, splitter);
let collection = pgml.newCollection("test_j_c_cvswle_2");
await collection.upsert_documents(generate_dummy_documents(3));
await collection.add_pipeline(pipeline);
let results = await collection.vector_search("Here is some query", pipeline);
expect(results).toHaveLength(3);
await collection.archive();
});

it("can vector search with remote embeddings", async() => {
let model = pgml.newModel("text-embedding-ada-002", "openai");
let splitter = pgml.newSplitter();
let pipeline = pgml.newPipeline("test_j_p_cvswre_0", model, splitter);
let collection = pgml.newCollection("test_j_c_cvswre_0");
await collection.upsert_documents(generate_dummy_documents(3));
await collection.add_pipeline(pipeline);
let results = await collection.vector_search("Here is some query", pipeline);
expect(results).toHaveLength(3);
});

it("can vector search with query builder", async() => {
let model = pgml.newModel();
let splitter = pgml.newSplitter();
let pipeline = pgml.newPipeline("test_j_p_cvswqb_0", model, splitter);
let collection = pgml.newCollection("test_j_c_cvswqb_0");
await collection.upsert_documents(generate_dummy_documents(3));
await collection.add_pipeline(pipeline);
let results = await collection.query().vector_recall("Here is some query", pipeline).limit(10).run();
expect(results).toHaveLength(3);
});

it("can vector search with query builder with remote embeddings", async() => {
let model = pgml.newModel("text-embedding-ada-002", "openai");
let splitter = pgml.newSplitter();
let pipeline = pgml.newPipeline("test_j_p_cvswqbwre_0", model, splitter);
let collection = pgml.newCollection("test_j_c_cvswqbwre_0");
await collection.upsert_documents(generate_dummy_documents(3));
await collection.add_pipeline(pipeline);
let results = await collection.query().vector_recall("Here is some query", pipeline).limit(10).run();
expect(results).toHaveLength(3);
});


///////////////////////////////////////////////////
Expand Down
2 changes: 1 addition & 1 deletion pgml-sdks/rust/pgml/pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,7 +13,7 @@ authors = [
homepage = "https://postgresml.org"
repository = "https://github.com/postgresml/postgresml"
documentation = "https://github.com/postgresml/postgresml/tree/master/pgml-sdks/python/pgml"
readme = "../../python/pgml/README.md"
readme = "./python/README.md"
keywords = ["postgres","machine learning","vector databases","embeddings"]
classifiers = [
"Programming Language :: Rust",
Expand Down
Loading