diff --git a/pgml-cms/docs/resources/benchmarks/ggml-quantized-llm-support-for-huggingface-transformers.md b/pgml-cms/docs/resources/benchmarks/ggml-quantized-llm-support-for-huggingface-transformers.md index 86e48ed91..59bd35af4 100644 --- a/pgml-cms/docs/resources/benchmarks/ggml-quantized-llm-support-for-huggingface-transformers.md +++ b/pgml-cms/docs/resources/benchmarks/ggml-quantized-llm-support-for-huggingface-transformers.md @@ -58,7 +58,7 @@ SELECT pgml.transform( ## Quantization -_Discrete quantization is not a new idea. It's been used by both algorithms and artists for more than a hundred years._\\ +_Discrete quantization is not a new idea. It's been used by both algorithms and artists for more than a hundred years._ Going beyond 16-bit down to 8 or 4 bits is possible, but not with hardware accelerated floating point operations. If we want hardware acceleration for smaller types, we'll need to use small integers w/ vectorized instruction sets. This is the process of _quantization_. Quantization can be applied to existing models trained with 32-bit floats, by converting the weights to smaller integer primitives that will still benefit from hardware accelerated instruction sets like Intel's [AVX](https://en.wikipedia.org/wiki/Advanced\_Vector\_Extensions). A simple way to quantize a model can be done by first finding the maximum and minimum values of the weights, then dividing the range of values into the number of buckets available in your integer type, 256 for 8-bit, 16 for 4-bit. This is called _post-training quantization_, and it's the simplest way to quantize a model. diff --git a/pgml-cms/docs/resources/benchmarks/making-postgres-30-percent-faster-in-production.md b/pgml-cms/docs/resources/benchmarks/making-postgres-30-percent-faster-in-production.md index 508110db5..e58268f2e 100644 --- a/pgml-cms/docs/resources/benchmarks/making-postgres-30-percent-faster-in-production.md +++ b/pgml-cms/docs/resources/benchmarks/making-postgres-30-percent-faster-in-production.md @@ -20,7 +20,7 @@ This is not only a performance benefit, but also a usability improvement for cli ## Benchmark -\\ +
diff --git a/pgml-cms/docs/resources/benchmarks/mindsdb-vs-postgresml.md b/pgml-cms/docs/resources/benchmarks/mindsdb-vs-postgresml.md index 2faa141c3..581c846b8 100644 --- a/pgml-cms/docs/resources/benchmarks/mindsdb-vs-postgresml.md +++ b/pgml-cms/docs/resources/benchmarks/mindsdb-vs-postgresml.md @@ -44,7 +44,7 @@ Another difference is that PostgresML also supports embedding models, and closel The architectural implementations for these projects is significantly different. PostgresML takes a data centric approach with Postgres as the provider for both storage _and_ compute. To provide horizontal scalability for inference, the PostgresML team has also created [PgCat](https://github.com/postgresml/pgcat) to distribute workloads across many Postgres databases. On the other hand, MindsDB takes a service oriented approach that connects to various databases over the network. -\\ +
@@ -59,7 +59,7 @@ The architectural implementations for these projects is significantly different. | On Premise | ✅ | ✅ | | Web UI | ✅ | ✅ | -\\ + The difference in architecture leads to different tradeoffs and challenges. There are already hundreds of ways to get data into and out of a Postgres database, from just about every other service, language and platform that makes PostgresML highly compatible with other application workflows. On the other hand, the MindsDB Python service accepts connections from specifically supported clients like `psql` and provides a pseudo-SQL interface to the functionality. The service will parse incoming MindsDB commands that look similar to SQL (but are not), for tasks like configuring database connections, or doing actual machine learning. These commands typically have what looks like a sub-select, that will actually fetch data over the wire from configured databases for Machine Learning training and inference. @@ -287,7 +287,7 @@ PostgresML is the clear winner in terms of performance. It seems to me that it c | translation\_en\_to\_es | t5-base | 1573 | 1148 | 294 | | summarization | sshleifer/distilbart-cnn-12-6 | 4289 | 3450 | 479 | -\\ + There is a general trend, the larger and slower the model is, the more work is spent inside libtorch, the less the performance of the rest matters, but for interactive models and use cases there is a significant difference. We've tried to cover the most generous use case we could between these two. If we were to compare XGBoost or other classical algorithms, that can have sub millisecond prediction times in PostgresML, the 20ms Python service overhead of MindsDB just to parse the incoming query would be hundreds of times slower. diff --git a/pgml-cms/docs/use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml.md b/pgml-cms/docs/use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml.md index 8ed3a34f8..07ccbbefb 100644 --- a/pgml-cms/docs/use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml.md +++ b/pgml-cms/docs/use-cases/embeddings/generating-llm-embeddings-with-open-source-models-in-postgresml.md @@ -198,7 +198,7 @@ For comparison, it would cost about $299 to use OpenAI's cheapest embedding mode | GPU | 17ms | $72 | 6 hours | | OpenAI | 300ms | $299 | millennia | -\\ + You can also find embedding models that outperform OpenAI's `text-embedding-ada-002` model across many different tests on the [leaderboard](https://huggingface.co/spaces/mteb/leaderboard). It's always best to do your own benchmarking with your data, models, and hardware to find the best fit for your use case.