-
-
-Lev Kokotov
-October 3, 2022
-
- Lev Kokotov
-October 3, 2022
-
- Montana Low
-August 25, 2022
-
- Montana Low
-April 21, 2023
-pgml.embed(model_name, text). Prove the results in this series to your own satisfaction, for free, by [signing up](<%- crate::utils::config::signup_url() %>) for a GPU accelerated database.
-
-This article is the first in a multipart series that will show you how to build a post-modern semantic search and recommendation engine, including personalization, using open source models.
-
-1) [Generating LLM Embeddings with HuggingFace models](/blog/generating-llm-embeddings-with-open-source-models-in-postgresml)
-2) [Tuning vector recall with pgvector](/blog/tuning-vector-recall-while-generating-query-embeddings-in-the-database)
-3) [Personalizing embedding results with application data](/blog/personalize-embedding-vector-search-results-with-huggingface-and-pgvector)
-4) Optimizing semantic results with an XGBoost ranking model - coming soon!
-
-## Introduction
-
-In recent years, embeddings have become an increasingly popular technique in machine learning and data analysis. They are essentially vector representations of data points that capture their underlying characteristics or features. In most programming environments, vectors can be efficiently represented as native array datatypes. They can be used for a wide range of applications, from natural language processing to image recognition and recommendation systems.
-
-They can also turn natural language into quantitative features for downstream machine learning models and applications.
-
-
-Embeddings show us the relationships between rows in the database.
[intfloat/e5-small](https://huggingface.co/intfloat/e5-small) will be a good first attempt. The great thing about PostgresML is you can always regenerate your embeddings later to experiment with different embedding models.
-
-It takes a couple of minutes to download and cache the `intfloat/e5-small` model to generate the first embedding. After that, it's pretty fast.
-
-Note how we prefix the text we want to embed with either `passage: ` or `query: `, the e5 model requires us to prefix our data with `passage: ` if we're generating embeddings for our corpus and `query: ` if we want to find semantically similar content.
-
-```postgresql
-SELECT pgml.embed('intfloat/e5-small', 'passage: hi mom');
-```
-
-This is a pretty powerful function, because we can pass any arbitrary text to any open source model, and it will generate an embedding for us. We can benchmark how long it takes to generate an embedding for a single review, using client-side timings in Postgres:
-
-
-```postgresql
-\timing on
-```
-
-Aside from using this function with strings passed from a client, we can use it on strings already present in our database tables by calling pgml.embed on columns. For example, we can generate an embedding for the first review using a pretty simple query:
-
-!!! generic
-
-!!! code_block time="54.820 ms"
-
-```postgresql
-SELECT
- review_body,
- pgml.embed('intfloat/e5-small', 'passage: ' || review_body)
-FROM pgml.amazon_us_reviews
-LIMIT 1;
-```
-
-!!!
-
-!!! results
-
-```
-CREATE INDEX
-```
-
-!!!
-
-!!!
-
-Time to generate an embedding increases with the length of the input text, and varies widely between different models. If we up our batch size (controlled by `LIMIT`), we can see the average time to compute an embedding on the first 1000 reviews is about 17ms per review:
-
-!!! code_block time="17955.026 ms"
-
-```postgresql
-SELECT
- review_body,
- pgml.embed('intfloat/e5-small', 'passage: ' || review_body) AS embedding
-FROM pgml.amazon_us_reviews
-LIMIT 1000;
-```
-
-!!!
-
-## Comparing different models and hardware performance
-
-This database is using a single GPU with 32GB RAM and 8 vCPUs with 16GB RAM. Running these benchmarks while looking at the database processes with `htop` and `nvidia-smi`, it becomes clear that the bottleneck in this case is actually tokenizing the strings which happens in a single thread on the CPU, not computing the embeddings on the GPU which was only 20% utilized during the query.
-
-We can also do a quick sanity check to make sure we're really getting value out of our GPU by passing the device to our embedding function:
-
-!!! code_block time="30421.491 ms"
-
-```postgresql
-SELECT
- reviqew_body,
- pgml.embed(
- 'intfloat/e5-small',
- 'passage: ' || review_body,
- '{"device": "cpu"}'
- ) AS embedding
-FROM pgml.amazon_us_reviews
-LIMIT 1000;
-```
-
-!!!
-
-Forcing the embedding function to use `cpu` is almost 2x slower than `cuda` which is the default when GPUs are available.
-
-If you're managing dedicated hardware, there's always a decision to be made about resource utilization. If this is a multi-workload database with other queries using the GPU, it's probably great that we're not completely hogging it with our multi-decade-Amazon-scale data import process, but if this is a machine we've spun up just for this task, we can up the resource utilization to 4 concurrent connections, all running on a subset of the data to more completely utilize our CPU, GPU and RAM.
-
-Another consideration is that GPUs are much more expensive right now than CPUs, and if we're primarily interested in backfilling a dataset like this, high concurrency across many CPU cores might just be the price-competitive winner.
-
-With 4x concurrency and a GPU, it'll take about 6 hours to compute all 5 million embeddings, which will cost $72 on [PostgresML Cloud](<%- crate::utils::config::signup_url() %>). If we use the CPU instead of the GPU, we'll probably want more cores and higher concurrency to plug through the job faster. A 96 CPU core machine could complete the job in half the time our single GPU would take and at a lower hourly cost as well, for a total cost of $24. It's overall more cost-effective and faster in parallel, but keep in mind if you're interactively generating embeddings for a user facing application, it will add double the latency, 30ms CPU vs 17ms for GPU.
-
-For comparison, it would cost about $299 to use OpenAI's cheapest embedding model to process this dataset. Their API calls average about 300ms, although they have high variability (200-400ms) and greater than 1000ms p99 in our measurements. They also have a default rate limit of 200 tokens per minute which means it would take 1,425 years to process this dataset. You better call ahead.
-
-| Processor | Latency | Cost | Time |
-|-----------|---------|------|-----------|
-| CPU | 30ms | $24 | 3 hours |
-| GPU | 17ms | $72 | 6 hours |
-| OpenAI | 300ms | $299 | millennia |
-
-
- Montana Low
-September 4, 2023
-
-PostgresML is a composition engine that provides advanced AI capabilities.
- Silas Marvin
-July 11, 2023
-
- Santi Adavani
-June 01, 2023
-
-
-## Key Features
-
-### Automated Database Management
-The Python SDK automates the management of various database tables, eliminating the complexity of setting up and maintaining the data structure required for vector search applications. With this automated system, you can focus on building robust search functionalities while the SDK handles the underlying database management.
-
-### Embedding Generation from Open Source Models
-Leveraging the Python SDK, you gain access to a vast collection of open source models. These models have been trained on extensive datasets and capture the semantic meaning of text. With just a few lines of code, you can generate embeddings using these models, enabling powerful analysis and search capabilities in your application.
-
-### Flexible and Scalable Vector Search
-The Python SDK seamlessly integrates with PgVector, a PostgreSQL extension designed for efficient vector-based indexing and querying. By leveraging the power of PgVector, you can perform advanced searches, rank results by relevance, and retrieve accurate and meaningful information from your database. The SDK ensures that your vector search application scales effortlessly to handle increasing amounts of data.
-
-## How the Python SDK Works
-
-The Python SDK simplifies the development of vector search applications by abstracting away the complexities of database management and indexing. Here's an overview of how it works:
-
-### Document and Text Chunk Management
-The SDK simplifies the process of upserting documents and generating text chunks by offering a user-friendly interface. It allows you to effortlessly add and configure various text splitters to generate text chunks of different sizes, overlaps, and file formats, such as Python and Markdown.
-
-### Open Source Model Integration
-With the SDK, you can seamlessly incorporate a wide range of open source models from HuggingFace into your application. These models capture the semantic meaning of text and enable powerful analysis and search capabilities. Generating high-quality embeddings from these models is a breeze with the Python SDK.
-
-### Embedding Indexing
-The Python SDK utilizes the PgVector extension to efficiently index the embeddings generated by the open source models. This indexing process optimizes search performance and allows for fast and accurate retrieval of relevant results, even with large volumes of data.
-
-### Querying and Search
-Once the embeddings are indexed, the SDK provides intuitive methods for executing vector-based searches on the documents and text chunks stored in the PostgreSQL database. You can easily execute queries and retrieve search results with precise and relevant information.
-
-## Use Cases
-
-The Python SDK's embedding capabilities find applications in various scenarios, including:
-
-### Search
-By comparing embeddings of query strings and documents, you can retrieve search results ranked by their relevance or similarity to the query. This allows users to find the most relevant information quickly and effectively.
-
-### Clustering
-Utilizing embeddings, you can group text strings based on their similarity. By measuring the similarity between embeddings, you can identify clusters or groups of text strings that share common characteristics, providing valuable insights for data analysis.
-
-### Recommendations
-Embeddings play a crucial role in recommendation systems. By identifying items with related text strings based on their embeddings, you can deliver personalized recommendations to users, enhancing user experience and engagement.
-
-### Anomaly Detection
-Anomaly detection involves identifying outliers or anomalies in data. By quantifying the similarity between text strings using embeddings, you can identify anomalies that have little relatedness to the rest of the data, aiding in anomaly detection tasks.
-
-### Classification
-Embeddings are valuable in classification tasks, where text strings are classified based on their most similar label. By comparing the embeddings of text strings and labels, you can accurately classify new text strings into predefined categories.
-
-## Get Started with the Python SDK
-
-To get started with the Python SDK for scalable vector search on PostgreSQL, visit our [GitHub repository](https://github.com/postgresml/postgresml/tree/master/pgml-sdks/python/pgml). You'll find comprehensive documentation, code examples, and installation instructions to help you integrate the SDK into your projects seamlessly.
-
-We're excited to see how the Python SDK transforms your vector search applications, enabling fast, accurate, and scalable search functionalities. Should you have any questions or need assistance please do not hesitate to reach out to us on [Discord](https://discord.gg/DmyJP3qJ7U) or send an [email](mailto:team@postgresml.org).
-
-Happy coding and happy searching!
-
diff --git a/pgml-dashboard/content/blog/llm-based-pipelines-with-postgresml-and-dbt.md b/pgml-dashboard/content/blog/llm-based-pipelines-with-postgresml-and-dbt.md
deleted file mode 100644
index 336b5bf62..000000000
--- a/pgml-dashboard/content/blog/llm-based-pipelines-with-postgresml-and-dbt.md
+++ /dev/null
@@ -1,190 +0,0 @@
----
-author: Santi Adavani
-description: Unlock the Power of Large Language Models (LLM) in Data Pipelines with PostgresML and dbt. Streamline your text processing workflows and leverage the advanced capabilities of LLMs for efficient data transformation and analysis. Discover how PostgresML and dbt combine to deliver scalable and secure pipelines, enabling you to extract valuable insights from textual data. Supercharge your data-driven decision-making with LLM-based pipelines using PostgresML and dbt.
-image: https://postgresml.org/dashboard/static/images/blog/llm_based_pipeline_hero.png
-image_alt: "LLM based pipelines with PostgresML and dbt (data build tool)"
----
-# LLM based pipelines with PostgresML and dbt (data build tool)
-
- Santi Adavani
-July 13, 2023
-
-
-## PostgresML
-PostgresML, an open-source machine learning extension for PostgreSQL, is designed to handle text processing tasks using large language models. Its motivation lies in harnessing the power of LLMs within the familiar PostgreSQL ecosystem. By integrating LLMs directly into the database, PostgresML eliminates the need for data movement and offers scalable and secure text processing capabilities. This native integration enhances data governance, security, and ensures the integrity of text data throughout the pipeline.
-
-## dbt (data build tool)
-dbt is an open-source command-line tool that streamlines the process of building, testing, and maintaining data infrastructure. Specifically designed for data analysts and engineers, dbt offers a consistent and standardized approach to data transformation and analysis. By providing an intuitive and efficient workflow, dbt simplifies working with data, empowering organizations to seamlessly transform and analyze their data.
-
-## PostgresML and dbt
-The integration of PostgresML and dbt offers an exceptional advantage for data engineers seeking to swiftly incorporate text processing into their workflows. With PostgresML's advanced machine learning capabilities and dbt's streamlined data transformation framework, data engineers can seamlessly integrate text processing tasks into their existing pipelines. This powerful combination empowers data engineers to efficiently leverage PostgresML's text processing capabilities, accelerating the incorporation of sophisticated NLP techniques and large language models into their data workflows. By bridging the gap between machine learning and data engineering, PostgresML and dbt enable data engineers to unlock the full potential of text processing with ease and efficiency.
-
-- Streamlined Text Processing: PostgresML seamlessly integrates large language models into the data pipeline, enabling efficient and scalable text processing. It leverages the power of the familiar PostgreSQL environment, ensuring data integrity and simplifying the overall workflow.
-
-- Simplified Data Transformation: dbt simplifies the complexities of data transformation by automating repetitive tasks and providing a modular approach. It seamlessly integrates with PostgresML, enabling easy incorporation of large language models for feature engineering, model training, and text analysis.
-
-- Scalable and Secure Pipelines: PostgresML's integration with PostgreSQL ensures scalability and security, allowing organizations to process and analyze large volumes of text data with confidence. Data governance, access controls, and compliance frameworks are seamlessly extended to the text processing pipeline.
-
-## Tutorial
-By following this [tutorial](https://github.com/postgresml/postgresml/tree/master/pgml-extension/examples/dbt/embeddings), you will gain hands-on experience in setting up a dbt project, defining models, and executing an LLM-based text processing pipeline. We will guide you through the process of incorporating LLM-based text processing into your data workflows using PostgresML and dbt. Here's a high-level summary of the tutorial:
-
-### Prerequisites
-
-- [PostgresML DB](https://github.com/postgresml/postgresml#installation)
-- Python >=3.7.2,<4.0
-- [Poetry](https://python-poetry.org/)
-- Install `dbt` using the following commands
- - `poetry shell`
- - `poetry install`
-- Documents in a table
-
-### dbt Project Setup
-
-Once you have the pre-requisites satisfied, update `dbt` project configuration files.
-
-### Project name
-You can find the name of the `dbt` project in `dbt_project.yml`.
-
-```yaml
-# Name your project! Project names should contain only lowercase characters
-# and underscores. A good package name should reflect your organization's
-# name or the intended use of these models
-name: 'pgml_flow'
-version: '1.0.0'
-```
-
-### Dev and prod DBs
-Update `profiles.yml` file with development and production database properties. If you are using Docker based local PostgresML installation, `profiles.yml` will be as follows:
-
-```yaml
-pgml_flow:
- outputs:
-
- dev:
- type: postgres
- threads: 1
- host: 127.0.0.1
- port: 5433
- user: postgres
- pass: ""
- dbname: pgml_development
- schema:
- Lev Kokotov
-June 16, 2023
-
- Montana Low
-June 8, 2023
-
- Montana Low
-May 3, 2023
-
-Models can be trained on application data, to reach an objective.
- Lev Kokotov
-September 7, 2022
-
- Montana Low
-May 3, 2023
-
-Embeddings can be combined into personalized perspectives when stored as vectors in the database.
- Jason Dusek
-May 8, 2023
-
- Santi Adavani
-August 17, 2023
-
- Montana Low
-August 31, 2022
-
- Santi Adavani
-May 3, 2023
-
- Lev Kokotov
-October 18, 2022
-
- Montana Low
-September 19, 2022
-
- 
-
- Montana Low, CEO
-May 10, 2023
-
-
-## Our Serverless AI Cloud
-
-Building on the success of our open source database extension to Postgres, we’ve created a cloud with our own custom Postgres load balancer. PgCat is tailored for our machine learning workflows at scale and enables us to pool multiple machines and connections, creating a mesh of Postgres clusters that appear as independent Postgres databases. We can scale single tenant workloads across a large fleet of physical machines, beyond traditional replication, enabling efficient multi GPU inference workloads.
-
-Creating a new database in this cluster takes a few milliseconds. That database will have massive burst capacity, up to a full sized shard with 128 concurrent workers. Our scaling is so fast and efficient we are offering free databases with up to 5GB of data, and only charge if you’d like us to cache your custom models, data, and indexes, for maximum performance.
-
-Even though PgCat is barely a year old, there are already production workloads handling hundreds of thousands of queries per second at companies like Instacart and OneSignal. Our own deployment is already managing hundreds of independent databases, and launching many new ones every day.
-
-
-
-## Open Source is the Way Forward
-
-Our team moves quickly by working collaboratively within the larger open source community. Our technologies, both [PostgresML](https://github.com/postgresml/postgresml) and [PgCat](https://github.com/postgresml/pgcat), are MIT-licensed because we believe the opportunity size and efforts required to succeed safely are long term and global in scale.
-
-PostgresML is an extension for Postgres that brings models and algorithms into the database engine. You can load pretrained state-of-the-art LLMs and datasets directly from HuggingFace. Additionally, the Postgres community has created a treasure trove of extensions like pgvector. For example, combining the vector database, open source models, and input text in a single process is up to 40 times faster than alternative architectures for semantic search. The quality of those open source embeddings are also at the top of the leaderboards, which include proprietary models.
-
-By integrating all the leading machine learning libraries like Torch, Tensorflow, XGBoost, LightGBM, and Scikit Learn, you can go beyond a simple vector database, to training your own models for better ranking and recall using your application data and real user interactions, e.g personalizing vector search results by taking into account user behavior or fine-tuning open source LLMs using AB test results.
-
-Many amazing open and collaborative communities are shaping the future of our industry, and we will continue to innovate and contribute alongside them. If you’d like to see more of the things you can do with an AI application database, check out the [latest series of articles](/blog/generating-llm-embeddings-with-open-source-models-in-postgresml).
-
-
-
-## Thanks to Our Community
-
-We see a long term benefit to our community by building a company on top of our software that will push the boundaries of scale and edges of practicality that smaller independent teams running their own Postgres databases and AI workloads may not approach.
-
-Toward that end, we’ve raised $4.7M in seed funding led by Amplify Partners. Angels participating in the round include Max Mullen and Brandon Leonardo (Co-founders of Instacart), Jack Altman (Co-founder of Lattice), Rafael Corrales (Founding Investor at Vercel), Greg Rosen (Box Group), Jeremy Stanley (Co-founder of Anomalo) and James Yu (Co-founder of Parse).
-
-Our sincere thanks also goes out to all of the friends, family, colleagues and open source contributors who continue to support us on this journey. We’d love to have you join us as well, because the next decade in this sector is going to be a wild ride.
-
-## We’re Hiring
-
-If this sounds as interesting to you as it does to us, join us! We’re hiring experienced engineers familiar with Rust, Machine Learning, Databases and managing Infrastructure as a Service. The best way to introduce yourself is by submitting a pull request or reporting an issue on our open source projects [PostgresML](https://github.com/postgresml/postgresml), [PgCat](https://github.com/postgresml/pgcat) & [pg_stat_sysinfo](https://github.com/postgresml/pg_stat_sysinfo), or emailing us at team@postgresml.org.
diff --git a/pgml-dashboard/content/blog/scaling-postgresml-to-one-million-requests-per-second.md b/pgml-dashboard/content/blog/scaling-postgresml-to-one-million-requests-per-second.md
deleted file mode 100644
index 6086d878a..000000000
--- a/pgml-dashboard/content/blog/scaling-postgresml-to-one-million-requests-per-second.md
+++ /dev/null
@@ -1,256 +0,0 @@
----
-author: Lev Kokotov
-description: Addressing horizontal scalability concerns, we've benchmarked PostgresML and ended up with an incredible 1 million requests per second using commodity hardware.
-image: https://static.postgresml.org/benchmarks/Slow-Down-Sign.jpg
-image_alt: PostgresML at 1 million requests per second
----
-# Scaling PostgresML to 1 Million Requests per Second
-
-
- Lev Kokotov
-November 7, 2022
-
- - System Architecture -
-
- Silas Marvin
-October 2, 2023
-
-HNSW (hierarchical navigable small worlds) is an indexing method that greatly improves vector recall
- Montana Low
-April 28, 2023
-
-Embeddings show us the relationships between rows in the database, using natural language.
Yeah, well, that's just like, your opinion, man
- Lev Kokotov
-September 1, 2022
-In our example case, it's interesting that as `max_depth` increases, the "Test Score" on the key metric trends lower, so the smallest value of
Luckily, the smallest max_depth values also have the fastest "Fit Time", indicating that we pay less for training these higher quality models.
It's a little less obvious how the different values `n_estimators` and learning_rate impact the test score. We may want to rerun our search and zoom in on our the search space to get more insight.
pgml.digits table into the pgml schema, naming it pgml.snapshot_{id} where id is the primary key of the snapshot, and train a linear classification model on the snapshot using the target column as the label.
-
-!!!
-
-
-When used for the first time in a project, `pgml.train()` function requires the `task` parameter, which can be either `regression` or `classification`. The task determines the relevant metrics and analysis performed on the data. All models trained within the project will refer to those metrics and analysis for benchmarking and deployment.
-
-The first time it's called, the function will also require a `relation_name` and `y_column_name`. The two arguments will be used to create the first snapshot of training and test data. By default, 25% of the data (specified by the `test_size` parameter) will be randomly sampled to measure the performance of the model after the `algorithm` has been trained on the 75% of the data.
-
-
-!!! tip
-
-```postgresql
-SELECT * FROM pgml.train(
- 'My Classification Project',
- algorithm => 'xgboost'
-);
-```
-
-!!!
-
-Future calls to `pgml.train()` may restate the same `task` for a project or omit it, but they can't change it. Projects manage their deployed model using the metrics relevant to a particular task (e.g. `r2` or `f1`), so changing it would mean some models in the project are no longer directly comparable. In that case, it's better to start a new project.
-
-
-!!! tip
-
-If you'd like to train multiple models on the same snapshot, follow up calls to pgml.train() may omit the relation_name, y_column_name, test_size and test_sampling arguments to reuse identical data with multiple algorithms or hyperparameters.
-
-!!!
-
-
-
-## Getting Training Data
-
-A large part of the machine learning workflow is acquiring, cleaning, and preparing data for training algorithms. Naturally, we think Postgres is a great place to store your data. For the purpose of this example, we'll load a toy dataset, the classic handwritten digits image collection, from scikit-learn.
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.load_dataset('digits');
-```
-
-=== "Output"
-
-```
-pgml=# SELECT * FROM pgml.load_dataset('digits');
-NOTICE: table "digits" does not exist, skipping
- table_name | rows
--------------+------
- pgml.digits | 1797
-(1 row)
-```
-
-This `NOTICE` can safely be ignored. PostgresML attempts to do a clean reload by dropping the `pgml.digits` table if it exists. The first time this command is run, the table does not exist.
-
-===
-
-
-PostgresML loaded the Digits dataset into the `pgml.digits` table. You can examine the 2D arrays of image data, as well as the label in the `target` column:
-
-=== "SQL"
-
-```postgresql
-SELECT
- target,
- image
-FROM pgml.digits LIMIT 5;
-
-```
-
-=== "Output"
-
-```
-target | image
--------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------
- 0 | {{0,0,5,13,9,1,0,0},{0,0,13,15,10,15,5,0},{0,3,15,2,0,11,8,0},{0,4,12,0,0,8,8,0},{0,5,8,0,0,9,8,0},{0,4,11,0,1,12,7,0},{0,2,14,5,10,12,0,0},{0,0,6,13,10,0,0,0}}
- 1 | {{0,0,0,12,13,5,0,0},{0,0,0,11,16,9,0,0},{0,0,3,15,16,6,0,0},{0,7,15,16,16,2,0,0},{0,0,1,16,16,3,0,0},{0,0,1,16,16,6,0,0},{0,0,1,16,16,6,0,0},{0,0,0,11,16,10,0,0}}
- 2 | {{0,0,0,4,15,12,0,0},{0,0,3,16,15,14,0,0},{0,0,8,13,8,16,0,0},{0,0,1,6,15,11,0,0},{0,1,8,13,15,1,0,0},{0,9,16,16,5,0,0,0},{0,3,13,16,16,11,5,0},{0,0,0,3,11,16,9,0}}
- 3 | {{0,0,7,15,13,1,0,0},{0,8,13,6,15,4,0,0},{0,2,1,13,13,0,0,0},{0,0,2,15,11,1,0,0},{0,0,0,1,12,12,1,0},{0,0,0,0,1,10,8,0},{0,0,8,4,5,14,9,0},{0,0,7,13,13,9,0,0}}
- 4 | {{0,0,0,1,11,0,0,0},{0,0,0,7,8,0,0,0},{0,0,1,13,6,2,2,0},{0,0,7,15,0,9,8,0},{0,5,16,10,0,16,6,0},{0,4,15,16,13,16,1,0},{0,0,0,3,15,10,0,0},{0,0,0,2,16,4,0,0}}
-(5 rows)
-```
-
-===
-
-## Training a Model
-
-Now that we've got data, we're ready to train a model using an algorithm. We'll start with the default `linear` algorithm to demonstrate the basics. See the [Algorithms](/docs/guides/training/algorithm_selection/) for a complete list of available algorithms.
-
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.train(
- 'Handwritten Digit Image Classifier',
- 'classification',
- 'pgml.digits',
- 'target'
-);
-```
-
-=== "Output"
-
-```
-INFO: Snapshotting table "pgml.digits", this may take a little while...
-INFO: Snapshot of table "pgml.digits" created and saved in "pgml"."snapshot_1"
-INFO: Dataset { num_features: 64, num_labels: 1, num_rows: 1797, num_train_rows: 1348, num_test_rows: 449 }
-INFO: Training Model { id: 1, algorithm: linear, runtime: python }
-INFO: Hyperparameter searches: 1, cross validation folds: 1
-INFO: Hyperparams: {}
-INFO: Metrics: {
- "f1": 0.91903764,
- "precision": 0.9175061,
- "recall": 0.9205743,
- "accuracy": 0.9175947,
- "mcc": 0.90866333,
- "fit_time": 0.17586434,
- "score_time": 0.01282608
-}
- project | task | algorithm | deployed
-------------------------------------+----------------+-----------+----------
- Handwritten Digit Image Classifier | classification | linear | t
-(1 row)
-```
-
-===
-
-
-The output gives us information about the training run, including the `deployed` status. This is great news indicating training has successfully reached a new high score for the project's key metric and our new model was automatically deployed as the one that will be used to make new predictions for the project. See [Deployments](/docs/guides/predictions/deployments/) for a guide to managing the active model.
-
-## Inspecting the results
-Now we can inspect some of the artifacts a training run creates.
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.overview;
-```
-
-=== "Output"
-
-```
-pgml=# SELECT * FROM pgml.overview;
- name | deployed_at | task | algorithm | runtime | relation_name | y_column_name | test_sampling | test_size
-------------------------------------+----------------------------+----------------+-----------+---------+---------------+---------------+---------------+-----------
- Handwritten Digit Image Classifier | 2022-10-11 12:43:15.346482 | classification | linear | python | pgml.digits | {target} | last | 0.25
-(1 row)
-```
-
-===
-
-## More Examples
-
-See [examples](https://github.com/postgresml/postgresml/tree/master/pgml-extension/examples) in our git repository for more kinds of training with different types of features, algorithms and tasks.
diff --git a/pgml-dashboard/content/docs/guides/training/preprocessing.md b/pgml-dashboard/content/docs/guides/training/preprocessing.md
deleted file mode 100644
index 2d0e01c37..000000000
--- a/pgml-dashboard/content/docs/guides/training/preprocessing.md
+++ /dev/null
@@ -1,162 +0,0 @@
-# Preprocessing Data
-
-The training function also provides the option to preprocess data with the `preprocess` param. Preprocessors can be configured on a per-column basis for the training data set. There are currently three types of preprocessing available, for both categorical and quantitative variables. Below is a brief example for training data to learn a model of whether we should carry an umbrella or not.
-
-!!! note
-
-Preprocessing steps are saved after training, and repeated identically for future calls to pgml.predict().
-
-!!!
-
-### `weather_data`
-| **month** | **clouds** | **humidity** | **temp** | **rain** |
-|-----------|------------|--------------|----------|----------|
-| 'jan' | 'cumulus' | 0.8 | 5 | true |
-| 'jan' | NULL | 0.1 | 10 | false |
-| … | … | … | … | … |
-| 'dec' | 'nimbus' | 0.9 | -2 | false |
-
-In this example:
-- `month` is an ordinal categorical `TEXT` variable
-- `clouds` is a nullable nominal categorical `INT4` variable
-- `humidity` is a continuous quantitative `FLOAT4` variable
-- `temp` is a discrete quantitative `INT4` variable
-- `rain` is a nominal categorical `BOOL` label
-
-There are 3 steps to preprocessing data:
-
- - [Encoding](#categorical-encodings) categorical values into quantitative values
- - [Imputing](#imputing-missing-values) NULL values to some quantitative value
- - [Scaling](#scaling-values) quantitative values across all variables to similar ranges
-
-These preprocessing steps may be specified on a per-column basis to the [train()](/docs/guides/training/overview/) function. By default, PostgresML does minimal preprocessing on training data, and will raise an error during analysis if NULL values are encountered without a preprocessor. All types other than `TEXT` are treated as quantitative variables and cast to floating point representations before passing them to the underlying algorithm implementations.
-
-```postgresql title="pgml.train()"
-SELECT pgml.train(
- project_name => 'preprocessed_model',
- task => 'classification',
- relation_name => 'weather_data',
- target => 'rain',
- preprocess => '{
- "month": {"encode": {"ordinal": ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]}}
- "clouds": {"encode": "target", scale: "standard"}
- "humidity": {"impute": "mean", scale: "standard"}
- "temp": {"scale": "standard"}
- }'
-);
-```
-
-In some cases, it may make sense to use multiple steps for a single column. For example, the `clouds` column will be target encoded, and then scaled to the standard range to avoid dominating other variables, but there are some interactions between preprocessors to keep in mind.
-
-- `NULL` and `NaN` are treated as additional, independent categories if seen during training, so columns that `encode` will only ever `impute` novel when novel data is encountered during training values.
-- It usually makes sense to scale all variables to the same scale.
-- It does not usually help to scale or preprocess the target data, as that is essentially the problem formulation and/or task selection.
-
-!!! note
-
-`TEXT` is used in this document to also refer to `VARCHAR` and `CHAR(N)` types.
-
-!!!
-
-## Predicting with Preprocessors
-
-A model that has been trained with preprocessors should use a Postgres tuple for prediction, rather than a `FLOAT4[]`. Tuples may contain multiple different types (like `TEXT` and `BIGINT`), while an ARRAY may only contain a single type. You can use parenthesis around values to create a Postgres tuple.
-
-```postgresql title="pgml.predict()"
-SELECT pgml.predict('preprocessed_model', ('jan', 'nimbus', 0.5, 7));
-```
-
-## Categorical encodings
-Encoding categorical variables is an O(N log(M)) where N is the number of rows, and M is the number of distinct categories.
-
-| **name** | **description** |
-|-----------|-------------------------------------------------------------------------------------------------------------------------------------------------|
-| `none` | **Default** - Casts the variable to a 32-bit floating point representation compatible with numerics. This is the default for non-`TEXT` values. |
-| `target` | Encodes the variable as the average value of the target label for all members of the category. This is the default for `TEXT` variables. |
-| `one_hot` | Encodes the variable as multiple independent boolean columns. |
-| `ordinal` | Encodes the variable as integer values provided by their position in the input array. NULLS are always 0. |
-
-### `target` encoding
-Target encoding is a relatively efficient way to represent a categorical variable. The average value of the target is computed for each category in the training data set. It is reasonable to `scale` target encoded variables using the same method as other variables.
-
-```
-preprocess => '{
- "clouds": {"encode": "target" }
-}'
-```
-
-!!! note
-
-Target encoding is currently limited to the first label column specified in a joint optimization model when there are multiple labels.
-
-!!!
-
-### `one_hot` encoding
-One-hot encoding converts each category into an independent boolean column, where all columns are false except the one column the instance is a member of. This is generally not as efficient or as effective as target encoding because the number of additional columns for a single feature can swamp the other features, regardless of scaling in some algorithms. In addition, the columns are highly correlated which can also cause quality issues in some algorithms. PostgresML drops one column by default to break the correlation but preserves the information, which is also referred to as dummy encoding.
-
-```
-preprocess => '{
- "clouds": {"encode": "one_hot" }
-}
-```
-
-!!! note
-
-All one-hot encoded data is scaled from 0-1 by definition, and will not be further scaled, unlike the other encodings which are scaled.
-
-!!!
-
-### `ordinal` encoding
-Some categorical variables have a natural ordering, like months of the year, or days of the week that can be effectively treated as a discrete quantitative variable. You may set the order of your categorical values, by passing an exhaustive ordered array. e.g.
-
-```
-preprocess => '{
- "month": {"encode": {"ordinal": ["jan", "feb", "mar", "apr", "may", "jun", "jul", "aug", "sep", "oct", "nov", "dec"]}}
-}
-```
-
-## Imputing missing values
-`NULL` and `NaN` values can be replaced by several statistical measures observed in the training data.
-
-| **name** | **description** |
-|----------|---------------------------------------------------------------------------------------|
-| `error` | **Default** - will abort training or inference when a `NULL` or `NAN` is encountered |
-| `mean` | the mean value of the variable in the training data set |
-| `median` | the middle value of the variable in the sorted training data set |
-| `mode` | the most common value of the variable in the training data set |
-| `min` | the minimum value of the variable in the training data set |
-| `max` | the maximum value of the variable in the training data set |
-| `zero` | replaces all missing values with 0.0 |
-
-
-!!! example
-
-```
-preprocess => '{
- "temp": {"impute": "mean"}
-}'
-```
-
-!!!
-
-## Scaling values
-Scaling all variables to a standardized range can help make sure that no feature dominates the model, strictly because it has a naturally larger scale.
-
-| **name** | **description** |
-|------------|-----------------------------------------------------------------------------------------------------------------------|
-| `preserve` | **Default** - Does not scale the variable at all. |
-| `standard` | Scales data to have a mean of zero, and variance of one. |
-| `min_max` | Scales data from zero to one. The minimum becomes 0.0 and maximum becomes 1.0. |
-| `max_abs` | Scales data from -1.0 to +1.0. Data will not be centered around 0, unless abs(min) == abs(max). |
-| `robust` | Scales data as a factor of the first and third quartiles. This method may handle outliers more robustly than others. |
-
-!!! example
-
-```
-preprocess => '{
- "temp": {"scale": "standard"}
-}'
-```
-
-!!!
-
diff --git a/pgml-dashboard/content/docs/guides/transformers/embeddings.md b/pgml-dashboard/content/docs/guides/transformers/embeddings.md
deleted file mode 100644
index 1f0bf810c..000000000
--- a/pgml-dashboard/content/docs/guides/transformers/embeddings.md
+++ /dev/null
@@ -1,80 +0,0 @@
-# Embeddings
-Embeddings are a numeric representation of text. They are used to represent words and sentences as vectors, an array of numbers. Embeddings can be used to find similar pieces of text, by comparing the similarity of the numeric vectors using a distance measure, or they can be used as input features for other machine learning models, since most algorithms can't use text directly.
-
-Many pretrained LLMs can be used to generate embeddings from text within PostgresML. You can browse all the [models](https://huggingface.co/models?library=sentence-transformers) available to find the best solution on Hugging Face.
-
-PostgresML provides a simple interface to generate embeddings from text in your database. You can use the `pgml.embed` function to generate embeddings for a column of text. The function takes a transformer name and a text value. The transformer will automatically be downloaded and cached for reuse.
-
-## Long Form Examples
-For a deeper dive, check out the following articles we've written illustrating the use of embeddings:
-
-- [Generating LLM embeddings in the database with open source models](/blog/generating-llm-embeddings-with-open-source-models-in-postgresml)
-- [Tuning vector recall while generating query embeddings on the fly](/blog/tuning-vector-recall-while-generating-query-embeddings-in-the-database)
-
-## API
-
-```sql linenums="1" title="embed.sql"
-pgml.embed(
- transformer TEXT, -- huggingface sentence-transformer name
- text TEXT, -- input to embed
- kwargs JSON -- optional arguments (see below)
-)
-```
-
-## Example
-
-Let's use the `pgml.embed` function to generate embeddings for tweets, so we can find similar ones. We will use the `distilbert-base-uncased` model. This model is a small version of the `bert-base-uncased` model. It is a good choice for short texts like tweets.
-To start, we'll load a dataset that provides tweets classified into different topics.
-```postgresql linenums="1"
-SELECT pgml.load_dataset('tweet_eval', 'sentiment');
-```
-
-View some tweets and their topics.
-```postgresql linenums="1"
-SELECT *
-FROM pgml.tweet_eval
-LIMIT 10;
-```
-
-Get a preview of the embeddings for the first 10 tweets. This will also download the model and cache it for reuse, since it's the first time we've used it.
-```postgresql linenums="1"
-SELECT text, pgml.embed('distilbert-base-uncased', text)
-FROM pgml.tweet_eval
-LIMIT 10;
-```
-
-
-It will take a few minutes to generate the embeddings for the entire dataset. We'll save the results to a new table.
-```postgresql linenums="1"
-CREATE TABLE tweet_embeddings AS
-SELECT text, pgml.embed('distilbert-base-uncased', text) AS embedding
-FROM pgml.tweet_eval;
-```
-
-Now we can use the embeddings to find similar tweets. We'll use the `pgml.cosign_similarity` function to find the tweets that are most similar to a given tweet (or any other text input).
-
-```postgresql linenums="1"
-WITH query AS (
- SELECT pgml.embed('distilbert-base-uncased', 'Star Wars christmas special is on Disney') AS embedding
-)
-SELECT text, pgml.cosine_similarity(tweet_embeddings.embedding, query.embedding) AS similarity
-FROM tweet_embeddings, query
-ORDER BY similarity DESC
-LIMIT 50;
-```
-
-On small datasets (<100k rows), a linear search that compares every row to the query will give sub-second results, which may be fast enough for your use case. For larger datasets, you may want to consider various indexing strategies offered by additional extensions.
-
-- [Cube](https://www.postgresql.org/docs/current/cube.html) is a built-in extension that provides a fast indexing strategy for finding similar vectors. By default it has an arbitrary limit of 100 dimensions, unless Postgres is compiled with a larger size.
-- [PgVector](https://github.com/pgvector/pgvector) supports embeddings up to 2000 dimensions out of the box, and provides a fast indexing strategy for finding similar vectors.
-
-```
-CREATE EXTENSION vector;
-CREATE TABLE items (text TEXT, embedding VECTOR(768));
-INSERT INTO items SELECT text, embedding FROM tweet_embeddings;
-CREATE INDEX ON items USING ivfflat (embedding vector_cosine_ops);
-WITH query AS (
- SELECT pgml.embed('distilbert-base-uncased', 'Star Wars christmas special is on Disney')::vector AS embedding
-)
-SELECT * FROM items, query ORDER BY items.embedding <=> query.embedding LIMIT 10;
-```
diff --git a/pgml-dashboard/content/docs/guides/transformers/fine_tuning.md b/pgml-dashboard/content/docs/guides/transformers/fine_tuning.md
deleted file mode 100644
index e172f8fed..000000000
--- a/pgml-dashboard/content/docs/guides/transformers/fine_tuning.md
+++ /dev/null
@@ -1,461 +0,0 @@
-# Fine Tuning
-
-Pre-trained models allow you to get up and running quickly, but you can likely improve performance on your dataset by fine tuning them. Normally, you'll bring your own data to the party, but for these examples we'll use datasets published on Hugging Face. Make sure you've installed the required data dependencies detailed in [setup](/docs/user_guides/transformers/setup).
-
-## Translation Example
-The [Helsinki-NLP](https://huggingface.co/Helsinki-NLP) organization provides more than a thousand pre-trained models to translate between different language pairs. These can be further fine tuned on additional datasets with domain specific vocabulary. Researchers have also created large collections of documents that have been manually translated across languages by experts for training data.
-
-### Prepare the data
-The [kde4](https://huggingface.co/datasets/kde4) dataset contains many language pairs. Subsets can be loaded into your Postgres instance with a call to `pgml.load_dataset`, or you may wish to create your own fine tuning dataset with vocabulary specific to your domain.
-
-```postgresql
-SELECT pgml.load_dataset('kde4', kwargs => '{"lang1": "en", "lang2": "es"}');
-```
-
-You can view the newly loaded data in your Postgres database:
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.kde4 LIMIT 5;
-```
-
-=== "Result"
-
-```postgresql
-id |
-translation
-
------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-99 | {"en": "If you wish to manipulate the DOM tree in any way you will have to use an external script to do so.", "es": "Si desea manipular el árbol DOM deberá utilizar un script externo para hacerlo."}
-100 | {"en": "Credits", "es": "Créditos"}
-101 | {"en": "The domtreeviewer plugin is Copyright & copy; 2001 The Kafka Team/ Andreas Schlapbach kde-kafka@master. kde. org schlpbch@unibe. ch", "es": "Derechos de autor de la extensión domtreeviewer & copy;. 2001. El equipo de Kafka/ Andreas Schlapbach kde-kafka@master. kde. org schlpbch@unibe. ch."}
-102 | {"en": "Josef Weidendorfer Josef. Weidendorfer@gmx. de", "es": "Josef Weidendorfer Josef. Weidendorfer@gmx. de"}
-103 | {"en": "ROLES_OF_TRANSLATORS", "es": "Rafael Osuna rosuna@wol. es Traductor"}
-(5 rows)
-```
-
-===
-
-This huggingface dataset stores the data as language key pairs in a JSON document. To use it with PostgresML, we'll need to provide a `VIEW` that structures the data into more primitively typed columns.
-
-=== "SQL"
-
-```postgresql
-CREATE OR REPLACE VIEW kde4_en_to_es AS
-SELECT translation->>'en' AS "en", translation->>'es' AS "es"
-FROM pgml.kde4
-LIMIT 10;
-```
-
-=== "Result"
-
-```
-CREATE VIEW
-```
-
-===
-
-Now, we can see the data in more normalized form. The exact column names don't matter for now, we'll specify which one is the target during the training call, and the other one will be used as the input.
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM kde4_en_to_es LIMIT 10;
-```
-
-=== "Result"
-
-```postgresql
- en | es
-
---------------------------------------------------------------------------------------------+--------------------------------------------------------------------------
-------------------------------
- Lauri Watts | Lauri Watts
- & Lauri. Watts. mail; | & Lauri. Watts. mail;
- ROLES_OF_TRANSLATORS | Rafael Osuna rosuna@wol. es Traductor Miguel Revilla Rodríguez yo@miguelr
-evilla. com Traductor
- 2006-02-26 3.5.1 | 2006-02-26 3.5.1
- The Babel & konqueror; plugin gives you quick access to the Babelfish translation service. | La extensión Babel de & konqueror; le permite un acceso rápido al servici
-o de traducción de Babelfish.
- KDE | KDE
- kdeaddons | kdeaddons
- konqueror | konqueror
- plugins | extensiones
- babelfish | babelfish
-(10 rows)
-```
-
-===
-
-
-### Tune the model
-Tuning is very similar to training with PostgresML, although we specify a `model_name` to download from Hugging Face instead of the base `algorithm`.
-
-```postgresql
-SELECT pgml.tune(
- 'Translate English to Spanish',
- task => 'translation',
- relation_name => 'kde4_en_to_es',
- y_column_name => 'es', -- translate into spanish
- model_name => 'Helsinki-NLP/opus-mt-en-es',
- hyperparams => '{
- "learning_rate": 2e-5,
- "per_device_train_batch_size": 16,
- "per_device_eval_batch_size": 16,
- "num_train_epochs": 1,
- "weight_decay": 0.01,
- "max_length": 128
- }',
- test_size => 0.5,
- test_sampling => 'last'
-);
-```
-
-### Generate Translations
-
-!!! note
-
-Translations use the `pgml.generate` API since they return `TEXT` rather than numeric values. You may also call `pgml.generate` with a `TEXT[]` for batch processing.
-
-!!!
-
-=== "SQL"
-
-```postgresql
-
-SELECT pgml.generate('Translate English to Spanish', 'I love SQL')
-AS spanish;
-```
-
-=== "Result"
-
-```postgresql
- spanish
-----------------
-Me encanta SQL
-(1 row)
-
-Time: 126.837 ms
-```
-
-===
-
-See the [task documentation](https://huggingface.co/tasks/translation) for more examples, use cases, models and datasets.
-
-
-## Text Classification Example
-
-DistilBERT is a small, fast, cheap and light Transformer model based on the BERT architecture. It can be fine tuned on specific datasets to learn further nuance between positive and negative examples. For this example, we'll fine tune `distilbert-base-uncased` on the IMBD dataset, which is a list of movie reviews along with a positive or negative label.
-
-Without tuning, DistilBERT classifies every single movie review as `positive`, and has a F1 score of 0.367, which is about what you'd expect for a relatively useless classifier. However, after training for a single epoch (takes about 10 minutes on an Nvidia 1080 TI), the F1 jumps to 0.928 which is a huge improvement, indicating DistilBERT can now fairly accurately predict sentiment from IMDB reviews. Further training for another epoch only results in a very minor improvement to 0.931, and the 3rd epoch is flat, also at 0.931 which indicates DistilBERT is unlikely to continue learning more about this particular dataset with additional training. You can view the results of each model, like those trained from scratch, in the dashboard.
-
-Once our model has been fine tuned on the dataset, it'll be saved and deployed with a Project visible in the Dashboard, just like models built from simpler algorithms.
-
-
-
-### Prepare the data
-The IMDB dataset has 50,000 examples of user reviews with positive or negative viewing experiences as the labels, and is split 50/50 into training and evaluation datasets.
-
-```postgresql
-SELECT pgml.load_dataset('imdb');
-```
-
-You can view the newly loaded data in your Postgres database:
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.imdb LIMIT 1;
-```
-
-=== "Result"
-
-```postgresql
- text | label
------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-------
- This has to be the funniest stand up comedy I have ever seen. Eddie Izzard is a genius, he picks in Brits, Americans and everyone in between. His style is completely natural and completely hilarious. I doubt that anyone could sit through this and not laugh their a** off. Watch, enjoy, it's funny. | 1
-(1 row)
-```
-
-===
-
-### Tune the model
-
-Tuning has a nearly identical API to training, except you may pass the name of a [model published on Hugging Face](https://huggingface.co/models) to start with, rather than training an algorithm from scratch.
-
-```postgresql
-SELECT pgml.tune(
- 'IMDB Review Sentiment',
- task => 'text-classification',
- relation_name => 'pgml.imdb',
- y_column_name => 'label',
- model_name => 'distilbert-base-uncased',
- hyperparams => '{
- "learning_rate": 2e-5,
- "per_device_train_batch_size": 16,
- "per_device_eval_batch_size": 16,
- "num_train_epochs": 1,
- "weight_decay": 0.01
- }',
- test_size => 0.5,
- test_sampling => 'last'
-);
-```
-
-### Make predictions
-
-=== "SQL"
-
-```postgresql
-SELECT pgml.predict('IMDB Review Sentiment', 'I love SQL')
-AS sentiment;
-```
-
-=== "Result"
-
-```
-sentiment
------------
-1
-(1 row)
-
-Time: 16.681 ms
-```
-
-===
-
-The default for predict in a classification problem classifies the statement as one of the labels. In this case, 0 is negative and 1 is positive. If you'd like to check the individual probabilities associated with each class you can use the `predict_proba` API:
-
-=== "SQL"
-
-```postgresql
-SELECT pgml.predict_proba('IMDB Review Sentiment', 'I love SQL')
-AS sentiment;
-```
-
-=== "Result"
-
-```
- sentiment
--------------------------------------------
-[0.06266672909259796, 0.9373332858085632]
-(1 row)
-
-Time: 18.101 ms
-```
-
-===
-
-This shows that there is a 6.26% chance for category 0 (negative sentiment), and a 93.73% chance it's category 1 (positive sentiment).
-
-See the [task documentation](https://huggingface.co/tasks/text-classification) for more examples, use cases, models and datasets.
-
-## Summarization Example
-At a high level, summarization uses similar techniques to translation. Both use an input sequence to generate an output sequence. The difference being that summarization extracts the most relevant parts of the input sequence to generate the output.
-
-### Prepare the data
-[BillSum](https://huggingface.co/datasets/billsum) is a dataset with training examples that summarize US Congressional and California state bills. You can pass `kwargs` specific to loading datasets, in this case we'll restrict the dataset to California samples:
-
-```postgresql
-SELECT pgml.load_dataset('billsum', kwargs => '{"split": "ca_test"}');
-```
-
-You can view the newly loaded data in your Postgres database:
-
-=== "SQL"
-
-```postgresql
-SELECT * FROM pgml.billsum LIMIT 1;
-```
-
-=== "Result"
-
-```
- text | summary | title
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+-----------------------------------------------------------------------------------------------------------------------------
-The people of the State of California do enact as follows: +| Existing property tax law establishes a veterans’ organization exemption under which property is exempt from taxation if, among other things, that property is used exclusively for charitable purposes and is owned by a veterans’ organization. +| An act to amend Section 215.1 of the Revenue and Taxation Code, relating to taxation, to take effect immediately, tax levy.
- +| This bill would provide that the veterans’ organization exemption shall not be denied to a property on the basis that the property is used for fraternal, lodge, or social club purposes, and would make specific findings and declarations in that regard. The bill would also provide that the exemption shall not apply to any portion of a property that consists of a bar where alcoholic beverages are served.+|
- +| Section 2229 of the Revenue and Taxation Code requires the Legislature to reimburse local agencies annually for certain property tax revenues lost as a result of any exemption or classification of property for purposes of ad valorem property taxation. +|
-SECTION 1. +| This bill would provide that, notwithstanding Section 2229 of the Revenue and Taxation Code, no appropriation is made and the state shall not reimburse local agencies for property tax revenues lost by them pursuant to the bill. +|
-The Legislature finds and declares all of the following: +| This bill would take effect immediately as a tax levy. |
-(a) (1) Since 1899 congressionally chartered veterans’ organizations have provided a valuable service to our nation’s returning service members. These organizations help preserve the memories and incidents of the great hostilities fought by our nation, and preserve and strengthen comradeship among members. +| |
-(2) These veterans’ organizations also own and manage various properties including lodges, posts, and fraternal halls. These properties act as a safe haven where veterans of all ages and their families can gather together to find camaraderie and fellowship, share stories, and seek support from people who understand their unique experiences. This aids in the healing process for these returning veterans, and ensures their health and happiness. +| |
-(b) As a result of congressional chartering of these veterans’ organizations, the United States Internal Revenue Service created a special tax exemption for these organizations under Section 501(c)(19) of the Internal Revenue Code. +| |
-(c) Section 501(c)(19) of the Internal Revenue Code and related federal regulations provide for the exemption for posts or organizations of war veterans, or an auxiliary unit or society of, or a trust or foundation for, any such post or organization that, among other attributes, carries on programs to perpetuate the memory of deceased veterans and members of the Armed Forces and to comfort their survivors, conducts programs for religious, charitable, scientific, literary, or educational purposes, sponsors or participates in activities of a patriotic nature, and provides social and recreational activities for their members. +| |
-(d) Section 215.1 of the Revenue and Taxation Code stipulates that all buildings, support and so much of the real property on which the buildings are situated as may be required for the convenient use and occupation of the buildings, used exclusively for charitable purposes, owned by a veterans’ organization that has been chartered by the Congress of the United States, organized and operated for charitable purposes, when the same are used solely and exclusively for the purpose of the organization, if not conducted for profit and no part of the net earnings of which ensures to the benefit of any private individual or member thereof, are exempt from taxation. +| |
-(e) The Chief Counsel of the State Board of Equalization concluded, based on a 1979 appellate court decision, that only parts of American Legion halls are exempt from property taxation and that other parts, such as billiard rooms, card rooms, and similar areas, are not exempt. +| |
-(f) In a 1994 memorandum, the State Board of Equalization’s legal division further concluded that the areas normally considered eligible for exemptions are the office areas used to counsel veterans and the area used to store veterans’ records, but that the meeting hall and bar found in most of the facilities are not considered used for charitable purposes. +| |
-(g) Tax-exempt status is intended to provide economic incentive and support to veterans’ organizations to provide for the social welfare of the community of current and former military personnel. +| |
-(h) The State Board of Equalization’s constriction of the tax exemption has resulted in an onerous tax burden on California veteran service organizations posts or halls, hinders the posts’ ability to provide facilities for veterans, and threatens the economic viability of many local organizations. +| |
-(i) The charitable activities of a veteran service organizations post or hall are much more than the counseling of veterans. The requirements listed for qualification for the federal tax exemption clearly dictate a need for more than just an office. +| |
-(j) Programs to perpetuate the memory of deceased veterans and members of the Armed Forces and to comfort their survivors require the use of facilities for funerals and receptions. +| |
-(k) Programs for religious, charitable, scientific, literary, or educational purposes require space for more than 50 attendees. +| |
-(l) Activities of a patriotic nature need facilities to accommodate hundreds of people. +| |
-(m) Social and recreational activities for members require precisely those areas considered “not used for charitable purposes” by the State Board of Equalization. +| |
-(n) The State Board of Equalization’s interpretation of the Revenue and Taxation Code reflects a lack of understanding of the purpose and programs of the veterans service organizations posts or halls and is detrimental to the good works performed in support of our veteran community.
-
- +|
- +| (g) Tax-exempt status is intended to provide economic incentive and support to veterans’ organizations to provide for the social welfare of the community of current and former military personnel. +| |
-(h) The State Board of Equalization’s constriction of the tax exemption has resulted in an onerous tax burden on California veteran service organizations posts or halls, hinders the posts’ ability to provide facilities for veterans, and threatens the economic viability of many local organizations. +| |
-(i) The charitable activities of a veteran service organizations post or hall are much more than the counseling of veterans. The requirements listed for qualification for the federal tax exemption clearly dictate a need for more than just an office. +| |
-(j) Programs to perpetuate the memory of deceased veterans and members of the Armed Forces and to comfort their survivors require the use of facilities for funerals and receptions. +| |
-(k) Programs for religious, charitable, scientific, literary, or educational purposes require space for more than 50 attendees. +| |
-(l) Activities of a patriotic nature need facilities to accommodate hundreds of people. +| |
-(m) Social and recreational activities for members require precisely those areas considered “not used for charitable purposes” by the State Board of Equalization. +| |
-(n) The State Board of Equalization’s interpretation of the Revenue and Taxation Code reflects a lack of understanding of the purpose and programs of the veterans service organizations posts or halls and is detrimental to the good works performed in support of our veteran community. +| |
-SECTION 1. +| |
-SEC. 2. +| |
-Section 215.1 of the Revenue and Taxation Code is amended to read: +| |
-215.1. +| |
-(a) All buildings, and so much of the real property on which the buildings are situated as may be required for the convenient use and occupation of the buildings, used exclusively for charitable purposes, owned by a veterans’ organization that has been chartered by the Congress of the United States, organized and operated for charitable purposes, and exempt from federal income tax as an organization described in Section 501(c)(19) of the Internal Revenue Code when the same are used solely and exclusively for the purpose of the organization, if not conducted for profit and no part of the net earnings of which inures to the benefit of any private individual or member thereof, shall be exempt from taxation.+| |
-(b) The exemption provided for in this section shall apply to the property of all organizations meeting the requirements of this section, subdivision (b) of Section 4 of Article XIII of the California Constitution, and paragraphs (1) to (4), inclusive, (6), and (7) of subdivision (a) of Section 214. +| |
-(c) (1) The exemption specified by subdivision (a) shall not be denied to a property on the basis that the property is used for fraternal, lodge, or social club purposes. +| |
-(2) With regard to this subdivision, the Legislature finds and declares all of the following: +| |
-(A) The exempt activities of a veterans’ organization as described in subdivision (a) qualitatively differ from the exempt activities of other nonprofit entities that use property for fraternal, lodge, or social club purposes in that the exempt purpose of the veterans’ organization is to conduct programs to perpetuate the memory of deceased veterans and members of the Armed Forces and to comfort their survivors, to conduct programs for religious, charitable, scientific, literary, or educational purposes, to sponsor or participate in activities of a patriotic nature, and to provide social and recreational activities for their members. +| |
-(B) In light of this distinction, the use of real property by a veterans’ organization as described in subdivision (a), for fraternal, lodge, or social club purposes is central to that organization’s exempt purposes and activities. +| |
-(C) In light of the factors set forth in subparagraphs (A) and (B), the use of real property by a veterans’ organization as described in subdivision (a) for fraternal, lodge, or social club purposes, constitutes the exclusive use of that property for a charitable purpose within the meaning of subdivision (b) of Section 4 of Article XIII of the California Constitution. +| |
-(d) The exemption provided for in this section shall not apply to any portion of a property that consists of a bar where alcoholic beverages are served. The portion of the property ineligible for the veterans’ organization exemption shall be that area used primarily to prepare and serve alcoholic beverages. +| |
-(e) An organization that files a claim for the exemption provided for in this section shall file with the assessor a valid organizational clearance certificate issued pursuant to Section 254.6. +| |
-(f) This exemption shall be known as the “veterans’ organization exemption.”
-
- +|
- |
-SEC. 2.
-
- +|
- |
-SEC. 3.
-
- +|
- |
-Notwithstanding Section 2229 of the Revenue and Taxation Code, no appropriation is made by this act and the state shall not reimburse any local agency for any property tax revenues lost by it pursuant to this act.
-
- +|
- |
-SEC. 3.
-
- +|
- |
-SEC. 4.
-
- +|
- |
-This act provides for a tax levy within the meaning of Article IV of the Constitution and shall go into immediate effect.
-
- |
- |
-(1 row)
-```
-
-===
-
-This dataset has 3 fields, but summarization transformers only take a single input to produce their output. We can create a view that simply omits the `title` from the training data:
-
-```postgresql
-CREATE OR REPLACE VIEW billsum_training_data
-AS SELECT "text", summary FROM pgml.billsum;
-```
-
-Or, it might be interesting to concat the title to the text field to see how relevant it actually is to the bill. If the title of a bill is the first sentence, and doesn't appear in summary, it may indicate that it's a poorly chosen title for the bill:
-
-```postgresql
-CREATE OR REPLACE VIEW billsum_training_data
-AS SELECT title || '\n' || "text" AS "text", summary FROM pgml.billsum
-LIMIT 10;
-```
-
-### Tune the model
-
-Tuning has a nearly identical API to training, except you may pass the name of a [model published on Hugging Face](https://huggingface.co/models) to start with, rather than training an algorithm from scratch.
-
-```postgresql
-SELECT pgml.tune(
- 'Legal Summarization',
- task => 'summarization',
- relation_name => 'billsum_training_data',
- y_column_name => 'summary',
- model_name => 'sshleifer/distilbart-xsum-12-1',
- hyperparams => '{
- "learning_rate": 2e-5,
- "per_device_train_batch_size": 2,
- "per_device_eval_batch_size": 2,
- "num_train_epochs": 1,
- "weight_decay": 0.01,
- "max_length": 1024
- }',
- test_size => 0.2,
- test_sampling => 'last'
-);
-```
-
-
-### Make predictions
-
-=== "SQL"
-
-```postgresql
-SELECT pgml.predict('IMDB Review Sentiment', 'I love SQL') AS sentiment;
-```
-
-=== "Result"
-
-```
-sentiment
------------
-1
-(1 row)
-
-Time: 16.681 ms
-```
-
-===
-
-The default for predict in a classification problem classifies the statement as one of the labels. In this case 0 is negative and 1 is positive. If you'd like to check the individual probabilities associated with each class you can use the `predict_proba` API
-
-=== "SQL"
-
-```postgresql
-SELECT pgml.predict_proba('IMDB Review Sentiment', 'I love SQL') AS sentiment;
-```
-
-=== "Result"
-
-```
- sentiment
--------------------------------------------
-[0.06266672909259796, 0.9373332858085632]
-(1 row)
-
-Time: 18.101 ms
-```
-
-===
-
-This shows that there is a 6.26% chance for category 0 (negative sentiment), and a 93.73% chance it's category 1 (positive sentiment).
-
-See the [task documentation](https://huggingface.co/tasks/text-classification) for more examples, use cases, models and datasets.
-
-
-
-## Text Generation
-
-```postgresql
-SELECT pgml.load_dataset('bookcorpus', "limit" => 100);
-
-SELECT pgml.tune(
- 'GPT Generator',
- task => 'text-generation',
- relation_name => 'pgml.bookcorpus',
- y_column_name => 'text',
- model_name => 'gpt2',
- hyperparams => '{
- "learning_rate": 2e-5,
- "num_train_epochs": 1
- }',
- test_size => 0.2,
- test_sampling => 'last'
-);
-
-SELECT pgml.generate('GPT Generator', 'While I wandered weak and weary');
-```
diff --git a/pgml-dashboard/content/docs/guides/transformers/pre_trained_models.md b/pgml-dashboard/content/docs/guides/transformers/pre_trained_models.md
deleted file mode 100644
index 7f164e2dc..000000000
--- a/pgml-dashboard/content/docs/guides/transformers/pre_trained_models.md
+++ /dev/null
@@ -1,228 +0,0 @@
-
-# Pre-Trained Models
-PostgresML integrates [🤗 Hugging Face Transformers](https://huggingface.co/transformers) to bring state-of-the-art models into the data layer. There are tens of thousands of pre-trained models with pipelines to turn raw inputs into useful results. Many state of the art deep learning architectures have been published and made available for download. You will want to browse all the [models](https://huggingface.co/models) available to find the perfect solution for your [dataset](https://huggingface.co/dataset) and [task](https://huggingface.co/tasks).
-
-We'll demonstrate some of the tasks that are immediately available to users of your database upon installation: [translation](#translation), [sentiment analysis](#sentiment-analysis), [summarization](#summarization), [question answering](#question-answering) and [text generation](#text-generation).
-
-## Examples
-All of the tasks and models demonstrated here can be customized by passing additional arguments to the `Pipeline` initializer or call. You'll find additional links to documentation in the examples below.
-
-The Hugging Face [`Pipeline`](https://huggingface.co/docs/transformers/main_classes/pipelines) API is exposed in Postgres via:
-
-```sql linenums="1" title="transformer.sql"
-pgml.transform(
- task TEXT OR JSONB, -- task name or full pipeline initializer arguments
- call JSONB, -- additional call arguments alongside the inputs
- inputs TEXT[] OR BYTEA[] -- inputs for inference
-)
-```
-
-This is roughly equivalent to the following Python:
-
-```python
-import transformers
-
-def transform(task, call, inputs):
- return transformers.pipeline(**task)(inputs, **call)
-```
-
-Most pipelines operate on `TEXT[]` inputs, but some require binary `BYTEA[]` data like audio classifiers. `inputs` can be `SELECT`ed from tables in the database, or they may be passed in directly with the query. The output of this call is a `JSONB` structure that is task specific. See the [Postgres JSON](https://www.postgresql.org/docs/14/functions-json.html) reference for ways to process this output dynamically.
-
-!!! tip
-
-Models will be downloaded and stored locally on disk after the first call. They are also cached per connection to improve repeated calls in a single session. To free that memory, you'll need to close your connection. You may want to establish dedicated credentials and connection pools via [pgcat](https://github.com/levkk/pgcat) or [pgbouncer](https://www.pgbouncer.org/) for larger models that have billions of parameters. You may also pass `{"cache": false}` in the JSON `call` args to prevent this behavior.
-
-!!!
-
-### Translation
-There are thousands of different pre-trained translation models between language pairs. They generally take a single input string in the "from" language, and translate it into the "to" language as a result of the call. PostgresML transformations provide a batch interface where you can pass an array of `TEXT` to process in a single call for efficiency. Not all language pairs have a default task name like this example of English to French. In those cases, you'll need to specify [the desired model](https://huggingface.co/models?pipeline_tag=translation) by name. You can see how to specify a model in the [next example](#sentiment-analysis). Because this is a batch call with 2 inputs, we'll get 2 outputs in the JSONB.
-
-For a translation from English to French with the default pre-trained model:
-
-=== "SQL"
-
-```sql linenums="1"
-SELECT pgml.transform(
- 'translation_en_to_fr',
- inputs => ARRAY[
- 'Welcome to the future!',
- 'Where have you been all this time?'
- ]
-) AS french;
-```
-
-=== "Result"
-
-```sql linenums="1"
- french
-------------------------------------------------------------
-[
- {"translation_text": "Bienvenue à l'avenir!"},
- {"translation_text": "Où êtes-vous allé tout ce temps?"}
-]
-```
-
-===
-
-See [translation documentation](https://huggingface.co/docs/transformers/tasks/translation) for more options.
-
-### Sentiment Analysis
-Sentiment analysis is one use of `text-classification`, but there are [many others](https://huggingface.co/tasks/text-classification). This model returns both a label classification `["POSITIVE", "NEUTRAL", "NEGATIVE"]`, as well as the score where 0.0 is perfectly negative, and 1.0 is perfectly positive. This example demonstrates specifying the `model` to be used rather than the task. The [`roberta-large-mnli`](https://huggingface.co/roberta-large-mnli) model specifies the task of `sentiment-analysis` in it's default configuration, so we may omit it from the parameters. Because this is a batch call with 2 inputs, we'll get 2 outputs in the JSONB.
-
-=== "SQL"
-
-```sql linenums="1"
-SELECT pgml.transform(
- '{"model": "roberta-large-mnli"}'::JSONB,
- inputs => ARRAY[
- 'I love how amazingly simple ML has become!',
- 'I hate doing mundane and thankless tasks. ☹️'
- ]
-) AS positivity;
-```
-
-=== "Result"
-
-```sql linenums="1"
- positivity
-------------------------------------------------------
-[
- {"label": "NEUTRAL", "score": 0.8143417835235596},
- {"label": "NEUTRAL", "score": 0.7637073993682861}
-]
-```
-
-===
-
-See [text classification documentation](https://huggingface.co/tasks/text-classification) for more options and potential use cases beyond sentiment analysis. You'll notice the outputs are not great in this example. RoBERTa is a breakthrough model, that demonstrated just how important each particular hyperparameter is for the task and particular dataset regardless of how large your model is. We'll show how to [fine tune](/docs/guides/transformers/fine_tuning/) models on your data in the next step.
-
-### Summarization
-Sometimes we need all the nuanced detail, but sometimes it's nice to get to the point. Summarization can reduce a very long and complex document to a few sentences. One studied application is reducing legal bills passed by Congress into a plain english summary. Hollywood may also need some intelligence to reduce a full synopsis down to a pithy blurb for movies like Inception.
-
-=== "SQL"
-
-```sql linenums="1"
-SELECT pgml.transform(
- 'summarization',
- inputs => ARRAY['
- Dominic Cobb is the foremost practitioner of the artistic science
- of extraction, inserting oneself into a subject''s dreams to
- obtain hidden information without the subject knowing, a concept
- taught to him by his professor father-in-law, Dr. Stephen Miles.
- Dom''s associates are Miles'' former students, who Dom requires
- as he has given up being the dream architect for reasons he
- won''t disclose. Dom''s primary associate, Arthur, believes it
- has something to do with Dom''s deceased wife, Mal, who often
- figures prominently and violently in those dreams, or Dom''s want
- to "go home" (get back to his own reality, which includes two
- young children). Dom''s work is generally in corporate espionage.
- As the subjects don''t want the information to get into the wrong
- hands, the clients have zero tolerance for failure. Dom is also a
- wanted man, as many of his past subjects have learned what Dom
- has done to them. One of those subjects, Mr. Saito, offers Dom a
- job he can''t refuse: to take the concept one step further into
- inception, namely planting thoughts into the subject''s dreams
- without them knowing. Inception can fundamentally alter that
- person as a being. Saito''s target is Robert Michael Fischer, the
- heir to an energy business empire, which has the potential to
- rule the world if continued on the current trajectory. Beyond the
- complex logistics of the dream architecture of the case and some
- unknowns concerning Fischer, the biggest obstacles in success for
- the team become worrying about one aspect of inception which Cobb
- fails to disclose to the other team members prior to the job, and
- Cobb''s newest associate Ariadne''s belief that Cobb''s own
- subconscious, especially as it relates to Mal, may be taking over
- what happens in the dreams.
- ']
-) AS result;
-```
-
-=== "Result"
-
-```sql linenums="1"
- result
---------------------------------------------------------------------------
-[{"summary_text": "Dominic Cobb is the foremost practitioner of the
-artistic science of extraction . his associates are former students, who
-Dom requires as he has given up being the dream architect . he is also a
-wanted man, as many of his past subjects have learned what Dom has done
-to them ."}]
-```
-
-===
-
-See [summarization documentation](https://huggingface.co/tasks/summarization) for more options.
-
-
-### Question Answering
-Question Answering extracts an answer from a given context. Recent progress has enabled models to also specify if the answer is present in the context at all. If you were trying to build a general question answering system, you could first turn the question into a keyword search against Wikipedia articles, and then use a model to retrieve the correct answer from the top hit. Another application would provide automated support from a knowledge base, based on the customers question.
-
-=== "SQL"
-
-```sql linenums="1"
-SELECT pgml.transform(
- 'question-answering',
- inputs => ARRAY[
- '{
- "question": "Am I dreaming?",
- "context": "I got a good nights sleep last night and started a simple tutorial over my cup of morning coffee. The capabilities seem unreal, compared to what I came to expect from the simple SQL standard I studied so long ago. The answer is staring me in the face, and I feel the uncanny call from beyond the screen to check the results."
- }'
- ]
-) AS answer;
-```
-
-=== "Result"
-
-```sql linenums="1"
- answer
------------------------------------------------------
-{
- "end": 36,
- "score": 0.20027603209018707,
- "start": 0,
- "answer": "I got a good nights sleep last night"
-}
-```
-
-===
-
-See [question answering documentation](https://huggingface.co/tasks/question-answering) for more options.
-
-### Text Generation
-If you need to expand on some thoughts, you can have AI complete your sentences for you:
-
-=== "SQL"
-
-```sql linenums="1"
-SELECT pgml.transform(
- 'text-generation',
- '{"num_return_sequences": 2}',
- ARRAY['Three Rings for the Elven-kings under the sky, Seven for the Dwarf-lords in their halls of stone']
-) AS result;
-```
-
-=== "Result"
-
-```sql linenums="1"
- result
------------------------------------------------------------------------------
-[[
- {
- "generated_text": "Three Rings for the Elven-kings under the sky,
- Seven for the Dwarf-lords in their halls of stone, and five for
- the Elves.\nWhen, from all that's happening, he sees these things,
- he says to himself,"
- },
- {
- "generated_text": "Three Rings for the Elven-kings under the sky,
- Seven for the Dwarf-lords in their halls of stone, Eight for the
- Erogean-kings in their halls of stone -- \"and so forth;\" and
- \"of these"
- }
-]]
-```
-
-===
-
-### More
-There are many different [tasks](https://huggingface.co/tasks) and tens of thousands of state-of-the-art [models](https://huggingface.co/models) available for you to explore. The possibilities are expanding every day. There can be amazing performance improvements in domain specific versions of these general tasks by fine tuning published models on your dataset. See the next section for [fine tuning](/docs/guides/transformers/fine_tuning/) demonstrations.
diff --git a/pgml-dashboard/content/docs/guides/transformers/setup.md b/pgml-dashboard/content/docs/guides/transformers/setup.md
deleted file mode 100644
index 94b81cfa9..000000000
--- a/pgml-dashboard/content/docs/guides/transformers/setup.md
+++ /dev/null
@@ -1,51 +0,0 @@
-# 🤗 Transformers
-PostgresML integrates [🤗 Hugging Face Transformers](https://huggingface.co/transformers) to bring state-of-the-art models into the data layer. There are tens of thousands of pre-trained models with pipelines to turn raw inputs into useful results. Many state of the art deep learning architectures have been published and made available for download. You will want to browse all the [models](https://huggingface.co/models) available to find the perfect solution for your [dataset](https://huggingface.co/dataset) and [task](https://huggingface.co/tasks).
-
-## Setup
-We include all known huggingface model dependencies in [pgml-extension/requirements.txt](https://github.com/postgresml/postgresml/blob/master/pgml-extension/requirements.txt), which is installed in the docker image by default.
-You may also install only the machine learning dependencies on the database for the transformers you would like to use:
-
-=== "PyTorch"
-
-See the [Pytorch docs](https://pytorch.org/) for more information.
-
-```bash
-$ sudo pip3 install torch
-```
-
-=== "Tensorflow"
-
-See the [Tensorflow docs](https://www.tensorflow.org/install/) for more information.
-
-```bash
-$ sudo pip3 install tensorflow
-```
-
-=== "Flax"
-
-See the [Flax docs](https://flax.readthedocs.io/en/latest/installation.html) for more information.
-
-```bash
-$ sudo pip3 install flax
-```
-
-===
-
-Models will be downloaded and cached on the database for repeated usage. View the [Transformers installation docs](https://huggingface.co/docs/transformers/installation) for cache management details and offline deployments.
-
-You may also want to [install GPU support](/docs/guides/setup/gpu_support/) when working with larger models.
-
-## Standard Datasets
-Many datasets have been published to stimulate research and benchmark architectures, but also to help demonstrate API usage in the tutorials. The Datasets package provides a way to load published datasets into Postgres:
-
-```bash
-$ sudo pip3 install datasets
-```
-
-## Audio Processing
-Torch Audio is required for many models that process audio data. You can install the additional dependencies with:
-
-```bash
-$ sudo pip3 install torchaudio
-```
-
diff --git a/pgml-dashboard/content/docs/guides/vector_operations/overview.md b/pgml-dashboard/content/docs/guides/vector_operations/overview.md
deleted file mode 100644
index 992ea0ea5..000000000
--- a/pgml-dashboard/content/docs/guides/vector_operations/overview.md
+++ /dev/null
@@ -1,171 +0,0 @@
-# Vector Operations
-
-PostgresML adds optimized vector operations that can be used inside SQL queries. Vector operations are particularly useful for dealing with embeddings that have been generated from other machine learning algorithms, and can provide functions like nearest neighbor calculations using various distance functions.
-
-Embeddings can be a relatively efficient mechanism to leverage the power of deep learning, without the runtime inference costs. These functions are fast with the most expensive distance functions computing upwards of ~100k per second for a memory resident dataset on modern hardware.
-
-The PostgreSQL planner will also [automatically parallelize](https://www.postgresql.org/docs/current/parallel-query.html) evaluation on larger datasets, if configured to take advantage of multiple CPU cores when available.
-
-Vector operations are implemented in Rust using `ndarray` and BLAS, for maximum performance.
-
-## Element-wise Arithmetic with Constants
-
-Looks like the page you're looking for doesn't exist. It may have been moved, or it never existed, we truly don't know. Try looking in our documentation or in our blog. We're also hanging out in Discord and are happy to answer any questions!
+Looks like the page you're looking for doesn't exist. It may have been moved, or it never existed, we truly don't know. Try looking in our documentation or in our blog. We're also hanging out in Discord and are happy to answer any questions!