* [07/23/2024] [Snowflake Teams Up with Meta to Host and Optimize New Flagship Model Family in Snowflake Cortex AI

* [Achieve Low-Latency and High-Throughput Inference with Meta's Llama 3.1 405B using Snowflake’s Optimized AI Stack](https://www.snowflake.com/engineering-blog/optimize-LLMs-with-llama-snowflake-ai-stack/)

* [Fine-Tune Llama 3.1 405B on a Single Node using Snowflake’s Memory-Optimized AI Stack](https://www.snowflake.com/engineering-blog/fine-tune-llama-single-node-snowflake/)

The Snowflake AI Research team is conducting open, foundational research to advance the field of AI while making enterprise AI easy, efficient, and trusted. This repo contains several artifacts to help efficiently train and inference popular LLMs in practice. We released [Arctic](https://www.snowflake.com/blog/arctic-open-efficient-foundation-language-models-snowflake/) in April of 2023 and are proud to announce the release of our Massive LLM inference and fine-tuning stacks specifically tailored to Llama 3.1 405B.

In collaboration with DeepSpeed, Hugging Face, vLLM, and the broader AI community we are excited to open-source our inference and fine-tuning stacks optimized for Llama 3.1 405B. For inference we support a massive 128K context window from day one, while enabling real-time inference with up to 3x lower end-to-end latency and 1.4x higher throughput than existing open source solutions. Please see our blog, [Achieve Low-Latency and High-Throughput Inference with Meta's Llama 3.1 405B using Snowflake’s Optimized AI Stack](https://www.snowflake.com/engineering-blog/optimize-LLMs-with-llama-snowflake-ai-stack/), that deep dive into all of these innovations. For fine-tuning we support training on a single and multi-node training environments using the latest in memory efficient training techniques such as parameter-efficient fine-tuning, FP8 quantization, ZeRO-3-inspired sharding, and targeted parameter offloading (when necessary). Please see our blog, [Fine-Tune Llama 3.1 405B on a Single Node using Snowflake’s Memory-Optimized AI Stack](https://www.snowflake.com/engineering-blog/fine-tune-llama-single-node-snowflake/), for a deep dive into how we did this.

* [Inference deployment and benchmarks with vLLM](inference/llama3.1)

* [Fine-Tuning Support for Llama 3.1 405B](training/llama3.1)

**Inference API Providers**

**Model Weights**

**Inference**

* [Basic Hugging Face setup](inference/arctic)

* [vLLM Deployment](inference/arctic/vllm/)

**Cookbooks/Tutorials**

This tutorial covers how to use Llama 3.1 405B with vLLM and what performance you should expect when running it. We are actively

working with the vLLM community to upstream Llama 3.1 405B support, but until then please use the repos detailed below.

Hardware assumptions of this tutorial. We are using two 8xH100 instances (i.e., [p5.48xlarge](https://aws.amazon.com/ec2/instance-types/p5/))

for this tutorial but similar hardware should provide similar results.

For the steps going forward we highly recommend that use `hf_transfer` when downloading any of the Llama 3.1 405B checkpoints

from Hugging Face to get the best throughput. On an AWS instance we are seeing the checkpoint will download in about 20-30 minutes. In vLLM

this should be enabled by default if the package is installed (https://github.com/vllm-project/vllm/pull/3817).

We recommend setting up a virtual environment to get all of your dependencies isolated to avoid potential conflicts.

On each node:

virtualenv llama3-venv

source llama3-venv/bin/activate

pip install huggingface_hub[hf_transfer]

pip install git+https://github.com/Snowflake-Labs/vllm.git@llama3-staging-rebase

pip install git+https://github.com/Snowflake-Labs/DeepSpeed.git@add-fp8-gemm

On node 1:

ray start --head

On node 2:

ray start --address <node 1 ip>:6379

On node 1:

pip install dataclasses_json

python benchmark_trace.py \

--backend vllm \

--trace synth-1k.jsonl \

--arrival-rate 1 \

--model meta-llama/Meta-Llama-3.1-405B \

--pipeline-parallel-size 2 \

--tensor-parallel-size 8 \

--enable-chunked-prefill \

--max-num-seqs 64 \

--max-num-batched-tokens 512 \

--gpu-memory-utilization 0.95 \

--use-allgather-pipeline-comm \

--disable-log-requests