DSPy: A Framework for

Programming with LLMs

Authored By

Rohit Sroch
Sr. AI Scientist at AI Labs
C5i
The rapid evolution of artificial intelligence, particularly with the emergence of LLMs, has spurred
the development of several frameworks focused on enhancing interactions with these large
language models, marking a significant technological advancement.

While numerous frameworks concentrate on prompting LLMs, DSPy is a frontrunner in this

advancement, introducing a novel approach to programming LLMs. This article delves deep into
the intricacies of DSPy, analyzing its programming framework, comparing it with other
frameworks, and offering practical demonstrations to highlight its capabilities.

Introduction to DSPy
DSPy, which stands for Declarative Sequencing Python framework, signifies a fundamental
change in how developers engage with LLMs. Conventional approaches involve manually creating
prompts, a task prone to being time-consuming and inaccurate.

The DSPy framework is designed to offer building blocks for developing LLM applications through
programming, similar to frameworks like PyTorch, in which we build neural networks by selecting
required network layers and optimizers to minimize training loss for specific metrics we aim to
optimize and learn model parameters.

But programming with foundational models follows a distinct paradigm, particularly emphasized
in the context of in-context learning. Within this paradigm, we identify tasks, write instructions
using natural language prompts, adjust the wording, give examples as few-shot of desired model
outcomes, provide the right context, and refine as needed.

Challenges faced with prompting LLMs:

1. Complexity of Breaking Down Problems: The process of breaking down problems into
isolated steps and effectively prompting an LLM for each step can be complex and
time-consuming.
2. Integration and Cohesion of Steps: Once individual steps are established, integrating, and
ensuring the smooth operation of these steps together can present challenges, requiring
careful tweaking and adjustments.

3. Manual Identification of Few-shot Examples: The manual identification of few-shot

examples to fine-tune each step can be labor-intensive and may require significant
domain expertise.

4. Maintenance and Adaptability: Maintaining prompt coherence becomes challenging when

there are changes in the pipeline, the LLM itself, or the data, potentially necessitating
frequent updates to prompts or fine-tuning steps which is further time-consuming.

© C5i 01
DSPy tackles these challenges by introducing a programmable interface that allows for
algorithmic optimization of prompts and model weights, particularly beneficial when LLMs are
employed multiple times within a pipeline. This enhances the efficiency and effectiveness of
interactions with language models.

Why DSPy? Comparison to other

Frameworks
To appreciate the uniqueness of DSPy, it is essential to understand how it contrasts with similar
technologies:

LangChain and LlamaIndex: LangChain aims to chain language models for application building
while LlamaIndex focuses on improving search capabilities within texts. However, both
programmable approach offers a distinct advantage in precision and adaptability. DSPy’s
niche is in optimizing prompt construction for better interaction with LLMs.

PyTorch: PyTorch is a comprehensive framework for a wide range of deep learning

applications. In analogy, DSPy offers specialized functionality for working with pre-trained
language models, providing general-purpose modules, optimizers, metrics, etc. for developers
looking to harness the power of natural language processing.

DSPy Programming Components

Training Data

DSPy Program

Optimizer & Metric

Write DSPy Program

Specify Evaluation
Logic & Optimizer
LLMs
Metrics
Signatures Compile
Accuracy DSPy Program
Inline
Collect Exact-Match
Training Data Class-Based DSPy Compiler

Optimizer
Modules
LabeledFewShot
Chain-of-thought
BootstrapFewShot
ReAct
Human
Annotator

Iterate

© C5i 02
At the heart of DSPy’s innovation is its unique programming components, which include several
key components:

Language Models: The framework provides easy access to several large language models like
AzureOpenAI, GoogleVertexAI, Amazon Bedrock, Claude, OpenAI, Mistral, etc. for building LLM
based applications.

Signatures: A declarative specification that outlines the input/output behavior of a DSPy

module which specifies LLM what it needs to do rather than how to do it. While defining
signatures, we provide descriptions of fields, and these field names hold semantic
significance, forming the basis for constructing prompts.

The framework provides two ways to define signature:

• Inline Signature: Defined as a short string for common tasks like question answering,
sentiment classification, etc.

• Class-based Signature: Defined as a custom class for advanced tasks that need more
verbose signature.

Modules: This constitutes the core aspect of the DSPy program responsible for managing
flow logic. DSPy offers pre-built modules for fundamental tasks such as Predict, Chain of
Thought, ReAct, etc. Moreover, we also have the option to craft custom modules and combine
multiple ones as needed.

Metrics: The framework provides common metrics like (accuracy, exact match, F1-score,
etc.) for evaluation and optimization. We can define a custom metric as it is a function that
will take examples from your data, the output of your system and return a score that
quantifies how good the output is.

Optimizers (formerly Teleprompters): Optimizers automatically tune prompts (by adding

examples through the selection procedure) and model parameters (like temperature). Also,
optimizers assess performance based on the metric being optimized. The framework offers
several optimizers like, LabeledFewShot and BootstrapFewShot, etc.

Compiler: The module's instructions are optimized to obtain relevant and efficient examples
for the task at hand. The compiled program can then be saved to disk and reloaded,
functioning similarly to checkpoints.

Note that DSPy is an open-source framework with an active community that is continuously
evolving in terms of more components, further reducing the time/effort required to build LLM
applications.

© C5i 03
 Text to SQL Implementation using DSPy
Let’s consider building a basic LLM application that allows users to ask natural questions
and generate an SQL query based on the provided database schema.

Using Langchain framework

a) Define the access to AzureOpenAI LLM (GPT4)

© C5i 04
b) Define the prompt for the Text to SQL task

c) Define a basic chain

© C5i 05
d) Make Predictions by executing LLM Chain

Using DSPy framework

a) Define the access to AzureOpenAI LLM (GPT4)

© C5i 06
b) Define class-based Signature for our Text to SQL task

c) Define the custom Text to SQL module

© C5i 07
d) Define the optimizer & evaluation metric and load labeled data, which will be used to identify
the few shot examples algorithmically.

e) Compile the DSPy program, which algorithmically tunes the prompt by identifying the best
few-shot examples

© C5i 08
f) Make Predictions by executing DSPy program

The above shows how we can use the Langchain or DSPy framework to implement Text to SQL
tasks. In contrast, DSPy provides the following advantages:
DSPy framework leverages optimizers to algorithmically tune the input prompt using
provided labeled data, significantly reducing the manual effort required for prompt
optimization compared to Langchain.

Unlike Langchain, which requires manual prompt optimization to align the model with the
desired output, DSPy can algorithmically identify the best few-shot examples, enhancing
accuracy without manual intervention.

DSPy framework includes a compiler that seamlessly recompiles the entire pipeline in
response to any changes in the LLM version or type. In contrast, Langchain necessitates
manual prompt retuning when such changes occur.

DSPy's automated processes contrast Langchain's manual methods, making it more efficient
and less labor-intensive to achieve desired model outputs and adapt to model variations.

© C5i 09
 Conclusion
DSPy enables future interaction with LLMs by allowing
users to build LLM applications time-efficiently by
programming (not prompting) with LLMs. Instead of
focusing on hand-crafted prompts that target specific
applications, DSPy has general-purpose modules that
learn to prompt (or finetune) a language model. This
approach to programming LLMs fixes the problem of
too-fragile prompts, as you can recompile the pipeline
when making changes to your code, data, assumptions,
or metric. DSPy will then automatically create new
effective prompts that fit your changes.

References
https://dspy-docs.vercel.app/docs/intro
https://github.com/stanfordnlp/dspy
https://arxiv.org/pdf/2310.03714
https://python.langchain.com/v0.1/docs/get_started/introduction/
https://docs.llamaindex.ai/en/stable/
https://pytorch.org/docs/stable/index.html

© C5i 10
About Us
C5i is a pure-play AI & Analytics provider that combines the
power of human perspective with AI technology to deliver
trustworthy intelligence. The company drives value through a
comprehensive solution set, integrating multifunctional teams
that have technical and business domain expertise with a robust suite of
products, solutions, and accelerators tailored for various horizontal and
industry-specific use cases. At the core, C5i’s focus is to deliver business
impact at speed and scale by driving adoption of AI-assisted decision-making.

C5i caters to some of the world’s largest enterprises, including many Fortune
500 companies. The company’s clients span Technology, Media, and Telecom
(TMT), Pharma & Lifesciences, CPG, Retail, Banking, and other sectors. C5i has
been recognized by leading industry analysts like Gartner and Forrester for its
Analytics and AI capabilities and proprietary AI-based platforms.

www.c5i.ai

© C5i