2023

ChatGPT
Chat Generative Pre-trained Transformer

Submitted By: Under Guidance Of:

TARANG SINGH (2002164) DR. A. CHARAN KUMARI
B.TECH (ELECTRICAL) ASST. PROFESSOR
3RD YEAR DPMNT. OF ELECTRICAL ENGINEERING
D.E.I

Email: tarangsingh03@gmail.com
 ChatGPT

Abstract—ChatGPT (Chat Generative Pre-trained

Transformer) is a chatbot developed by OpenAI and II. ARCHITECTURE OF CHATGPT
launched in November 2022. It is built on top of The architecture of ChatGPT is based on the Transformer
OpenAI's GPT-3 family of large language models and architecture, which was introduced in the paper "Attention
has been fine-tuned (an approach to transfer learning) Is All You Need" by Vaswani et al. (2017). The
using both supervised and reinforcement Transformer architecture uses self-attention mechanisms to
learning techniques. model the relationships between different words in a
sentence or sequence, and has proven to be highly effective
in natural language processing tasks.
I. INTRODUCTION
ChatGPT is an AI language model developed by OpenAI Specifically, ChatGPT is based on a variant of the
that is capable of generating human-like text in response to a Transformer architecture called the GPT (Generative Pre-
wide range of prompts and questions. The model is based on trained Transformer) architecture. The GPT architecture is
the GPT (Generative Pre-trained Transformer) architecture, designed to generate coherent and natural language text
which uses self-attention mechanisms to model the based on a prompt or seed input.
relationships between different words in a sentence or ChatGPT uses a variant of the GPT architecture that has
sequence. been pre-trained on a massive dataset of text from the
internet, using a technique called unsupervised pre-training.
ChatGPT was pre-trained on a massive dataset of diverse During pre-training, the model learns to predict the next
text from the internet using unsupervised learning, which word in a sequence of text, based on the previous words in
involved predicting the next word in a sequence of text the sequence. This process allows the model to learn a deep
based on the previous words. This pre-training process understanding of the structure and patterns of natural
allowed the model to learn a deep understanding of the language.
structure and patterns of natural language.
Once the model has been pre-trained, it can be fine-tuned on
ChatGPT has been used for a wide range of applications, specific natural language processing tasks, such as question
including language translation, text generation, chatbots, answering, text classification, or text generation. During
and more. The model has achieved state-of-the-art fine-tuning, the model is trained on a smaller dataset of
performance on many natural language processing tasks and labeled examples, which allows it to adapt to the specific
has been widely adopted in industry and academia. task at hand.
Overall, the combination of the Transformer architecture,
In addition, OpenAI has released multiple versions of the unsupervised pre-training, and fine-tuning has proven to be
GPT model, including GPT-2 and GPT-3, which have highly effective for a wide range of natural language
increased in size and complexity and have further improved processing tasks, and ChatGPT is one example of the power
the capabilities of the model. These models have been of this approach.
trained on even larger datasets and have achieved even
higher levels of performance, allowing them to generate
even more complex and diverse text. III. ALGORITHMS TO TRAIN CHATGPT
The GPT architecture uses several key algorithms and
The original version of ChatGPT, which was released in techniques to train the model and generate text, including:
2019, was trained on a large corpus of English language
text, and is optimized for generating English language Self-attention: The transformer architecture uses self-
responses. However, since then, OpenAI has released attention to allow the model to focus on different parts of
additional versions of the GPT model that have been trained the input sequence when generating output. This helps the
on other languages, such as Chinese, Arabic, and German. model capture long-term dependencies and context in the
input sequence.
Overall, ChatGPT is a large language model created by
OpenAI. It was trained on a massive dataset of diverse test Pre-training and fine-tuning: ChatGPT, like other language
from the internet, which allows it to generate human-like models, is pre-trained on a large dataset of text using
responses to a wide range of questions and prompts. It can unsupervised learning techniques, such as masked language
understand and generate text in multiple languages and can modeling and next sentence prediction. The pre-trained
assist with a variety of tasks, such as answering questions, model can then be fine-tuned on a smaller dataset of task-
providing recommendations, generating creative writing specific data to adapt it to a particular application.
prompts, and more.
Beam search: When generating text, the model uses a search
algorithm called beam search to find the most likely
sequence of words based on the probabilities assigned by
the model.
 virtually indistinguishable from text written by
Gradient descent: The model is trained using an humans, and has been shown to perform well on a
optimization algorithm called stochastic gradient descent range of natural language processing tasks, including
(SGD), which adjusts the parameters of the model to question answering, language translation, and text
minimize the difference between the predicted output and completion.
the actual output. • Multilingual ChatGPT: OpenAI has also developed
multilingual versions of the ChatGPT model that are
capable of generating text in multiple languages.
IV. DEVELOPMENT OF CHATGPT These models have been trained on large datasets of
The development of ChatGPT involved a combination of text in multiple languages and have shown promising
data results in generating high-quality text in different
collection, model architecture design, and training. languages.

A. Collection of Dataset B. Ongoing Work

The first step in developing ChatGPT was to collect a There is ongoing work on ChatGPT, both by OpenAI
massive dataset of diverse text from the internet. This dataset and by researchers and developers in the wider natural
was used to pre-train the GPT architecture using
language processing community. Some of the current areas
unsupervised learning, which involved predicting the next
of focus include:
word in a sequence of text based on the previous words.
• Scaling up: There is ongoing work on scaling up
ChatGPT and other language models to even larger
B. Fine Tuning sizes and more complex architectures, which is
After pre-training, the model was fine-tuned on specific expected to further improve the quality and diversity
natural language processing tasks, such as question of the text they can generate.
answering, text classification, or text generation. Fine-tuning
involved training the, model on a smaller dataset of labelled • Fine-tuning: Researchers are exploring new
examples, which allowed it to adapt to the specific task at methods for fine-tuning ChatGPT on specific natural
hand. language processing tasks, with the goal of
improving the accuracy and efficiency of the model
Throughout the development process, the model was in various applications.
evaluated on
a variety of metrics, such as accuracy, perplexity, and human • Multimodal learning: There is increasing interest
evaluation. These evaluations helped to identify areas where in combining language models like ChatGPT with
other types of neural networks, such as those that
the model could be improved, and guided the design of new
process images, audio, or other modalities. This is
training strategies and architectures.
expected to enable more sophisticated and integrated
natural language processing systems that can analyze
Overall, the development of ChatGPT involved a and generate text in context with other forms of data.
combination of data collection, model architecture design,
training, and iterative evaluation and refinement, with the • Ethical concerns: There is growing awareness of
goal of creating a highly effective and flexible natural the potential ethical and social implications of large-
language processing model scale language models like ChatGPT, including
concerns about bias, fairness, privacy, and the
impact on employment and society. Researchers are
V. PRGOGRESS OF CHATGPT exploring ways to address these concerns and
develop responsible approaches to the development
A. Progress Till Date and deployment of language models.
The progress of ChatGPT has been rapid and significant • Domain-specific applications: There is increasing
since its initial release in 2019. The following are some of interest in applying ChatGPT and other language
the major milestones and developments in the evolution of models to domain-specific applications, such as
ChatGPT: medical diagnosis, financial analysis, and legal
reasoning. This requires fine-tuning the models on
• ChatGPT-2: In 2019, OpenAI released a larger specific domains and developing specialized
version of the original ChatGPT model called was architectures and training data.
trained on a much larger dataset and was capable of Overall, the ongoing work on ChatGPT and other
generating even more sophisticated and coherent text. language models is expected to lead to continued
ChatGPT-2 achieved impressive results on a range of advances in natural language processing and to open up
natural language processing tasks and generated new possibilities for the use of AI in various domains and
widespread interest and attention in the research applications
community and the media.
• ChatGPT-3: In 2020, OpenAI released ChatGPT-3, C. Future ChatGPT
which is even larger and more complex than The future of ChatGPT is expected to be marked by
ChatGPT-2 and has significantly expanded further advances in natural language processing and the
capabilities. ChatGPT-3 can generate text that is
integration of AI technology into various industries and Propensity for biases: Language models like ChatGPT can
domains. Here are some potential developments and areas of inadvertently reflect and amplify biases present in their
focus for ChatGPT in the future: training data, leading to problematic or discriminatory
responses. This is a particular concern given the large scale
Advanced capabilities: ChatGPT and other language models and breadth of data used to train these models.
are expected to continue to improve in terms of their ability
to generate high-quality and diverse text, and to understand Difficulty with long-term coherence: While ChatGPT is
and respond to natural language input in more sophisticated capable of generating high-quality text, it may struggle with
ways. This may include developing models that can reason maintaining long-term coherence or structure in longer
about complex topics and engage in more nuanced pieces of text. This can lead to disjointed or confusing
conversations with users. responses in certain contexts.

Multimodal integration: As language models like ChatGPT Lack of understanding of context: ChatGPT is primarily a
are integrated with other types of AI systems, such as those language model, which means it may not always have a
that process images or video, they are expected to become deep understanding of the context in which it is generating
more powerful and versatile in their applications. This may or responding to text. This can lead to inaccurate or
include developing systems that can generate text irrelevant responses in some situations.
descriptions of images or videos, or that can analyze natural
language in context with other forms of data. Overall, while ChatGPT is a powerful and versatile
language model, it also has several limitations and
Domain-specific applications: ChatGPT is likely to be challenges that must be addressed as the technology is
applied to a growing range of domain-specific applications, developed and applied in various contexts.
such as medical diagnosis, legal reasoning, and financial
analysis. This will require developing specialized training
data and architectures that are tailored to specific domains
and tasks. VII. CHATGPT VS GOOGLE BARD

Ethical considerations: As the use of ChatGPT and other A. ChatGPT Vs. Google
language models becomes more widespread, there will be ChatGPT and Google are two different entities that serve
increasing attention paid to the ethical and social different purposes.
implications of these systems. This may include developing
guidelines and best practices for the responsible use of AI, ChatGPT is a large language model trained by OpenAI that
as well as developing methods for mitigating potential can understand and respond to natural language queries
biases or harmful effects of these systems. from users. It uses machine learning algorithms to generate
human-like responses to user input.
New architectures and techniques: As the field of natural
language processing continues to evolve, there may be new On the other hand, Google is a multinational technology
architectures and techniques developed that go beyond the company that provides a range of services including search
current paradigm of language modeling. This could include engine, email, cloud computing, online advertising, and
new types of neural networks, or novel approaches to many other products. Google's search engine is one of the
training and fine-tuning models. most widely used tools on the internet, and it uses complex
algorithms to deliver relevant results to users based on their
VI. LIMITATIONS OF CHATGPT search queries.
While ChatGPT has shown impressive performance in
generating high-quality text and engaging in conversational While ChatGPT and Google may share some similarities in
dialogue, there are also several limitations and challenges terms of their use of natural language processing and
associated with the technology. OpenAI acknowledged that machine learning, they serve different purposes and have
ChatGPT "sometimes writes plausible-sounding but different areas of expertise.
incorrect or nonsensical answers". Here are some of the key
limitations of ChatGPT: CHATGPT GOOGLE
ChatGPT is an AI-powered Google is an internet search
Limited domain knowledge: ChatGPT and other language dialogue-based chatbot engine, which is owned by
models are trained on large datasets of text, but they may developed by OpenAI. It is a Google LLC.
lack specific knowledge about certain domains or topics. fine-tuned version of GPT-
This can lead to inaccuracies or errors in generating text 3.5 large language model.
related to those domains or topics. It is good at understanding It excels at helping users
prompts and providing find helpful information and
Lack of common-sense knowledge: Similarly, ChatGPT responses using natural websites on the internet
may lack common sense knowledge that humans take for language processing.
granted, which can lead to unexpected or nonsensical Currently, the research Google’s search engine is
responses in some situations. preview (prototype) of totally free to use for
ChatGPT is available for everyone. derivatives before long. Google is also expected to open up
free now. We can expect a Google Bard to third-party developers in the future.
pro version of ChatGPT
soon. VIII. ADVANTAGES AND DISADVANTAGES
The prototype of ChatGPT Google index contains over
is based on only 570 hundreds of billions of web A. Advantages
gigabytes of textual data pages amounting to over 1. Imitates Human Conversation: The core feature of
1,00,000,000 gigabytes of ChatGPT centers on providing human-like conversation
data. based on user-placed queries or commands. It is generally
The prototype of ChatGPT Leading search engine similar to virtual assistant technologies and software
has limited knowledge of the Google keeps its index applications such as Siri from Apple and Alexa from
world’s events only up to afresh to pace up with the Amazon. However, considering its capabilities, it mimics
September 2021 latest events and happenings real-life conversation because it is based on more advanced
around the world. supervised learning and reinforcement learning using large
Although having limited Google relies on the language models.
knowledge, ChatGPT is available data in its index to
insanely good at answer the user’s queries 2. Built Based on GPT-3 Model: GPT-3 or Generative
understanding the user’s and searches. Pre-trained Transformer 3 is an autoregressive language and
prompts and generating language prediction model developed by OpenAI. It is the
responses and solutions largest non-sparse language model and has been considered
through NLP. one of the most important AI systems ever produced. The
The dialogue-based Being an internet search quality of texts it generates makes it difficult to ascertain
conversations on ChatGPT engine, Google offers whether or not it is written by a human.
do not provide any details multiple options to choose
about the sources of from for the search results, 3. Expansive Applications and Benefits: The chatbot is
information, which may put and also provides a complete versatile. It can write outputs similar to commercial AI
a question mark on its source of information on the copywriters. Experiments have shown that it can even
reliability. result pages. compose music and produce works of fiction such as short
As of now, ChatGPT only Google offers results in the stories. It can help content creators or technical writers
offers textual answers to most-suitable form — text, produce an outline. The text. Another interesting application
queries. images, videos, QnA, of ChatGPT is that it can also write and debug computer
products, etc. programs.

4. Open For Further Fine-Tuning: Another advantage of

B. ChatGPT Vs. Google Bard ChatGPT is that its responses and overall performance can
be fine-tuned. It banks on existing large language models
ChatGPT and Google Bard are very similar natural language while also having room for further improvements through
AI chatbots, but they have some differences, and are active training using supervised learning and reinforcement
designed to be used in slightly different ways — at least for learning. A user can upvote or downvote a particular
now. ChatGPT has been used for answering direct questions response while also providing additional feedback.
with direct answers, mostly correctly, but it’s caused a lot of
consternation among white collar workers, like writers, SEO
B. Disadvantages
advisors, and copy editors, since it has also demonstrated an 1. Inaccuracies and Ambiguities: One of the biggest
impressive ability to write creatively ,even if it has faced criticisms and limitations of ChatGPT is that it sometimes
a few problems with accuracy and plagiarism. tends to produce texts that sound plausible or convincing but
are incorrect or nonsensical under the surface. This
Still, Microsoft has integrated ChatGPT into its Bing search phenomenon is called “hallucination” and it is common in
engine to give users the ability to ask direct questions of the language models. Furthermore, when it comes to obtaining
search engine, rather than searching for terms of keywords information, it does not provide references or citations.
to find the best results. It has also built it into its Teams Using this chatbot alone for research purposes and
communications tool, and it’s coming to the Edge browser electronic trailing is not ideal.
in a limited form. The Opera browser has also pledged to
integrate ChatGPT in the future. 2. Limited Knowledge of Recent Events: The version
launched in November 2022 can only provide information
Google Bard was mainly designed around augmenting about events occurring in 2021 and earlier. It will soon
Google’s own search tool, however it is also destined to provide more recent events as it continues to feed on data
become an automated support tool for businesses without based on human-generated texts. Nevertheless, considering
the funds to pay for human support teams. It will be offered this drawback, users should keep in mind that it has limited
to customers through a trained AI responder. It is likely to knowledge of facts because it uses datasets that are not
be integrated into the Chrome browser and its Chromium updated.
3. Ethical Issues and Concerns: Another disadvantage of [4].https://openai.com/blog/chatgpt/
ChatGPT is that it has been subjected to scrutiny. Several [5]. https://en.wikipedia.org/wiki/ChatGPT
educational institutions have banned its use. Researchers [6]. https://www.pcguide.com/apps/what-is-chat-gpt/
and creatives have worried about copyright infringement
because its outputs are based on human-generated texts. It
also raises the question of whether it is ethical to use it as a
substitute for services that require human interactions such
as customer service representation and even therapeutic
counseling.

IX. REFERENCES
[1]. The Curious Case Of Neural Text Degeneration: Holtzman,
A., Buys, J., Du, L., Forbes, M. and Choi, Y., 2019. The curious
case of neural text degeneration. arXiv preprint arXiv:1904.09751.
https://arxiv.org/abs/1904.09751
[2].Is ChatGPT a Good Translator? A Preliminary Study: Jiao, W.,
Wang, W., Huang, J.T., Wang, X. and Tu, Z., 2023. Is ChatGPT a
good translator? A preliminary study. arXiv preprint
arXiv:2301.08745.
https://arxiv.org/abs/2301.08745
[3].https://www.profolus.com/topics/advantages-and-
disadvantages-of-chatgpt/