AI Content Self-Detection for Transformer-based

Large Language Models

Antônio Junior Alves Caiado and Michael Hahsler
Deptartment of Computer Science
Lyle School of Engineering, Southern Methodist University
Dallas, Texas
acaiado@smu.edu

Abstract—The usage of generative artificial intelligence (AI) Recent research works have explored novel approaches
arXiv:2312.17289v1 [cs.CL] 28 Dec 2023

tools based on large language models, including ChatGPT, Bard, to identify the specific source AI system responsible for
and Claude, for text generation has many exciting applications text generation rather than detecting plagiarism only. These
with the potential for phenomenal productivity gains. One issue
is authorship attribution when using AI tools. This is especially approaches try to identify artifacts produced by the generation
important in an academic setting where the inappropriate use of process and range from analyzing statistical patterns to stylistic
generative AI tools may hinder student learning or stifle research cues [2] and authorship attribution [1]. However, even these
by creating a large amount of automatically generated derivative detection techniques encounter difficulties when AI-generated
work. Existing plagiarism detection systems can trace the source content is paraphrased or modified to disguise its origins.
of submitted text but are not yet equipped with methods to
accurately detect AI-generated text. This paper introduces the This paper proposes a novel method for origin detection
idea of direct origin detection and evaluates whether generative called self-detection. It involves using a generative AI system’s
AI systems can recognize their output and distinguish it from capability to distinguish between its own output and human-
human-written texts. We argue why current transformer-based written texts. We will argue why current transformer-based
models may be able to self-detect their own generated text and language models should be able to identify their own work.
perform a small empirical study using zero-shot learning to
investigate if that is the case. Results reveal varying capabilities Then, we will use a set of controlled text samples to assess
of AI systems to identify their generated text. Google’s Bard if leading language models, such as OpenAI’s ChatGPT,
model exhibits the largest capability of self-detection with an Google’s Bard, and Anthropic’s Claude can accurately detect
accuracy of 94%, followed by OpenAI’s ChatGPT with 83%. their own output. The results demonstrate the limitations of
On the other hand, Anthropic’s Claude model seems to be not self-detection by AI systems and the potential for evading
able to self-detect.
plagiarism checks through paraphrasing techniques. These
Index Terms—generative AI, plagiarism, paraphrasing, origin findings emphasize the need to reconsider plagiarism and
detection develop more robust techniques for identifying AI-generated
content.
I. I NTRODUCTION To summarise, the contribution of this paper is as follows:
Generative AI models like the large language models Chat- • We address the struggle of plagiarism detection methods
GPT have become very popular and are applied for many tasks to identify text generated using AI tools.
that involve question answering and generating text in general. • We propose the novel idea of self-detection, where the
Many exciting applications show the potential for phenomenal tool itself is used to detect AI-generated text.
productivity gains. Such tasks include text summarization, • We provide a small study to examine the ability of AI
generating explanations, and answering questions in general. systems to differentiate between human-written and AI-
These models aim to complete a text prompt by mimicking a generated text.
human-like response.
This paper first summarizes the background and discusses
While legitimate use of the see models increases, inap-
related studies. We then introduce self-detection and discuss
propriate use also grows. This is an issue in many areas
why transformer-based models should have the capability of
and especially problematic in cases of academic dishonesty
detecting their own generated text, and we describe several
where AI-generated text is presented as authentic intellectual
hypotheses. In the experiments section, we evaluate the hy-
work by students or researchers. As the models get closer
potheses. The paper closes with a discussion of the findings.
to producing human-like text, detecting AI-generated text
becomes increasingly challenging [1]. Conventional plagiarism II. BACKGROUND
detection methods rely on text similarity with known sources,
which is inadequate for identifying AI-generated content that A. Generative AI
represents a new, paraphrased, and integrated version of multi- Generative models are statistical models that learn the joint
ple sources. This limitation requires profound reconsideration probability distribution of the data-generating process. Such
of what constitutes plagiarism in the age of generative AI. models are often used in machine learning for classification
tasks [3], but they can also generate new data following their Notable results can be can be found in [10], [11], [12] and,
model. The research of generative models in AI [4] accelerated [13].
after the invention of Variational Autoencoders (VAE) [5] Gua et al [14] introduce the Human ChatGPT Compari-
in 2013, and Generative Adversarial Networks (GAN) [6] son Corpus used to compare ChatGPT compared to human-
in 2014. A milestone for text data was the development generated content. They found that part-of-speech (POS) and
of the transformer architecture [7] which is the basis for dependency analysis demonstrate that ChatGPT uses more
models, including OpenAI’s family of generative pretrained determination, conjunction, and auxiliary relations, producing
transformers (GPT) [8] and other large language models. longer dependency distances for certain relations. On the other
This technology enables the capability to produce realistic hand, Busch Hausvik [15] found that ChatGPT can generate
human-like text. An offspring of GPT, ChatGPT pushed the exam answers indistinguishable from human-written text. This
boundaries of natural text generation, enabling the capabilities raises concerns about academic misconduct. Khalil and Er [16]
to produce contextually relevant text. indicate students could potentially use ChatGPT to bypass
Generative AI is also used to create other types of content plagiarism detection. This indicates that plagiarism detection
including images, but we focus on text only in this paper. will need to shift its focus to verifying the origin of the content.
Yu et al [17] focus on finding ChatGPT-written content in
B. Detection of AI-generated Text academic paper abstracts by developing the CHatGPT-writtEn
While detection of AI-generated content can be important AbsTract (CHEAT) dataset to support the development of
in many settings, the emergence of generative AI creates detection algorithms. Weber-Wulff et al [18] compare multiple
especially complicated ethical challenges for academic in- tools for testing AI-generated text detection, such as Check For
tegrity. Much work had already been done to detect plagiarism, AI, Turnitin, ZeroGPT, PlagiarismCheck. The results indicate
which can lead to students not learning by copying assignment significant limitations in detecting AI-generated content with
solutions or researchers taking credit for someone else’s work many false positives and negatives. Detection tools often
and ideas. misclassify AI-generated content as human-written and strug-
AI-generated content creates a new challenge since it does gle with obfuscated texts. The conclusion from a study by
not directly copy existing content but generates new text. Ventayen [19] shows a similar results. These studies show that
Traditional methods that identify similarities between a new detecting AI-generated text is a new and very difficult problem
document and a database of existing documents may fall short where new AI models are presented regularly.
of distinguishing AI-generated content from new human work.
Large language models aim to create natural, human-like text, III. AI S ELF -D ETECTION BY T RANSFORMER - BASED
making it increasingly hard to differentiate generated from M ODELS
human-created text.
Most detection tools focus on training a classifier that learns
Many tools to detect AI-generated text are now offered.
to detect artifacts introduced by the generative model when
Some popular tools geared toward educators are Copyleaks AI
generating text. While some types of artifacts may result from
Content Detector, Crossplag, GPTZero, Hugging Face OpenAI
the used base technology, the transformer, many more will be
Detector, Originality.ai, Turnitin AI Detection and ZeroGPT.
due to model training, including the chosen training data and
The list of detectors and their capability is constantly changing
the performed fine-tuning. Since every model can be trained
following the fast-paced changes seen in the development of
differently, creating one detector tool to detect the artifacts
large language models.
created by all possible generative AI tools is hard to achieve.
Most tools are based on detecting artifacts of the text gen-
Here, we develop a different approach called self-detection,
eration process, including word choice, writing style, sentence
where we use the generative model itself to detect its own
length, and many more. A report by Open AI [9] lays out three
artifacts to distinguish its own generated text from human-
AI content detection strategies, including a simple classifier
written text. This would have the advantage that we do not
learned from scratch, a classifier resulting from fine-tuning an
need to learn to detect all generative AI models, but we only
existing language model, or using the probabilities assigned by
need access to a generative AI model for detection. This is a
the model to strings. Many existing tools follow the first two
big advantage in a world where new models are continuously
approaches. For example, the Hugging Face Open AI detector
developed and trained. We start with an argument about why
is a transformer-based classifier that is fine-tuned to detect
large language models may have the capability to detect their
GPT-2 text. Self-detection introduced in this paper is most
own artifacts.
closely related to the third approach. However, it does not
Current large language models use the decoder of the trans-
require access to the model parameters to assess probabilities.
former architecture as their basic building block (see [8], [20],
It relies on the model itself to perform the detection.
[21]). These models are pre-trained using the unsupervised
task of predicting the next word token on a large text corpus.
C. Generative AI and Academic Integrity
The model learns the following function
Many studies have addressed the ethical implications of
AI-generated content in academic contexts in recent years. P (ui+1 |ui−k , ..., ui ) = f (ui−k , ..., ui , ui+1 )
to predict the probability for each possible next token for the in the prompts focused on ensuring the comparability be-
next position i+1. ui is the i-th word token in the sequence and tween AI-generated texts in terms of content and length. This
k is the context length. The model will then predict the token process resulted in 50 AI-generated essays produced with a
with the highest probability or randomly choose among the short prompt. Following the initial generation of essays, each
most likely tokens. During this training phase, the model learns original essay underwent a paraphrasing process by the same
the language’s grammar and acquires facts and knowledge AI system. We prompted the AI system with the original
vital to performing well on the next-word prediction task. The essay and the instruction to rewrite it (see Appendix A). This
popular Chatbot models are then fine-tuned (typically using procedure resulted in 50 modified versions of those essays.
reinforcement learning [22]) to produce suitable responses to For comparison, we also collected 50 human-written essays
user requests. An example of this approach is ChatGPT, which of similar length from bbc.com by manually searching for
is a fine-tuned GPT model [8]. recent news on the given topics and extracting text passages
Generating text using a trained model consists of the fol- of about 250 words. Statistics of the generated dataset are in
lowing steps: Appendix B.
1) Tokenize the input text. After the creation of the essay dataset, we used zero-shot
2) Embed the tokens as numeric vectors and add positional prompting to ask the AI system to perform self-detection. This
information. is a very convenient approach because it can be quickly done
3) Apply multiple transformer blocks using self-attention with any model and does not require extra steps like fine-
and predict the next token using the transformer’s output. tuning. We created a new instance of each AI system initiated
4) Add the new token to the input sequence and go back and posed with a specific query: ”If the following text matches
to step 2 till a special end token is produced. its writing pattern and choice of words.” The procedure is
5) Convert the generated token sequence back to text. repeated for the original, paraphrased, and human essays, and
the results are recorded. We also added the result of the AI
This approach is autoregressive since it adds one token at a
detection tool ZeroGPT. We do not use this result to compare
time and the next token depends on the previously generated
performance but as a baseline to show how challenging the
tokens. The most important innovation of transformers is
detection task is. The complete dataset with the results of
attention [7], where the model learns to modify tokens to
self-detection is available for research at https://github.com/
attend to other tokens in the sequence. For example, in the
antoniocaiado1/ai-self-detection-study-dataset/.
sentence ”it is not hot,” hot can attend to the word not
modifying hot to look more similar to the word cold. V. R ESULTS
The typical use case for a chatbot is that a user provides
the prompt consisting of a request, and the model generates
For hypotheses H1 and H2, we compare how well AI
the answer. During the text generation process, the model will
systems can self-detect their own text compared to the human-
attend to the tokens in the prompt to be relevant for the prompt
written texts. Each comparison involves 50 AI-generated and
and to the tokens generated so far to create a consistent answer.
50 human-written texts. The results are shown in Table I.
This means that if the complete prompt and the generated text
The accuracy results are visualized in Figure 1. The charts
are available, the model can check if the complete sequence
also include error bars indicating the 95% confidence interval
is consistent with its learned function.
around the estimated accuracy. Note that the dataset is always
In the following, we will perform several experiments to
balanced, meaning an accuracy of 50% means random guess-
investigate the following hypotheses:
ing by a model with no detection power. The AI systems show
H1: Generative AI models based on transformers can self- varying abilities to recognize their generated and paraphrased
detection their own generated text. texts.
H2: Generative AI models based on transformers can self- Hypothesis H1 proposes that generative AI models based on
detect text they have paraphrased. transformers can self-detect their own generated text. This can
H3: Generative AI models cannot detect other model’s gen- be analyzed using the results in Figure 1a. The chart shows
erated text. that Bard and ChatGPT perform well in distinguishing their
generated text from human-written text with high accuracy
IV. E XPERIMENTAL S ETUP
values. The confidence intervals do not span 50% indicating
For the experiments in this paper, we use three mod- that they can self-detect. Claude, however, lacks this ability
els: Open AI’s ChatGPT-3.5, Google’s Bard and Anthropic’s with a confidence interval spanning an accuracy of 50%, so
Claude (all using their September 2023 version). We created it is not able to self-detect. Note that the ability or inability
a new dataset consisting of texts about 50 different topics. We to self-detect results from two reasons. The ability to self-
use each model for each topic to generate essays containing detect given the transformer approach and how well the models
approximately 250 words each. The experimental procedure mimic human writing. To look into this, we also applied
maintained consistency by providing each AI system with an ZeroGPT as a baseline detector. The chosen detector and its
identical prompt (see Appendix A), which instructed them actual performance are not so important. Still, it is important
to write an essay based on the given topic. The uniformity that ZeroGPT performed much better for text generated by
 Generator Paraphrased Detector Accuracy PValue (Accuracy > .5) Detection Rate Precision
1 ChatGPT No Self-Detection 0.83 0.00 0.90 0.79
2 ChatGPT No ZeroGPT 0.66 0.00 0.64 0.67
3 Bard No Self-Detection 0.94 0.00 0.96 0.92
4 Bard No ZeroGPT 0.78 0.00 0.88 0.73
5 Claude No Self-Detection 0.45 0.86 0.04 0.22
6 Claude No ZeroGPT 0.40 0.98 0.12 0.27
7 ChatGPT Yes Self-Detection 0.58 0.07 0.40 0.62
8 ChatGPT Yes ZeroGPT 0.50 0.54 0.32 0.50
9 Bard Yes Self-Detection 0.92 0.00 0.92 0.92
10 Bard Yes ZeroGPT 0.72 0.00 0.76 0.70
11 Claude Yes Self-Detection 0.83 0.00 0.80 0.85
12 Claude Yes ZeroGPT 0.38 0.99 0.08 0.20

TABLE I: Results of AI self-detection.

(a) Self-detection accuracy on prompted essays. (b) Self-detection accuracy on AI-paraphrased essays.
Fig. 1: AI self-detection accuracy with 95% confidence interval.

Bard and ChatGPT and could not detect Claude’s generated while it was unable to detect its original essay. The finding
text. This may indicate that Claude produced output with that paraphrasing prevents ChatGPT from self-detecting while
harder-to-detect artifacts, which also would make it harder for increasing Claude’s ability to self-detect is very interesting
Claude to self-detect. and may be the result of the inner workings of these two
Hypothesis H2 proposes that generative AI models based on transformer models. There is promise for hypothesis H2.
transformers can self-detect text they have paraphrased. The However, the fact that a long prompt and the corresponding
reason for this hypothesis is that the artifacts created by the attendance values are missing seem to make it a far more
model should also be present when it rewrites text. However, difficult problem.
the prompting process differs since it includes the original To investigate hypothesis H3, which proposes that AI mod-
text, which may lead to different self-detection performances. els cannot detect text generated by other models, we ask each
Figure 1b shows the accuracy of paraphrased text versus model to determine if the other model’s output is human-
human-written text. The ZeroGPT baseline shows that the per- written or AI-generated. The results are shown in Table II
formance on the paraphrased essays is largely similar to that and Figure 2. We see again that Bard’s text is the easiest to
on the original essays. The results for Bard’s self-detection are detect. Bard’s self-detection if 94%. The other model also can
slightly lower than on the original essays. ChatGPT performs detect some of Bard’s text, but at a level just above random
way worse with the 95% confidence band covering 50%, but guessing. ChatGPT can self-detect its generated text, but the
Claude seems to be able to self-detect its paraphrased content other models cannot. For Claude, the situation is very different.
 Generator Detector Accuracy PValue (Accuracy > .5) Detection Rate Precision
1 ChatGPT ChatGPT 0.83 0.00 0.90 0.79
2 ChatGPT Bard 0.48 0.69 0.04 0.33
3 ChatGPT Claude 0.48 0.69 0.10 0.42
4 Bard ChatGPT 0.63 0.01 0.50 0.68
5 Bard Bard 0.94 0.00 0.96 0.92
6 Bard Claude 0.63 0.01 0.40 0.74
7 Claude ChatGPT 0.68 0.00 0.60 0.71
8 Claude Bard 0.46 0.82 0.00 0.00
9 Claude Claude 0.45 0.86 0.04 0.22

TABLE II: Results of AI self-detection vs. detection by other models.

Fig. 2: AI self-detection vs. detection by other models accuracy with 95% confidence interval.

Claude cannot self-detect its text, and Bard can also not detect VI. D ISCUSSION
Claude’s text, but ChatGPT can detect some of Claude’s text.
Detecting the use of the currently leading AI systems is a
difficult task. The results in this paper demonstrate varying
If the assumption is correct that self-detection relies on
capabilities of leading AI systems to self-detect their own
the model’s knowledge of the model parameters used in the
generated text. Bard performs the best on its own work,
transformer, the H3 should hold. However, the study shows
including originally created essays from a short prompt and
a mixed result for H3. It seems like Bard introduces artifacts
after paraphrasing a longer given essay. ChatGPT performs
that are relatively easy to identify by other models, which also
reasonably well on essays it has created after a short prompt
explains the good performance of the feature-based AI-content
but cannot reliably detect essays it has paraphrased. Claude
detector ZeroGPT on Bard’s output. The other models have no
is not able to detect its own created text. This seemingly
access to Bards’s model parameter and, therefore, must also
inconclusive result needs more consideration since it is driven
be able to pick up these artifacts. For ChatGPT, H3 seems
by two conflated causes.
to apply as expected. Claude’s generated text is generally
the hardest to detect, which may indicate fewer artifacts. 1) The ability of the model to create text with very few
Interestingly, Claude cannot self-detect, but ChatGPT can detectable artifacts. Since the goal of these systems is to
detect either Claude’s artifacts or knows Claude’s generating generate human-like text, fewer artifacts that are harder
model. An explanation could be that either ChatGPT or Claude to detect means the model gets closer to that goal.
either shared a significant portion of their training sets or could 2) The inherent ability of the model to self-detect can be
train on each other’s generated text. However, this is hard to affected by the used architecture, the prompt, and the
determine from the outside. applied fine-tuning.
 We use the external AI content detector ZeroGPT to address other popular AI content detection tools exist (Turnitin,
the first cause. ZeroGPT states on its website1 that it works PlagiarismCheck, GPT Zero, etc.).
accurately for text created by models including GPT-4, GPT- While AI content detection tools have the advantage that
3, GPT-2, Claude AI, and Google Bard. We use its results as they can be trained to identify the artifacts of multiple gen-
a proxy for how difficult it is to detect the text generated by erative AI tools, they need to be updated to add detection
different models. The results in Figure 1a show that Bard’s capabilities for a new model or when models change. A
generated text is the easiest to detect, followed by ChatGPT. significant disadvantage is that self-detection can only detect
Only Claude cannot be detected. This indicates that Claude its own work by using knowledge of its generation process
might produce fewer detectable artifacts than the other models. and the artifacts that it creates. However, in a world where
The detection rate of self-detection follows the same trend, new models are introduced at a break-neck pace, it may be
indicating that Claude creates text with fewer artifacts, making easier and faster to add this new model to the set of models
it harder to distinguish from human writing. Self-detection that are asked to self-detect instead of creating a large amount
shows similar detection power compared to ZeroGPT, but note of data with the models and retraining a standard AI content
that the goal of this study is not to claim that self-detection is detector.
superior to other methods, which would require a large study
to compare to many state-of-the-art AI content detection tools. VII. C ONCLUSION
Here, we only investigate the models’ basic ability of self- Detecting AI-generated content, which includes proper attri-
detection. bution of authorship and addressing questions of remuneration
In general, the self-detection performance decreases for of the creator of the content used to train these models,
AI-paraphrased text (shown in Figure 1b). This may be af- is becoming increasingly important for many applications.
fected by the inherent ability of the transformer-based models Especially in academia, generative AI has many uses that can
to self-detect. An important part of why transformer-based improve learning by generating explanations for students, but
large language models process the prompt and generate text it can also detract from learning by enabling students to let
so successfully is using the attention mechanism. Attention AI solve their exercises.
allows the model to learn how to modify tokens based on This study’s unique contribution lies in introducing self-
previously seen tokens to include context information before detection, a step forward in addressing the challenges posed
it uses a learned function to predict the next word. Since the by AI systems. We describe why transformer-based systems
transformer has access to its own attention mechanism and the should have the capability to self-detect and demonstrate this
prediction function, we have hypothesized that transformer- capability in a first small study. We identify the main limitation
based generative AI models can self-detect their own generated of self-detection as the unavailability of the original prompt.
text. An important issue is that the prompt text is available The presented first study is very limited. Here are some
during text generation and is included in the attention calcu- topics to explore in future studies.
lation. The used prompt is typically not available during self-
• Use a larger dataset with more diverse generated text.
detection. This means that the attention to the tokens in the
• Explore more different generative AI models.
prompt cannot be calculated, reducing the ability to self-detect.
• Compare the performance of self-detection with the cur-
A counter-intuitive finding is that Claude has difficulties in
rently best detectors.
self-detecting its originally generated content but can detect
• Explore how prompt engineering affects self-detection.
content that it has paraphrased with a high degree of accuracy
For example, use few-shot prompting for self-detection.
while the baseline detector still cannot detect it.
This initial study has several important limitations. R EFERENCES
• This study is limited by the small dataset containing a [1] A. Uchendu, T. Le, K. Shu, and D. Lee, “Authorship attribution for
randomized set of topics and a simplified paraphrasing neural text generation,” in Proceedings of the 2020 Conference on
Empirical Methods in Natural Language Processing (EMNLP). Online:
approach. Association for Computational Linguistics, Nov. 2020, pp. 8384–8395.
• This experiment only utilizes three popular AI systems— [Online]. Available: https://aclanthology.org/2020.emnlp-main.673
ChatGPT, BARD, and Claude. [2] Y. Arase, Z. Chen, X. Li, J. Zhang, and T. Zhang, “A style-aware
generative model for paraphrasing,” Transactions of the Association for
• Generative AI systems are constantly evolving, and the Computational Linguistics, vol. 9, pp. 1060–1075, 2021.
systems are changing quickly (e.g., by training on addi- [3] T. M. Mitchell, Machine learning. McGraw-hill New York, 1997, vol. 1,
tional data, changes in pre-prompts, and changes in the no. 9.
[4] S. J. Russell, Artificial intelligence a modern approach. Pearson
used architecture). This makes comparisons difficult, and Education, Inc., 2010.
detailed results may quickly become irrelevant. [5] D. P. Kingma and M. Welling, “Auto-encoding variational bayes,” 2022.
• Only a single conventional plagiarism detection tool, [6] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. Warde-
Farley, S. Ozair, A. Courville, and Y. Bengio, “Generative
ZeroGPT, has been used as a baseline to reason about the adversarial nets,” in Advances in Neural Information Processing
artifacts present in the output of different models. Many Systems, Z. Ghahramani, M. Welling, C. Cortes, N. Lawrence,
and K. Weinberger, Eds., vol. 27. Curran Associates, Inc.,
2014. [Online]. Available: https://proceedings.neurips.cc/paper files/
1 ZeroGPT website: https://zerogpt.cc/ paper/2014/file/5ca3e9b122f61f8f06494c97b1afccf3-Paper.pdf
 [7] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, [14] B. Guo, X. Zhang, Z. Wang, M. Jiang, J. Nie, Y. Ding, J. Yue, and
A. N. Gomez, L. Kaiser, and I. Polosukhin, “Attention is all Y. Wu, “How close is ChatGPT to human experts? comparison corpus,
you need,” CoRR, vol. abs/1706.03762, 2017. [Online]. Available: evaluation, and detection,” arXiv preprint arXiv:2301.07597, 2023.
http://arxiv.org/abs/1706.03762 [15] P. A. Busch and G. I. Hausvik, “Too good to be true? an empirical
[8] A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever, “Improving study of ChatGPT capabilities for academic writing and implications
language understanding by generative pre-training,” OpenAI, Tech. Rep., for academic misconduct,” 2023.
2018. [16] M. Khalil and E. Er, “Will ChatGPT get you caught? rethinking of
[9] Irene Solaimanet al, “Release strategies and the social impacts of plagiarism detection,” arXiv preprint arXiv:2302.04335, 2023.
language models,” arXiv preprint arXiv:1908.09203, 2019. [17] P. Yu, J. Chen, X. Feng, and Z. Xia, “CHEAT: A large-scale
[10] D. R. Cotton, P. A. Cotton, and J. R. Shipway, “Chatting and cheating: dataset for detecting ChatGPT-writtEn AbsTracts,” arXiv preprint
Ensuring academic integrity in the era of ChatGPT,” Innovations in arXiv:2304.12008, 2023.
Education and Teaching International, pp. 1–12, 2023. [18] D. Weber-Wulff, A. Anohina-Naumeca, S. Bjelobaba, T. Foltýnek,
[11] Z. Liu, Z. Yao, F. Li, and B. Luo, “Check me if you can: Detecting J. Guerrero-Dib, O. Popoola, P. Šigut, and L. Waddington, “Testing of
ChatGPT-generated academic writing using CheckGPT,” arXiv preprint detection tools for ai-generated text,” 2023.
arXiv:2306.05524, 2023. [19] R. J. M. Ventayen, “OpenAI ChatGPT generated results: Similarity index
[12] S. Mitrović, D. Andreoletti, and O. Ayoub, “ChatGPT or human? detect of artificial intelligence-based contents,” Available at SSRN 4332664,
and explain. explaining decisions of machine learning model for de- 2023.
tecting short chatgpt-generated text,” arXiv preprint arXiv:2301.13852, [20] Romal Thoppilan et al, “LaMDA: Language models for dialog applica-
2023. tions,” arXiv preprint arXiv:2201.08239, 2022.
[13] N. Anderson, D. L. Belavy, S. M. Perle, S. Hendricks, L. Hespanhol, [21] Rohan Anil et al, “PaLM 2 technical report,” arXiv preprint
E. Verhagen, and A. R. Memon, “AI did not write this manuscript, or did arXiv:2305.10403, 2023.
it? can we trick the AI text detector into generated texts? the potential [22] D. M. Ziegler, N. Stiennon, J. Wu, T. B. Brown, A. Radford, D. Amodei,
future of ChatGPT and AI in sports & exercise medicine manuscript P. Christiano, and G. Irving, “Fine-tuning language models from human
generation,” p. e001568, 2023. preferences,” 2020.
 A PPENDIX A
P ROMPTS USED IN THIS S TUDY

Task Prompt
Essay Generation Write an essay within 250 words regarding (topic name) in one paragraph
Paraphrased Essay Generation Paraphrase the following paragraph: (previously generated paragraph)
AI Self-Detection Check if the following paragraph matches your text patterns and choice of words for generating the
response. If it matches, respond TRUE; otherwise, FALSE.
AI Detecting other AI Content Check if the given paragraph matches or contains AI jargon or Human written patterns. Give your answer
as either Human or AI.

TABLE III: Prompts used in this study

A PPENDIX B
S TATISTICS OF THE DATA S ET

Source Model Paraphrased Number of Essays Word Count Sentence Count Words/Sentence Newline Count
1 AI ChatGPT No 50 193.90 7.98 24.30 0.00
2 AI ChatGPT Yes 50 194.36 8.10 24.00 0.00
3 AI Bard No 50 292.30 6.06 48.23 10.86
4 AI Bard Yes 50 251.48 5.24 47.99 9.32
5 AI Claude No 50 206.38 11.76 17.55 0.00
6 AI Claude Yes 50 187.40 11.06 16.94 0.00
7 Human None 50 348.26 11.04 31.55 6.24

TABLE IV: Average statistics for different groups in the dataset

 A PPENDIX C
E XAMPLE DATA
The complete dataset can be found at https://github.com/antoniocaiado1/ai-self-detection-study-dataset/.
A. ChatGPT’s Essay Identification Data

Topic Generated Answer Paraphrased Answer

James Webb The James Webb Space Telescope (JWST) stands as a The James Webb Space Telescope (JWST) represents a
Telescope monumental advancement in humanity’s quest to unravel significant leap in humanity’s quest to comprehend the
the mysteries of the universe. Scheduled for launch, this universe’s enigmas. With a launch date on the horizon,
successor to the Hubble Space Telescope promises to rev- this successor to the Hubble Space Telescope pledges to
olutionize our understanding of celestial phenomena and transform our grasp of celestial occurrences and deepen
deepen our insights into the cosmos. Designed collabo- our cosmic insights. Collaboratively crafted by NASA,
ratively by NASA, the European Space Agency (ESA), the European Space Agency (ESA), and the Canadian
and the Canadian Space Agency (CSA), the JWST boasts Space Agency (CSA), the JWST showcases an array
an array of cutting-edge technologies that enable it to of state-of-the-art technologies, enabling it to observe
observe the universe in ways previously unattainable. the cosmos in unprecedented ways. Primarily function-
Operating primarily in the infrared spectrum, the tele- ing in the infrared range, the telescope’s intricate tools
scope’s intricate instruments will allow it to peer through will facilitate peering through interstellar dust, capturing
cosmic dust clouds, capture the glow of distant galaxies, remote galaxy glows, and scrutinizing exoplanet atmo-
and study the atmospheres of exoplanets. Moreover, the spheres. Additionally, the JWST’s expansive sunshield
JWST’s colossal sunshield will protect its sensitive in- will safeguard its delicate equipment from solar heat and
struments from the Sun’s heat and light, ensuring precise brilliance, ensuring meticulous and lucid observations.
and clear observations. By probing the universe’s earliest By investigating the universe’s infancy, examining star
moments, studying the formation of stars and galaxies, and galaxy formation, the JWST is poised to deliver
the JWST is poised to provide unprecedented insights unparalleled revelations into the fundamental queries
into the fundamental questions surrounding our existence surrounding human existence and the wider cosmos.
and the cosmos.
Global Warming Global warming, an alarming consequence of human Global warming, a concerning outcome of human actions,
activities, has emerged as a pressing threat to our planet. has arisen as an imminent menace to our planet. Un-
The unrestrained burning of fossil fuels, deforestation, controlled combustion of fossil fuels, deforestation, and
and industrial processes have unleashed an influx of industrial operations have released a surge of greenhouse
greenhouse gases into the atmosphere, intensifying the gases into the atmosphere, intensifying the innate green-
natural greenhouse effect and causing Earth’s tempera- house effect and leading to a rise in Earth’s temperatures.
tures to rise. Consequently, ice sheets and glaciers are This has led to the accelerated melting of ice sheets and
melting at an accelerated pace, fueling sea level rise and glaciers, contributing to the elevation of sea levels and
endangering coastal regions. The frequency and intensity jeopardizing coastal areas. The frequency and severity
of extreme weather events have surged, resulting in of extreme weather occurrences have risen substantially,
devastating hurricanes, prolonged droughts, and erratic causing catastrophic hurricanes, prolonged periods of
rainfall patterns. These changes disrupt ecosystems, en- drought, and irregular patterns of precipitation. These al-
dangering various species and diminishing biodiversity. terations disrupt ecosystems, imperil diverse species, and
Urgent global action is imperative to curb this crisis. Gov- lessen biodiversity. Swift worldwide measures are vital to
ernments, industries, and individuals must transition to alleviate this predicament. Governments, industries, and
sustainable energy sources, enhance forest conservation individuals should transition toward sustainable energy
efforts, and adopt eco-friendly practices. Furthermore, sources, amplify endeavors for forest preservation, and
raising awareness about the impacts of global warming embrace environmentally conscious practices. Moreover,
can drive collective efforts to reduce carbon emissions heightening awareness about the repercussions of global
and promote environmental stewardship. By recognizing warming can stimulate collective endeavors to decrease
the gravity of the situation and taking comprehensive carbon emissions and foster ecological guardianship.
measures, we can mitigate global warming’s adverse By acknowledging the seriousness of the situation and
effects and secure a sustainable future for the generations implementing comprehensive steps, we can mitigate the
to come. detrimental repercussions of global warming and ensure
a sustainable future for future generations.

TABLE V: ChatGPT’s Essay Identification Data

B. BARD’s Essay Identification Data

Topic Generated Answer Paraphrased Answer

James Webb The James Webb Space Telescope (JWST) is the most The James Webb Space Telescope is a powerful telescope
Telescope powerful telescope ever built. It was launched on De- that can see objects that are too old, distant, or faint
cember 25, 2021, and is currently orbiting the Sun about for other telescopes to see. It was launched in 2021
a million miles away from Earth. JWST is designed to and is currently orbiting the Sun about a million miles
study the infrared universe, which allows it to see objects away from Earth. JWST is expected to make ground-
that are too old, distant, or faint for other telescopes to breaking discoveries about the universe, including seeing
see. JWST’s primary mirror is 6.5 meters in diameter, the first stars and galaxies that formed after the Big
which is about six times larger than the Hubble Space Bang. JWST’s primary mirror is 6.5 meters in diameter,
Telescope’s mirror. This larger mirror allows JWST to which is about six times larger than the Hubble Space
collect more light and see fainter objects. JWST also Telescope’s mirror. This larger mirror allows JWST to
has four instruments that are specifically designed to collect more light and see fainter objects. JWST also has
study the infrared universe. JWST is also expected to four instruments that are specifically designed to study
provide new insights into the formation of stars and the infrared universe. The James Webb Space Telescope
galaxies, and the evolution of the universe. The James is a truly remarkable telescope that has the potential
Webb Space Telescope is a truly remarkable telescope to revolutionize our understanding of the universe. It
that has the potential to revolutionize our understanding is a testament to the ingenuity and perseverance of the
of the universe. It is a testament to the ingenuity and scientists and engineers who built it.
perseverance of the scientists and engineers who built it.
Global Warming Global warming is the long-term heating of Earth’s Global warming is the long-term heating of Earth’s
climate system observed since the pre-industrial period climate system. It is caused by human activities, such
(between 1850 and 1900) due to human activities, pri- as burning fossil fuels, which release greenhouse gases
marily fossil fuel burning, which increases heat-trapping into the atmosphere. These gases trap heat, causing the
greenhouse gas levels in Earth’s atmosphere. The term planet to warm.The effects of global warming are already
is frequently used interchangeably with the term climate being felt around the world. Sea levels are rising, glaciers
change, though the latter refers to both human- and are melting, and extreme weather events are becoming
naturally produced warming and the effects it has on our more common. Global warming is a serious threat to
planet. The main cause of global warming is the release our planet and its inhabitants.We need to take action
of greenhouse gases into the atmosphere. Greenhouse to reduce greenhouse gas emissions and mitigate the
gases trap heat from the sun, causing the planet to effects of climate change. This includes switching to
warm. The primary greenhouse gases are carbon dioxide, renewable energy sources, improving energy efficiency,
methane, nitrous oxide, and ozone.The effects of global and reducing deforestation. Individuals can also make a
warming are already being felt around the world. These difference by making changes in their own lives, such as
effects include rising sea levels, melting glaciers, more driving less, using less energy, and eating less meat.By
extreme weather events, and changes in plant and animal taking action now, we can help to protect our planet for
life. Global warming is a serious threat to our planet future generations.
and its inhabitants. We need to take urgent action to
reduce greenhouse gas emissions and mitigate the effects
of climate change.

TABLE VI: BARD’s Essay Identification Data

C. Claude’s Essay Identification Data

Topic Generated Answer Paraphrased Answer

James Webb The James Webb Space Telescope (JWST) is a large, The James Webb Space Telescope, launched in 2021, is
Telescope infrared-optimized space telescope launched by NASA, the most powerful space telescope ever built. Stationed
ESA, and the Canadian Space Agency in 2021. JWST is far from Earth, it uses specialized infrared technology
the most powerful space telescope ever built, acting as to observe extremely distant cosmic objects, allowing it
the successor to the Hubble Space Telescope. Stationed to see farther back in time than any previous telescope.
nearly 1 million miles from Earth at the second Lagrange As Hubble’s successor, one of Webb’s main goals is
point, JWST utilizes a 6.5-meter primary mirror and four studying the earliest galaxies that emerged over 13 billion
scientific instruments to observe extremely distant celestial years ago right after the Big Bang, to understand the
objects. Its specialized infrared technology allows JWST origins of the universe. It will also analyze atmospheres
to penetrate cosmic dust clouds and detect the faint light of exoplanets orbiting other stars to potentially find signs
from the earliest stars and galaxies in the universe over of life outside our solar system. The precise engineering
13.5 billion years ago. One of JWST’s primary scientific and technology behind Webb’s successful launch and
goals is observing these primordial galaxies during the deployment mark major space achievements. With at
reionization era to understand the origins of the universe. least a 5-10 year operational lifetime, Webb is expected
JWST will also analyze the atmospheres of exoplanets to revolutionize astronomy by unveiling unprecedented
to search for the potential chemical indications of life views of the early infrared universe.
outside our solar system. The precise launch and orbital
insertion of JWST represent major achievements in space
engineering and technology. With a minimum lifetime of
5-10 years, JWST is expected to revolutionize many areas
of astronomy by revealing unprecedented views of the
infrared universe.
Global Warming Global warming is the steady rise in Earth’s average The planet is heating up at an alarming rate largely
temperature over the past century primarily due to the due to human activities like burning fossil fuels. This is
burning of fossil fuels like coal, oil and gas. This human- causing major shifts in weather patterns, melting of ice
caused climate change is leading to alterations in global sheets, and rising oceans. If we stay on the current path,
weather patterns, melting glaciers and polar ice caps, the future impacts could be devastating - more extreme
and rising sea levels. Unless drastic action is taken, weather events, spreading of diseases, food shortages,
these trends will continue and intensify in the coming ecosystem destruction, and so on. To avoid catastrophic
decades. The consequences of unchecked global warming climate change, we need to quickly transition our energy
could be catastrophic, including more extreme weather systems to renewables like solar and wind while also
like droughts, floods and storms, the spread of diseases, conserving resources and replanting forests. Individuals
reduced agricultural yields leading to food shortages, and can help by reducing their carbon footprint, but large-
the destruction of ecosystems unable to adapt quickly scale change by governments and companies will be
enough. To prevent the most severe impacts, the world essential for real progress. There is an urgent need to
must transition from fossil fuels to renewable energy cut emissions and limit additional warming to 1.5°C over
sources like solar and wind power while also practicing pre-industrial levels, the target of the Paris Agreement, to
conservation and reforestation. Individuals can help by avoid triggering feedback loops that could spiral out of
making lifestyle changes to reduce their carbon footprint, control. With global teamwork, it may still be possible to
but systemic change by governments and corporations will create a sustainable world, but time is running out fast.
be necessary for major progress. Urgent action is required Decisive action now is critical.
to curb greenhouse gas emissions and limit further warm-
ing to 1.5 degrees Celsius above pre-industrial levels,
the target set in the Paris Climate Agreement to avoid
environmental tipping points leading to runaway climate
change. With global cooperation, it may still be possible
to create a sustainable future, but the window for effective
action is closing rapidly.

TABLE VII: Claude’s Essay Identification Data

 A PPENDIX D
AI S YSTEMS ’ H UMAN W RITTEN T EXT I DENTIFICATION DATA
In this particular task, the AI systems were asked to write an essay about a particular topic and paraphrase it. Later, it was
checked whether the system could detect its own generation. For this purpose, the same prompts, shown in III, were given in
every iteration.
Topic Human Text
Global Warming Climate change is already shaping what the future will look like and plunging the world into crisis. Cities
are adapting to more frequent and intense extreme weather events, like superstorms and heatwaves. People
are already battling more destructive wildfires, salvaging flooded homes, or migrating to escape sea level
rise. Policies and economies are also changing as world leaders and businesses try to cut down global
greenhouse gas emissions. How energy is produced is shifting, too — from fossil fuels to carbon-free
renewable alternatives like solar and wind power. New technologies, from next-generation nuclear energy
to devices that capture carbon from the atmosphere, are in development as potential solutions. The Verge
is following it all as the world reckons with the climate crisis.

James Webb Telescope NASA’s James Webb Space Telescope’s high resolution, near-infrared look at Herbig-Haro 211 reveals
exquisite detail of the outflow of a young star, an infantile analogue of our Sun. Herbig-Haro objects are
formed when stellar winds or jets of gas spewing from newborn stars form shock waves colliding with
nearby gas and dust at high speeds. The image showcases a series of bow shocks to the southeast (lower-
left) and northwest (upper-right) as well as the narrow bipolar jet that powers them in unprecedented
detail. Molecules excited by the turbulent conditions, including molecular hydrogen, carbon monoxide
and silicon monoxide, emit infrared light, collected by Webb, that map out the structure of the outflows.

Genetic Engineering Religion provides the strongest grounds for protesting genetic engineering. So it is not surprising that
most of the resistance to all new reproductive technologies comes from people with religious beliefs. This
resistance is deeply rooted in fundamental religious norms. According to the Judeo-Christian tradition,
humans were created in the “image” and “likeness” of God (Genesis 1:26-27), which, according to some
interpreters, means both the given nature of man and their perfection, the goal towards which they must
strive; and from the point of view of others, “image” and “likeness” are synonymous. Humans are likened
to God, first of all, in that they were given power over nature (Ps. 8), and also in that they received from
the Creator the “breath of life.” Thanks to this, a person becomes a “living soul.” This concept means
a living personality, the unity of vital forces, the “I” of a person. Soul and flesh are characterized by
organic unity (in contrast to the Greek philosophical dualism, which contrasted spirit and flesh). Some
people believe genetic engineering is morally wrong because it interferes with God’s plan for humanity.
They believe that we are playing with fire by altering the genes of living organisms and that this could
have catastrophic consequences for both humans and the environment.

TABLE VIII: Human Written Essay Identification Data