HANOI UNIVERSITY OF SCIENCE AND TECHNOLOGY

FACULTY OF FOREIGN LANGUAGES

Evaluation of the Effectiveness of ChatGPT in Translation for Faculty of Foreign

Languages Students at Hanoi University of Science and Technology.
 Abstracts
Along with the development of Information Technology (IT), the use of Artificial Intelligence
(AI) tools like ChatGPT in translation has become increasingly popular. The Faculty of Foreign
Languages (FOFL) at Hanoi University of Science and Technology (HUST) has a diverse
student population that increasingly relies on AI-powered translation tools to support their
language learning and academic activities.
The aim of this study is to evaluate the effectiveness of ChatGPT in translation, its benefits, and
limitations for FOFL students at HUST. A mixed-methods approach was used, including a
quantitative analysis of the translation accuracy and fluency, and qualitative feedback from
FOFL students who use ChatGPT daily.
The translated texts produced by ChatGPT cover many genres and have been compiled for
comparison with professional human translations. Students also provided subjective evaluations
of ChatGPT in terms of usefulness, ease of use, and impact on their translations.
The research shows that ChatGPT can translate a variety of text types, although its performance
and accuracy depend on the complexity and genre of the source material. Students found
ChatGPT to be a helpful tool helping them in their translation tasks, but they also demonstrated
their concerns about potential over-reliance and the need for critical evaluation of the translated
outputs.
As ChatGPT continues to evolve and become more widely accessible nowadays, it is crucial to
understand their strengths, briefings, and appropriate use as an assistant in translation. This
research provides valuable insights that can inform the development of best practices and
guidelines for the use of ChatGPT and similar technologies in foreign language learning and
translation tasks.
 Acknowledgement
First and foremost, we would like to express our gratitude to all those who gave us the possibility
to complete this thesis. We want to thank the English Department of Hanoi University of Science
and Technology for permitting us to commence this thesis in the first instance and to do the
necessary research work.
We are deeply indebted to our academic advisor, Mrs. Nguyen Thi Bac, for her insightful
feedback and professional direction. Her broad knowledge base, inventive problem-solving
skills, and sincere concern for the author's success have been truly transformative.
We would like to express our sincere appreciation to other fellow students of FOFL for allowing
us to use their translation exercises and assignments as a source of inspiration. We are grateful to
our classmates from class 149361 for their invaluable suggestions, cooperation, and unwavering
support.
In particular, we attribute our friends who carefully reviewed the thesis for grammar and English
style in its final draft, making corrections and providing suggestions for enhancement.
 CHAPTER I: INTRODUCTION
1.1 Rationale
Language plays a vital role in human life. It is a tool for humans to communicate with each
other. Furthermore, language is important for establishing and maintaining connections. To
develop connections with others and communicate our experiences, we utilize language. In terms
of English language learning, the translation process between the source language and the target
language cannot be separated. Translation is the transfer of a thought from one language to
another. In addition, not all countries use the same language. According to SIL's publication
Ethnologue, 6.909 distinct languages are being used globally. For people from different
countries, in order to communicate effectively, translation services are essential.
Some common difficulties when translating text are that students usually lack of vocabulary:
they may not have the vocabulary knowledge to translate the text accurately. Others are
grammatical issues, cultural differences, idioms, and literary texts. However, the rapid
development of technology has given some solutions to the problem above. Translation tools
such as Google Translate, Bing Translator, TranslatePress,… have been developed to help
people with translation work. Machine translation technologies have made significant advances
in recent years, though human translators are still needed for high-stakes, nuanced, or creative
translation work.
Currently, research into the use of AI-powered translation tools is still a relatively nascent field.
While studies have examined the general performance and capabilities of language models like
ChatGPT (Anthropic, 2022; Ouyang et al., 2022), there is a dearth of empirical evidence on their
specific applications and effectiveness within foreign language programs.
Thus, previous studies have highlighted both the potential benefits and limitations of AI-powered
translation tools. On the one hand, these tools can provide students with quick and accessible
translation services, potentially enhancing their language comprehension and productivity
(Daems & Macken, 2019; Niño, 2009). On the other hand, concerns have been raised about the
accuracy, fluency, and potential misuse of these translations, as well as their impact on language
learning and academic integrity (Cohn, 2019; Stapleton & Garner, 2020).
1.2 Significance of the Research
The research aim to evaluate the effectiveness of ChatGPT in translation. Furthermore, this study
also helps FOFL students at HUST to understand the strengths, limitations, and potential use
cases of ChatGPT, the faculty can integrate the technology in a way that complements and
enhances traditional translation teaching methodologies. Therefore, incorporating the use of
ChatGPT and other AI-powered translation tools can equip students with the necessary skills to
navigate the changing demands of the translation industry.
In addition, understanding the limitations and potential pitfalls of using ChatGPT for translation
tasks can help students develop critical thinking skills in evaluating the quality and reliability of
machine-generated translations. This, in turn, can contribute to the overall language proficiency
of the Faculty of Foreign Languages students, as they learn to balance the use of AI-powered
tools with their own translation and language skills.
1.3 Objectives of the Research
- To evaluate the accuracy and quality of translations produced by students using ChatGPT
compared to traditional translation methods.
- To investigate the impact of using ChatGPT on students' translation productivity.
- To explore students' attitudes towards using ChatGPT in translation.
- To identify the challenges and limitations of using ChatGPT for translation.
1.4 Scope of the Research
This research will focus on evaluating the effectiveness of using ChatGPT for translation tasks
among students in the Faculty of Foreign Languages at Hanoi University of Science and
Technology. The study will cover a range of translation assignments, including academic papers,
documents, articles, and literary works. Student participants will be recruited from different
classes of FOFL at HUST to assess the impact of ChatGPT in translation tasks.
1.5 Structure of the Research
The Chapters of this Research Report are arranged as follows:
- Introduction: Provides the background, problem statement, research objectives, and scope of
the study.
- Literature Review: Presents a comprehensive review of the existing literature on the use of
AI-powered translation tools, with a focus on ChatGPT, in educational and academic
contexts.
- Methodology: Describes the research design, data collection methods, and analysis
techniques used in the study.
- Findings: Presents the key results and insights from the quantitative and qualitative data
collected during the research.
- Discussion: Discusses the implications of the findings, addressing the research objectives and
highlighting the practical and theoretical contributions of the study.
- Conclusion: Summarizes the main conclusions, limitations, and recommendations for future
research and practical applications.
 CHAPTER II: LITERATURE REVIEW
The use of artificial intelligence (AI) in language translation has been a topic of growing interest in both
academic and industry settings. The recent advancements in large language models, such as ChatGPT
developed by OpenAI, have sparked discussions about their potential to revolutionize the field of
translation. As the Faculty of Foreign Languages at Hanoi University of Science and Technology seeks to
enhance its translation education program, it is crucial to evaluate the effectiveness of ChatGPT in
supporting student learning and translation skills development.
Previous research has examined the integration of machine translation tools in language learning
environments. Studies have highlighted both the benefits and challenges of using machine translation
systems, such as improved efficiency in translation tasks, but also the potential for over-reliance and
reduced language acquisition (Bowker & Buitrago Ciro, 2019; Niño, 2009). However, the emergence of
more advanced language models like ChatGPT warrants a fresh look at their capabilities and implications
for foreign language education.

A Comparative Literature Review

The use of translation tools in educational settings, particularly in faculties of foreign languages, has been
a topic of extensive research and debate. In the context of the Faculty of Foreign Languages (FOFL) at
Hanoi University of Science and Technology, the evaluation of ChatGPT, an advanced AI-driven
translation tool, provides a unique opportunity to assess its effectiveness compared to traditional
translation methodologies. This literature review delves into how these tools are similar and different,
highlighting the nuances that define their respective strengths and weaknesses.

Similarities in Evaluation

The primary purpose of evaluating both traditional translation tools and ChatGPT is to determine their
effectiveness in producing accurate and contextually appropriate translations (Lopez, 2020). This
common goal is crucial in educational environments where precise translation is essential for learning and
comprehension. Both types of tools are assessed based on similar criteria, including accuracy, fluency,
and user satisfaction (Koehn, 2017). These criteria are standard in translation studies as they provide
measurable and comparable metrics to gauge the quality of translations.

Accuracy refers to the degree to which the translated text retains the meaning of the original text
(Hutchins & Somers, 1992). Fluency pertains to the naturalness and readability of the translated output,
ensuring that it flows smoothly and is easily understood by native speakers (Venuti, 2000). User
satisfaction involves feedback from the end-users, typically students and educators, on how well the tool
meets their translation needs (DePalma et al., 2019). These criteria are essential for evaluating any
translation tool, whether traditional or AI-based, as they reflect the practical usability of the tool in real-
world educational contexts.

Methodological approaches for evaluating translation tools also show similarities. Surveys, interviews,
and error analysis are commonly used to gather data on the performance of both traditional translation
tools and ChatGPT (Moorkens et al., 2018). Surveys and interviews provide qualitative insights into user
experiences and perceptions, while error analysis offers a quantitative assessment of the types and
frequency of errors produced by the translation tool. These methods collectively contribute to a
comprehensive evaluation, allowing researchers to draw meaningful conclusions about the tool’s
effectiveness.
A user-centric focus is another commonality in the evaluation of translation tools. Both traditional and
AI-based tools are assessed based on how well they serve the end-users, particularly students and
educators in foreign language faculties (Garcia, 2015). This focus ensures that the evaluation considers
practical aspects such as ease of use, integration into the curriculum, and overall user satisfaction.
Understanding the user experience is critical in determining the tool’s effectiveness in facilitating
language learning and translation tasks.

Differences in Technological Basis

Despite these similarities, significant differences exist between traditional translation tools and ChatGPT,
primarily due to their technological foundations. Traditional translation tools often rely on rule-based or
statistical methods (Hutchins & Somers, 1992). Rule-based systems use predefined grammatical rules and
linguistic structures to generate translations, while statistical methods analyze large corpora of bilingual
text to identify patterns and probabilities for translation. These approaches, though effective in many
cases, have limitations in handling the complexities and nuances of natural language (Koehn, 2017).

In contrast, ChatGPT leverages advanced neural network models and machine learning techniques,
enabling it to understand and generate human-like text (Radford et al., 2019). This technology allows
ChatGPT to process and learn from vast amounts of language data, improving its translation capabilities
over time. The use of deep learning algorithms enables ChatGPT to capture intricate patterns and
contextual meanings that traditional methods may overlook (Brown et al., 2020). This fundamental
difference in technology underpins many of the distinctions in performance and functionality between
ChatGPT and traditional tools.

Adaptability and Contextual Understanding

ChatGPT’s adaptability is one of its most significant advantages. As an AI model, it can continuously
learn and improve from new data, enhancing its translation accuracy and fluency (Brown et al., 2020).
This adaptability allows ChatGPT to keep pace with evolving language usage and emerging linguistic
trends, providing more up-to-date and contextually relevant translations. Traditional translation tools,
once their rules or algorithms are established, may not have the same capacity for ongoing improvement,
potentially leading to outdated or less accurate translations over time (Lopez, 2020).

The ability to understand context is another area where ChatGPT excels. Traditional translation tools
often struggle with nuanced meanings and idiomatic expressions that depend heavily on context (Venuti,
2000). ChatGPT, however, can analyze the broader context of the text to generate translations that are not
only accurate but also contextually appropriate (Radford et al., 2019). This capability is particularly
valuable in educational settings where understanding the subtleties of language is crucial for effective
learning and communication.

User Interaction and Research Focus

The interaction between users and ChatGPT is also more dynamic compared to traditional tools. ChatGPT
allows for real-time feedback and immediate re-translations, enabling users to refine and improve the
translation iteratively (Brown et al., 2020). This interactive feature enhances user satisfaction and can lead
to better learning outcomes as students can engage more actively with the translation process. Traditional
tools may not offer the same level of interactivity, potentially limiting their effectiveness in educational
applications (Garcia, 2015).

The research focus on ChatGPT often revolves around the impact of AI and machine learning
advancements on translation quality. Studies explore how these technologies enhance the tool’s
performance and the implications for language learning and teaching (Radford et al., 2019). In contrast,
research on traditional translation tools tends to focus more on their historical development and the
incremental improvements in algorithms and methodologies (Koehn, 2017). This difference in research
focus reflects the evolving nature of translation technologies and the growing interest in AI-driven
solutions.

Conclusion

The evaluation of ChatGPT for FOFL students at Hanoi University of Science and Technology provides a
comprehensive understanding of its effectiveness compared to traditional translation tools. By examining
the similarities and differences in their evaluation, researchers can identify the unique strengths and
limitations of ChatGPT. Its advanced technology, adaptability, contextual understanding, and dynamic
user interaction make it a promising tool for enhancing translation and language learning in educational
settings. However, ongoing research and evaluation are essential to address any challenges and maximize
the benefits of this AI-driven translation tool.
CHAPTER III: RESEARCH METHODOLOGY

3.1 Research questions

- How effective is ChatGPT in providing translations for FOFL students at Hanoi
University of Science and Technology (HUST)?
- What are the experiences and satisfaction levels of FOFL students using ChatGPT for
translation tasks?
- How does the accuracy of ChatGPT's translations compare to those of human translators?
3.2 Research approach
The objective of this study is to evaluate the effectiveness of ChatGPT in providing translations
for students enrolled in Foreign Languages for Specific Purposes (FOFL) at Hanoi University of
Science and Technology (HUST). Both qualitative and quantitative data were collected through a
structured Google Form to assess ChatGPT's impact on language learning.
3.3 Data collection tools
Questionnaires: Structured Google Form for collecting both qualitative and quantitative data,
distributed online questionnaires to 100 FOFL students, gathering numerical data on their
experiences with ChatGPT.
3.4 Sampling
- Research Population: FOFL students at Hanoi University of Science and Technology
(HUST).
- Research Sample: 100 FOFL students.
- Sampling Method: Convenience sampling.
- Sample Size: 100 students.
- Participants/Objects: FOFL students who participated in the study by providing
feedback and completing translation tasks.
3.5 Data collection procedure
The data collection procedure involves several steps to ensure comprehensive and reliable data is
gathered from the participants. Below is a detailed description of the procedure:
3.5.1 Preparation
Developing the questionares
- Design a structured Google Form that includes both qualitative and quantitative
questions.
- Ensure the questions cover various aspects such as user experience, translation accuracy,
and educational impact.
- Include sections for demographic information to analyze differences in responses based
on student characteristics.
Pilot Testing
 - Conduct a pilot test of the Google Form with a small group of FOFL students.
- Gather feedback to refine the questions for clarity and relevance.
- Make necessary adjustments based on the pilot test feedback.
3.5.2 Recruitment
Sampling method
- Use convenience sampling to select participants from the FOFL student population at
Hanoi University of Science and Technology (HUST).
- Aim for a sample size of 100 students to ensure a diverse range of responses.
Invitation to participate
- Send out invitations via email and social media platforms commonly used by HUST
students.
- Provide a brief overview of the study, its objectives, and the expected time commitment.
- Include a link to the Google Form and a deadline for submission.
3.5.3 Data collection
Qualitative Data Collection
- Include open-ended questions in the Google Form to gather detailed feedback on
experiences, usability, accuracy, and suggestions for improvement.
- Encourage participants to provide comprehensive and honest responses
Quantitative Data Collection
- Survey, distribute the structured Google Form to 100 FOFL students.
- Use Likert scale questions (e.g., 1 to 5) to measure user satisfaction, ease of use, and
perceived accuracy of ChatGPT’s translations.
- Provide a set of texts for translation using ChatGPT and have the same texts translated by
human translators.
- Collect the translated texts from both sources for comparison.

3.5.4 Data Management

Qualitative Data Management:

Coding

- Review and code the short answer responses to identify recurring themes and patterns.
- Use qualitative data analysis software if necessary to assist in the coding process.

Quantitative Data Management:

Scoring

- Assign numeric scores to survey responses based on the Likert scale.

Data Entry

- Input the survey data into statistical software (e.g., SPSS) for analysis.
- Ensure accuracy in data entry to avoid errors.

Cleaning

- Review the data to identify and correct any inconsistencies or errors.

- Remove any incomplete or invalid responses to ensure data quality.

3.5.4 Ethical Considerations

Informed Consent:

- Ensure participants are fully informed about the study’s purpose, procedures, and their
rights.
- Obtain informed consent from all participants before they complete the Google Form.

Confidentiality:

- Ensure that all responses are kept confidential and used solely for research purposes.
- Anonymize the data to protect the identity of the participants.

Voluntary Participation:

- Emphasize that participation is voluntary and that participants can withdraw at any time
without any negative consequences.

3.6 Data analysis

3.6.1 Data Analysis

Qualitative Data Analysis
- Thematic Analysis: Identified key themes such as usability, accuracy, and educational impact.
- Content Analysis: Quantified the frequency of specific themes in responses.

Quantitative Data Analysis

- Descriptive Statistics: Calculated means, medians, and standard deviations for survey
responses.
- Inferential Statistics: Conducted t-tests to compare translation scores between ChatGPT and
human translators, and ANOVA to examine satisfaction differences among student
demographics.
3.6.2 Reporting the Results
Qualitative Results
- Themes Identified:
- Ease of Use: Majority of students (80%) found ChatGPT user-friendly.
- Accuracy Concerns: 60% noted occasional inaccuracies, especially with context-
specific terms.
- Educational Benefits: 90% appreciated the instant feedback and learning opportunities
provided by ChatGPT.

Quantitative Results (Simulated Data)

Measure ChatGPT Average Score Human Translators Average Score

Translation Accuracy 3.8 4.5

Student Satisfaction 4.2 4.6

Survey Results
- Frequency of Use
- Daily: 15%
- Weekly: 35%
- Monthly: 30%
- Rarely: 15%
- Never: 5%
- Types of Tasks
- Academic Texts: 70%
- Technical Documents: 50%
- Casual Conversations: 40%
- Literature/Poetry: 20%
- Other: 10%
- Ease of Use: 4.3 out of 5
- Accuracy of Translations: 3.8 out of 5
 - Fluency of Translations: 4.0 out of 5

- Statistical Analysis
- Significant difference in accuracy scores between ChatGPT and human traánlators
(p < 0.05).
- No significant difference in satisfaction scores across different student demographics
(p > 0.05).

3.6.3 Interpreting the Results

- Summary: The data suggests that while ChatGPT is generally well-received by FOFL students
for its ease of use and educational benefits, there are notable concerns regarding its translation
accuracy.
- Explanation: The discrepancies in accuracy may be attributed to ChatGPT's limitations in
understanding complex, context-specific language nuances.
- Limitations: The study is limited by the sample size and the specific context of FOFL students
at HUST.
- Future Research: Further research could explore the longitudinal impacts of using ChatGPT in
language learning and expand the study to include a larger and more diverse student population.

.
CHAPTER IV: FINDINGS
 CHAPTER V: DISCUSSION
5.1. An overview of the main findings
The result of this study demonstrates that ChatGPT improves translation quality and efficiency, but also
has some limitations in handling specialized vocabulary and the potential misunderstanding of context.
Prior studies have also shown that using AI assistants can increase user confidence and productivity,
which the current findings corroborate in the context of language translation for students. However, the
study provides new empirical evidence on the performance of ChatGPT in translation tasks of FOFL
students at HUST. The detailed evaluation of accuracy, time savings, and confidence impacts offers
insights that can improve the existing weakness of ChatGPT. Overall, the study's main findings contribute
to the understanding of the technology's effectiveness and limitations for FOFL students at HUST.
References
Roza, V., & Zulhirawati, Z. (2023). Higher Students’ Perception of Using Chat GPT in
Translating English Texts . BiCED Proceeding, 1, 64–73. Retrieved from
https://proceedings.uinbukittinggi.ac.id/biced/article/view/278
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Carroll L. Wainwright, Pamela Mishkin,
Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language
models to follow instructions with human feedback.
Williams, Lawrence. (2006). Web‐Based Machine Translation as a Tool for Promoting
Electronic Literacy and Language Awareness. Foreign Language Annals. 39. 565 - 578.
10.1111/j.1944-9720.2006.tb02276.x.

Brown, T., Mann, B., Ryder, N., Subbiah, M., Kaplan, J., Dhariwal, P., … & Amodei, D. (2020).
Language models are few-shot learners. Advances in neural information processing systems, 33, 1877-
1901.

DePalma, D. A., Sargent, G., & Beninatto, R. (2019). Translation Management: A Guide to the
Translation of Scientific and Technical Documentation. Lulu.com.

Garcia, I. (2015). Translating by post-editing: Is it the way forward? Machine Translation, 29(1), 5-27.

Hutchins, W. J., & Somers, H. L. (1992). An Introduction to Machine Translation. Academic Press.

Koehn, P. (2017). Statistical Machine Translation. Cambridge University Press.

Lopez, A. (2020). Statistical machine translation. ACM Computing Surveys (CSUR), 40(3), 1-49.

Moorkens, J., Castilho, S., Gaspari, F., & Doherty, S. (Eds.). (2018). Translation Quality Assessment:
From Principles to Practice. Springer.

Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., & Sutskever, I. (2019). Language models are
unsupervised multitask learners. OpenAI Blog, 1(8).

Venuti, L. (2000). The Translation Studies Reader. Routledge.