VIETNAM NATIONAL UNIVERSITY HANOI

UNIVERSITY OF ENGINEERING AND TECHNOLOGY

TECHNOLOGY PROJECT REPORT

Major: Computer Science

Augmented Reality Application for Exploring Cultural

Heritage Sites: A Case Study of Cultural Complexes

Student : Nguyen Duc Tung

Student ID : 20021469
Class : K65-CA-CLC2
Mentors : Dr. Ngo Thi Duyen

HANOI - 2023
 TABLE OF CONTENTS
IMAGE CATALOG .................................................................................................... iii
Chapter 1: Introduction ............................................................................................... 1
1.1. Question ............................................................................................................ 1
1.2. About the application ........................................................................................ 1
1.2.1. The objective of the application ................................................................. 1
1.2.2. Scope of the application ............................................................................. 2
1.2.3. Technical overview .................................................................................... 2
Chapter 2: Theoretical background ............................................................................ 3
2.1. Augmented Reality ............................................................................................ 3
2.1.1. What is AR? ............................................................................................... 3
2.1.2. AR classification ........................................................................................ 4
2.2. ARKit ................................................................................................................ 6
2.2.1. Definition .................................................................................................... 6
2.2.2. Working mechanism ................................................................................... 6
2.2.3. Main features .............................................................................................. 7
2.3. ARCore.............................................................................................................. 8
2.3.1. Definition .................................................................................................... 8
2.3.2. Working mechanism ................................................................................... 8
2.3.3. Main features .............................................................................................. 9
Chapter 3: Design ....................................................................................................... 12
3.1. Select cultural heritage areas ........................................................................... 12
3.1.1. Temple of Literature Overview ................................................................ 12
3.1.2. Rationale for Selection ............................................................................. 14
3.1.3. Considerations for AR Integration ........................................................... 14
3.2. Models design ................................................................................................. 15
3.2.1. Scholar model ........................................................................................... 16
3.2.2. Model of the overseer ............................................................................... 17

i
 3.2.3. Model of the soldier.................................................................................. 17
3.2.4. A few other models .................................................................................. 18
3.3. Application design........................................................................................... 18
3.4. AR Interaction and Features ............................................................................ 20
Chapter 4: Conclusion and Future Work ................................................................ 21
4.1. Summary of Achievements ............................................................................. 21
4.2. Challenges and Lessons Learned .................................................................... 21
4.3. Future Enhancements ...................................................................................... 22

ii
 IMAGE CATALOG
Figure 1: Number of Mobile AR users worldwide from 2019 to 2023 .......................... 4
Figure 2: An app that uses marker-based AR ................................................................. 6
Figure 3: Oxyz coordinate system .................................................................................. 7
Figure 4: Operation diagram of ARKit ........................................................................... 7
Figure 5: Feature points (green dots) .............................................................................. 9
Figure 6: Feature point tracking in ARCore ................................................................. 10
Figure 7: Google image showing meshing in action .................................................... 11
Figure 8: Google image of demo ARCore app showing off subdued lighting ............. 12
Figure 9: A corner in the Temple of Literature ............................................................ 13
Figure 10: Scene of a reconstructed exam .................................................................... 14
Figure 11: QR (lower left corner) is being used at the Temple of Literature, but only
provides written content and audio commentary .......................................................... 15
Figure 12: Scholar model .............................................................................................. 16
Figure 13: Model of the overseer .................................................................................. 17
Figure 14: Model of the soldier .................................................................................... 17
Figure 15: A few other models ..................................................................................... 18
Figure 16: Operational process each time the application is run .................................. 18
Figure 17: The scene of the exam hall in the Temple of Literature is expected to be
displayed ....................................................................................................................... 20
Figure 18: An example of an application that includes the above functions ................ 21

iii
Chapter 1: Introduction
1.1. Question
Historical relics are valuable heritage of the country and people. Those relics bring us
scenes of bygone eras, and traditional cultural features of the nation in each design,
pattern, and motif. When combined with records, antiques, etc., we can get a panoramic
view of a historical period, and the development of the country's cultural history.
But with what remains of history, not everyone can imagine the images that once existed
of relics and relics. There are relics with patterns that have faded over time. Like ceramic
pieces that once carried unique images, their original colors and lines have been erased,
making them appear to be no different from ordinary broken pieces. Or the relics were
destroyed by war and bombs, with almost no old architecture or decoration left. And
especially, the activities of ancient people, there are things that have been erased from
the normal activities of modern life, thereby making it difficult for us to relate.
With the help of technology, there have been many ideas to learn and recreate old images
of history, using modern analysis and reconstruction. With modern scanning
technology, faded patterns on relics can be traced and recreated on computer screens.
With graphic applications and careful research, images of ancient architecture and daily
life can be recreated with real-life effects and actions. And especially AR technology
helps bring old images to real life, giving a vivid visualization of architecture and daily
life as if we were living in that era.
AR is a groundbreaking technology that is increasingly being used to enrich and enhance
real-world user experiences. In the context of exploring cultural relics, AR offers the
ability to subtly combine the virtual world and the real world, helping users interact and
gain a deep understanding of the relics in a unique way. Using AR not only enhances
the user experience but also opens up new opportunities to preserve and showcase
cultural relics.
1.2. About the application
1.2.1. The objective of the application
The application aims to create a unique and interactive cultural site exploration
experience, using Augmented Reality (AR) technology on mobile devices. be mobile.
Our main goals are:

1
 - Enhance User Experience: Create an interactive and scholarly experience,
helping users better understand the history, art, and cultural value of the relics
explored.
- Preserving and Promoting Cultural Values: Support the conservation of cultural
relics by presenting them creatively and attracting interest from the community.
- Creating a Foundation for Research: Provide high-quality data and information
to support research on the conservation and development of cultural relics.
1.2.2. Scope of the application
The application will focus on a number of special cultural areas, selected for their
diversity and historical importance. The scope of application includes:
- Explore Cultural Diversity: Focus on cultural areas with strong characteristics,
from historical sites to traditional arts, to meet the diversity of cultural heritage.
- Provide Detailed Information: The application will provide detailed and multi-
dimensional information about each monument, helping users clearly understand
the historical context and cultural significance.
- Cross-Platform Support: Develop applications on popular mobile platforms, such
as iOS and Android, to ensure widespread utility and reach from the user
community. I hope that this application will not only be an educational tool but
also a bridge between the current generation and our precious cultural heritage.
1.2.3. Technical overview
- App name: TimeTravelerHub
- Description: TimeTravelerHub combines the interactivity of AR with the
authenticity of cultural relics, providing a limitless exploration experience. With
detailed information, animations, and 3D interactivity, the app helps users
understand the surroundings and cultural values of historical areas.
- Technology Used:
o Flutter: Flutter is used for cross-platform app development, minimizing
concurrency and development costs, while providing a consistent
experience across both iOS and Android.
o ARCore and ARKit: Google's ARCore and Apple's ARKit are the main
Virtual Reality platforms for Android and iOS. They provide real-world
motion tracking and location capabilities, creating quality and smooth AR
experiences.

2
 o Blender: Blender is used to create and manage 3D models for cultural
monuments. Blender is a powerful tool for creating high-quality
animations and 3D models.
o Swift (iOS) and Java (Android): Swift is used for iOS development, while
Java is used for Android development. This ensures that the application
can be perfectly optimized and compatible with each individual operating
system.

Chapter 2: Theoretical background

2.1. Augmented Reality
2.1.1. What is AR?
Augmented reality (AR) is a digital technology that allows users to experience the real
world and its objects supplemented and enhanced with multimedia information such as
images, sounds, videos and 3D graphics. This information is displayed directly in the
real world through devices such as smartphones, tablets or AR glasses.
AR technology helps create a "virtual" layer of information in the real world, helping
users interact with information and objects naturally. Through sensors such as cameras,
location sensors, and motion sensors, AR can determine the user's location and
surrounding objects, creating an interactive and intuitive experience.
The applications of AR are very diverse, from entertainment, advertising, and education
to healthcare and industry. This technology is becoming increasingly popular and
continues to open up new opportunities in improving user experience and interaction
with the surrounding environment.

3
 Figure 1: Number of Mobile AR users worldwide from 2019 to 2023

2.1.2. AR classification
There are two techniques in Augmented Reality to calculate the pose, marker-based and
markerless methods:
- Marker-based AR: Marker-based AR, is an image or a small object such as a QR
code that is recognized by the AR device. When the AR device recognizes a
marker, it displays additional information on top of that object. Applications of
this type of AR include virtual makeup try-ons, viewing product information in
stores, interactive educational experiences, etc. A Marker-based application
requires the prior registration of an image for it to be recognized. This method is
the most widely used because the markers are easy to recognize and track, and
does not require powerful devices.
- Markerless AR: AR is not based on markers, additional information is displayed
in the real world using technologies such as GPS positioning, cameras, or other
image recognition technologies. Applications of this type of AR include giving
directions, searching for information about attractions, and helping architects and
designers check 3D models of buildings in actual locations,... A Markerless
application recognizes objects without having been registered beforehand. The

4
 implementation of this method is quite complicated. The algorithm that identifies
the objects must recognize different things, such as patterns, colors and other
characteristics that exist in the scene.
The marker-based AR technology will be used for the project for the following reasons:
- Easy to Deploy and Use: Marker-based AR systems are often easy to deploy and
use compared to other complex AR methods. Users can easily understand how to
use your application through identifying markers.
- High precision: Marker-based AR typically provides high accuracy in locating
virtual objects. Markers can be designed to have specific characteristics, helping
the system determine the position and direction of the viewpoint.
- Stable Performance: Compared to some other AR methods, marker-based AR
often has more stable performance, especially in poor or non-ideal lighting
conditions.
- Utilities For Exploring Cultural Relics: In your project exploring cultural relics,
the stability and accuracy of Marker-based AR can be an important factor.
Markers can be naturally integrated into symbols or objects within cultural areas,
creating an engaging experience for users.
- Platform Compatibility: Many application development platforms support
Marker-based AR, making it easy to integrate into popular mobile devices.
- Low cost: Marker-based AR can be a low-cost solution compared to some other
complex AR methods, helping to reduce your project development and
implementation costs.
- Good User Experience: Marker-based AR can create stable and interactive user
experiences, especially in cultural and tourism environments.

5
 Figure 2: An app that uses marker-based AR

2.2. ARKit
2.2.1. Definition
ARKit is Apple's Augmented Reality Application Development Kit released in June
2017. This kit is dedicated to iOS and allows to create augmented reality applications
for iPhone and iPad under iOS 11 or higher. The Apple ARKit SDK is available for
download for all iOS developers who have an Apple Developer Account.
2.2.2. Working mechanism
ARKit uses the Oxyz coordinate system according to the right-hand rule.

6
 Figure 3: Oxyz coordinate system

The diagram below describes The operation of ARKit:

Figure 4: Operation diagram of ARKit

2.2.3. Main features

a) Tracking

7
 Apple ARKit allows you to track the smartphone's position in the real world in real
time. To do this, the SDK combines the VIO (Visual Inertial Odometer) with data
from the camera and the motion sensor of the device.
b) Understanding the environment
With ARKit, iPhone and iPad can identify real-world vertical and horizontal flat
surfaces in real time for realistic integration of virtual elements.
c) Static 2D image recognition
This feature allows to store images in the application and to detect them in a real
environment, to display and superimpose virtual information on these images.
d) Brightness estimation
ARKit is also able to use the brightness sensor to determine the amount of light in
the surrounding room to apply the degree of brightness to the virtual object. ARKit
can also create reflections of real objects on virtual metal.
2.3. ARCore
2.3.1. Definition
In 2014, Google launched its first Tango SDK dedicated to Augmented Reality, it
required a smartphone equipped with a depth sensing camera to work. Tango's
particularity was to allow its users to map environments in 2D and 3D. Lenovo phab 2
pro was the first smartphone to integrate Tango technology, enabling its users to benefit
from functionalities related to augmented reality. This project did not last very long
because it was limited to devices with essential sensors that no smartphone had by
default. Google is relaunching the adventure of augmented reality by launching the
ARCore project in March 2018. Unlike Tango, ARCore is usable on several
smartphones with Android Nougat (7.0) and IOS devices that support ARKit. ARCore
is based on the Tango project with fewer features that do not depend on specialized
cameras, allowing a larger number of devices to be compatible with its technology.
2.3.2. Working mechanism
ARCore's working mechanism essentially consists of two processes: tracking the
location of the mobile device as it moves and building its own understanding of the real
world.
ARCore's motion tracking technology uses the phone's camera to identify “feature
points” and track how they move over time. Combining the movements of these points

8
and readings from the phone's inertial sensors, ARCore determines both position and
direction as the phone moves through space.
By identifying “feature points”, ARCore can detect flat surfaces, such as tables or floors,
and can also estimate the average lighting in the area around it. These capabilities
combine to enable ARCore to build its own understanding of the world around it.
ARCore's understanding of the real world allows us to place objects, annotations, or
other information in a way that is seamless with the real world. With mobile device
location and movement tracking, we can move around and view these objects from any
angle. Even if we leave that location, when we return, the virtual objects will still remain
in the same position.

Figure 5: Feature points (green dots)

2.3.3. Main features

a) Motion tracking
ARCore uses the phone's camera to observe the characteristic points of the scene and
the data from the Inertial Measurement Unit (IMU) sensor, it determines the position
and orientation of the device as it moves. This allows the virtual objects to be
properly positioned. As your phone moves in the real world, ARCore measures
concurrent mapping, or COM, to understand how the phone relates to the world
around it. ARCore detects distinct features visually through the camera, called
feature points, and uses these points to calculate changes in position. Visual
information combined with inertial measurements from the IMU estimates the pose
(position and orientation) of the camera relative to the real world over time.
By aligning the pose of the virtual camera that will display your 3D content with the
camera pose provided by ARCore, programmers will be able to render virtual assets

9
from an accurate perspective. The rendered virtual image can be overlaid on top of
the image captured from the camera, making it appear as if the virtual content is part
of the real world.
Tracking a user's motion and ultimately their position in 2D and 3D space is
fundamental to any AR application. ARCore allows us to track position changes by
identifying and tracking visual feature points from the device's camera image. An
example of how this works is shown in this figure:

Figure 6: Feature point tracking in ARCore

b) Environmental understanding
The better an AR application understands the user's reality or the
environment around them, the more successful the immersion. We already saw how
ARCore uses feature identification in order to track a user's motion. Yet, tracking
motion is only the first part. What we need is a way to identify physical objects or
surfaces in the user's reality. ARCore does this using a technique called meshing.
ARCore continuously improves its understanding of the real world by detecting
characteristic points and planes. ARCore looks for clusters of feature points that
appear on horizontal or vertical surfaces, such as tables or walls, and makes these
surfaces appear in the application as planes. ARCore can also determine the
boundaries of planes and transmit that information to the application. We can use
that information to place virtual objects on flat surfaces.
Because ARCore uses feature points to define planes, planes without textures, such
as white walls, may not be identified accurately.

10
 Figure 7: Google image showing meshing in action

c) Light estimation
The light estimation that allows the smartphone to estimate the ambient light in order
to perfectly integrate the 3D elements into the real world. The goal of light estimation
in ARCore is to make virtual elements feel like they belong in the real world, creating
a more convincing and natural AR experience. This feature is particularly important
for applications where accurate lighting and shading can significantly enhance the
user's perception of virtual objects.

11
 Figure 8: Google image of demo ARCore app showing off subdued lighting

Chapter 3: Design
3.1. Select cultural heritage areas
In the design phase of our augmented reality (AR) application for exploring cultural
heritage sites, the careful selection of cultural areas plays a pivotal role. The chosen
cultural site for this study is the renowned Temple of Literature.
3.1.1. Temple of Literature Overview
The Temple of Literature, located in Hanoi, Vietnam, is a historical and cultural symbol
renowned for its architectural and educational significance. Built in 1070 during the Ly
Dynasty, it originally functioned as a Confucian temple and later became Vietnam's first
national university. The site is dedicated to Confucius and honors scholars and literary
achievements.

12
 Figure 9: A corner in the Temple of Literature

The Temple of Literature - Quoc Tu Giam is the most typical symbol of the studious
spirit of Vietnamese people. This place is considered the country's first university,
training many generations of outstanding people of the ancient Dai Viet country. Here,
we worship the sages and saints of Asian Culture in general and Vietnam in particular,
such as Confucius - the founder of Confucianism, Chu Van An - the eternal teacher of
the Vietnamese people. In particular, Khue Van Cac (Khue Van Attic) - a building inside
the Temple of Literature - Quoc Tu Giam - was chosen as a symbol of the thousand-
year-old civilization capital of Hanoi. As the center of learning throughout the country,
Quoc Tu Giam is the place where many important competitions are held, selecting the
nation's talents. With the desire to bring visitors a vivid experience of the solemn,
ancient atmosphere of ancient exams, I decided to use AR technology to recreate the
exam scene in the Temple of Literature.

13
 Figure 10: Scene of a reconstructed exam

3.1.2. Rationale for Selection

Several factors contribute to the selection of the Temple of Literature as the focal point
for our AR application:
a) Historical and Cultural Richness
The Temple of Literature embodies a wealth of historical and cultural elements,
providing users with a diverse and immersive experience. Its centuries-old
architecture, statues, and inscriptions offer a unique glimpse into Vietnam's cultural
heritage.
b) Educational Significance
As a former center of learning, the Temple of Literature aligns with the educational
theme of the AR application. Exploring this site allows users to connect with the rich
intellectual history of Vietnam and fosters an appreciation for the pursuit of
knowledge.
c) Tourist Attraction
Being a popular tourist destination, selecting the Temple of Literature ensures that
the AR application has a broad audience. This choice aims to enhance the overall
visitor experience and contribute to the promotion of cultural tourism.
3.1.3. Considerations for AR Integration
When selecting the Temple of Literature, considerations for seamless AR integration
were paramount. Factors such as site accessibility, and the availability of historical data

14
influenced the decision-making process. Additionally, partnerships with local
authorities and experts were established to ensure accurate representation and
interpretation of the cultural site within the AR application.
At the Temple of Literature, many guide signs with QR codes have been used.
Therefore, we can effectively utilize available resources, as well as create favorable
conditions for tourists because most people are now familiar with the experience of
using QR.

Figure 11: QR (lower left corner) is being used at the Temple of Literature, but only provides written
content and audio commentary

In the subsequent sections, we will delve into the specifics of designing 3D models
representing the Temple of Literature and the overall application architecture,
respectively. These components aim to create an engaging and informative AR
experience for users exploring this cultural heritage site.
3.2. Models design
The simulation models were designed on Blender, an effective 3D design software for
three-dimensional graphics work. To optimize storage, download and display, the
following designs need to be compressed to maximum capacity. This makes rendering
the arena scene faster and smoother because low-capacity models load faster and

15
consume fewer processing resources. There are many models designed to simulate
people and things at the competition site. These models, after being designed, will be
uploaded to the web and downloaded and displayed by the application if necessary.
3.2.1. Scholar model

Figure 12: Scholar model

16
3.2.2. Model of the overseer

Figure 13: Model of the overseer

3.2.3. Model of the soldier

Figure 14: Model of the soldier

17
3.2.4. A few other models

Figure 15: A few other models

3.3. Application design

The application is a place that gives users real-life experiences, it will recreate the entire
exam scene designed on the web and bring it into the real environment. To synchronize
with web design, the application must have the function of automatically updating data,
and then displaying it to the user. Basically, an application life cycle will include the
following steps:

Figure 16: Operational process each time the application is run

18
Once the user opens the app, it calls the API to check for updates of exam fields stored
on the web. If new updates are available, the app will synchronize with the web part by
applying updates to the position, rotation, and size of the models. If a model needs to be
displayed in the test field but is not yet stored in the application, it will call the API that
helps download these missing models. This way, whenever the web test field changes,
the application will automatically update the changes and apply them to the actual
display environment.
After checking and updating the application, the user will next scan the QR code placed
in real space. When put into practical use, the QR code will be placed by the Temple of
Literature at a fixed position in space, at a height above the ground and the directions of
rotation relative to the three axes x,y,z. The position and direction of rotation of this QR
code must be synchronized with the position and direction of rotation of the QR code
simulated on the web environment. QR codes in the real environment are completely
changeable, we can change their position and rotation direction, however in the web
environment, we must these parameters to match reality. This can be done extremely
easily because the web part already supports changing the position of the QR code to
simulate the parameters of its position, size, and rotation direction in real space.
The reason why the position and direction of rotation of the QR code are important is
that it will be set as the origin for display in real space. When the user scans the QR
code, its position and rotation direction relative to the phone will be set as the origin
coordinates to display models according to this coordinate system. So, if in reality the
QR code is placed one meter above the ground, then on the web it should also be one
meter above the xz plane because the web environment is completely simulating the real
environment. Once you have the origin coordinates, the application will use the
parameters of the model's position, size, and rotation direction to display and recreate
the exam scene in the real environment.

19
 Figure 17: The scene of the exam hall in the Temple of Literature is expected to be displayed

3.4. AR Interaction and Features

Each 3D model within the application represents a significant element within the
Temple of Literature, such as the Scholar, Overseer, and Soldier statues. These models
are not merely static representations but are enriched with interactive features:
Information Display Button
- Button Functionality: Each 3D model includes an information display button.
- User Interaction: Users can interact with the models by tapping on the
information button, triggering the display of additional information.
Multi-Media Content Integration
- Text, Images, and Sounds: The information displayed upon interaction is not
limited to text. Users will encounter a variety of multimedia content, including
historical context in text form, archival images, and audio clips relevant to the
specific model.
Contextual Information
- Context-Aware Content: The information presented is context-aware, offering
users historical details about the particular statue or element within the Temple
of Literature. This approach provides a richer and more meaningful experience.

20
 Figure 18: An example of an application that includes the above functions

Chapter 4: Conclusion and Future Work

4.1. Summary of Achievements
Although the development of the augmented reality (AR) application is still in its initial
stages, the project has outlined several key achievements that form the foundation for
future development:
- Vision and Planning: Establishment of Project Goals: A clear vision has been
outlined, with a focus on developing an AR application to explore the Temple of
Literature and immerse users in its cultural richness.
- Preliminary Design: Conceptualization of Interactive Models: The
conceptualization phase has defined the structure of the application, with a plan
to incorporate interactive 3D models and engage users with informative content.
- Stakeholder Engagement: Initial Collaboration: Preliminary steps have been
taken to engage with local authorities and cultural experts, setting the stage for
fruitful collaboration in the future.
4.2. Challenges and Lessons Learned
Despite the project being in its infancy, the planning and preparatory stages have already
presented valuable insights and lessons:

21
 - Anticipated Technical Challenges: Foreseeing Integration Complexity:
Anticipated challenges in integrating AR models and ensuring cross-device
compatibility highlight the need for a robust technical strategy.
- User Experience Considerations: Early Emphasis on UX: Initial considerations
for user interface design and information presentation underscore the importance
of user-centric design from the project's outset.
- Importance of Collaboration: Coordination with Stakeholders: Early efforts in
collaboration with stakeholders emphasize the vital role of engaging with local
authorities and experts for accurate cultural representation.
4.3. Future Enhancements
Looking forward, the project holds significant potential for growth and improvement.
The following are envisioned as areas for future enhancements:
- Detailed 3D Models: A priority will be given to refining and expanding the 3D
models to accurately represent various cultural elements within the Temple of
Literature.
- Educational Features: Incorporation of Learning Modules: Future development
will focus on integrating educational modules to cater to diverse audiences,
including students and those seeking a deeper understanding of the historical site.
- Accessibility Integration: Ensuring Inclusivity: Special attention will be given to
incorporating accessibility features to make the AR application inclusive and
accessible to users with varying needs.
- Expansion to Additional Cultural Heritage Sites: As the AR application matures, consider
the possibility of expanding beyond the Temple of Literature to incorporate more cultural
heritage sites.

22
References
[1] Oufqir, Z., El Abderrahmani, A., & Satori, K. (2020). ARKit and ARCore in serve
to augmented reality. 2020 International Conference on Intelligent Systems and
Computer Vision (ISCV)
[2] Micheal Lanham (2018). Learn ARCore – Fundamentals of Google ARCore. Learn
to build augmented reality apps for Android, Unity, and the web with Google ARCore
1.0
[3] Carmigniani, J., & Furht, B. (2011). Augmented Reality: An Overview. Handbook
of Augmented Reality
[4] Cranmer, E. E., tom Dieck, M. C., & Fountoulaki, P. (2020). Exploring the value of
augmented reality for tourism. Tourism Management Perspectives
The project was built and developed based on the experimental and research results of
previous students as well as those who followed:
[5] Nguyen Van Phong, Giap Van Huy, Nguyen Quang Huy, Phan Thanh Ngoc, Duong
Hieu, 2023, Applying AR technology to re-recreate the examination school at the Quoc
Tu Giam Temple of Literature

23