Text-to-Image Generation Using XLNet

Abstract—Text-to-image generation is a rapidly advancing dependencies in textual descriptions. XLNet, a transformer-

field within the realm of artificial intelligence. The transformation based autoregressive model, offers improvements over prior
of textual descriptions into realistic images holds significant models by capturing bidirectional context without the
potential across various domains, including creative industries, limitations of traditional transformers like BERT [3]. GANs
design automation, and human-computer interaction. This paper have been widely used for image generation, achieving
explores a novel approach using XLNet, an autoregressive remarkable results in producing realistic images. Variants like
language model, for text encoding and GAN-based image Pix2Pix and CycleGAN have been developed for specific tasks
generation. The proposed architecture integrates the XLNet like image-to-image translation.
model with a Conditional Generative Adversarial Network
(CGAN) to create detailed images from textual descriptions. III. METHODOLOGY
Experimental results demonstrate the effectiveness of this hybrid
architecture in generating realistic images and point to potential XLNet comprises input embeddings, multiple Transformer
improvements for future research. blocks with self-attention, position-wise feedforward networks,
layer normalization, and residual connections. Its multi-head
Keywords—xlnet, cgan, neural networks. self-attention differs by allowing each token to attend to itself,
enhancing contextual understanding compared to other models
I. INTRODUCTION [4]. XLNet captures the semantic meaning of the text and
Text-to-image generation has been a subject of considerable generates high-dimensional vectors representing the textual
research due to its ability to convert human language into visual content. These vectors are then used to guide the image
content. Traditional models have largely focused on recurrent generation process, ensuring the generated images accurately
networks or basic transformer architectures. However, recent reflect the input descriptions.
advancements in language models, such as XLNet, offer new
possibilities for encoding textual data in a more contextualized
and robust manner. This research investigates how XLNet,
combined with a GAN architecture, can enhance the text-to-
image pipeline.
A. Problem Definition
The task of text-to-image generation is to create a visual
representation of a given natural language description. Unlike
image captioning, where an image is already available, this task
requires the system to learn the semantics of text and synthesize
images from scratch. The challenge lies in the ability of the
model to understand textual nuances and map them to complex
visual structures.
B. Objectives

 To implement a text-to-image generation model using Figure 1: The model consists of three main components.
XLNet as a text encoder and a GAN-based
architecture for image synthesis. The generator network takes the encoded text vectors as input
and generates images that match the textual descriptions. The
 To evaluate the model's performance on a dataset of
discriminator evaluates the generated images, distinguishing
flower images and corresponding descriptions.
between real and synthetic images [5]. The generator and
 To compare the effectiveness of XLNet-based encoding
discriminator are trained adversarial, with the generator
against other text encoding techniques.
improving its ability to create realistic images and the
discriminator enhancing its ability to identify fakes. Training
II. RELATED WORK our model involves alternating between optimizing the
Previous efforts in text-to-image synthesis have utilized generator and the discriminator. XLNet excels in capturing
various deep learning techniques, notably GANs, coupled with bidirectional context and has achieved state-of-the-art
recurrent neural networks (RNNs) [1] or simple transformer performance on various NLP tasks.
models like BERT. While GANs such as StackGAN and
AttnGAN [2] showed promising results, their reliance on older
text models limited their ability to effectively capture long-range

XXX-X-XXXX-XXXX-X/XX/$XX.00 ©20XX IEEE

 Figure 4: Upsampling process in generator.

Figure 2: XLNet comprises input embeddings, multiple

Transformer blocks
Figure 5: XLNet generator code
IV. DATASET
We utilized a dataset of high-quality flower images, each
accompanied by descriptive textual information. The dataset Discriminator: The discriminator is tasked with
was preprocessed to standardize image sizes (64x64) and distinguishing between real images and fake images
tokenize the textual descriptions using the XLNet tokenizer. generated by the generator. It takes both the image and the
encoded text as input, and classifies whether the image is real
A. Model Architecture or fake. We utilize a convolutional neural network (CNN)
The model consists of three main components: architecture with LeakyReLU activations and Batch
Normalization [7] for this purpose. The discriminator is
Text Encoder (XLNet-based): XLNet is a state-of-
optimized using binary cross-entropy loss.
the-art transformer-based language model known for its
autoregressive nature and ability to handle longer text
sequences. In our architecture, XLNet is utilized to encode the
input textual descriptions into a meaningful vector
representation. This encoded context serves [6] as input for the
GAN model, which conditions the image generation process on
the textual features.
The XLNet text encoder is implemented using the XLNetModel
from the Hugging Face library. The model outputs hidden states,
which are averaged across the sequence dimension to obtain a
single context vector.
Figure 6: Discriminator distinguishing between real images
and fake images generated by the generator

Figure 7: XLNet discriminator code

Figure 3: Text encoder of the XLNet B. Training Strategy

The model is trained in a standard adversarial manner. The
Generator: The generator in our model follows a typical generator aims to minimize the classification error made by
convolutional transpose architecture, conditioned on the the discriminator, while the discriminator attempts to
encoded text produced by XLNet. The generator network is correctly classify real and fake images. The models are
designed to take both a noise vector and the text embedding as optimized using Adam optimizers with learning rates of 1e-
input and generate a 64x64 pixel image. The model uses a series 4 for the text encoder and discriminator, and 5e-4 for the
of convolutional transpose layers to up sample the input, along generator.
with batch normalization and activation layers.
C. Evaluation metrices
The model's performance is evaluated based on two metrics:
 1. Discriminator Loss: Measures how well the B. Result
discriminator distinguishes between real and generated After three epochs, the model demonstrated promising results
images. in generating realistic images from text. The generator loss
2. Generator Loss: Measures the generator's ability to decreased significantly, indicating that the images produced
produce convincing images. were increasingly difficult for the discriminator to distinguish
as fake. Additionally, the discriminator maintained a balanced
To improve the training stability and performance of the GAN ability to identify real images from generated ones.
[8], we also include auxiliary loss functions in addition to these
primary losses. Perceptual loss, for instance, is employed [9] to Throughout training, we monitored the losses of both the
guarantee that images that are generated retain high-level generator and the discriminator to ensure stable and balanced
features and semantic content that is comparable to actual training. We observed that our model successfully generated
photographs. By aligning the intermediate features of generated images that accurately reflected the given descriptions, with
images with those of genuine images [10], a different technique notable improvements in visual quality over time. Examples of
called feature matching loss helps to improve the overall generated images at various epochs are provided, illustrating
consistency and quality of images. Our goal is to create a more the model's progress. Comparative analysis with baseline
powerful and reliable model that can produce visually models demonstrates the superiority of our approach in terms
appealing and semantically accurate high-quality photos by of both visual fidelity and alignment with textual descriptions.
incorporating these extra loss functions.

V. EXPERIMENTS AND RESULTS

A. Experimental Setup:
The model was trained on a GPU for 100 epochs with a
batch size of 32. We used the Adam optimizer with learning
rates adjusted through a step-learning rate scheduler. The
dataset was split into training and validation sets using stratified
sampling.
Figure 10: Train loss vs Validation loss

Figure 8: Fake vs real image comparison result during the

training
Figure 11: Generated image from the user input

We conducted a qualitative analysis of the generated images,

examining their visual coherence and relevance to the input
descriptions. Our model produces images with high visual
fidelity and accurately reflects the semantic content of the text,
indicating the success of our approach.
VI. CONCLUSION
This research explored the use of XLNet for text encoding in a
text-to-image generation task. By leveraging the rich contextual
understanding of XLNet, the model was able to create more
realistic and contextually accurate images. Future work could
explore scaling the model to more complex datasets and
integrating attention mechanisms to enhance the fidelity of the
Figure 9: Training Loss and Accuracy Progression Across generated images.
Epochs for XLNet Model
 REFERENCES

[1] Z. Akata, S. Reed, D. Walter, H. Lee, and B. Schiele, [7] J. Y. Zhu, T. Park, P. Isola, and A. A. Efros, "Unpaired
"Evaluation of Output Embeddings for Fine-Grained Image Image-to-Image Translation Using Cycle-Consistent.
Classification," in Proc. CVPR, 2015.
Adversarial Networks," in Proc. IEEE International
[2] K. Gregor, I. Danihelka, A. Graves, D. Rezende, and D. Conference on Computer Vision, pp. 2223-2232, 2017.
Wierstra, "Draw: A Recurrent Neural Network for Image
Generation," in Proc. ICML, 2015. [8] S. E. Reed, Z. Akata, X. Yan, L. Logeswaran, B. Schiele,
and H. Lee, "Generative Adversarial Text to Image
[3] J. Yang, S. Reed, M.-H. Yang, and H. Lee, "Weakly- Synthesis," arXiv preprint arXiv:1605.05396, 2016.
Supervised Disentangling with Recurrent Transformations for
3D View Synthesis," in Proc. NIPS, 2015. [9] K. Sohn, W. Shang, and H. Lee, "Improved Multimodal
Deep Learning with Variation of Information," in Proc. NIPS,
[4] I. Goodfellow, J. Pouget-Abadie, M. Mirza, B. Xu, D. 2014.
Warde-Farley, S. Ozair, ... and Y. Bengio, "Generative
Adversarial Nets," Advances in Neural Information [10] O. Vinyals, A. Toshev, S. Bengio, and D. Erhan, "Show
Processing Systems, vol. 27, pp. 2672-2680, 2014. and Tell: A Neural Image Caption Generator," in Proc. CVPR,
2015.
[5] Z. Yang, D. Dai, Y. Yang, J. Carbonell, R. R.
Salakhutdinov, and Q. V. Le, "Generalized Autoregressive
Pretraining for Language Understanding," Advances in Neural
Information Processing Systems, vol. 32, pp. 5754-5764,
2019.

[6] P. Isola, J. Y. Zhu, T. Zhou, and A. A. Efros, "Image-to-

Image Translation with Conditional Adversarial Networks," in
Proc. IEEE Conf. on Computer Vision and Pattern
Recognition, pp. 1125-1134, 2017.