Landing Page Personalization at Expedia Group

Pavlos Mitsoulis-Ntompos, Dionysios Varelas, Travis Brady, J. Eric Landry, Robert F. Dickerson,
Timothy Renner, Chris Harris, Shuqin Ye, Abbas Amirabadi, Lisa Jones, Javier Luis Cardo
{pntompos,dvarelas,tbrady,johlandry,rdickerson,trenner,chrharris,sye,aamirabadi,ljones1,jlcardo}@expediagroup.com
Expedia Group

ABSTRACT For example:

Recommender systems have become critical tools for e-commerce • Vrbo Home Page: Travelers can land on this page by en-
businesses in recent years and online travel platforms such as Expe- tering vrbo.com directly on their browser. Figure 1 shows
dia Group have made heavy use of them in production. Contempo- an example of Vrbo’s home page.
rary travel platforms benefit greatly from the use of recommender • Hotels.com Home Page: Travelers can land on this page
systems as very often the space of products (trips) is quite large by entering hotels.com directly on their browser.
and shopping cycles often stretch into the weeks. • Vrbo Landing Pages: Landing pages refer to destination
Expedia Group has successfully trained and deployed multiple landing pages. Travelers usually land on destination landing
types of recommender systems in order to help our travelers find pages via searching on search engines.
the perfect destination and property for them. In recommender sys- • Hotels.com Landing Pages: Similar to Vrbo, landing pages
tems literature, much attention is paid to the mathematical aspects refer to destination landing pages. Figure 2 depicts a landing
of the field but here we focus on best practices in applying recom- page for San Diego. Travelers usually land on destination
mender systems in large-scale e-commerce for the improvement landing pages via searching on search engines. In this ex-
of browsing and shopping experiences. ample, a traveler probably searched for “san diego vacation
In this paper, we describe how we personalize the user experi- homes”.
ence on a number of our core pages by exploiting existing inter- Prior to the introduction of ML-driven personalization techniques
nal recommender systems and relevant recommender system liter- our landing and home pages did not recognise or change the expe-
ature. Additionally we note several critical lessons learned about rience for travelers based on what we already knew about them.
the importance of a robust machine learning platform, the need to Returning travelers would often have to go through the long pro-
apply engineering best practices and how best to integrate and test cess of restarting their search, find properties they liked, and look
recommender systems in production. for comparisons and alternatives themselves.
Our motivation for this work was to make the booking experi-
KEYWORDS ence effortless and to enable travelers to make progress based on
Recommender Systems, Machine Learning, Travel, E-Commerce, their previous visits. We hypothesize that by making contextually
Machine Learning Platform, Product Development relevant destination and property recommendations to travelers
based upon their recent browsing behavior we can substantially
1 INTRODUCTION increase the likelihood of matching them with their ideal trip. We
Typically users of our sites (e.g. expedia.com, hotels.com, vrbo.com) provide evidence to support this hypothesis by exploiting our in-
begin their trip planning process on one of two pages: the home ternal Machine Learning (ML) platform and two systems already
page or a “destination landing page” (DLP), a page created for a deployed within the organization for other internal use cases: a
specific destination such as Paris, New York or Tokyo. Both home Property Recommender System (PRS) and a Destination Recom-
pages and destination landing pages contain separate sets of rec- mender System (DRS).
ommendations of destinations and properties in the form of horizontal- The proposed solution must address the following challenges:
scrolling carousels. Not all travelers need both sets of recommen- (1) Software engineers from landing pages and home page team
dations since many already know their destination and are then should easily integrate the two recommender systems.
determining which property to book. Yet others have some ideas (2) The recommender systems should provide predictions in a
about the type of trip they want to take (e.g. beaches in Texas), but synchronous and an asynchronous fashion.
are flexible as to the specific destination. (3) The recommender systems should be able to leverage very
Many travelers will make multiple visits to our site before mak- recent history for each traveler.
ing a booking. In fact, repeat travelers make up a significant frac-
tion of traffic to both landing and home pages. We can use re- 1.1 Use Cases
cent signals such as properties viewed and destinations searched
Figure 3 shows a high-level abstraction of Traveler’s Shopping Map.
to reduce the shopping time for repeat travelers by recommend-
The shopping journey of travelers is non-linear, and there are many
ing properties and destinations similar to those they have already
different traveler journeys. Some land on our websites and belong
searched for1 . But for travelers identified as new to the site, we of-
to Discovery phase, then they move to Shop phase and, finally, Book-
ten have no historical data for them and therefore cannot use these
ing phase. However, there are many travelers who know already
strategies.
where and when to travel, and at which property they want to stay
1 Long term signals are much less useful, as they are often for a different trip. and they start in the Shop phase. Others already have a shortlist of
KDD ’20, August 22–27, 2020, San Diego, CA Mitsoulis-Ntompos, Varelas, Brady, Landry, Dickerson, Renner, Harris et al.

Figure 2: Hotels.com Landing Page

potential properties in mind, but haven’t made a final selection and

they begin in the Booking phase. The purpose of home and landing
pages is to fulfill the needs of all travelers that are in Discovery
and Shop phase. These are some examples of selection constraints
for travelers in the Discovery and Shop phase:
(1) Traveler has not selected the date, destination or property
(Discovery),
(2) Traveler knows the date and destination, but has not se-
Figure 1: Vrbo Home Page lected the property (Discovery),
(3) Traveler knows the destination, but has not selected the
property and is flexible on the dates (Discovery),
Landing Page Personalization at Expedia Group KDD ’20, August 22–27, 2020, San Diego, CA

(4) Traveler has selected the destination and date, and has a of our home and landing pages contain carousels showing recent
shortlist of properties (Shop), searching and browsing behavior as well as separate recommen-
(5) Traveler has selected the destination and property (or has a dation carousels based on said recent behavior. We proved the
shortlist of either / both) and dates (Shop). hypothesis that recent activity is correlated with shortlisted prop-
erties and/or destinations using online experiments. Typically we
choose the simplest possible solution to start as we believe that sim-
ple solutions can be very powerful in terms of providing better user
experience and provide baselines for future ML based approaches.

2 IMPLEMENTING PERSONALIZATION FOR

LANDING PAGES
To implement landing and home page personalization, we first tried
a couple of heuristics. Applying simple methods allowed us to both
establish an experimental baseline and address a number of techni-
cal issues before adding the complexity of machine learning. With
a baseline established, the heuristic logic can then be replaced with
machine learning models and tested for effectiveness and end-user
Figure 3: Traveler Shopping Map. In the Discovery phase, impact. The aforementioned heuristic of most recent activity was
travelers are open to any of the following: dates, destination employed for both Discovery and Shop phases.
and/or properties. In the Shop phase, travelers are creating After employing these heuristic approaches, we utilized recom-
a shortlist of destinations and properties, and are conduct- mender systems to further enhance personalization. These recom-
ing further research on them. The traveler books a specific mender systems are explained in the following subsections.
property for specific dates in the Book phase, and physically
stays at the booked property during the Stay phase. 2.1 Property Recommender System
The goal of a Property Recommender System (PRS) is to identify
1.1.1 Discovery. During the Discovery phase travelers can either the most similar properties to recent properties travelers have in-
be looking for a destination to travel to and/or a property to book teracted with. Formally, let 𝑃 = {𝑝 1, ..., 𝑝 𝑀 } denote the set of prop-
within a destination they already decided to visit. erties where 𝑀 is the number of properties and 𝑇 = {𝑡 1, ..., 𝑡 𝑁 } de-
At Hotels.com and Vrbo we aim to facilitate this process by pro- note the set of travelers where 𝑁 is the number of travelers. PRS
viding travelers destination and property recommendations in or- can then be formulated as a problem of conditional probability dis-
der to narrow down the plethora of available destinations and prop- tribution estimation
erties. ( )
𝑃 traveler will click 𝑝𝑘 |traveler already clicked 𝑝𝑖 1 , ..., 𝑝𝑖𝑚𝑖
On the Hotels.com homepage, recommender systems are used
on:
where the indices 1 ≤ 𝑖 1 ,…,𝑖𝑚𝑖 ≤ 𝑀 represent an exhaustive list of
(1) The search box where travelers receive recommendations indices for traveler 𝑡𝑖 ’s previous clicks. So, 𝑚𝑖 is the total number
for nearby destinations based on their geographical loca- of clicks by 𝑡𝑖 .
tion.
(2) The properties carousel where travelers are shown person- 2.1.1 Model. We decided to model the aforementioned conditional
alized recommendations based on their previous browsing probability distribution using Word2Vec and specifically the Con-
sessions. tinuous Bag-of-Words (CBoW) [10], [9] model. Similar approaches
(3) The destination panel where travelers are shown person- that used Word2Vec for item recommendations can be found in [5],
alized recommendations based on their previous browsing [6], [11], [14]. Applied to property recommendations, CBoW pre-
sessions. dicts the target property 𝑝𝑘 based on the properties 𝑝𝑘−𝑐 and 𝑝𝑘+𝑐
Destination recommenders are used on destination landing pages that appear around it in a given corpus (described in the following
where recommended destinations similar to the one users searched section), where 𝑐 is the window size representing property context
for are shown. (Figure 4). The architecture and optimization targets are the same
In the same manner, Vrbo’s home page and landing pages show as in [9] but our use case differs from language modeling problems
personalized property recommendations based on their previous in the choice of corpus specifying the sequential data to which the
browsing sessions. model is applied.

1.1.2 Shop. In this phase, travelers have already shortlisted a num- 2.1.2 Dataset and Application. For the experiments, anonymized
ber of properties and destinations. Many of them have even se- clickstream data is collected for millions of travelers and properties.
lected their destination, and they just need to make decision be- Specifically, for each traveler 𝑡𝑖 we create training instances by col-
tween a handful of properties. lecting all the previously seen properties into sequences 𝑝 𝑗1 , ..., 𝑝 𝑗𝑚𝑖 .
In order to facilitate travelers in the Shop phase, we initially im- From those training instances we slide a window in order to create
plemented and deployed a couple of heuristic algorithms. Many training examples [10].
KDD ’20, August 22–27, 2020, San Diego, CA Mitsoulis-Ntompos, Varelas, Brady, Landry, Dickerson, Renner, Harris et al.

of each property they have visited and average them to create a

“traveler profile”. We then search the embedding space using co-
sine similarity and retrieve the properties closest to the traveler
profile.
We compared these recommendations with the recommenda-
tions generated by a popularity-based recommender as a simple
baseline. The popularity-based recommender calculates for each
pair of properties their Jaccard similarity which for two properties
𝑓 (𝑝 ,𝑝 )
𝑝𝑖 , 𝑝 𝑗 is defined as 𝐽𝑆 (𝑝𝑖 , 𝑝 𝑗 ) = 𝑓 (𝑝 )+𝑓 (𝑝 𝑖 )−𝑓𝑗 (𝑝 ,𝑝 ) where 𝑓 re-
𝑖 𝑗 𝑖 𝑗
turns the number of times a property or two properties appear in
all the traveler sequences. In order to generate personalized rec-
Figure 4: Training Dataset Batching ommendations, we score each property by taking the average of
its similarity score with all the properties a traveler has clicked on
and return the top 𝑘 properties based on that score. For example, if
In our case, the order in which properties appear in the sets is
a traveler has clicked on properties 𝑝 1, 𝑝 2 then the score we assign
not important as it is in many NLP tasks. Given that a traveler’s
to a property 𝑝 3 would be 12 (𝐽𝑆 (𝑝 1, 𝑝 3 ) + 𝐽𝑆 (𝑝 2, 𝑝 3 ))
choice to view a property is affected by the website’s modules they
Table 1 compares the offline evaluation results of these two meth-
interacted with during their shopping journey, viewing a property
ods.
before another does not imply sequential preference. For this rea-
son we wanted to ensure that our model is trained with as many Table 1: Comparison between Model Settings
possible permutations of the original sequences as possible. In of-
fline experiments this resulted in a significant increase in normal-
ized discounted cumulative gain (NDCG), precision, and recall. Algorithm NDCG@20 Precision@20 Recall@20
Popularity based Baseline 0.16441 0.03766 0.23977
CBoW 0.18412 0.03966 0.27548

2.2 Destination Recommender System

Using similar data collection methods as in section 2.1, we lever-
aged session-based search data to build a co-search matrix, with
the goal of recommending relevant destinations to travelers in the
Discovery phase who have not yet chosen a destination.
Figure 5: Training Dataset Shuffling. We shuffle the proper-
On the homepage, the target is traveler’s last destination in search
ties within the traveler sequence after each epoch. This sig-
history. We want to recommend similar locations with similar des-
nificantly improved offline performance.
tination type (for example city, point of interest, or neighborhood).
For example, if someone searches for Austin, TX, we would recom-
Finally, for each target property, 2048 properties are randomly mend destinations like San Antonio and Houston which are both
sampled based on the probability [10]: cities in Texas similar to Austin.
On a destination landing page, the target is the destination that
𝑓 (𝑝𝑖 ) 3/4 the traveler typed in the search engine. Since we know that the
𝑃 (𝑝𝑖 ) = ∑
𝑛
𝑗=0 𝑓 (𝑝 𝑗 ) 3/4 travelers coming from a search engine are for a particular desti-
Where: nation, we recommend smaller locations that are within the tar-
get destination. For example, if someone searched for France, we
𝑓 (𝑝𝑖 ) denotes the frequency property 𝑖 appears
would like to help the traveler know more about the place by rec-
2.1.3 Experiments and Results. For all experiments the optimiza- ommending the most relevant cities/places in France such as Paris
tion setup consists of: (a city) or Southern France (a region).
(1) The window size was 3. Powering both of the above use-cases, the destination recom-
(2) Stochastic gradient descent was used for the minimization mender system (DRS) aims at identifying similar destinations to a
of the loss function. set of destinations travelers have interacted with.
(3) The batch size was 128. 2.2.1 Model. Formally, we are trying to model the conditional
(4) The learning rate was kept constant at 1. probability distribution 𝑃 (𝑑 𝑗 |𝑑𝑖 ) where 𝑑𝑖 is the destination that
(5) The model ran for 63 epochs. a traveler searched for and 𝑑 𝑗 is any destinations where 𝑑 𝑗 ≠ 𝑑𝑖 .
(6) The embeddings weights were initialized using uniform dis- The model we use on the homepage, destination landing page and
tribution from -1 to 1. Hotels.com search box is a matrix decomposition in which we de-
After training, we extract the embedding layer from the result- compose a very sparse preference matrix. The specifics of the pref-
ing CBoW model. For each traveler, we look up the embeddings erence matrix vary according to the application. On the home page
Landing Page Personalization at Expedia Group KDD ’20, August 22–27, 2020, San Diego, CA

the preference matrix is traveler-destination, corresponding com- 3 RESULTS

monly to user-item matrices in wider literature [8], [15]. On desti- Our top priority is to improve the experience of our users, and
nation pages the preference matrix is destination-destination, cor- the machine learning models are only useful if they improve the
responding to item-item matrices. For the search box, the prefer- traveler experience. Sophisticated models are not valuable for their
ence matrix is expressed between latitude/longitude coordinates own sake, they must provide significant value as measured by an
of travelers and destinations. online controlled experiment. In fact, it was shown in [2] that im-
The model decomposes the sparse preference matrix into two provements in offline metrics do not necessarily drive an improved
matrices of traveler and destination factors. These factors are dense user experience for a whole host of reasons. At Vrbo and Hotels.com
latent space representations of each traveler and destination, de- we have observed the same. We measure the impact of each new
noted 𝑥𝑢 ∈ R 𝑓 and 𝑦𝑖 ∈ R 𝑓 , respectively. To use these factors for algorithm or system using our internal A/B test platform. For each
recommendations, we simply take the dot product of the traveler model discussed in Section 2, the improvements on one of our core
and destination vectors to obtain a score 𝑠 = 𝑥𝑢=𝑘 ·𝑦𝑖=𝑙 , for traveler metrics are presented in Table 2. The definition of the core met-
𝑘 and destination 𝑙, and rank the destinations by that score. This ric and the actual percentages are considered sensitive and cannot
yields the most relevant destinations for the traveler. be revealed. The recommended destinations model at Hotels.com’s
For the destination-destination preference matrix, the process home page had the lower improvement and is considered the bench-
is identical except that the preference signal is expressed between mark. The Table 2 shows the relative improvements against this
destinations (described in the next section), and the latent traveler benchmark.
factor is now an “origin” destination vector. For the latitude/longitude
preference matrix, the preference signal is expressed between the
traveler’s location and the destination’s location, and the latent Table 2: Online Experiment Results
traveler factor is now a “geographic origin” vector.
Irrespective of how it is populated, the size and sparseness of the
preference matrix presents a significant computational challenge. Brand Model Page Improvement
To solve the decomposition we use Alternating Least Squares (ALS) Hotels.com “Similar Properties” home page 3.25
Hotels.com “Recommended destinations” home page 1.0
[12], which is a highly scalable, efficient, and well understood method
Hotels.com “Similar destinations” landing page 1.975
for solving this type of problem. Hotels.com “Search box recommendation” home page 2
Vrbo “Recently viewed properties” home page 4.25
Vrbo “Recent activity” landing page 2.25
2.2.2 Dataset and Application. On the homepage, we use recent Vrbo “Recommended for you” home page 5
Vrbo “Based on your recent searches” landing page 2.425
destinations that travelers have searched for in order to build the
traveler-destination preference matrix. Each destination searched
by a traveler receives a positive preference. Factorizing this ma-
trix into traveler and destination features as described above we
There is a significant difference in orders of magnitude in re-
generate personalized recommendations for each traveler that has
sults between Hotels.com and Vrbo. At Hotels.com, typical trip
searched for at least one destination in the last year.
planning is relatively short in duration and with many travelers
On destination landing pages, we know the original intent desti-
booking last minute trips. Vrbo on the other hand is a vacation
nation of travelers. We leverage session-based search data in order
rental focused site, where travelers are more likely to have long
to build a destination-destination preference matrix. Two destina-
shopping cycles and may search for a wider variety of properties
tions searched by one traveler in a single session receive a posi-
and destinations before making a final selection. This gives the rec-
tive preference. Using the destination-destination preference ma-
ommendation systems much more exposure and therefore higher
trix decomposition we provide recommendations for similar desti-
potential impact.
nations.
On the Hotels.com search box, travelers are shown destination
recommendations based on their geolocation. The preference ma- 4 CHALLENGES AND SOLUTIONS
trix for this application is built almost identically to the traveler- The models themselves, though important, are a small part of the
destination matrix, except that the traveler rows are now replaced overall system that provides and surfaces the recommendations to
with traveler location rows. This enables recommendations to be travelers. The complete system consists of multiple backend ser-
provided for a traveler who has no search history. vices and data stores as well as a front-end client. Additionally
components for experimentation (the aforementioned A/B testing
platform), instrumentation and monitoring are also required to en-
2.2.3 Setup. The deployed ALS model on Homepage has a num- sure consistent uptime and technical correctness. In this section
ber of hyperparameters tuned via cross-validation: we present our initial attempts at machine learning integrations
and the lessons we learned from those. We then describe our inter-
(1) Number of latent factors, 𝑓 , often on the order of tens nal machine learning platform and how it addresses many of the
(2) Number of iterations represents the number of training loops challenges we faced early on. We also describe the machine learn-
performed, often tens will suffice ing platform’s impact on overall data science efficiency, company-
(3) Lambda, 𝜆 is problem dependent but always <1 wide.
KDD ’20, August 22–27, 2020, San Diego, CA Mitsoulis-Ntompos, Varelas, Brady, Landry, Dickerson, Renner, Harris et al.

4.1 History Uber 3 , and FBLearner 4 from Facebook. We also developed a ma-
Initially at both Vrbo and Hotels.com, machine learning models chine learning platform that serves models in real-time, stream-
were directly integrated with the product’s codebase. At the time ing and batch contexts. We decided to favor a microservice ap-
there were few machine learning models in production and there- proach since the systems ML is integrated into are mostly Java
fore specialized infrastructure wasn’t economical. Absent machine or Scala based. The microservice approach also solves the scaling
learning specific infrastructure, embedding the model code directly / system requirements mismatch, as the ML model can be scaled
in the backend software stack is simpler. independently from the integrating system. For the purposes of
There are a number of drawbacks to this approach that com- recommendation systems there are three relevant patterns to dis-
pound complexity as model updates become more frequent, how- cuss: real-time synchronous, real-time precomputed (offline), and
ever. Firstly, any model retraining or update would require rede- real-time precomputed (online). These are diagrammed in Figure 6.
ploying the entire stack. For home and landing pages, this is not Here real-time refers to how the inferences are accessed, meaning
trivial. Those services are critical and must have near-perfect up- large numbers of small one-at-a-time requests as opposed to bulk
time, or users will be unable to search for properties and therefore requests of the entire dataset (via SQL or Apache Spark 5 , for ex-
unable to book. This slows down the speed of iteration and exper- ample). The platform was built in stages, usually driven by data
imentation, which is critical for machine learning success. scientist / software engineering integration requirements, rather
Machine learning models are very different from traditional back- than dictated top-down all at once. The result is a series of loosely
end systems. The code for training ML models is usually written coupled components that were developed based on real-world use
in Python, Scala or R, while most of the backend systems at Ho- cases.
tels.com and Vrbo are built in Java. This necessitates a model “hand- 4.2.1 Real-time Synchronous Inference (RTSI). This is a straight-
of” from one runtime to the next, and makes apples-to-apples test- forward request / response pattern. The caller sends the payload
ing extremely difficult, particularly if there’s data processing code containing the model features, then the service invokes the model
required. Compute and memory requirements are also significantly in-memory and performs the inference, sending the result back to
higher than traditional backend systems. This means that adding the caller. This pattern is very simple and follows a RESTful way
machine learning directly into a backend system will likely change of calling the prediction logic of models. Deployment is entirely
the requirements for the whole system, even if it’s only needed for self-service, and requires only a small amount of code on the data
the ML model. Monitoring machine learning systems is more chal- scientist’s part for instrumentation and packaging. It can support
lenging as well. All the usual monitoring for uptime and latency ap- any Python, Java or Scala machine learning framework.
ply, but it’s also important to monitor predictions (as noted in [2]) The obvious disadvantage to this pattern is clear when the model
as a means of detecting bias, train-test skew, and model drift. Fi- requires a significant amount of computational power, or when
nally, the skill sets between data scientists and backend developers the required latency is low. Since the model is being invoked per
are also very different. Managing software in production requires a request, latency can be quite high depending on the model.
completely different skill set than building a powerful model. With
the model integrated directly into the backend, the data scientists 4.2.2 Real-time Precomputed (Offline) (RTPOf). If the inference scores
need to know a lot about that system, and the backend developers can be precomputed in an offline environment, they can be loaded
must understand the computational and memory requirements of into a key-value store as long as there’s a suitable key. Examples
the model. of common keys at Vrbo and Hotels.com are property ID, traveler
All of these differences slow down iteration and experimenta- ID and destination ID. Then callers would only need to know the
tion. Redeploying the entire backend stack for a critical system is key to retrieve the scores. This transforms what was a complicated
slow and risky, even with appropriate application infrastructure. inference per-prediction into a lookup, reducing the latency by at
Translating the model from one runtime to another takes time, and least an order of magnitude.
testing is very tedious. The single most important thing a data sci- This approach has drawbacks too. If the key space has a very
entist can do with a model to increase it’s economic impact is to high cardinality then precomputation and storage become imprac-
test it online as much as possible, yet at the start of these efforts tical thus necessitating a move to the aforementioned synchronous
that was extremely difficult. This necessitated an investment on the inference. Another drawback is the frequency of updates to the
part of Vrbo and Hotels.com to develop specialized infrastructure key-value store. In many cases a simple daily or hourly update will
specifically for deploying and monitoring ML models. suffice, however some use cases require the precomputed values to
be updated continuously as new data arrives in the system. Updat-
ing traveler “profiles” based on their searches as they search on the
4.2 Machine Learning Infrastructure site is an example of such a use case.
Most companies with significant machine learning requirements
and products have a platform to accelerate ML development and 4.2.3 Real-time Precomputed (Online) - (RTPOn). By combining
deployment. Some concrete examples include Tensorflow Extended the real-time synchronous services with the key-value store, we
(TFX) [1] from Google, Bighead from Airbnb 2 , Michaelangelo from are able to listen to a stream of values (via Apache Kafka6 ) and
3 https://eng.uber.com/michelangelo/
4 https://engineering.fb.com/core-data/introducing-fblearner-flow-facebook-s-ai-
backbone/
2 https://databricks.com/session/bighead-airbnbs-end-to-end-machine-learning- 5 https://spark.apache.org/

platform 6 https://kafka.apache.org/
Landing Page Personalization at Expedia Group KDD ’20, August 22–27, 2020, San Diego, CA

Real-time Real-time
Real-time
Precomputed Precomputed
Synchronous
(Offline) (Online)

Model service Stream Processor

Batch Model
Job service

Figure 6: (a) Real-time synchronous, (b) real-time precomputed (offline), (c) real-time precomputed (online)

perform the predictions as the inputs arrive, rather than when the our methods for personalizing landing pages and the home page
predictions are needed. The assumption of course is that the pre- are always evolving. We present ongoing and future work in this
dictions won’t be needed immediately, but some time after the fea- section.
tures become available on the stream.
When the model input (including key for lookup) arrives on the
stream, a stream processor service issues a synchronous request to
the model service to retrieve the score. Then the stream processor
inserts the score into the key-value store, where going forward it 5.1 Content Augmented Recommendation
will be available at low latency without having to recompute the System - CaRS
predictions. This results in continual updates to the key-value store Building upon the work done on Property Embeddings (section 2.1)
at high frequency. Should the number of input events become large and inspired by [4] we started developing a new recommendations
for a short period of time, it is possible to scale the model service to framework in order to incorporate additional meaningful signals
handle the traffic, or if a delay is acceptable, wait until the volume such as price or destination attributes (historic, romantic, etc.).
of stream events returns to normal levels. The aim of CaRS is to provide a recommender system that can
be used for both property and destination recommendations. We’re
Table 3: Mapping of Models to Inference Type developing CaRS in a way that allows us to generate both person-
alized (traveler to item) and non-personalized (item to item) rec-
Model - Use Case Inference Type ommendations while maintaining only one model.
For the rest of this subsection, an item will refer to both prop-
Hotels.com “Similar Properties” RTSI erties and destinations. We consider a destination “booked” if a
Hotels.com “Recommended destinations” RTPOf traveler booked a property within the destination.
Hotels.com “Similar destinations” RTPOf The model uses sequences of clicked items as context and booked
Hotels.com “Search box recommendation” RTPOf items as targets from traveler sessions. Travelers are represented
Vrbo “Recently viewed properties” RTPOn as an aggregation (e.g. average) of the features of the items they
Vrbo “Recent activity” RTPOn have clicked on. We train a neural network to generate a score
Vrbo “Recommended for you” RTSI which represents how likely travelers are to book an item given
Vrbo “Based on your recent searches” RTSI the previous items they have clicked on. The function learned by
the neural network can be used both with multiple items as input
so as to personalize results and with a single item if no personal-
ization is desired.
4.3 Productivity We use a pairwise loss; for each positive example we draw neg-
The combination of straightforward deployment, standardized in- ative examples and learn the weights such that a higher score is
frastructure toolkits, flexible ML framework support, and separa- assigned to positive examples than the negative ones. Inspired by
tion from the integrating system has significantly improved the [3], [13] we used our pretrained embeddings in order to identify
iteration speed (and therefore effectiveness) of data scientists at similar items that weren’t viewed by the travelers. This allowed
Vrbo and Hotels.com. Since the development and adoption of the us to improve the performance of the model with negative sam-
ML platform, the number of ML models deployed has grown ap- pling based on the already estimated conditional probabilities pro-
proximately linearly. vided by the embeddings. By sampling negatives near the decision
boundary we can improve ranking metrics like NDCG.
5 DISCUSSION AND FUTURE WORK Combining content features with decision boundary negative
At Vrbo and Hotels.com, we are always looking for new and better sampling, we saw an improvement in offline metrics like NDCG
ways to lead travelers to their dream vacation. Correspondingly, and recall when we compared CaRS to CBoW.
KDD ’20, August 22–27, 2020, San Diego, CA Mitsoulis-Ntompos, Varelas, Brady, Landry, Dickerson, Renner, Harris et al.

5.2 Multi-armed Bandits REFERENCES

Inspired by a paper on using multi-armed bandits for real-time [1] Denis Baylor, Eric Breck, Heng-Tze Cheng, Noah Fiedel, Chuan Yu Foo, Zakaria
Haque, Salem Haykal, Mustafa Ispir, Vihan Jain, Levent Koc, Chiu Yuen Koo,
optimization of web page layouts [7], we have recently adopted Lukasz Lew, Clemens Mewald, Akshay Naresh Modi, Neoklis Polyzotis, Sukriti
methods to optimize traveler experience via Reinforcement Learn- Ramesh, Sudip Roy, Steven Euijong Whang, Martin Wicke, Jarek Wilkiewicz,
Xin Zhang, and Martin Zinkevich. 2017. TFX: A TensorFlow-Based Production-
ing (RL). The “arms” in this multi-armed bandit are various layouts Scale Machine Learning Platform. In Proceedings of the 23rd ACM SIGKDD Inter-
or other visual attributes of the page. Since we are looking to serve national Conference on Knowledge Discovery and Data Mining (KDD ’17). ACM,
layouts that will more often lead travelers to find a property, our New York, NY, USA, 1387–1395. https://doi.org/10.1145/3097983.3098021
[2] Lucas Bernardi, Themistoklis Mavridis, and Pablo Estevez. 2019. 150 Successful
reward function is a simple binary reward offering positive value Machine Learning Models: 6 Lessons Learned at Booking.Com. In Proceedings of
whenever the traveler clicks. As previously stated in Section 2 not the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data
all travelers have the same behavior and so we seek to learn how Mining (KDD ’19). Association for Computing Machinery, New York, NY, USA,
1743–1751. https://doi.org/10.1145/3292500.3330744
traveler context influences preferences. We make use of contex- [3] Long Chen, Fajie Yuan, Joemon M. Jose, and Weinan Zhang. 2018. Improving
tual bandits to attempt to automatically discover ideal layouts tai- Negative Sampling for Word Representation Using Self-Embedded Features. In
Proceedings of the Eleventh ACM International Conference on Web Search and Data
lored to users all exhibiting some shared characteristics such as Mining (WSDM ’18). Association for Computing Machinery, New York, NY, USA,
traveler segmentation, purchase history, or device type. The model 99–107. https://doi.org/10.1145/3159652.3159695
we build for the click-through-rate then has a certain amount of [4] Paul Covington, Jay Adams, and Emre Sargin. 2016. Deep Neural Networks
for YouTube Recommendations. In Proceedings of the 10th ACM Conference on
personalization built in by using features which are not only de- Recommender Systems. New York, NY, USA.
pendent on the particular layout that was served but also on some [5] Kallirroi Dogani, Matteo Tomassetti, Saúl Vargas, Benjamin Paul Chamberlain,
context of the traveler. and Sofie De Cnudde. 2019. Learning Embeddings for Product Size Recommen-
dations. In Proceedings of the SIGIR 2019 Workshop on eCommerce, co-located
RL is fundamentally different in several ways from more tradi- with the 42st International ACM SIGIR Conference on Research and Development
tional ML approaches. With RL, there is a feedback loop in which in Information Retrieval, eCom@SIGIR 2019, Paris, France, July 25, 2019 (CEUR
Workshop Proceedings), Jon Degenhardt, Surya Kallumadi, Utkarsh Porwal, and
the model provides the front-end application with a layout and Andrew Trotman (Eds.), Vol. 2410. CEUR-WS.org. http://ceur-ws.org/Vol-
then observations of what results were obtained from that lay- 2410/paper13.pdf
out must be fed back into the model for it to update its parame- [6] Mihajlo Grbovic and Haibin Cheng. 2018. Real-Time Personalization Using Em-
beddings for Search Ranking at Airbnb. In Proceedings of the 24th ACM SIGKDD
ters. This is a stateful, real-time coupling to both the backend and International Conference on Knowledge Discovery & Data Mining (KDD ’18). As-
front-end systems serving the layouts which grows in complex- sociation for Computing Machinery, New York, NY, USA, 311–320. https:
ity as the scope of observations increases. Reinforcement learning //doi.org/10.1145/3219819.3219885
[7] Daniel N. Hill, Houssam Nassif, Yi Liu, Anand Iyer, and S.V.N. Vishwanathan.
presents many new challenges for machine learning infrastructure 2017. An Efficient Bandit Algorithm for Realtime Multivariate Optimization. Pro-
that was designed primarily with supervised learning use cases ceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discov-
ery and Data Mining - KDD ’17 (2017). https://doi.org/10.1145/3097983.3098184
in mind. However, as before simplicity is the best approach. By [8] Y. Hu, Y. Koren, and C. Volinsky. 2008. Collaborative Filtering for Implicit Feed-
starting with initial experiments focused on leaRning we have back Datasets. In 2008 Eighth IEEE International Conference on Data Mining. 263–
shown the ability to identify the patterns likely to succeed at larger 272. https://doi.org/10.1109/ICDM.2008.22
[9] Tomas Mikolov, Kai Chen, Greg Corrado, and Jeffrey Dean. 2013. Efficient Esti-
scale whether related to deployement, offline evaluation or system mation of Word Representations in Vector Space. arXiv:cs.CL/1301.3781
health monitoring. [10] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. 2013.
Distributed representations of words and phrases and their compositionality.
(2013), 3111–3119.
6 CONCLUSION [11] Pavlos Mitsoulis-Ntompos, Meisam Hejazinia, Serena Zhang, and Travis Brady.
In this paper, we introduced a way to implement personalization 2019. A Simple Deep Personalized Recommendation System. In Proceedings of
the Workshop on Recommenders in Tourism co-located with the 13th ACM Con-
for returning travelers while they are in the process of trip plan- ference on Recommender Systems (RecSys 2019), Copenhagen, Denmark, Septem-
ning. Our contribution here is two-fold: first we share details on ber 16-20, 2019. (CEUR Workshop Proceedings), Vol. 2435. CEUR-WS.org, 22–26.
http://ceur-ws.org/Vol-2435/paper4.pdf
how the system was initially created and then ultimately run. Next [12] Gábor Takács and Domonkos Tikk. 2012. Alternating least squares for person-
we note lessons learned on the ways in which ML platforms can alized ranking. (09 2012). https://doi.org/10.1145/2365952.2365972
be especially valuable for large e-commerce companies. We empha- [13] Ugo Tanielian and Flavian Vasile. 2019. Relaxed Softmax for PU Learning. In
Proceedings of the 13th ACM Conference on Recommender Systems (RecSys ’19).
size the importance of simple solutions and of democratizing ML Association for Computing Machinery, New York, NY, USA, 119–127. https:
model predictions to a large and varied organization. And finally //doi.org/10.1145/3298689.3347034
we conclude with details on extensions to personalization via Re- [14] Flavian Vasile, Elena Smirnova, and Alexis Conneau. 2016. Meta-Prod2Vec: Prod-
uct Embeddings Using Side-Information for Recommendation. In Proceedings of
inforcement Learning. Here we have begun to explore the usage the 10th ACM Conference on Recommender Systems (RecSys ’16). Association for
of Multi-Arm Bandits to determine the best possible layouts for Computing Machinery, New York, NY, USA, 225–232. https://doi.org/10.1145/
2959100.2959160
travelers that are in Discovery or Shop phases as well as improve [15] Yunhong Zhou, Dennis Wilkinson, Robert Schreiber, and Rong Pan. 2008. Large-
the current Property Recommender System (PRS) with content fea- Scale Parallel Collaborative Filtering for the Netflix Prize. In Proceedings of the
tures. 4th International Conference on Algorithmic Aspects in Information and Manage-
ment (AAIM ’08). Springer-Verlag, Berlin, Heidelberg, 337–348. https://doi.org/
10.1007/978-3-540-68880-8_32
ACKNOWLEDGMENTS
The authors would like to thank Fedor Alexandre Parfenov, Kyler
Eastman, Vasileios Vryniotis, Seiyang Chua for their contribution
to this paper.