Investigating Recommendation Algorithms for Escape Rooms

Accepted manuscript to appear in VJCS Accepted Manuscript Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. Vietnam Journal of Computer Science Article Title: Investigating Recommendation Algorithms for Escape Rooms Author(s): Sagi Bazinin, Guy Shani DOI: 10.1142/S2196888819500209 Received: 15 May 2019 Accepted: 28 June 2019 To be cited as: Sagi Bazinin, Guy Shani, Investigating Recommendation Algorithms for Escape Rooms, Vietnam Journal of Computer Science, doi: 10.1142/S2196888819500209 Link to final version: https://doi.org/10.1142/S2196888819500209 This is an unedited version of the accepted manuscript scheduled for publication. It has been uploaded in advance for the benefit of our customers. The manuscript will be copyedited, typeset and proofread before it is released in the final form. As a result, the published copy may differ from the unedited version. Readers should obtain the final version from the above link when it is published. The authors are responsible for the content of this Accepted Article. IPT Accepted manuscript to appear in VJCS CR Click here to download Manuscript (PDF) Investigating_Recommendations_for_Escape_Rooms (1).pdf Investigating Recommendation Algorithms for Escape Rooms US Sagi Bazinin and Guy Shani Abstract AN An escape room is a physical puzzle solving game, where participants solve a series of riddles within a limited time to exit a locked room. Escape rooms differ in their theme, environment, and difficulty, and people hence often vary on their preferences over escape rooms. As such, recommendation systems can help people in deciding which room to visit. In this paper we describe the DM properties of the escape rooms recommendation problem, with respect to other popular recommendation problems. We describe a dataset of reviews collected within a current system. We provide an empirical comparison between a set of recommendation algorithms over two problems, top-N recommendation and rating prediction. In both cases a KNN method performed the best. Keywords: Recommender Systems, Collaborative Filtering, Escape Room, Empirical Evaluation TE 1. Introduction Escape rooms [29, 40, 19] have become a popular entertainment throughout the world. In an escape room a group of participants is locked in a room, and must solve a series of riddles in order to unlock the room and escape within a EP limited time (typically an hour). Rooms vary in theme, from space adventures to prisons, in their mood, from comedy to horror, and in their difficulty level. As such, different rooms may appeal to different people. Some people, for example, expect the room to be scary, while others avoid all horror rooms. It is AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. Software and Information Systems Engineering, Ben Gurion University Preprint submitted to Elsevier May 15, 2019 IPT CR Accepted manuscript to appear in VJCS also common for escape room fans to develop their skills, initially preferring simpler rooms, and later moving to more challenging rooms. Indeed, it is unlikely that an experienced user will enjoy a simple room, or vice versa. US The price for playing is typically about 40$ per person, and often a group of often travel to other cities to visit a specific escape room. As such, an escape room incurs a significant cost for the participants. The escape room experience is constructed around the element of surprise AN — the visitor should know nothing about the structure of the room, the types of puzzles, and most importantly, any hint about the solution of these puzzles. As such, the available description of a room is very limited, and a person looking for an escape room to visit must rely on the opinions of previous visitors. It is very common to post a review of a given room, but these reviews are also limited to discussing the general attributes of the room, such as whether it was the experience. DM scary, the level of difficulty, and the very general feelings of the reviewer towards Given all the above, it is a challenging task for a user to identify an appropriate room among the dozens of rooms in a major city. Recommendation systems [34], systems that recommend items to users, can be a valuable tool in helping users to choose which room to visit next. Recommendations may be computed by content similarity [23]. For exam- TE ple, one can recommend movies of a given genre, or starring a preferred actor. For escape rooms, however, there is no agreed upon content classification that can help us to identify rooms with similar content. In addition, much of the similarity between rooms is based on abstract qualities, such as the types of the puzzles within the room, which is difficult to define. EP Alternatively, in the collaborative filtering approach [10, 7] an item is recommended to the active user based on users with similar behavior. In this paper we take the collaborative filtering approach, computing similarity between users and items based on previous user ratings of rooms. We describe a dataset of user ratings for rooms given in a website designed to AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. 4 or more people go to an escape room together. Furthermore, escape room fans 2 IPT CR Accepted manuscript to appear in VJCS help users in choosing an appropriate room to visit. The website provides users with room descriptions, and allows users to search for available rooms in an area. The website also allows users to review rooms both in a textual description and We compare a number of collaborative filtering algorithms implemented in open source libraries over the dataset. We used algorithms for two different AN tasks — rating prediction and top-N recommendation. 2. Background Recommender systems actively suggest items to users, to help them to rapidly discover relevant items, and to increase item consumption [35]. Such systems can be found in many applications, including TV streaming services DM [1], online e-commerce [39], smart tutoring [9], and many more [24]. We focus here on two important recommendation tasks [37] — rating prediction, and top-N recommendation. In the rating prediction task the system is given a user u and an item i, and must compute a predicted rating r̂u,i that u would give to i. In the top-N task the system computes a list of N recommended item that the user may choose. There are two dominant approaches for computing recommendations for the active user — the user that is currently interacting with the application and the TE recommender system. First, the collaborative filtering approach [4, 10] assumes that users who agreed on preferred items in the past will tend to agree in the future too. Many such methods rely on a matrix of user-item ratings to predict unknown matrix entries, and thus to decide which items to recommend. A simple approach in this family [30], commonly referred to as user based EP collaborative filtering, identifies a neighborhood of users that are similar to the active user. This set of neighbors is based on the similarity of observed preferences between these users and the active user. Then, items that were preferred by users in the neighborhood are recommended to the active user. Another AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. recommendation system for this website. US using a numerical rating. We have recently constructed a collaborative filtering 3 IPT CR Accepted manuscript to appear in VJCS approach [36, 2], known as item based collaborative filtering recommends items also preferred by users that prefer a particular active item to other users that also prefer that active item. In collaborative filtering approaches, the system US only has access to the item and user identifiers, and no additional information tions titled “users who preferred this item also prefer” typically use some type of collaborative filtering algorithm. A second popular approach is the content-based recommendation [23]. In AN this approach, the system has access to a set of item features. The system then learns the user preferences over features, and uses these computed preferences to recommend new items with similar features. Such recommendations are typically titled “similar items”. User’s features, if available, such as demographics (e.g. gender, age, geographic location) can also provide valuable information. As content information is not available for the escape room data that we DM collected, we focus here on the collaborative (CF) filtering approach. Recently, many collaborative filtering algorithms were implemented in off-the-shelf libraries, allowing us to easily compare a large set of algorithms for a particular problem. Specifically, in this paper we compared algorithms from several families. The user-based KNN method [8] directly identifies a set of similar users to the active user, based on the similarity of past ratings, and computes recommendations TE based on the items favored by these similar users. Alternatively, the item-based KNN method [36] identifies a set of similar items to items that the active user has rated. The matrix factorization (MF) approach [22] computes for each user and item a vector of latent features, and recommends an item if its latent vector is similar to the user latent vector. Many MF algorithms were suggested EP in the past, and MF is widely used in many recommendation applications. kMarkov models [38] utilize the temporal order of past ratings. Clustering models [14] attempt to group together users or items that have similar behavior, and compute recommendations based on other members of the cluster. AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. over items or users is used. For example, websites that present recommenda- 4 IPT CR Accepted manuscript to appear in VJCS 3. The Escape Room Domain We now review escape rooms with respect to other recommendation domains. We focus on various aspects influencing user behavior, as well as the decision on US domains — recommendations for movies [15], e-commerce [17], and hotels [3]. First, when a user makes a decision about which item to choose, the user is exposed to various information sources. In many cases there is available information about the item, such as hotel amenities, the genres and the description AN of the movie, or the specification of an electronic gadget. In these domains there is an attempt to provide as much information as possible, to avoid bad experience from the user due to unfulfilled expectations. In escape rooms, on the other hand, mystery plays an important role in the user experience. As such, escape rooms provide description only about the DM general theme of the room, such as whether it takes place in ancient Egypt or in space, but discloses no information as to how many rooms the user has to go through, or how many riddles need to be solved, which may influence the user experience much more than the theme. As such, making informed decisions about which room to choose becomes much harder for the user. It is also important that user reviews of escape rooms refrain from revealing such information. These reviews are hence limited to comments on the general difficulty, theme, and quality of the room. Many users even avoid reading these TE reviews, fearing that they may contain “spoilers”. An additional source of information can be reviews over items [5]. These are common in all 4 domains, but originate from different sources. In movies, reviews are typically written by expert critics which review many, if not all, new movies. A user that identifies a critic that is aligned with her may trust EP the critic’s opinions over new movies [32]. For electronic products, such as cell phones or laptops, one may find available reviews by experts that may compare several items, allowing the users to make informed decisions. In escape rooms, as well as hotels, users rely on reviews provided by other AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. which item should be recommended. We compare escape rooms to three other 5 IPT CR Accepted manuscript to appear in VJCS users, which become an essential component in decision making. It is common to read reviews by other users, both positive and negative before making a de- cision [28]. The mystery component is important here as well, and user reviews US for escape rooms make an effort not to disclose any details about the room. general information about the riddles (difficulty). In hotel reviews, on the other hand, the reviewers conveys as much information as possible concerning their experience. AN Price also plays an important role in making a decision about escape rooms. In hotels and e-commerce price may vary greatly. Hotels can be found in major cities in a wide price range, from perhaps $50 to many hundreds of dollars per night. Reasonable cell phones as well can be found in a wide price range. Escape rooms, like movies, are typically offered at a fixed price. However, the same movie is typically shown in many theaters, while escape rooms are unique. DM As such, one can go to the nearest movie theater, but may have to travel far to a specific location for an escape room. As such, traveling imposes an additional cost in terms of money and time. Escape rooms must also be booked in advance, while one can typically purchase movie tickets at a short notice. The availability of escape rooms is also different than other domains. There are many thousands of movies that one can watch in VOD, and a few dozen that play at local cinemas, with new movies released weekly. For many electronic TE gadgets, such as cell phones, there is also a wide variety of items to choose from. Escape rooms are much less common. In London, for example, there are only about 100 escape rooms. Although in major tourist attractions hotels are much more abundant, one may claim that the availability of hotels is also quite limited in many locations. However, an escape room fan may visit all rooms in her local EP city, while it is unlikely that people would visit all hotels in a city. Moreover, people who frequently travel to the same city may find a suitable hotel to use in all visits [27], while returning an escape room is pointless. In escape rooms, a major consideration is the group that is playing together. Recommendations for groups are important in many domains [26]. It is often AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. Reviews may report the company (number of people, level of expertise), and 6 IPT CR Accepted manuscript to appear in VJCS the case that people go with a group of friends to a movie together, and it is important to find a movie that would fit the preferences of everybody in the group. This is even more crucial in escape rooms, where the experience US in interactive and everybody should contribute. As such, escape rooms are on the other hand, people typically travel with family or alone, and visiting a hotel with a group of friends is perhaps less common. In e-commerce, people typically purchase items for a specific individual, and groups are less relevant. AN Another interesting domain that bears some resemblance to escape rooms is the area of point of interest (POI) recommendations. More specifically, while some researchers consider restaurants and stores as POIs, escape rooms are more similar to attractions POIs, such as museums and landmarks [6]. As opposed to escape rooms, many POIs can be visited many times. However, some work in POI recommendation focuses on recommending only new POIs, which is more DM similar to escape rooms [11]. POI recommendations are often of importance for touristic applications. In this context, in many cases a tourist wishes to visit several POIs during the same day [18]. As such, many POI recommenders consider the order by which a set of POIs should be visited, based on properties such as geographical location and type, so as to reduce the required distance to travel, and increase diversity. In escape rooms, it is highly unlikely that one TE would visit several escape rooms during a single day. 4. Empirical Comparison of CF Algorithms We now review an experimental study that we conducted using data collected by an escape room booking website. EP 4.0.1. Dataset We now describe a dataset obtained from 3 years of reviews written by escape-rooms fans. The website provides a platform for companies to publish their escape rooms, and for users to provide reviews for escape rooms they visited. The website is not associated with any specific escape room company, AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. perhaps a major application for research in group recommendations. In hotels, 7 Dataset Users Items Ratings IPT Density Git (Django) 790 1757 13,165 Escape Rooms 20,197 375 41,256 Movielens 20M 138,493 27,278 20,000,263 0.52% Last.fm 1892 17632 92,834 0.28% Book-Crossing 92,107 271,379 1,031,175 0.00% 0.95% DM AN US 0.54% Figure 1: A histogram in log scale of the number of reviews per user for the escape-rooms dataset and escape rooms pay a commission for each user referred to them from the TE website. There are between 15, 000 to 20, 000 monthly visits to the website. Users can write reviews for escape rooms, both numeric ratings, in the range of 1 through 10, and optional textual reviews describing their experience. The dataset contains 20, 197 users who uploaded 41, 256 reviews for 375 rooms. Figure 1 shows a histogram of the number of reviews per user. As can be seen, EP about half of the users rate only a single room, but there are about 800 users who reviewed 5 or more rooms, 54 of which reviewed more than 50 rooms. Table 1 shows the properties of our dataset with respect to other well known AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. Table 1: Datasets statistics CR Accepted manuscript to appear in VJCS 8 IPT CR Accepted manuscript to appear in VJCS datasets1 . As can be seen, our dataset has relatively low sparsity. This is due to the relatively low ratio of items to users, but can also be because relatively many users of the website are eager to express their opinions concerning their US experience. average rating of previous users. This non-personalized method provided an incentive for room owners to solicit fake reviews. Indeed, there are many users in the system that provided a single review giving the maximal rating (10 stars) to AN a room, which we suspect to be fake. In most collaborative filtering approaches, though, a user that provided a single review has little to no influence over the predicted ratings for other users. As such, we took no specific steps towards identifying and removing these reviews. In our experiments below we used a temporal train-test split, using ratings from the last 2 months as a test set, and all other ratings as training data. We DM remove new users from the test set, as CF algorithms cannot provide recommendations or predictions for such users. 4.1. Suspected Reviews Prior to implementing a collaborative filtering approach in our website, the rooms were displayed to a user ranked by their average grade. As such, there is a significant incentive for escape room operators to solicit positive fake reviews, TE that would increase the room’s average rating and improve its rank and hence, its observability. To analyze this, we tried to identify what constitutes as a fake review. First, as there was no collaborative filtering engine, there was no incentive to rate more than a single room. As such, we focused on users who rated only a single EP room, giving it a perfect rating of 10 stars. To avoid detection, these reviews also contained textual descriptions, and are linked to a facebook account. We find such suspected fake reviews, i.e., a single rating by a user of 10 1 https://bit.ly/2MJE9ed AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. During the collection of the data, the website displayed for each room the 9 IPT US AN Figure 2: A histogram of the number of rooms with a given ratio of suspected reviews to other reviews. stars, for all escape rooms. Figure 2 shows the ratio between suspected reviews and other reviews. We can see that for some rooms, the amount of suspected DM reviews greatly outnumbers the amount of other reviews. For about half the rooms, however, the number of suspected reviews is no more than half the number of other reviews. The suspected reviews also greatly influence the ranking of the rooms following the average rating. As can be seen in Figure 3 rooms with more than two times the number of fake reviews to the number of other reviews increase their ranking by an average of 84.8 positions in the list. On the other hand, rooms who did not have many suspected reviews dropped 37.7 places in the list TE on average. We believe that this is sufficient evidence to remove these suspected reviews from consideration in our empirical evaluation. 4.2. Collaborative Filtering Algorithms EP In our experiments we used algorithms implemented in two popular rec- ommendation frameworks available online — MyMediaLite2 and Surprise3 . In 2 http://www.mymedialite.net/[13] 3 https://surprise.readthedocs.io/en/stable/[16] AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. CR Accepted manuscript to appear in VJCS 10 IPT US AN Figure 3: Average increase in the room ranking given the ratio of suspected reviews. addition, we implemented the k-Markov algorithm [38], which is not provided by either package. We experimented with many algorithms implemented by the two libraries, but, due to space restrictions, report below only the best perform- DM ing algorithms. Our implementation and dataset are available online, along with the framework built for joint evaluation 4 . We used the MyMediaLite API also to evaluate the recommendations given by all algorithms. We evaluated the all algorithms using AUC, Precision@5, Precision@10, MAP, Recall@5, Recall@10 and NDCG. Due to space constrains, however, we report only precision, RMSE, and MAP below. AUC, Recall and NDCG ranked the algorithms roughly in the same order as precision and MAP. TE 4.3. Results We experimented with two relevant tasks — rating prediction, where the system presents to the user a personalized predicted rating for a given room that she is considering, and top-N recommendations, where the system presents EP to the user a personalized list of N rooms that she may want to visit. Tables 2 and 3 show a selected set of results for the best techniques that we experimented with. In addition, we show results for a random algorithm, and 4 https://github.com/Sharpen6/EscapeRoomsRecSys AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. CR Accepted manuscript to appear in VJCS 11 US IPT Table 2: Results for the rating prediction task . (M) denotes a MyMediaLite implementation AN and (S) denotes a Surprise implementation. RMSE (M) KNN user - cosine 1.226 (M) SCAF [31] 1.247 (M) SVD++ 1.257 1.285 (M) User-Item bias 1.303 (M) Matrix Factorization 1.336 (M) Co-Clustering 1.386 (M) KNN item - cosine 1.397 (S) Base model 1.399 (S) KNN item - pearson 1.431 (M) SigmoidSVD++ [20] 1.952 TE DM (M) SlopeOne Average (Current) 2.289 Random 4.636 EP AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. CR Accepted manuscript to appear in VJCS 12 IPT US Table 3: Results for the top-10 recommendation task. (M) denotes a MyMediaLite implemen- (S) KNN user - cosine 0.087 (M) KNN item - cosine 0.072 k-Markov(k=2) [38] 0.061 (S) Co-Clustering [14] 0.054 AN tation and (S) denotes a Surprise implementation. 0.077 0.039 0.130 0.235 (S) User-Item bias [21] 0.046 0.047 0.044 0.069 0.147 (S) Base model [22] 0.039 0.046 0.033 0.059 0.118 (S) SVD++ [20] 0.038 0.039 0.034 0.077 0.118 (S) NMF [25] 0.036 0.034 0.035 0.058 0.093 (M) MostPopular 0.026 0.027 0.022 0.03 0.051 (M) WRMF [12] 0.025 0.026 0.026 0.04 0.072 (M) BPRMF [33] 0.014 0.014 0.015 0.037 0.046 prec@5 prec@10 recall@5 recall@10 0.076 0.055 0.24 0.308 0.061 0.044 0.184 0.215 0.061 0.054 0.108 0.191 TE DM MAP 0.01 0.011 0.01 0.026 0.041 Random 0.007 0.007 0.007 0.01 0.017 EP Average (Current) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. CR Accepted manuscript to appear in VJCS 13 IPT CR Accepted manuscript to appear in VJCS for the current average rating prediction. For top-N recommendations, the traditional KNN model over either users or items performed well in this domain, possibly due to the relatively low di- US mensionality of the problem, with only 375 items. The k-Markov model also of this domain. The Co-Clustering model [14], recommending based on a user cluster, item cluster, and a user-item cluster, also provided good results. This is somewhat surprising, because in many domains clustering algorithms do not AN produce good results. This may be attributed to the relatively low sparsity in our domain, compared with other well known CF datasets [7]. Popular matrix factorization approaches, such as BPR, SVD variants, and others, produced less accurate recommendations. While the precision of all algorithms may seem low, this is not untypical for top-N recommendations in similar domains. For example, for the new POI DM problem, Feng et al. [11] report similar precision values. For rating prediction, the user-based KNN model again produced the best results, but matrix factorization method performed very well for this problem. The average item rating that was shown on the website prior to the installation of the recommendation engine, performed much worse, with an RMSE almost twice as much as the user-based KNN method. TE 5. Conclusion In this paper we described a recommendation system for the growing area of escape rooms which can now be found all around the world. We discussed the characteristics of escape rooms, and their similarity to other popular recommendation system domains. We then reported an evaluation of many collaborative EP filtering algorithms for two problems — rating prediction and top-N recommendations over a real dataset of an escape room website. Our system has only been recently installed in the website, and in the future we will report user response to the recommended items. In addition, given data AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. produced good results here, which can be attributed to the sequential nature 14 Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. 15 DM AN US CR Accepted manuscript to appear in VJCS TE about how users interact with the recommender system, we may be able to EP design better algorithms for this domain. AC C IPT IPT CR Accepted manuscript to appear in VJCS [1] Amat, F., Chandrashekar, A., Jebara, T., Basilico, J.: Artwork personal- ization at netflix. In: Proceedings of the 12th ACM Conference on Recommender Systems, pp. 487–488. ACM (2018) US orative filtering. In: 2016 IEEE 26th International Workshop on Machine Learning for Signal Processing (MLSP), pp. 1–6. IEEE (2016) [3] Borràs, J., Moreno, A., Valls, A.: Intelligent tourism recommender systems: AN A survey. Expert Systems with Applications 41(16), 7370–7389 (2014) [4] Breese, J.S., Heckerman, D., Kadie, C.: Empirical analysis of predictive algorithms for collaborative filtering. In: Proceedings of the Fourteenth conference on Uncertainty in artificial intelligence, pp. 43–52. Morgan Kaufmann Publishers Inc. (1998) DM [5] Chen, L., Chen, G., Wang, F.: Recommender systems based on user reviews: the state of the art. User Modeling and User-Adapted Interaction 25(2), 99–154 (2015) [6] Cheng, C., Yang, H., Lyu, M.R., King, I.: Where you like to go next: Successive point-of-interest recommendation. In: IJCAI, vol. 13, pp. 2605– 2611 (2013) [7] Desrosiers, C., Karypis, G.: A comprehensive survey of neighborhood- TE based recommendation methods. In: Recommender systems handbook, pp. 107–144. Springer (2011) [8] Desrosiers, C., Karypis, G.: A comprehensive survey of neighborhoodbased recommendation methods. In: Recommender systems handbook, EP pp. 107–144. Springer (2011) [9] Drachsler, H., Verbert, K., Santos, O.C., Manouselis, N.: Panorama of recommender systems to support learning. In: Recommender systems handbook, pp. 421–451. Springer (2015) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. [2] Barkan, O., Koenigstein, N.: Item2vec: neural item embedding for collab- 16 IPT CR Accepted manuscript to appear in VJCS [10] Ekstrand, M.D., Riedl, J.T., Konstan, J.A., et al.: Collaborative filtering recommender systems. Foundations and Trends R in Human–Computer Interaction 4(2), 81–173 (2011) US ranking metric embedding for next new poi recommendation. In: IJCAI, vol. 15, pp. 2069–2075 (2015) [12] Gantner, Z., Drumond, L., Freudenthaler, C., Schmidt-Thieme, L.: Person- Cup 2011, pp. 231–247 (2012) AN alized ranking for non-uniformly sampled items. In: Proceedings of KDD [13] Gantner, Z., Rendle, S., Freudenthaler, C., Schmidt-Thieme, L.: MyMediaLite: A free recommender system library. In: Proceedings of the 5th ACM Conference on Recommender Systems (RecSys 2011) (2011) [14] George, T., Merugu, S.: A scalable collaborative filtering framework based DM on co-clustering. In: Data Mining, Fifth IEEE international conference on, pp. 4–pp. IEEE (2005) [15] Harper, F.M., Konstan, J.A.: The movielens datasets: History and context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5(4), 19 (2016) [16] Hug, N.: Surprise, a Python library for recommender systems. http: //surpriselib.com (2017) TE [17] Jannach, D., Lerche, L., Jugovac, M.: Adaptation and evaluation of recommendations for short-term shopping goals. In: Proceedings of the 9th ACM Conference on Recommender Systems, pp. 211–218. ACM (2015) [18] Jiang, S., Qian, X., Mei, T., Fu, Y.: Personalized travel sequence recom- EP mendation on multi-source big social media. IEEE Transactions on Big Data 2(1), 43–56 (2016) [19] Kolar, T.: Conceptualising tourist experiences with new attractions: the case of escape rooms. International Journal of Contemporary Hospitality Management 29(5), 1322–1339 (2017) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. [11] Feng, S., Li, X., Zeng, Y., Cong, G., Chee, Y.M., Yuan, Q.: Personalized 17 IPT CR Accepted manuscript to appear in VJCS [20] Koren, Y.: Factorization meets the neighborhood: a multifaceted collaborative filtering model. In: Proceedings of the 14th ACM SIGKDD international conference on Knowledge discovery and data mining, pp. 426–434. US [21] Koren, Y.: Factor in the neighbors: Scalable and accurate collaborative filtering. ACM Transactions on Knowledge Discovery from Data (TKDD) 4(1), 1 (2010) AN [22] Koren, Y., Bell, R.: Advances in collaborative filtering. In: Recommender systems handbook, pp. 77–118. Springer (2015) [23] Lops, P., De Gemmis, M., Semeraro, G.: Content-based recommender systems: State of the art and trends. In: Recommender systems handbook, pp. 73–105. Springer (2011) DM [24] Lu, J., Wu, D., Mao, M., Wang, W., Zhang, G.: Recommender system application developments: a survey. Decision Support Systems 74, 12–32 (2015) [25] Luo, X., Zhou, M., Xia, Y., Zhu, Q.: An efficient non-negative matrixfactorization-based approach to collaborative filtering for recommender systems. IEEE Transactions on Industrial Informatics 10(2), 1273–1284 (2014) [26] Masthoff, J.: Group recommender systems: Combining individual models. TE In: Recommender systems handbook, pp. 677–702. Springer (2011) [27] Mattila, A.S.: How affective commitment boosts guest loyalty (and promotes frequent-guest programs). Cornell Hotel and Restaurant Administration Quarterly 47(2), 174–181 (2006) EP [28] Mauri, A.G., Minazzi, R.: Web reviews influence on expectations and purchasing intentions of hotel potential customers. International Journal of Hospitality Management 34, 99–107 (2013) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. ACM (2008) 18 IPT CR Accepted manuscript to appear in VJCS [29] Nicholson, S.: Peeking behind the locked door: A survey of escape room facilities. White Paper available online at http://scottnicholson. com/pubs/erfacwhite. pdf (2015) US A comprehensive survey of neighborhood-based recommendation methods. In: Recommender systems handbook, pp. 37–76. Springer (2015) [31] Paterek, A.: Improving regularized singular value decomposition for col- AN laborative filtering. In: Proceedings of KDD cup and workshop, vol. 2007, pp. 5–8 (2007) [32] Ponnamma Divakaran, P.K., Nørskov, S.: Are online communities on par with experts in the evaluation of new movies? evidence from the fandango community. Information Technology & People 29(1), 120–145 (2016) DM [33] Rendle, S., Freudenthaler, C., Gantner, Z., Schmidt-Thieme, L.: Bpr: Bayesian personalized ranking from implicit feedback. In: Proceedings of the twenty-fifth conference on uncertainty in artificial intelligence, pp. 452–461. AUAI Press (2009) [34] Ricci, F., Rokach, L., Shapira, B.: Introduction to recommender systems handbook. In: Recommender systems handbook, pp. 1–35. Springer (2011) [35] Ricci, F., Rokach, L., Shapira, B.: Recommender systems: IntroducIn: Recommender Systems Handbook, pp. 1–34 TE tion and challenges. (2015). DOI 10.1007/978-1-4899-7637-6\ 1. URL https://doi.org/10. 1007/978-1-4899-7637-6\_1 [36] Sarwar, B., Karypis, G., Konstan, J., Riedl, J.: Item-based collaborative EP filtering recommendation algorithms. In: Proceedings of the 10th international conference on World Wide Web, pp. 285–295. ACM (2001) [37] Shani, G., Gunawardana, A.: Evaluating recommendation systems. In: Recommender systems handbook, pp. 257–297. Springer (2011) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. [30] Ning, X., Desrosiers, C., Karypis, G.: 19 IPT CR Accepted manuscript to appear in VJCS [38] Shani, G., Heckerman, D., Brafman, R.I.: An mdp-based recommender system. Journal of Machine Learning Research 6(Sep), 1265–1295 (2005) [39] Smith, B., Linden, G.: Two decades of recommender systems at amazon. US [40] Wiemker, M., Elumir, E., Clare, A.: Escape room games. Game Based EP TE DM AN Learning 55 (2015) AC C Vietnam J. Comp. Sci. Downloaded from www.worldscientific.com by 198.252.58.113 on 07/06/19. Re-use and distribution is strictly not permitted, except for Open Access articles. com. Ieee internet computing 21(3), 12–18 (2017) 20