Recbench: Benchmarks for evaluating performance of recommender system architectures

Justin J. Levandoski, Michael D. Ekstrand, Michael J. Ludwig, Ahmed Eldawy, Mohamed Mokbel, John T. Riedl

Research output: Contribution to journalArticle

8 Citations (Scopus)

Abstract

Traditionally, recommender systems have been "hand-built", implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the "hand-built" MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: "hand-built" recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.

Original languageEnglish
Pages (from-to)911-920
Number of pages10
JournalProceedings of the VLDB Endowment
Volume4
Issue number11
Publication statusPublished - 1 Aug 2011
Externally publishedYes

Fingerprint

Recommender systems

ASJC Scopus subject areas

  • Computer Science (miscellaneous)
  • Computer Science(all)

Cite this

Levandoski, J. J., Ekstrand, M. D., Ludwig, M. J., Eldawy, A., Mokbel, M., & Riedl, J. T. (2011). Recbench: Benchmarks for evaluating performance of recommender system architectures. Proceedings of the VLDB Endowment, 4(11), 911-920.

Recbench : Benchmarks for evaluating performance of recommender system architectures. / Levandoski, Justin J.; Ekstrand, Michael D.; Ludwig, Michael J.; Eldawy, Ahmed; Mokbel, Mohamed; Riedl, John T.

In: Proceedings of the VLDB Endowment, Vol. 4, No. 11, 01.08.2011, p. 911-920.

Research output: Contribution to journalArticle

Levandoski, JJ, Ekstrand, MD, Ludwig, MJ, Eldawy, A, Mokbel, M & Riedl, JT 2011, 'Recbench: Benchmarks for evaluating performance of recommender system architectures', Proceedings of the VLDB Endowment, vol. 4, no. 11, pp. 911-920.
Levandoski JJ, Ekstrand MD, Ludwig MJ, Eldawy A, Mokbel M, Riedl JT. Recbench: Benchmarks for evaluating performance of recommender system architectures. Proceedings of the VLDB Endowment. 2011 Aug 1;4(11):911-920.
Levandoski, Justin J. ; Ekstrand, Michael D. ; Ludwig, Michael J. ; Eldawy, Ahmed ; Mokbel, Mohamed ; Riedl, John T. / Recbench : Benchmarks for evaluating performance of recommender system architectures. In: Proceedings of the VLDB Endowment. 2011 ; Vol. 4, No. 11. pp. 911-920.
@article{97fe25125a4e4d419afd84f9c2d40275,
title = "Recbench: Benchmarks for evaluating performance of recommender system architectures",
abstract = "Traditionally, recommender systems have been {"}hand-built{"}, implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the {"}hand-built{"} MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: {"}hand-built{"} recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.",
author = "Levandoski, {Justin J.} and Ekstrand, {Michael D.} and Ludwig, {Michael J.} and Ahmed Eldawy and Mohamed Mokbel and Riedl, {John T.}",
year = "2011",
month = "8",
day = "1",
language = "English",
volume = "4",
pages = "911--920",
journal = "Proceedings of the VLDB Endowment",
issn = "2150-8097",
publisher = "Very Large Data Base Endowment Inc.",
number = "11",

}

TY - JOUR

T1 - Recbench

T2 - Benchmarks for evaluating performance of recommender system architectures

AU - Levandoski, Justin J.

AU - Ekstrand, Michael D.

AU - Ludwig, Michael J.

AU - Eldawy, Ahmed

AU - Mokbel, Mohamed

AU - Riedl, John T.

PY - 2011/8/1

Y1 - 2011/8/1

N2 - Traditionally, recommender systems have been "hand-built", implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the "hand-built" MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: "hand-built" recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.

AB - Traditionally, recommender systems have been "hand-built", implemented as custom applications hard-wired to a particular recommendation task. Recently, the database community has begun exploring alternative DBMS-based recommender system architectures, whereby a database both stores the recommender system data (e.g., ratings data and the derived recommender models) and generates recommendations using SQL queries. In this paper, we present a comprehensive experimental comparison of both architectures. We define a set of benchmark tasks based on the needs of a typical recommender-powered e-commerce site. We then evaluate the performance of the "hand-built" MultiLens recommender application against two DBMS-based implementations: an unmodified DBMS and RecStore, a DBMS modified to improve efficiency in incremental recommender model updates. We employ two non-trivial data sets in our study: the 10 million rating MovieLens data, and the 100 million rating data set used in the Netflix Challenge. This study is the first of its kind, and our findings reveal an interesting trade-off: "hand-built" recommenders exhibit superior performance in model-building and pure recommendation tasks, while DBMS-based recommenders are superior at more complex recommendation tasks such as providing filtered recommendations and blending text-search with recommendation prediction scores.

UR - http://www.scopus.com/inward/record.url?scp=84863795647&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84863795647&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:84863795647

VL - 4

SP - 911

EP - 920

JO - Proceedings of the VLDB Endowment

JF - Proceedings of the VLDB Endowment

SN - 2150-8097

IS - 11

ER -