Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks

Bryan Carpenter; Christina Pierrette Abilali Diedhiou; Aamir Shafi; Soumabha Sarkar; Ramazan Esmeli; Ryan Gadsdon

doi:10.1109/CAHPC.2018.8645926

Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks

Bryan Carpenter, Christina Pierrette Abilali Diedhiou, Aamir Shafi, Soumabha Sarkar, Ramazan Esmeli, Ryan Gadsdon

Imam Abdulrahman Bin Faisal University

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

449 Downloads (Pure)

Abstract

One of the challenges our society faces is the ever-increasing amount of data, requiring systems to analyse large data sets without compromising their performances, and humans to navigate through a deluge of irrelevant material. Among existing platforms that address the system requirements, Hadoop is a framework widely used to store and analyse “big data”. On the human side, one of the aids to finding the things people really want is recommendation systems. This paper evaluates approaches to highly scalable parallel algorithms for recommendation systems with application to very large data sets. A particular goal is to evaluate an open source Java message passing library for parallel computing called MPJ Express, which has been integrated with Hadoop. As a demonstration we use MPJ Express to implement collaborative filtering on various data sets using the algorithm ALSWR (Alternating-Least-Squares with Weighted-λ-Regularization). We benchmark the performance and demonstrate parallel speedup on Movielens and Yahoo Music data sets. We then compare our results with two other frameworks: Mahout and Spark. Our results indicate that MPJ Express implementation of ALSWR has very competitive performance and scalability in comparison with the two other frameworks.

Original language	English
Title of host publication	2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)
Publisher	Institute of Electrical and Electronics Engineers Inc.
Number of pages	8
ISBN (Electronic)	978-1-5386-7769-8
ISBN (Print)	978-1-5386-7770-4
DOIs	https://doi.org/10.1109/CAHPC.2018.8645926
Publication status	Published - 21 Feb 2019
Event	High Performance Machine Learning Workshop - Lyon, France Duration: 24 Nov 2018 → …

Publication series

Name	IEEE SBAC-PAD Proceedings Series
Publisher	IEEE
ISSN (Print)	1550-6533

Conference

Conference	High Performance Machine Learning Workshop
Abbreviated title	HMPL 2018
Country/Territory	France
City	Lyon
Period	24/11/18 → …

Keywords

HPC
Mahout
Spark
YARN
MapReduce
Hadoop
MPJExpress

UN SDGs

This output contributes to the following UN Sustainable Development Goals (SDGs)

Access to Document

10.1109/CAHPC.2018.8645926

HPML_Conference (6)
© © 2019 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.
Accepted author manuscript (Post-print), 294 KB

Cite this

Carpenter, B., Diedhiou, C. P. A., Shafi, A., Sarkar, S., Esmeli, R., & Gadsdon, R. (2019). Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks. In 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD) (IEEE SBAC-PAD Proceedings Series). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/CAHPC.2018.8645926

Carpenter, Bryan ; Diedhiou, Christina Pierrette Abilali ; Shafi, Aamir et al. / Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks. 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). Institute of Electrical and Electronics Engineers Inc., 2019. (IEEE SBAC-PAD Proceedings Series).

@inproceedings{cccb62b18867443db66e806de7fa7811,

title = "Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks",

abstract = "One of the challenges our society faces is the ever-increasing amount of data, requiring systems to analyse large data sets without compromising their performances, and humans to navigate through a deluge of irrelevant material. Among existing platforms that address the system requirements, Hadoop is a framework widely used to store and analyse “big data”. On the human side, one of the aids to finding the things people really want is recommendation systems. This paper evaluates approaches to highly scalable parallel algorithms for recommendation systems with application to very large data sets. A particular goal is to evaluate an open source Java message passing library for parallel computing called MPJ Express, which has been integrated with Hadoop. As a demonstration we use MPJ Express to implement collaborative filtering on various data sets using the algorithm ALSWR (Alternating-Least-Squares with Weighted-λ-Regularization). We benchmark the performance and demonstrate parallel speedup on Movielens and Yahoo Music data sets. We then compare our results with two other frameworks: Mahout and Spark. Our results indicate that MPJ Express implementation of ALSWR has very competitive performance and scalability in comparison with the two other frameworks.",

keywords = "HPC, Mahout, Spark, YARN, MapReduce, Hadoop, MPJExpress",

author = "Bryan Carpenter and Diedhiou, {Christina Pierrette Abilali} and Aamir Shafi and Soumabha Sarkar and Ramazan Esmeli and Ryan Gadsdon",

year = "2019",

month = feb,

day = "21",

doi = "10.1109/CAHPC.2018.8645926",

language = "English",

isbn = "978-1-5386-7770-4",

series = "IEEE SBAC-PAD Proceedings Series",

publisher = "Institute of Electrical and Electronics Engineers Inc.",

booktitle = "2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)",

address = "United States",

note = "High Performance Machine Learning Workshop, HMPL 2018 ; Conference date: 24-11-2018",

}

Carpenter, B, Diedhiou, CPA, Shafi, A, Sarkar, S, Esmeli, R & Gadsdon, R 2019, Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks. in 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). IEEE SBAC-PAD Proceedings Series, Institute of Electrical and Electronics Engineers Inc., High Performance Machine Learning Workshop, Lyon, France, 24/11/18. https://doi.org/10.1109/CAHPC.2018.8645926

Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks. / Carpenter, Bryan; Diedhiou, Christina Pierrette Abilali; Shafi, Aamir et al.
2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). Institute of Electrical and Electronics Engineers Inc., 2019. (IEEE SBAC-PAD Proceedings Series).

Research output: Chapter in Book/Report/Conference proceeding › Conference contribution

TY - GEN

T1 - Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks

AU - Carpenter, Bryan

AU - Diedhiou, Christina Pierrette Abilali

AU - Shafi, Aamir

AU - Sarkar, Soumabha

AU - Esmeli, Ramazan

AU - Gadsdon, Ryan

PY - 2019/2/21

Y1 - 2019/2/21

N2 - One of the challenges our society faces is the ever-increasing amount of data, requiring systems to analyse large data sets without compromising their performances, and humans to navigate through a deluge of irrelevant material. Among existing platforms that address the system requirements, Hadoop is a framework widely used to store and analyse “big data”. On the human side, one of the aids to finding the things people really want is recommendation systems. This paper evaluates approaches to highly scalable parallel algorithms for recommendation systems with application to very large data sets. A particular goal is to evaluate an open source Java message passing library for parallel computing called MPJ Express, which has been integrated with Hadoop. As a demonstration we use MPJ Express to implement collaborative filtering on various data sets using the algorithm ALSWR (Alternating-Least-Squares with Weighted-λ-Regularization). We benchmark the performance and demonstrate parallel speedup on Movielens and Yahoo Music data sets. We then compare our results with two other frameworks: Mahout and Spark. Our results indicate that MPJ Express implementation of ALSWR has very competitive performance and scalability in comparison with the two other frameworks.

AB - One of the challenges our society faces is the ever-increasing amount of data, requiring systems to analyse large data sets without compromising their performances, and humans to navigate through a deluge of irrelevant material. Among existing platforms that address the system requirements, Hadoop is a framework widely used to store and analyse “big data”. On the human side, one of the aids to finding the things people really want is recommendation systems. This paper evaluates approaches to highly scalable parallel algorithms for recommendation systems with application to very large data sets. A particular goal is to evaluate an open source Java message passing library for parallel computing called MPJ Express, which has been integrated with Hadoop. As a demonstration we use MPJ Express to implement collaborative filtering on various data sets using the algorithm ALSWR (Alternating-Least-Squares with Weighted-λ-Regularization). We benchmark the performance and demonstrate parallel speedup on Movielens and Yahoo Music data sets. We then compare our results with two other frameworks: Mahout and Spark. Our results indicate that MPJ Express implementation of ALSWR has very competitive performance and scalability in comparison with the two other frameworks.

KW - HPC

KW - Mahout

KW - Spark

KW - YARN

KW - MapReduce

KW - Hadoop

KW - MPJExpress

UR - https://hpml2018.github.io/

U2 - 10.1109/CAHPC.2018.8645926

DO - 10.1109/CAHPC.2018.8645926

M3 - Conference contribution

SN - 978-1-5386-7770-4

T3 - IEEE SBAC-PAD Proceedings Series

BT - 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD)

PB - Institute of Electrical and Electronics Engineers Inc.

T2 - High Performance Machine Learning Workshop

Y2 - 24 November 2018

ER -

Carpenter B, Diedhiou CPA, Shafi A, Sarkar S, Esmeli R, Gadsdon R. Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks. In 2018 30th International Symposium on Computer Architecture and High Performance Computing (SBAC-PAD). Institute of Electrical and Electronics Engineers Inc. 2019. (IEEE SBAC-PAD Proceedings Series). doi: 10.1109/CAHPC.2018.8645926

Performance comparison of a parallel recommender algorithm across three Hadoop-based frameworks

Abstract

Publication series

Conference

Keywords

UN SDGs

Access to Document

Fingerprint

Cite this