Byung-Gon Chun, Tyson Condie, Yingda Chen, Brian Cho, Andrew Chung, Carlo Curino, Chris Douglas, Matteo Interlandi, Beomyeol Jeon, Joo Seong Jeong, Gyewon Lee, Yunseong Lee, Tony Majestro, Dahlia Malkhi, Sergiy Matusevych, Brandon Myers, Mariia Mykhailova, Shravan Narayanamurthy, Joseph Noor, Raghu Ramakrishnan, Sriram Rao, Russell Sears, Beysim Sezgin, Taegeon Um, Julia Wang, Markus Weimer, Youngseok Yang

Abstract

Resource Managers like YARN and Mesos have emerged as a critical layer in the cloud computing system stack, but the developer abstractions for leasing cluster resources and instantiating application logic are very lowlevel. This flexibility comes at a high cost in terms of developer effort, as each application must repeatedly tackle the same challenges (e.g., fault tolerance, task scheduling and coordination) and reimplement common mechanisms (e.g., caching, bulk-data transfers). This article presents REEF, a development framework that provides a control plane for scheduling and coordinating task-level (data-plane) work on cluster resources obtained from a Resource Manager. REEF provides mechanisms that facilitate resource reuse for data caching and state management abstractions that greatly ease the development of elastic data processing pipelines on cloud platforms that support a Resource Manager service. We illustrate the power of REEF by showing applications built atop: a distributed shell application, a machine-learning framework, a distributed in-memory caching system, and a port of the CORFU system. REEF is currently an Apache top-level project that has attracted contributors from several institutions and it is being used to develop several commercial offerings such as the Azure Stream Analytics service.

Download PDF

BibTeX

@article{Chun:2017:ARR:3129286.3132037,
author = {Chun, Byung-Gon and Condie, Tyson and Chen, Yingda and Cho, Brian and Chung, Andrew and Curino, Carlo and Douglas, Chris and Interlandi, Matteo and Jeon, Beomyeol and Jeong, Joo Seong and Lee, Gyewon and Lee, Yunseong and Majestro, Tony and Malkhi, Dahlia and Matusevych, Sergiy and Myers, Brandon and Mykhailova, Mariia and Narayanamurthy, Shravan and Noor, Joseph and Ramakrishnan, Raghu and Rao, Sriram and Sears, Russell and Sezgin, Beysim and Um, Taegeon and Wang, Julia and Weimer, Markus and Yang, Youngseok},
title = {Apache REEF: Retainable Evaluator Execution Framework},
journal = {ACM Trans. Comput. Syst.},
issue_date = {October 2017},
volume = {35},
number = {2},
month = oct,
year = {2017},
issn = {0734-2071},
pages = {5:1--5:31},
articleno = {5},
numpages = {31},
url = {http://doi.acm.org/10.1145/3132037},
doi = {10.1145/3132037},
acmid = {3132037},
publisher = {ACM},
address = {New York, NY, USA},
keywords = {Data processing, resource management},
}