The Apache Software Foundation (ASF), the all-volunteer developers, stewards, and incubators of nearly 150 Open Source projects and initiatives, announced today that Apache Mesos has graduated from the Apache Incubator to become a Top-Level Project (TLP), signifying that the project’s community and products have been well-governed under the ASF’s meritocratic process and principles.
Apache Mesos is a cluster manager that provides efficient resource isolation and sharing across distributed applications, or frameworks. It can run multiple frameworks, including Apache Hadoop, MPI, Hypertable, Jenkins, Storm, and Spark, as well as other applications and custom frameworks.
“It was our goal all along to see Mesos become a kernel of the infrastructure stack of the future,” said Benjamin Hindman, Vice President of Apache Mesos. “The project’s graduation from the Apache Incubator is recognition that the software is mature and has brought together a diverse community to sustain it in the future.”
Initially created at the University of California at Berkeley’s AMPLab (the research center also responsible for the original development of Apache Spark) to manage resource sharing and isolation in data centers, Mesos acts as a layer of abstraction between applications and pools of servers. Mesos helps avoid the necessity of creating separate clusters to run individual frameworks and instead making it possible to optimize how jobs are executed across shared machines.
Whilst in the Apache Incubator, Mesos had four releases, and established an Open Source community according to The Apache Way of governance. Additional improvements to the project includes its flexibility to support several application framework languages, and scalability that has been production tested to thousands of nodes and simulated to tens of thousands of nodes and hundreds of frameworks.
Apache Mesos has proven to be reliable for use in production, and has already been adopted by several organizations for cluster management.
“Mesos is the cornerstone of our elastic compute infrastructure,” explained Chris Fry, Senior Vice President of Engineering at Twitter. “It’s how we build all our new services and is critical for Twitter’s continued success at scale … one of the primary keys to our data infrastructure efficiency.”
“We’re using Mesos to manage cluster resources for most of our data infrastructure,” said Brenden Matthews, Engineer at Airbnb and Apache Mesos Committer. “We run Chronos, Storm, and Hadoop on top of Mesos in order to process petabytes of data.” (Chronos is an Airbnb-developed Mesos framework as a replacement for cron, and an example of how custom frameworks can be developed