Hadoop Yarn

Yet Another Resource Negotiator

• Global resource manager

• Per-application application master for job scheduling and monitoring

Hadoop 1.0 limitations

• Single Point of Failure – JobTracker and NameNode are the single point of failure. If it fails, the entire Job will fail.

• Limitation in running applications – Hadoop 1.0 is limited to run only a MapReduce application and supports only the batch mode of processing.

• Imbalance in Resource Utilization – Each TaskTracker is allocated predefined numbers of map and reduce slots, and hence resources may not be utilized completely when the map slots are performing tasks and might be full while the reduce slots are available to perform tasks and vice versa.

YARN NodeManager & Resource Manager

Commands

yarn jar
yarn node -list
yarn node -status <nodeId>
yarn top

Apache Hadoop YARN

The fundamental idea of YARN is to split up the functionalities of resource management and job scheduling/monitoring into separate daemons. The idea is to have a global ResourceManager ( RM) and per-application ApplicationMaster ( AM). An application is either a single job or a DAG of jobs.

https://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html