we have hadoop cluster configured cloudera cdh 5.4.2 distribution yarn fair scheduler scheduling , managing resources. have 18 worker nodes total amounting 963gb main memory , 288 vcores.
now problem able run 1 application @ time no matter how small resource requirement of application is. example here dynamic resource pool status:
status yarn using 288 vcores , 942 gib of memory. resource pools usage
this table , charts on right contain metrics yarn only. resource pool name allocated memory allocated vcores allocated containers pending containers
root 0 b 0 0 0
user1 0 b 0 0 1
default 0 b 0 0 0
user2 22 gib 11 11 1
for user 2 though 22gb , 11 vcores allocated out of 288 vcores , 942 gib of memory, there still pending container! idea going on? not case days before, not sure setting causing this?
here yarn-site.xml content
<?xml version="1.0" encoding="utf-8"?> <!--autogenerated cloudera manager--> <configuration> <property> <name>yarn.acl.enable</name> <value>true</value> </property> <property> <name>yarn.admin.acl</name> <value>*</value> </property> <property> <name>yarn.resourcemanager.address</name> <value>hadoopmaster:8032</value> </property> <property> <name>yarn.resourcemanager.admin.address</name> <value>hadoopmaster:8033</value> </property> <property> <name>yarn.resourcemanager.scheduler.address</name> <value>hadoopmaster:8030</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.address</name> <value>hadoopmaster:8031</value> </property> <property> <name>yarn.resourcemanager.webapp.address</name> <value>hadoopmaster:8088</value> </property> <property> <name>yarn.resourcemanager.webapp.https.address</name> <value>hadoopmaster:8090</value> </property> <property> <name>yarn.resourcemanager.client.thread-count</name> <value>50</value> </property> <property> <name>yarn.resourcemanager.scheduler.client.thread-count</name> <value>50</value> </property> <property> <name>yarn.resourcemanager.admin.client.thread-count</name> <value>1</value> </property> <property> <name>yarn.scheduler.minimum-allocation-mb</name> <value>2048</value> </property> <property> <name>yarn.scheduler.increment-allocation-mb</name> <value>512</value> </property> <property> <name>yarn.scheduler.maximum-allocation-mb</name> <value>65536</value> </property> <property> <name>yarn.scheduler.minimum-allocation-vcores</name> <value>1</value> </property> <property> <name>yarn.scheduler.increment-allocation-vcores</name> <value>1</value> </property> <property> <name>yarn.scheduler.maximum-allocation-vcores</name> <value>24</value> </property> <property> <name>yarn.resourcemanager.amliveliness-monitor.interval-ms</name> <value>1000</value> </property> <property> <name>yarn.am.liveness-monitor.expiry-interval-ms</name> <value>600000</value> </property> <property> <name>yarn.resourcemanager.am.max-attempts</name> <value>2</value> </property> <property> <name>yarn.resourcemanager.container.liveness-monitor.interval-ms</name> <value>600000</value> </property> <property> <name>yarn.resourcemanager.nm.liveness-monitor.interval-ms</name> <value>1000</value> </property> <property> <name>yarn.nm.liveness-monitor.expiry-interval-ms</name> <value>600000</value> </property> <property> <name>yarn.resourcemanager.resource-tracker.client.thread-count</name> <value>50</value> </property> <property> <name>yarn.application.classpath</name> <value>$hadoop_client_conf_dir,$hadoop_conf_dir,$hadoop_common_home/*,$hadoop_common_home/lib/*,$hadoop_hdfs_home/*,$hadoop_hdfs_home/lib/*,$hadoop_yarn_home/*,$hadoop_yarn_home/lib/*</value> </property> <property> <name>yarn.resourcemanager.scheduler.class</name> <value>org.apache.hadoop.yarn.server.resourcemanager.scheduler.fair.fairscheduler</value> </property> <property> <name>yarn.scheduler.fair.user-as-default-queue</name> <value>true</value> </property> <property> <name>yarn.scheduler.fair.preemption</name> <value>true</value> </property> <property> <name>yarn.scheduler.fair.sizebasedweight</name> <value>false</value> </property> <property> <name>yarn.scheduler.fair.assignmultiple</name> <value>true</value> </property> <property> <name>yarn.resourcemanager.max-completed-applications</name> <value>10000</value> </property> </configuration>
can please check fair-scheduler.xml file value "maxrunningapps" defines number of concurrent application can run in queue.
please refer https://hadoop.apache.org/docs/r2.4.1/hadoop-yarn/hadoop-yarn-site/fairscheduler.html
Comments
Post a Comment