Ver Fonte

perf(manual/trd): 订单 re-init rebucket 加 Spark 资源调优 SET(集群闲置放大)

L2 SET 覆盖默认:executor 20×4核/8g + overhead 2g、driver 6g、shuffle/parallelism 800
(默认 15×2核/6g/200);动态分配关,杠杆是 executor.instances。按集群容量可增减。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tianyu.chu há 2 semanas atrás
pai
commit
f9d8367392

+ 11 - 0
manual/backfill/20260723_ods_trd_card_group_order_info_reinit_rebucket.sql

@@ -17,6 +17,17 @@ SET hive.exec.dynamic.partition.mode=nonstrict;
 SET hive.exec.max.dynamic.partitions=10000;
 SET hive.exec.max.dynamic.partitions.pernode=10000;
 
+-- 资源放大(集群闲置,一次性重灌;L2 SET 覆盖 spark-tuning.conf 默认 15×2核/6g/shuffle200)。
+-- 动态分配是关的,杠杆是 executor.instances。按集群实际容量增减:单 executor 别超 YARN 容器上限
+-- (executor.memory+overhead ≤ yarn.scheduler.maximum-allocation-mb;cores ≤ maximum-allocation-vcores)。
+SET spark.executor.instances=20;
+SET spark.executor.cores=4;
+SET spark.executor.memory=8g;
+SET spark.executor.memoryOverhead=2g;
+SET spark.driver.memory=6g;
+SET spark.sql.shuffle.partitions=800;
+SET spark.default.parallelism=800;
+
 INSERT OVERWRITE TABLE ods.ods_trd_card_group_order_info_inc_d PARTITION (dt)
 SELECT
     CAST(id                        AS BIGINT)        AS id,