Просмотр исходного кода

fix(raw): 埋点 backfill raw 调用加 memoryOverhead=4g 防 UDF off-heap 撑爆 container

311 文件全量回算时 Python UDF mask_source 的 Python worker off-heap
内存超过默认 512MB overhead,YARN 持续杀 executor 累计 26 次,
最终单 task 累计 4 次失败导致 stage abort。只在 backfill 路径加,
daily 单文件不需要。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tianyu.chu 1 месяц назад
Родитель
Сommit
e497bb5a92
1 измененных файлов с 2 добавлено и 1 удалено
  1. 2 1
      jobs/raw/usr/raw_usr_traces_apd_d_backfill.py

+ 2 - 1
jobs/raw/usr/raw_usr_traces_apd_d_backfill.py

@@ -57,7 +57,8 @@ def main():
     start, stop = min(put), max(put)
     print('回算 %d 天,范围 [%s, %s]' % (len(put), start, stop))
 
-    rc_raw = run(['python3', STARTER, '-f', RAW_SQL, '-u', UDF], cwd=PROJECT_ROOT)
+    rc_raw = run(['python3', STARTER, '-f', RAW_SQL, '-u', UDF,
+                  '-sc', 'spark.executor.memoryOverhead=4g'], cwd=PROJECT_ROOT)
     rc_ods = run(['python3', STARTER, '-f', ODS_SQL,
                   '-p', 'start_date=%s' % start, '-p', 'stop_date=%s' % stop], cwd=PROJECT_ROOT)
     run(['hdfs', 'dfs', '-rm', '-r', '-f', '-skipTrash', STAGING])