Kaynağa Gözat

fix(datax): datax.py 显式 -Xmx6g,避免多通道 ORC 写 GC 风暴/OOM

默认堆 ~1G,6 通道 + ORC writer 攒 stripe 打满老年代、Full GC 风暴甚至 OOM;
runner 起 datax.py 时传 --jvm=-Xms1g -Xmx6g(-Xmx 仅上限,小任务不浪费)。

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
tianyu.chu 2 hafta önce
ebeveyn
işleme
34aecd381d
1 değiştirilmiş dosya ile 3 ekleme ve 0 silme
  1. 3 0
      dw_base/datax/runner.py

+ 3 - 0
dw_base/datax/runner.py

@@ -122,9 +122,12 @@ def run_job(ini_path: str,
         print('[datax] {ini} skip_datax=True,跳过执行'.format(ini=ini_path))
         return 0
 
+    # DataX 默认堆 ~1G;多通道 + ORC writer 各自攒 stripe 很吃内存,老年代易打满 →
+    # Full GC 风暴甚至 OOM。显式抬到 6G(-Xmx 是上限,小任务只用到 -Xms 不浪费)。
     exec_argv = [
         python3_path, '-u',
         os.path.join(datax_home, 'bin', 'datax.py'),
+        '--jvm=-Xms1g -Xmx6g',
         json_path,
     ]
     print('[datax] {ini} 开始执行 datax.py @ worker={w}'.format(ini=ini_path, w=worker_host))