Morph MORPH
SIGN IN SIGN UP
PaddlePaddle / Paddle UNCLAIMED

PArallel Distributed Deep LEarning: Machine Learning Framework from Industrial Practice （『飞桨』核心框架，深度学习&机器学习高性能单机、分布式训练和跨平台部署）

0 0 1 C++
Normal View History Raw
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
# Copyright (c) 2022 PaddlePaddle Authors. All Rights Reserved.
【code format check upgrade】 step2：yapf (#42944) * use yapf to format all python file * yapf exclude two unittests file for they rely on writing and reading file, and format will break them * disable diff_py_file because too many diff files cause command following failed
2022-06-05 10:58:58 +08:00
#
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
【code format check upgrade】 step2：yapf (#42944) * use yapf to format all python file * yapf exclude two unittests file for they rely on writing and reading file, and format will break them * disable diff_py_file because too many diff files cause command following failed
2022-06-05 10:58:58 +08:00
#
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
# http://www.apache.org/licenses/LICENSE-2.0
【code format check upgrade】 step2：yapf (#42944) * use yapf to format all python file * yapf exclude two unittests file for they rely on writing and reading file, and format will break them * disable diff_py_file because too many diff files cause command following failed
2022-06-05 10:58:58 +08:00
#
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
remove fluid (#47959) * remove fluid * update public * core * public * public1 * ci
2022-11-28 11:43:47 +08:00
import paddle
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
from paddle import base
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
from paddle.distributed.fleet.base.private_helper_function import (
wait_server_ready,
)
[CodeStyle][F401] remove unused imports in python/paddle/distributed (#46758) * [CodeStyle][F401] remove unused import in python/paddle/distributed * remove pass * empty commit * Fix ValueError: list.remove(x): x not in list for meta_optimizer_names. Fix ValueError: list.remove(x): x not in list for meta_optimizer_names. * Fix split import. Fix split import. * add noqa after meta_optimizers in factory * restort collective ops * expand `import *` * add noqa after required imports * try to fix APIs without core.ops * Revert "try to fix APIs without core.ops" This reverts commit 6172beaf601e84bf61f2490c12c4739f0edaa5eb. * fix an increment * empty commit * add noqa after required imports * expand `import *`, fix ci error Co-authored-by: Shuangchi He <34329208+Yulv-git@users.noreply.github.com>
2022-10-12 10:52:31 +08:00
from paddle.distributed.passes import new_pass
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
[CodeStyle][isort] introduce isort (part4) (#48402) * isort all files * revert conflicting files * revert conflicting files * revert conflicting files
2022-11-29 18:50:04 +08:00
from .public import * # noqa: F403
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
[CodeStyle][py2][U004] unecessary explicit `object` inheritance in class definition (#47642) * [CodeStyle][py2][U004] unecessary explicit `object` inheritance in class definition * fix an increment
2022-11-08 11:29:41 +08:00
class PsProgramBuilder:
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def __init__(self, pass_ctx):
self.pass_ctx = pass_ctx
self.attrs = self.pass_ctx._attrs
self.loss = self.attrs['loss']
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
self.origin_startup_program = self.attrs['origin_startup_program']
self.main_program = self.attrs['origin_main_programs']
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
self.cloned_main = self.attrs['cloned_main']
self.cloned_startup = self.attrs['cloned_startup']
self.use_ps_gpu = self.attrs['use_ps_gpu']
self.use_heter_ps = self.attrs['is_heter_ps_mode']
self.is_worker = self.attrs['is_worker']
self.is_heter_worker = self.attrs['is_heter_worker']
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
self.is_server = self.attrs['is_server']
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
self.ps_mode = self.attrs['ps_mode']
self.launch_barrier = self.attrs['launch_barrier']
self.launch_barrier_flag = self.attrs['launch_barrier_flag']
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
self.server_endpoints = self.attrs[
'role_maker'
]._get_pserver_endpoints()
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
def _build_trainer_desc(self):
opt_info = self.loss.block.program._fleet_opt
opt_info = {} if opt_info is None else opt_info
delete commonsparsetable and communicator from gpups (#40973) * trainer and worker * delete commonsparsetable from gpups * delete vlog * codestyle * delete communicator from gpups
2022-03-28 20:36:15 +08:00
opt_info["trainer"] = opt_info.get("trainer", "MultiTrainer")
opt_info["device_worker"] = opt_info.get("device_worker", "Hogwild")
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
self.cloned_main._fleet_opt = opt_info
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _optimize_programs(self):
pass
def _build_trainer_programs(self):
统一 ps 开发 - python (#39431) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . * cpu-async-ps minimize test ok & gpu minimize test ok Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-11 13:08:06 +08:00
raise NotImplementedError
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_pserver_programs(self):
is_sgd_adam = False
ops = get_optimize_ops(self.attrs['origin_main_program'])
if len(ops) == 0:
return
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_lr_decay_table_pass = new_pass(
'add_lr_decay_table_pass', self.attrs
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
add_lr_decay_table_pass.apply([], [], self.pass_ctx)
for op in ops:
if op.type in ["sgd", "adam"]:
is_sgd_adam = True
break
if is_sgd_adam:
return
def _build_programs(self):
if self.attrs['is_worker']:
self._build_trainer_programs()
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
base.framework.switch_startup_program(self.cloned_startup)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
print(
[CodeStyle][ruff] fix v0.3.3 UP032 (#63111)
2024-04-01 10:20:33 +08:00
f"paddle.static.default_startup_program: {paddle.static.default_startup_program}"
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
)
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
# print("ps_program_build before =", id(self.loss.block.program))
self._build_trainer_desc()
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
self.loss.block.program = self.cloned_main
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
# print("ps_program_build after =", id(self.loss.block.program))
# print("ps_program_build clone after =", id(self.cloned_main))
# print("ps_program_build after trainer_desc",
# id(self.loss.block.program))
# print("ps_program build trainer desc",
# self.loss.block.program._fleet_opt)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
elif self.attrs['is_server']:
self._build_pserver_programs()
self.loss.block.program = self.attrs['_main_server']
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
base.framework.switch_startup_program(self.attrs['_startup_server'])
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
class GeoPsProgramBuilder(PsProgramBuilder): # 仅 CPU 模式
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
if self.ps_mode != DistributedMode.GEO:
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
raise ValueError(
[CodeStyle] Fix pass multiple arg to error message (#65575) --------- Co-authored-by: gouzil <66515297+gouzil@users.noreply.github.com>
2024-06-30 16:37:22 +08:00
f"ps mode: {self.ps_mode} not matched GeoPsProgramBuilder",
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_trainer_programs(self):
append_send_ops_pass = new_pass("append_send_ops_pass", self.attrs)
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
self.attrs['origin_main_program'] = self.cloned_main
if self.launch_barrier and self.launch_barrier_flag:
Ps optimizer multi programs (#39883) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * the one ps merge * fix bug in test
2022-03-05 00:11:53 +08:00
wait_server_ready(self.server_endpoints)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
def _build_pserver_programs(self):
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_listen_and_serv_pass = new_pass(
'add_listen_and_serv_pass', self.attrs
)
add_listen_and_serv_pass.apply(
[self.attrs['_main_server']], [None], self.pass_ctx
)
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
class NuPsProgramBuilder(PsProgramBuilder):
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
if not self.attrs['local_sparse']:
raise ValueError("No local sparse params")
def _build_trainer_programs(self):
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_lr_decay_table_pass = new_pass(
"add_lr_decay_table_pass", self.attrs
)
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
add_lr_decay_table_pass.apply([], [], self.pass_ctx)
distributed_ops_pass = new_pass("distributed_ops_pass", self.attrs)
distributed_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
delete_optimizer_pass = new_pass("delete_optimizer_pass", self.attrs)
delete_optimizer_pass.apply([self.cloned_main], [None], self.pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
append_send_ops_pass = new_pass(
"append_send_ops_pass", self.attrs
) # fleet->PushDenseVarsAsync
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
delete_extra_optimizer_pass = new_pass(
"delete_extra_optimizer_pass", self.attrs
)
delete_extra_optimizer_pass.apply(
[self.attrs['origin_main_program']],
[self.cloned_startup],
self.pass_ctx,
)
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
fake_init_ops_pass = new_pass("fake_init_ops_pass", self.attrs)
fake_init_ops_pass.apply([None], [self.cloned_startup], self.pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
append_send_ops_pass = new_pass(
"append_send_ops_pass", self.attrs
) # communicator->Send
fl-ps: support split sparse params in local & remote (#44864) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * . * fix bug * . * . * fl-ps with coordinator ready * merge dev * update message parse only * update fl client scheduler * fix bug * update multithreads sync * fix ci errors * update role_maker.py * update role_maker.py * fix ci error: windows py import error * fix ci error: windows py import error * fix windows ci pylib import error * add dump fields & params * try to fix windows import fleet error * fix ps FLAGS error * fix logging risk * fix logging possible risk * write trainer_desc file * support split sparse params in local & remote * fix import paddle.fluid.core.PSGPU * fix import paddle.fluid.core.PSGPU * add remote_sparse & local_sparse config * fix unittest * fix test_dist_fleet_geo table error * fix PADDLE_ENFORCE error * fix other's pr conflict
2022-08-13 08:27:37 +08:00
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
self.attrs['origin_main_program'] = self.cloned_main
self.attrs['origin_startup_program'] = self.cloned_startup
if self.launch_barrier and self.launch_barrier_flag:
wait_server_ready(self.server_endpoints)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
class CpuSyncPsProgramBuilder(PsProgramBuilder):
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
if (
self.ps_mode != DistributedMode.SYNC
and self.ps_mode != DistributedMode.ASYNC
):
raise ValueError(
[CodeStyle] Fix pass multiple arg to error message (#65575) --------- Co-authored-by: gouzil <66515297+gouzil@users.noreply.github.com>
2024-06-30 16:37:22 +08:00
f"ps mode: {self.ps_mode} not matched PsProgramBuilder"
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_trainer_programs(self):
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
# print("build trainer program entry")
# print("before ps program builder program:", self.cloned_main)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_lr_decay_table_pass = new_pass(
"add_lr_decay_table_pass", self.attrs
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
add_lr_decay_table_pass.apply([], [], self.pass_ctx)
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
# print("before distributed op pass")
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
distributed_ops_pass = new_pass("distributed_ops_pass", self.attrs)
distributed_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
delete_optimizer_pass = new_pass("delete_optimizer_pass", self.attrs)
delete_optimizer_pass.apply([self.cloned_main], [None], self.pass_ctx)
append_send_ops_pass = new_pass("append_send_ops_pass", self.attrs)
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
delete_extra_optimizer_pass = new_pass(
"delete_extra_optimizer_pass", self.attrs
)
delete_extra_optimizer_pass.apply(
[self.attrs['origin_main_program']],
[self.cloned_startup],
self.pass_ctx,
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
fake_init_ops_pass = new_pass("fake_init_ops_pass", self.attrs)
fake_init_ops_pass.apply([None], [self.cloned_startup], self.pass_ctx)
self.attrs['origin_main_program'] = self.cloned_main
self.attrs['origin_startup_program'] = self.cloned_startup
two-phase training for ps (#40762) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * cvm & datanorm backend * fix dim * fix unittest * fix * the one ps merge * remove comm * add DownpourLiteWorker * all * fix * fix * device worker downpour lite * fix * fix bug in global shuffle * save inference model * fix & add log * fix * remove log * fix * fix save summary * fix * fix pscore * fix * fix * fix * fix * fix * remove logs * fix * fix * fix * fix * fix * add some comments * fix Co-authored-by: esythan <esythan@126.com>
2022-03-23 17:02:23 +08:00
# print("after ps program builder program:", self.cloned_main)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
if self.launch_barrier and self.launch_barrier_flag:
Ps optimizer multi programs (#39883) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * the one ps merge * fix bug in test
2022-03-05 00:11:53 +08:00
wait_server_ready(self.server_endpoints)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
class CpuAsyncPsProgramBuilder(CpuSyncPsProgramBuilder):
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
delete commonsparsetable and communicator from gpups (#40973) * trainer and worker * delete commonsparsetable from gpups * delete vlog * codestyle * delete communicator from gpups
2022-03-28 20:36:15 +08:00
def _build_trainer_desc(self):
opt_info = self.loss.block.program._fleet_opt
opt_info = {} if opt_info is None else opt_info
opt_info["trainer"] = opt_info.get("trainer", "DistMultiTrainer")
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
opt_info["device_worker"] = opt_info.get(
"device_worker", "DownpourLite"
)
delete commonsparsetable and communicator from gpups (#40973) * trainer and worker * delete commonsparsetable from gpups * delete vlog * codestyle * delete communicator from gpups
2022-03-28 20:36:15 +08:00
pid = str(id(self.cloned_main))
program_configs = {
pid: {
'pull_dense': [],
'push_dense': [],
'pull_sparse': [],
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
'push_sparse': [],
delete commonsparsetable and communicator from gpups (#40973) * trainer and worker * delete commonsparsetable from gpups * delete vlog * codestyle * delete communicator from gpups
2022-03-28 20:36:15 +08:00
}
}
dense_table_config = {}
send_ctx = get_the_one_send_context(self.attrs)
recv_ctx = get_the_one_recv_context(self.attrs)
for name, ctx in send_ctx.items():
if ctx.program_id() != id(self.loss.block.program):
continue
if ctx.is_sparse():
continue
if not ctx.is_tensor_table():
program_configs[pid]['pull_dense'].append(ctx.table_id())
program_configs[pid]['push_dense'].append(ctx.table_id())
dense_table_config[ctx.table_id()] = recv_ctx[ctx.table_id()]
opt_info['program_configs'] = program_configs
opt_info['dense_table_config'] = dense_table_config
self.cloned_main._fleet_opt = opt_info
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
统一 ps 开发 - python (#39431) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . * cpu-async-ps minimize test ok & gpu minimize test ok Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-11 13:08:06 +08:00
class GpuPsProgramBuilder(PsProgramBuilder):
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_trainer_programs(self):
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_lr_decay_table_pass = new_pass(
"add_lr_decay_table_pass", self.attrs
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
add_lr_decay_table_pass.apply([], [], self.pass_ctx)
distributed_ops_pass = new_pass("distributed_ops_pass", self.attrs)
distributed_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
fake_init_ops_pass = new_pass("fake_init_ops_pass", self.attrs)
fake_init_ops_pass.apply([None], [self.cloned_startup], self.pass_ctx)
ps_gpu_pass = new_pass("ps_gpu_pass", self.attrs)
ps_gpu_pass.apply([self.cloned_main], [None], self.pass_ctx)
Merge gpu graph to develop (#59000) pglbox version 3 and 4 into develop, support training big language model for ps mode; support multi-node sample and train in graph-based work.
2023-12-08 16:01:27 +08:00
if not getattr(self.attrs['user_defined_strategy'], "sharding", False):
ps_transpile_pass = new_pass("ps_transpile_pass", self.attrs)
ps_transpile_pass.apply(
[self.cloned_main], [self.cloned_startup], self.pass_ctx
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
self.attrs['origin_main_program'] = self.cloned_main
self.attrs['origin_startup_program'] = self.cloned_startup
if self.launch_barrier and self.launch_barrier_flag:
Ps optimizer multi programs (#39883) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * the one ps merge * fix bug in test
2022-03-05 00:11:53 +08:00
wait_server_ready(self.server_endpoints)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
class HeterAsyncPsProgramBuilder(PsProgramBuilder):
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_trainer_programs(self):
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
add_lr_decay_table_pass = new_pass(
"add_lr_decay_table_pass", self.attrs
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
add_lr_decay_table_pass.apply([], [], self.pass_ctx)
distributed_ops_pass = new_pass("distributed_ops_pass", self.attrs)
统一ps：heter ps 二阶段单测通过 (#39468) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . * cpu-async-ps minimize test ok & gpu minimize test ok * add heter 2stage unittest * add heter 2stage unittest * add heter 2stage unittest Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-14 14:18:41 +08:00
distributed_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
delete_optimizer_pass = new_pass("delete_optimizer_pass", self.attrs)
统一ps：heter ps 二阶段单测通过 (#39468) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . * cpu-async-ps minimize test ok & gpu minimize test ok * add heter 2stage unittest * add heter 2stage unittest * add heter 2stage unittest Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-14 14:18:41 +08:00
delete_optimizer_pass.apply([self.cloned_main], [None], self.pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
append_send_ops_pass = new_pass("append_send_ops_pass", self.attrs)
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
delete_extra_optimizer_pass = new_pass(
"delete_extra_optimizer_pass", self.attrs
)
delete_extra_optimizer_pass.apply(
[self.attrs['origin_main_program']],
[self.cloned_startup],
self.pass_ctx,
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
fake_init_ops_pass = new_pass("fake_init_ops_pass", self.attrs)
fake_init_ops_pass.apply([None], [self.cloned_startup], self.pass_ctx)
if self.is_heter_worker:
split_heter_worker_ops_pass = new_pass(
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
"split_heter_worker_ops_pass", self.attrs
)
split_heter_worker_ops_pass.apply(
[self.cloned_main], [None], self.pass_ctx
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
else:
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
split_trainer_ops_pass = new_pass(
"split_trainer_ops_pass", self.attrs
)
split_trainer_ops_pass.apply(
[self.cloned_main], [None], self.pass_ctx
)
set_heter_pipeline_opt_pass = new_pass(
'set_heter_pipeline_opt_pass', self.attrs
)
set_heter_pipeline_opt_pass.apply(
[self.cloned_main], [self.cloned_startup], self.pass_ctx
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
if self.launch_barrier and self.launch_barrier_flag:
Ps optimizer multi programs (#39883) * fix benchmark and communicator config * fix bugs of the_one_ps * multi program and fix bug in optimizer * multi program in the_one_ps * public commcontext * ps optimizer multi programs * the one ps merge * fix bug in test
2022-03-05 00:11:53 +08:00
wait_server_ready(self.server_endpoints)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_programs(self):
if self.attrs['is_worker'] or self.attrs['is_heter_worker']:
self._build_trainer_programs()
ps_set_heter_pipeline_opt_pass = new_pass(
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
"set_heter_pipeline_opt_pass", self.attrs
)
ps_set_heter_pipeline_opt_pass.apply(
[self.cloned_main], [self.cloned_startup], self.pass_ctx
)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
elif self.attrs['is_server']:
self._build_pserver_programs()
self.loss.block.program = self.attrs['_main_server']
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
base.framework.switch_startup_program(self.attrs['_startup_server'])
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
class FlPsProgramBuilder(HeterAsyncPsProgramBuilder):
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def __init__(self, pass_ctx):
[CodeStyle][py2][U008] remove unnecessary args in `super()` (#47549) * [CodeStyle][py2][U008] remove unnecessary args in `super()` * remove remained args * revert changes in test_pylayer_op * Revert "revert changes in test_pylayer_op" This reverts commit ff185a9ae738afac3b0264f61bde6c6b7f72e7c4. * revert some changes in example code
2022-11-03 14:33:00 +08:00
super().__init__(pass_ctx)
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_trainer_programs(self):
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
_main_file = ps_log_root_dir + '0_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
distributed_ops_pass = new_pass("distributed_ops_pass", self.attrs)
distributed_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
_main_file = ps_log_root_dir + '1_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
delete_optimizer_pass = new_pass("delete_optimizer_pass", self.attrs)
delete_optimizer_pass.apply([self.cloned_main], [None], self.pass_ctx)
_main_file = ps_log_root_dir + '2_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
append_send_ops_pass = new_pass("append_send_ops_pass", self.attrs)
append_send_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
_main_file = ps_log_root_dir + '3_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
delete_extra_optimizer_pass = new_pass(
"delete_extra_optimizer_pass", self.attrs
)
delete_extra_optimizer_pass.apply(
[self.attrs['origin_main_program']],
[self.cloned_startup],
self.pass_ctx,
)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
_main_file = ps_log_root_dir + '4_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# fake_init_ops_pass = new_pass("fake_init_ops_pass", self.attrs)
# fake_init_ops_pass.apply([None], [self.cloned_startup], self.pass_ctx)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
_main_file = ps_log_root_dir + '5_fl_worker_main_program.prototxt'
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
# debug_program(_main_file, self.cloned_main)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
split_trainer_ops_pass = new_pass("split_fl_ops_pass", self.attrs)
split_trainer_ops_pass.apply([self.cloned_main], [None], self.pass_ctx)
if not self.is_heter_worker:
self.part_a_program = self.pass_ctx._attrs['part_a_main_program']
self.cloned_main = self.part_a_program
_main_file = ps_log_root_dir + '8_fl_A_main_program.prototxt'
debug_program(_main_file, self.cloned_main)
else:
self.part_b_program = self.pass_ctx._attrs['part_b_main_program']
self.cloned_main = self.part_b_program
_main_file = ps_log_root_dir + '8_fl_B_main_program.prototxt'
debug_program(_main_file, self.cloned_main)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
set_heter_pipeline_opt_pass = new_pass(
'set_heter_pipeline_opt_pass', self.attrs
)
set_heter_pipeline_opt_pass.apply(
[self.cloned_main], [self.cloned_startup], self.pass_ctx
)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
self.attrs['origin_startup_program'] = self.cloned_startup
self.attrs['origin_main_program'] = self.cloned_main
if not self.is_heter_worker:
_main_file = ps_log_root_dir + 'final_fl_A_main_program.prototxt'
【code format check upgrade】 step2：yapf (#42944) * use yapf to format all python file * yapf exclude two unittests file for they rely on writing and reading file, and format will break them * disable diff_py_file because too many diff files cause command following failed
2022-06-05 10:58:58 +08:00
debug_program(
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
_main_file,
self.attrs['origin_main_program']._heter_pipeline_opt[
'section_program'
],
)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
else:
_main_file = ps_log_root_dir + 'final_fl_B_main_program.prototxt'
【code format check upgrade】 step2：yapf (#42944) * use yapf to format all python file * yapf exclude two unittests file for they rely on writing and reading file, and format will break them * disable diff_py_file because too many diff files cause command following failed
2022-06-05 10:58:58 +08:00
debug_program(
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
_main_file,
self.attrs['origin_main_program']._heter_pipeline_opt[
'section_program'
],
)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_pserver_programs(self):
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
self.loss.block.program = self.attrs['_main_server']
ps optimize refactor (#38982) * delete gloo connect retry * the_one_ps dirs reconstruct * . * . * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * create the_one_ps dirs * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * the one ps dirs modify * refactor ps optimize * refactor ps optimize * refactor ps optimize * . * . * . * . * . * . * refactor theoneps * the_one_ps * add ps pass unittest * add ps pass unittest * ps unitest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * ps unittest frame * add cpu_async_ps_mode test * add cpu_async_ps_mode test * add cpu_async_ps_mode test * ps unittest ready * ps unittest ready * solve dist_pass init conflict * solve import CommContext error * unittest ok * implement AllocateFrom * solve setup.py.in conflict * solve conflict * solve conflict * solve conflict * . * . Co-authored-by: zkh2016 <zhangkaihuo@baidu.com>
2022-02-08 17:22:06 +08:00
def _build_programs(self):
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
if not self.is_server:
self._build_trainer_programs()
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
base.framework.switch_startup_program(self.cloned_startup)
[Fluid clean]clean distributed fluid API (#49795)
2023-01-16 15:37:39 +08:00
paddle.framework.switch_main_program(self.cloned_main)
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
print(
[CodeStyle][ruff] fix v0.3.3 UP032 (#63111)
2024-04-01 10:20:33 +08:00
f"paddle.static.default_startup_program: {paddle.static.default_startup_program()._heter_pipeline_opt}"
[CodeStyle][black] format dy2static unittests (#47268) * [CodeStyle][black] format dy2static unittests * format some missing files * update lineno in test_origin_info * update lineno in test_error * update lineno
2022-10-24 15:26:50 +08:00
)
add federated learning parameter server(fl-ps) mode (#42682) * back fl * delete ssl cert * . * make warning * . * unittest paral degree * solve unittest * heter & multi cloud commm ready * . * . * fl-ps v1.0 * . * support N + N mode * . * . * . * . * delete print * . * . * . * .
2022-06-02 12:46:17 +08:00
else:
self._build_pserver_programs()
【Fluid Clean】 Rename paddle.fluid to paddle.base (#56576) * rename fluid to base * fix setup and rename paddle.fluid * fix protobuf generated paddle.fluid * fix some errors * fix cmake depends * fix some error * try to fix cache error * fix codestyle && some doc error * remove build change * fix some error --------- Co-authored-by: XiangGao <jeff41404@gmail.com>
2023-09-07 17:26:19 +08:00
base.framework.switch_startup_program(self.attrs['_startup_server'])
[Fluid clean]clean distributed fluid API (#49795)
2023-01-16 15:37:39 +08:00
paddle.framework.switch_main_program(self.attrs['_main_server'])
Reference in New Issue Copy Permalink