Repository navigation
fix: do not skip the last batch on Windows in train, eval and get_center - #18446
Open
sibisrinivasb wants to merge 1 commit into
Open
sibisrinivasb wants to merge 1 commit into
sibisrinivasb wants to merge 1 commit into
Conversation
Since April 2021 (PaddlePaddle#2654 and d5651df, for the crash reported in PaddlePaddle#2636) the loops in tools/program.py stop at len(dataloader) - 1 when platform.system() is Windows, so on Windows the last batch of every epoch is never trained on and the last batch of every evaluation is never scored. With PaddlePaddle 3.3.1 on Windows 11 the loops run to the end without the limit (GPU and CPU, 0 and 2 loader workers), so remove it and the now unused import.
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Fixes #18445.
What
tools/program.pystopped its loops atlen(dataloader) - 1whenplatform.system()is"Windows", in four places (train, twice; eval; get_center). This change setsmax_iter = len(dataloader)in all four and drops theplatformimport, which is not used anywhere else in the file.Why
The limit was added in April 2021 (#2654 for evaluation, d5651df for training) after #2636, where training on Windows 10 broke off at "eval model". Since then, on Windows the last batch of every epoch is never trained on and the last batch of every evaluation is never scored; a dataset that fits in one batch does nothing at all.
With PaddlePaddle 3.3.1 on Windows 11 the loops run to the end without the limit. A toy set of 40 word images, batch size 4 (10 iterations),
configs/rec/rec_icdar15_train.yml, 1 epoch, throughtools/train.pyandtools/eval.py:dab3fe3)dab3fe3)Not tested
get_centerpath.max_iterwas alreadylen(dataloader)there.If the limit is still needed for an older setup you support, I can restrict the change instead (for example keep it behind a version check) - tell me which you prefer.