Skip to content

fix: do not skip the last batch on Windows in train, eval and get_center - #18446

Open
sibisrinivasb wants to merge 1 commit into
PaddlePaddle:mainfrom
sibisrinivasb:fix-windows-last-batch
Open

sibisrinivasb wants to merge 1 commit into
PaddlePaddle:mainfrom
sibisrinivasb:fix-windows-last-batch

Conversation

@sibisrinivasb

Copy link
Copy Markdown

Fixes #18445.

What

tools/program.py stopped its loops at len(dataloader) - 1 when platform.system() is "Windows", in four places (train, twice; eval; get_center). This change sets max_iter = len(dataloader) in all four and drops the platform import, which is not used anywhere else in the file.

Why

The limit was added in April 2021 (#2654 for evaluation, d5651df for training) after #2636, where training on Windows 10 broke off at "eval model". Since then, on Windows the last batch of every epoch is never trained on and the last batch of every evaluation is never scored; a dataset that fits in one batch does nothing at all.

With PaddlePaddle 3.3.1 on Windows 11 the loops run to the end without the limit. A toy set of 40 word images, batch size 4 (10 iterations), configs/rec/rec_icdar15_train.yml, 1 epoch, through tools/train.py and tools/eval.py:

code device loader workers runs training steps logged evaluation batches scored exit code
main (dab3fe3) GPU 0 and 2 2 9 of 10 9 of 10 0
main (dab3fe3) CPU 0 and 2 2 9 of 10 9 of 10 0
this change GPU 0 and 2 6 10 of 10 10 of 10 0
this change CPU 0 and 2 6 10 of 10 10 of 10 0

Not tested

  • Any PaddlePaddle version other than 3.3.1, and Windows 10 (the setup of the 2021 report).
  • Detection and table configs (they share these loops), and the get_center path.
  • Linux and macOS behaviour does not change: max_iter was already len(dataloader) there.

If the limit is still needed for an older setup you support, I can restrict the change instead (for example keep it behind a version check) - tell me which you prefer.

Since April 2021 (PaddlePaddle#2654 and d5651df, for the crash reported in PaddlePaddle#2636) the loops in tools/program.py stop at len(dataloader) - 1 when platform.system() is Windows, so on Windows the last batch of every epoch is never trained on and the last batch of every evaluation is never scored. With PaddlePaddle 3.3.1 on Windows 11 the loops run to the end without the limit (GPU and CPU, 0 and 2 loader workers), so remove it and the now unused import.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

On Windows the last batch of every epoch and of every evaluation is skipped (len(dataloader) - 1 in tools/program.py, a workaround from 2021)

1 participant