Skip to content

fix: handle multi-node UnixFileLock race during dataset preparation - #10605

Open
mvanhorn wants to merge 1 commit into
modelscope:mainfrom
mvanhorn:fix/8234-multinode-dataset-filelock-race
Open

mvanhorn wants to merge 1 commit into
modelscope:mainfrom
mvanhorn:fix/8234-multinode-dataset-filelock-race

Conversation

@mvanhorn

Copy link
Copy Markdown

Multi-node training can stop before the first training step when workers share a dataset cache. A worker can raise FileExistsError while preparing the dataset. UnixFileLock checks whether the lock file exists before opening it. If another worker creates or removes the file between those operations, the open can raise FileExistsError or FileNotFoundError and interrupt the polling loop.

Swift wraps UnixFileLock._acquire once at import time. When either race occurs, the wrapper returns without acquiring the lock so filelock can retry with the caller's timeout. It propagates both errors if the parent directory is missing and propagates FileExistsError if the lock path is a symlink. A regression test simulates each error on the first open and checks that the lock can then be acquired and released.

Ran make test locally on macOS for the ms-swift change; it fails the same way on the base branch, so the failure predates this change (it fails the same way on upstream/main).
pytest -q tests/utils/test_filelock_patch.py stopped during collection because the datasets package was missing.
make test failed because its test script was missing.
repo_lint ran twice with no recorded output or exit status.
pre-commit run --all-files has no recorded output or exit status.

A run with the ms-swift change reverted gave no clear result, so no test is shown to catch this bug.
The new test tests/utils/test_filelock_patch.py did not run here: the datasets module is not installed.

PR type

  • Bug Fix
  • New Feature
  • Document Updates
  • More Models or Datasets Support

PR information

Fixes #8234

Write the detail information belongs to this PR.

Experiment results

Paste your experiment result here(if needed).

Checklist

  • I have linked the related issue above, if there is one.
  • I have reviewed and understand every changed line, and I can explain how the change works and why it is correct.
  • I have run the relevant tests or reproductions locally, and included the commands and results above (if applicable).
    Ran make test locally on macOS for the ms-swift change; it fails the same way on the base branch, so the failure predates this change (it fails the same way on upstream/main).
  • I am responsible for the content of this PR, regardless of how it was produced.

AI was used for assistance.

Swift wraps UnixFileLock._acquire once at import time. When either race
occurs, the wrapper returns without acquiring the lock so filelock can
retry with the caller's timeout. It propagates both errors if the parent
directory is missing and propagates FileExistsError if the lock path is
a symlink. A regression test simulates each error on the first open and
checks that the lock can then be acquired and released.

Fixes modelscope#8234

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Node crashes when preparing datasets on multi-node environment

1 participant