Repository navigation
Conversation
Swift wraps UnixFileLock._acquire once at import time. When either race occurs, the wrapper returns without acquiring the lock so filelock can retry with the caller's timeout. It propagates both errors if the parent directory is missing and propagates FileExistsError if the lock path is a symlink. A regression test simulates each error on the first open and checks that the lock can then be acquired and released. Fixes modelscope#8234
1 task done
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Multi-node training can stop before the first training step when workers share a dataset cache. A worker can raise FileExistsError while preparing the dataset. UnixFileLock checks whether the lock file exists before opening it. If another worker creates or removes the file between those operations, the open can raise FileExistsError or FileNotFoundError and interrupt the polling loop.
Swift wraps UnixFileLock._acquire once at import time. When either race occurs, the wrapper returns without acquiring the lock so filelock can retry with the caller's timeout. It propagates both errors if the parent directory is missing and propagates FileExistsError if the lock path is a symlink. A regression test simulates each error on the first open and checks that the lock can then be acquired and released.
Ran
make testlocally on macOS for the ms-swift change; it fails the same way on the base branch, so the failure predates this change (it fails the same way on upstream/main).pytest -q tests/utils/test_filelock_patch.pystopped during collection because the datasets package was missing.make testfailed because its test script was missing.repo_lintran twice with no recorded output or exit status.pre-commit run --all-fileshas no recorded output or exit status.A run with the ms-swift change reverted gave no clear result, so no test is shown to catch this bug.
The new test tests/utils/test_filelock_patch.py did not run here: the datasets module is not installed.
PR type
PR information
Fixes #8234
Write the detail information belongs to this PR.
Experiment results
Paste your experiment result here(if needed).
Checklist
Ran
make testlocally on macOS for the ms-swift change; it fails the same way on the base branch, so the failure predates this change (it fails the same way on upstream/main).AI was used for assistance.