You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Sanitizer integration in rust is still quite experimental and tends to produce false positives. I went through a bunch of logs and think the issues below could be real bugs. I've added a few points of interest to key points in the codebase after some initial disentangling of the error logs.
For reference those errors had nothing to do with the "unsafe" nature. It was purely in a circular reference where the RefStore owned an Arc of the StoreManager and the StoreManager owned an Arc of the RefStore.
As for the tsan errors, I'm having issues filtering out all the noise... By running: bazel run --config tsan //util:retry_test -- retry_success_after_2_runs
I am getting a bunch of errors that seem related to the std and nothing else.
Here's the base case command I'm using for this debug case: bazel run --config tsan //cas/store:filesystem_store_test -- filesystem_store_tests::file_continues_to_stream_on_content_replace_test 2>&1 | grep -z -P "\s(cas|config|util)/.*\.rs"
From what I can tell is happening is that the file is reading data from the filesystem here:
Internally tokio calls spawn_blocking, which often creates a new thread. The spawn_blocking call gets future with a poll() to when the data is done on the child thread. The spawn_blocking thread then yields back. The next action drops the future that owns the spawn_blocking. Then the new thread starts up and panics. This triggers an unroll which needs to read some global state, which is where the race is coming in.
Or something similar. I'm not sure this is an issue for us, because I'm about 75% sure all the race problems happen on the thread that is being destroyed, which it's panic data is not going to be used.
A data race happens when there are two memory accesses in a program where both:
target the same location
are performed concurrently by two threads
are not reads
are not synchronization operations
I think the issue is the way rust's Arc works and how tsan checks for race conditions. In our case we have 2 threads that are racing to delete the Arc. At this point: https://doc.rust-lang.org/src/alloc/sync.rs.html#1864
In any sense I'm 95% sure this has nothing to do with us or tokio. This looks like a case where rust's Arc is falling through the cracks because of some minor details of how tsan checks race conditions.
Lastly, I noticed we set RUST_TEST_THREADS=1. This pretty much means we are not testing the thread safety of our code. The only reason we are getting any errors is because tokio spawns threads when spawn_local is called and there's a few places in their code and ours where this happens, like the entire tokio::fs api.
The only way I can see to do this is to always compile the std ourselves. To be completely honest, I think I'd prefer this because I want to replace std::String and std::Vec with small-string/small-vec optimized versions, since many of our protos use small strings and vectors to pipe data around and we have to do a lot of heap allocations because of them even though they are only a few bytes.
As for overriding Vec and String, I'm not too convinced that overriding std is a good idea. It seems like we could just import this from some crate
Sadly we'd have to do a lot of hacking to overload a lot of places that do stuff with strings, especially tonic, which is where most of the strings are created in the first place.
Status check: ASan has been fixed and runs nightly (#187, #3029). TSan is in progress
in #2981 and #2982 but is blocked on rules_rust#2068 (no build-std). The filesystem_store_test item in the description refers to a test that no longer
exists. Should this issue track only the TSan CI lane now, with the description
updated to match?
Activity
allada commentedon Jul 16, 2023
I fixed all the asan bugs here and enabled it in CI: #187
allada commentedon Jul 16, 2023
For reference those errors had nothing to do with the "unsafe" nature. It was purely in a circular reference where the
RefStoreowned an Arc of theStoreManagerand theStoreManagerowned an Arc of theRefStore.allada commentedon Jul 16, 2023
As for the
tsanerrors, I'm having issues filtering out all the noise... By running:bazel run --config tsan //util:retry_test -- retry_success_after_2_runsI am getting a bunch of errors that seem related to the
stdand nothing else.aaronmondal commentedon Jul 16, 2023
For now the following seems to filter fairly well:
I'll check whether there is a way to do it without that extra file.
allada commentedon Jul 16, 2023
Aaah, I remember there was an environ somewhere that you had to set, I just couldn't remember what it is and my 2 min search didn't yield much.
allada commentedon Jul 16, 2023
Here's the base case command I'm using for this debug case:
bazel run --config tsan //cas/store:filesystem_store_test -- filesystem_store_tests::file_continues_to_stream_on_content_replace_test 2>&1 | grep -z -P "\s(cas|config|util)/.*\.rs"From what I can tell is happening is that the file is reading data from the filesystem here:
Internally tokio calls
spawn_blocking, which often creates a new thread. Thespawn_blockingcall gets future with apoll()to when the data is done on the child thread. Thespawn_blockingthread then yields back. The next action drops the future that owns thespawn_blocking. Then the new thread starts up and panics. This triggers an unroll which needs to read some global state, which is where the race is coming in.Or something similar. I'm not sure this is an issue for us, because I'm about 75% sure all the race problems happen on the thread that is being destroyed, which it's panic data is not going to be used.
allada commentedon Jul 16, 2023
I spent way more time than I probably should have on this, but here's what I've found...
https://blog.regehr.org/archives/490
I think the issue is the way rust's
Arcworks and how tsan checks for race conditions. In our case we have 2 threads that are racing to delete theArc. At this point:https://doc.rust-lang.org/src/alloc/sync.rs.html#1864
In any sense I'm 95% sure this has nothing to do with us or tokio. This looks like a case where rust's Arc is falling through the cracks because of some minor details of how tsan checks race conditions.
Lastly, I noticed we set
RUST_TEST_THREADS=1. This pretty much means we are not testing the thread safety of our code. The only reason we are getting any errors is because tokio spawns threads whenspawn_localis called and there's a few places in their code and ours where this happens, like the entiretokio::fsapi.allada commentedon Jul 16, 2023
Here's a minimum reproducable example to show the Arc issue:
https://github.com/allada/turbo-cache/blob/show-arc-tsan-issue/arc_test.rs
bazel run --config tsan //:arc_testallada commentedon Jul 16, 2023
I found it! It's
rules_rustcausing the issues and getting in the way...rules_rustalways bring in pre-built standard library by default:https://github.com/bazelbuild/rules_rust/blob/main/rust/private/repository_utils.bzl#L191
In order to use tsan we need to compile the
stdourselves. In cargo this is usually done usingcargo build -Zbuild-std:https://doc.rust-lang.org/beta/unstable-book/compiler-flags/sanitizer.html
The only way I can see to do this is to always compile the std ourselves. To be completely honest, I think I'd prefer this because I want to replace
std::Stringandstd::Vecwith small-string/small-vec optimized versions, since many of our protos use small strings and vectors to pipe data around and we have to do a lot of heap allocations because of them even though they are only a few bytes.aaronmondal commentedon Jul 16, 2023
Naive attempts at adding
build-stddidn't work. Since this is probably relevant to others as well I've opened the above linked issue in rules_rust.As for overriding
VecandString, I'm not too convinced that overridingstdis a good idea. It seems like we could just import this from some crate:allada commentedon Jul 16, 2023
Sadly we'd have to do a lot of hacking to overload a lot of places that do stuff with strings, especially
tonic, which is where most of the strings are created in the first place.peacess commentedon Oct 8, 2023
i have the same issues. so dont use the Hashmap in service than run long times
rajputdivyanshu81 commentedon Oct 31, 2026
Status check: ASan has been fixed and runs nightly (#187, #3029). TSan is in progress
in #2981 and #2982 but is blocked on rules_rust#2068 (no
build-std). Thefilesystem_store_testitem in the description refers to a test that no longerexists. Should this issue track only the TSan CI lane now, with the description
updated to match?