Repository navigation
[C++] Thread deadlock in ObjectOutputStream #41862
Description
Describe the bug, including details regarding any error messages, version, and platform.
I am seeing a deadlock when destructing an ObjectOutputStream. I have attached the stack trace.
I did some debugging and found that the issue seems to be that the mutex in question is already held by this thread (I checked the __owner field in the pthread_mutex_t which points to the hanging thread)
Unfortunately the stack trace doesn’t show exactly which mutex it is trying to lock. I wonder if someone more familiar with the IO code has some ideas what might be the issue and where to dig deeper?
arrow_object_output_stream_stacktrace.txt
Edit:
I am using Arrow 15.0.0
Component(s)
C++
Activity
pitrou commentedon May 29, 2024
Copying the temptative diagnosis I initially posted on the dev ML:
It seems the problem lies here:
arrow/cpp/src/arrow/filesystem/s3fs.cc
Line 1858 in 9f58990
The Future is marked finished with the ObjectOutputStream's mutex taken,
and the Future's callback then triggers a chain of event which leads to
calling the ObjectOutputStream destructor, which in turn tries to take
the lock.
pitrou commentedon May 29, 2024
@icexelloss If I submit a PR trying to fix this, will it be possible for you to validate it with your use case?
icexelloss commentedon May 29, 2024
Yep - I can cherry pick the patch internally and test.
icexelloss commentedon May 29, 2024
@pitrou I am curious how did you find the
HandleUploadOutcomemethod? I couldn't find it in the stacktrace.pitrou commentedon May 29, 2024
It's frame 68 in the stacktrace :-)
apacheGH-41862: [C++][Dataset] Try out possible fix number 1
pitrou commentedon May 29, 2024
Ok, I have not one but two potential fixes, it would be nice if you could try each of them independently: #41875 , #41876
Note that both seem desirable in any case.
icexelloss commentedon May 29, 2024
Thanks @pitrou! I will test both out in the next couple of days!
[-]Thread deadlock in ObjectOutputStream[/-][+][C++] Thread deadlock in ObjectOutputStream[/+]icexelloss commentedon May 30, 2024
FYI I am testing #41875 Today, I will let it run through out the day because the it doesn't always happen quickly.
icexelloss commentedon May 31, 2024
#41875 doesn't seem to work and with similar stacktrace :(
I am rerunning it to double check
6 remaining items
icexelloss commentedon Jun 3, 2024
This is on Arrow 15. The line numbers are slightly off but this is the line that stack 21 is referring to:
https://github.com/apache/arrow/blob/apache-arrow-15.0.0/cpp/src/arrow/dataset/dataset_writer.cc#L346
(or on master)
https://github.com/apache/arrow/blob/main/cpp/src/arrow/dataset/dataset_writer.cc#L346
pitrou commentedon Jun 3, 2024
Ok, thanks. That explains why the "fix" doesn't work then :-)
icexelloss commentedon Jun 3, 2024
Sorry I am slightly confused. The "fix" doesn't work because I am using Arrow 15 and the fix only works on newer Arrow version?
pitrou commentedon Jun 3, 2024
No, I mean the traceback better explains why the fix doesn't work. It probably wouldn't work on latest Arrow either (though you can still try if that's easy for you).
icexelloss commentedon Jun 3, 2024
I see. Trying #41876 then.
icexelloss commentedon Jun 6, 2024
@pitrou I ran tests with #41876 for two days and didn't see a deadlock. So I think #41876 fixes the issue that I was seeing.
pitrou commentedon Jun 6, 2024
Cool, thanks!
pitrou commentedon Jun 10, 2024
Issue resolved by pull request 41876
#41876
icexelloss commentedon Jun 10, 2024
Thank you! @pitrou