Skip to content

ConcurrentModificationException when leaving multiple channels #19

Closed

Description

@papylhomme

Leaving multiple channels at the same time seems to disagree with the current implementation !

The exception is always thrown from Socket.onMessage, and using the debugger I can observe calls made to remove(Channel) just "before" the exception.
I tried to secure every access to channels using synchronized() block, with no luck.

Eventually I ended up investigating threading issues and tried a lazy Java programmer solution: updating the type of channels to CopyOnWriteArrayList ; and it worked for me.

I can provide a PR with the related commit if needed

Activity

  1. eoinsha commented on Mar 25, 2016

    @eoinsha
    Owner

    Thanks for this.
    The implementation is not thread safe and requires the application to synchronize access to Sockets and Channels as required. Can you provide part of the ConcurrentModificationException stracktrace to show how this occurs. It may be that we need to provide protection against concurrent access for callbacks and I/O-triggered events.

  2. papylhomme commented on Mar 25, 2016

    @papylhomme
    ContributorAuthor

    Here is the stacktrace.

    FATAL EXCEPTION: OkHttp Dispatcher
    Process: org.papylhomme.habitat, PID: 21422
    java.util.ConcurrentModificationException
        at java.util.ArrayList$ArrayListIterator.next(ArrayList.java:573)
        at org.phoenixframework.channels.Socket$PhoenixWSListener.onMessage(Socket.java:87)
        at com.squareup.okhttp.internal.ws.RealWebSocket$1.onMessage(RealWebSocket.java:62)
        at com.squareup.okhttp.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:242)
        at com.squareup.okhttp.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:103)
        at com.squareup.okhttp.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
        at com.squareup.okhttp.ws.WebSocketCall.createWebSocket(WebSocketCall.java:153)
        at com.squareup.okhttp.ws.WebSocketCall.access$000(WebSocketCall.java:40)
        at com.squareup.okhttp.ws.WebSocketCall$1.onResponse(WebSocketCall.java:98)
        at com.squareup.okhttp.Call$AsyncCall.execute(Call.java:177)
        at com.squareup.okhttp.internal.NamedRunnable.run(NamedRunnable.java:33)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
        at java.lang.Thread.run(Thread.java:818)
    

    I investigated this a bit further, and it appears it's not a threading issue as I originally thought, but a problem with Java iterators not handling modifications very well.

    The proposed solution isn't appliable here as the removing is done using another iterator. Alternatively the list can be looped "old-style" with an integer, but I ran into other issues:

    • if initializing the size before the loop, the for loop ends up accessing indexes which doesn't exists anymore due to the removal
    int size = channels.size();
    for(int i = 0; i < size; i++) {
      //...
    }
    
    throws java.lang.IndexOutOfBoundsException: Invalid index 1, size is 1
    • if testing the size directly in the for loop, then no more exception, but a channel is skipped in the operation
    for(int i = 0; i < channels.size(); i++) {
      //say current index is 1, and we remove it
      //next item will be at index 2, but in the meantime the real next item has moved to index 1...
    }

    My former laziness might be of help here, as according to the doc of CopyOnWriteArrayList:

    When an iterator is created, it keeps a copy of the list's contents.

    It also mean that

    iterations may not reflect the latest state of the list.

    but after checking the code of Socket this should not lead to unwanted side effects.

  3. PierceZ commented on Apr 8, 2016

    @PierceZ

    I just ran into the same crash and this seemed to resolve it:

    if (socket != null) {
        socket.remove(channel1);
        socket.remove(channel2);
    }
    if (channel1 != null) {
        channel1.leave();
    }
    if (channel2 != null) {
        channel2.leave();
    }
    
  4. papylhomme commented on Apr 13, 2016

    @papylhomme
    ContributorAuthor

    This is indeed working. The workaround here is to remove the channels before leaving them, so the ack of leaving a channel will not trigger anything. I'm not especially fond of this, but it works :)

  5. bryanjos commented on Oct 1, 2016

    @bryanjos
    Collaborator

    Does #27 resolve this or is it a different issue?

  6. bryanjos commented on Nov 4, 2016

    @bryanjos
    Collaborator

    Going to close this since there hasn't been any activity on it

  7. vrcmarcos commented on Dec 13, 2016

    @vrcmarcos

    Folks,

    I'm facing this problem. If I connect than disconnect and repeat this action a lot of times, my app crashes and throws the following exceptions.

    12-13 17:50:48.781 19100-21889/tv.globosat.fightssdktestapp W/Socket: WebSocket connection error
                                                                          java.net.SocketTimeoutException
                                                                              at java.net.PlainSocketImpl.read(PlainSocketImpl.java:488)
                                                                              at java.net.PlainSocketImpl.access$000(PlainSocketImpl.java:37)
                                                                              at java.net.PlainSocketImpl$PlainSocketInputStream.read(PlainSocketImpl.java:237)
                                                                              at okio.Okio$2.read(Okio.java:138)
                                                                              at okio.AsyncTimeout$2.read(AsyncTimeout.java:238)
                                                                              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:325)
                                                                              at okio.RealBufferedSource.indexOf(RealBufferedSource.java:314)
                                                                              at okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.java:210)
                                                                              at okhttp3.internal.http.Http1xStream.readResponse(Http1xStream.java:186)
                                                                              at okhttp3.internal.http.Http1xStream.readResponseHeaders(Http1xStream.java:127)
                                                                              at okhttp3.internal.http.CallServerInterceptor.intercept(CallServerInterceptor.java:53)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                              at okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.java:45)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                              at okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.java:109)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                              at okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.java:93)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                              at okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.java:124)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:92)
                                                                              at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.java:67)
                                                                              at okhttp3.RealCall.getResponseWithInterceptorChain(RealCall.java:170)
                                                                              at okhttp3.RealCall.access$100(RealCall.java:33)
                                                                              at okhttp3.RealCall$AsyncCall.execute(RealCall.java:120)
                                                                              at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                                                                              at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                                                                              at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                                                                              at java.lang.Thread.run(Thread.java:818)
    12-13 17:50:48.801 19100-21889/tv.globosat.fightssdktestapp E/RoomChannel: java.net.SocketTimeoutException
    12-13 17:50:50.202 19100-23914/tv.globosat.fightssdktestapp E/AndroidRuntime: FATAL EXCEPTION: OkHttp Dispatcher
                                                                                  Process: tv.globosat.fightssdktestapp, PID: 19100
                                                                                  java.util.ConcurrentModificationException
                                                                                      at java.util.ArrayList$ArrayListIterator.next(ArrayList.java:573)
                                                                                      at org.phoenixframework.channels.Socket$PhoenixWSListener.onMessage(Socket.java:82)
                                                                                      at okhttp3.internal.ws.RealWebSocket$1.onMessage(RealWebSocket.java:62)
                                                                                      at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:242)
                                                                                      at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:108)
                                                                                      at okhttp3.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
                                                                                      at okhttp3.ws.WebSocketCall.createWebSocket(WebSocketCall.java:152)
                                                                                      at okhttp3.ws.WebSocketCall.access$000(WebSocketCall.java:41)
                                                                                      at okhttp3.ws.WebSocketCall$1.onResponse(WebSocketCall.java:97)
                                                                                      at okhttp3.RealCall$AsyncCall.execute(RealCall.java:126)
                                                                                      at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
                                                                                      at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1112)
                                                                                      at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:587)
                                                                                      at java.lang.Thread.run(Thread.java:818)

    I tried to call socket.removeAllChannels() before call socket.disconnect(), but did not have any success with that. Could you help me with this?

  8. vrcmarcos commented on Dec 13, 2016

    @vrcmarcos

    I could avoid this exception by calling socket.removeAllChannels() before I added the first channel to my fresh-new socket (I was receiving the same message twice 'cause, for an unknown reason, my channel still registered on socket after a reconnection from a server's restart).

    But now I'm thinking that the channels list at the Socket.java can be changed from ArrayList to CopyOnWriteArrayList, a thread-safe variant from ArrayList (Docs - https://docs.oracle.com/javase/7/docs/api/java/util/concurrent/CopyOnWriteArrayList.html). This could avoid this exception.

  9. jpage4500 commented on Feb 17, 2017

    @jpage4500
    Contributor

    I'm trying to get a fork of this project to build in Android Studio but in the meantime I noticed that every time you try to access channels it's wrapped in synchronized(channels) except this this case:

        private void triggerChannelError() {
            for (final Channel channel : channels) {
                channel.trigger(ChannelEvent.ERROR.getPhxEvent(), null);
            }
        }
    

    It just so happens that I'm also seeing a ConcurrentModificationException while connected to a socket+channel (just 1).. No messages are being sent (server or client) that I see at this time. But, I do see the onError callback called right before the app crashes with this Exception.

    Socket       WebSocket connection error
    Socket       java.io.EOFException
    Socket           at okio.RealBufferedSource.require(RealBufferedSource.java:59)
    Socket           at okio.RealBufferedSource.readByte(RealBufferedSource.java:72)
    Socket           at okhttp3.internal.ws.WebSocketReader.readHeader(WebSocketReader.java:115)
    Socket           at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:104)
    Socket           at okhttp3.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
    Socket           at okhttp3.ws.WebSocketCall.createWebSocket(WebSocketCall.java:152)
    Socket           at okhttp3.ws.WebSocketCall.access$000(WebSocketCall.java:41)
    Socket           at okhttp3.ws.WebSocketCall$1.onResponse(WebSocketCall.java:97)
    Socket           at okhttp3.RealCall$AsyncCall.execute(RealCall.java:126)
    Socket           at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    Socket           at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1133)
    Socket           at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:607)
    Socket           at java.lang.Thread.run(Thread.java:761)
    blr_WSServiceBase     onError: java.io.EOFException
    Socket       Failed to explicitly close following failure
    

    and then right after this:

    AndroidRuntime     FATAL EXCEPTION: OkHttp Dispatcher
    AndroidRuntime     Process: com.bleachr.saints, PID: 7722
    AndroidRuntime     java.util.ConcurrentModificationException
    AndroidRuntime         at java.util.ArrayList$Itr.next(ArrayList.java:831)
    AndroidRuntime         at org.phoenixframework.channels.Socket$PhoenixWSListener.onMessage(Socket.jav
    AndroidRuntime         at okhttp3.internal.ws.RealWebSocket$1.onMessage(RealWebSocket.java:62)
    AndroidRuntime         at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:2
    AndroidRuntime         at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:1
    AndroidRuntime         at okhttp3.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
    AndroidRuntime         at okhttp3.ws.WebSocketCall.createWebSocket(WebSocketCall.java:152)
    AndroidRuntime         at okhttp3.ws.WebSocketCall.access$000(WebSocketCall.java:41)
    AndroidRuntime         at okhttp3.ws.WebSocketCall$1.onResponse(WebSocketCall.java:97)
    AndroidRuntime         at okhttp3.RealCall$AsyncCall.execute(RealCall.java:126)
    AndroidRuntime         at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
    AndroidRuntime         at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1
    AndroidRuntime         at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:
    AndroidRuntime         at java.lang.Thread.run(Thread.java:761)
    
  10. voltazor commented on Mar 1, 2017

    @voltazor

    I have the same issue. When socket lost connection and trying to reconnect, it just crashes with an exception:

    FATAL EXCEPTION: OkHttp Dispatcher
    Process: , PID: 27997
    java.util.ConcurrentModificationException
          at java.util.ArrayList$ArrayListIterator.next(ArrayList.java:573)
          at org.phoenixframework.channels.Socket$PhoenixWSListener.onMessage(Socket.java:82)
          at okhttp3.internal.ws.RealWebSocket$1.onMessage(RealWebSocket.java:62)
          at okhttp3.internal.ws.WebSocketReader.readMessageFrame(WebSocketReader.java:242)
          at okhttp3.internal.ws.WebSocketReader.processNextFrame(WebSocketReader.java:108)
          at okhttp3.internal.ws.RealWebSocket.readMessage(RealWebSocket.java:97)
          at okhttp3.ws.WebSocketCall.createWebSocket(WebSocketCall.java:152)
          at okhttp3.ws.WebSocketCall.access$000(WebSocketCall.java:41)
          at okhttp3.ws.WebSocketCall$1.onResponse(WebSocketCall.java:97)
          at okhttp3.RealCall$AsyncCall.execute(RealCall.java:126)
          at okhttp3.internal.NamedRunnable.run(NamedRunnable.java:32)
          at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1113)
          at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:588)
          at java.lang.Thread.run(Thread.java:818)
    
  11. jpage4500 commented on Mar 1, 2017

    @jpage4500
    Contributor

    just FYI - I'm not seeing this issue anymore on my local version. It might be the synchronized() that I mentioned above but the bigger change I made was to use the official okhttp3 which has websockets built-in. The version used in this project wasn't considered stable as mentioned here: https://medium.com/square-corner-blog/web-sockets-now-shipping-in-okhttp-3-5-463a9eec82d1#.e6cbixd8k

    I'd be happy to try and create a pull request for these changes but the problem I had originally when I tried to fork was just getting the project to open and build in Android Studio. I don't remember exactly why anymore.. I'll take a look today and see if I can figure it out

  12. jpage4500 commented on Mar 1, 2017

    @jpage4500
    Contributor

    I created a new issue for this since this one's closed: #47

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions