Skip to content

OSX / Raspberry Pi issues #9

Closed
Closed

Description

@cjcliffe

Just trying out SoapyRTLSDR and SoapyRemote on a Raspberry Pi here, few items to note:

  • -std-c++11 in CMakeLists.txt isn't valid for GCC 4.6.3 on current Raspbian, -std-c++0x seems to work everywhere
  • SoapySDRServer --bind doesn't seem to work alone; just outputs the following with no reason after the 'FAIL: ':
pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5198&hid=8323329
Launching the server... tcp://[::]:55132
Server socket bind FAIL: 

Running it with a specific bind parameter seems to work:

pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################

upid://raspberrypi?pid=5202&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server

Activity

  1. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    I'm aware of the bind issue, probably its on platforms which are missing ipv6 support in the kernel. You can also use --bind="0.0.0.0" to get the default port. I think it needs to test and fallback to ipv4 automatically... seems more graceful that way.

    Regarding the flag, I think it need a test and fallback as well. I dont want a newer gcc to bomb out either just in case they finally take -std-c++0x away.

  2. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Aye, it looks IPV6 related for sure; not a huge issue in that regard. Having a fallback for c++0x would be a good idea but I'm guessing it'll be around for awhile to support the many projects that haven't updated yet.

    On another note might as well keep this issue going with the following:

    pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
    ######################################################
    ## Soapy Server -- Use any Soapy SDR remotely
    ######################################################
    
    upid://raspberrypi?pid=5241&hid=8323329
    Launching the server... tcp://0.0.0.0:55132
    Server bound to 0.0.0.0:55132
    Press Ctrl+C to stop the server
    SoapyServerListener::accept(192.168.1.101:63177)
    Segmentation fault
    

    Seems to crash right away once I connect from desktop; I'll do some more debugging on the Pi to figure out what's going on.

  3. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    Might be the same problem, not sure since he said the module wasn't all filled out: https://groups.google.com/forum/#!topic/pothos-users/w_ABDy-xOU0

    But arm to/from amd64 should be OK in principal. So I guess a backtrace will be very telling.

  4. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    Hopefully the first issues have been fixed now in master, fixed flags, strerror, and ipv6 bind

  5. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    This may be the fix for the crash: 2fbb9ac

  6. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Excellent, was just about to do some more testing on the Pi

  7. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Just a typo found so far:

            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std-c++0x")
    

    should be

            set(CMAKE_CXX_FLAGS "${CMAKE_CXX_FLAGS} -std=c++0x")
    
  8. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    doh! and i copied that around quite a bit -- fixed now

  9. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Another quick typo, looks like you didn't mean to return here, compiles when removed:

    SoapyRemote/common/SoapyRPCSocket.cpp:214:12: error: cannot initialize return object of type 'const char *' with an rvalue of type 'int'
        return strerror_r(errno, buff, sizeof(buff));
               ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    

    (when compiled on OSX)

  10. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    Unfortunately there are two versions of strerror_r, and one of them returns a char pointer, which I have to return, the other doesn't. The ifdefs were supposed to be the way to figure it out: http://linux.die.net/man/3/strerror_r

    It looks like (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE should become ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__

    Thats a mess :-)

  11. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Seems to be working properly with SoapySDRUtil; probes, finds, makes remotely; but when I actually try to start a stream:

    cjmacbook:build ccliffe$ SoapySDRServer --bind
    ######################################################
    ## Soapy Server -- Use any Soapy SDR remotely
    ######################################################
    
    upid://cjmacbook.local?pid=23325&hid=0
    Launching the server... tcp://[::]:55132
    Server bound to [::]:55132
    Press Ctrl+C to stop the server
    SoapyServerListener::accept([::ffff:127.0.0.1]:53537)
    SoapyServerListener::accept([::ffff:127.0.0.1]:53538)
    Found Rafael Micro R820T tuner
    SoapyServerListener::accept([::ffff:127.0.0.1]:53539)
    SoapyServerListener::close()
    SoapyServerListener::close()
    SoapyServerListener::accept([::ffff:127.0.0.1]:53540)
    Found Rafael Micro R820T tuner
    [R82XX] PLL not locked!
    Floating point exception: 8
    

    This isn't specific to the Pi; that's output from my OSX console connecting to localhost.

    On the CubicSDR side:

    [DEBUG] Found RTL-SDR Device using device index parameter 'rtl' = 0
    [DEBUG] RTL-SDR opening device 0
    [DEBUG] RTL-SDR Using buffer length 16384
    [DEBUG] RTL-SDR Using 6 buffers
    [DEBUG] RTL-SDR direct sampling mode 0
    [DEBUG] RTL-SDR I/Q swap: No
    [DEBUG] RTL-SDR offset_tune mode: No
    [DEBUG] RTL-SDR PPM: 0
    [DEBUG] RTL-SDR Tuner type: Rafael Micro R820T
    [DEBUG] Setting sample rate: 2400000
    [DEBUG] Setting center freq: 106072873
    [DEBUG] Setting RTL-SDR AGC: Automatic
    [INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
    [INFO] Client side stream bound to 127.0.0.1:60019
    [INFO] Client side status bound to 127.0.0.1:60019
    [DEBUG] Generating RTL-SDR lookup tables
    [INFO] Using format CF32.
    [INFO] Server side stream bound to [::ffff:127.0.0.1]:63781
    [INFO] Server side stream connected to [::ffff:127.0.0.1]:60019
    [INFO] Server side status connected to [::ffff:127.0.0.1]:63736
    

    Connect seems fine, but as soon as I call:

        SoapySDR::Stream *stream = device->setupStream(SOAPY_SDR_RX,"CF32");
    

    Immediately throws a runtime error:

    �[1m�[31m[ERROR] SoapyLogAcceptor::handlerLoop() �[0m
    libc++abi.dylib: terminating with uncaught exception of type std::runtime_error: SoapyRPCUnpacker::recv(header) FAIL: Undefined error: 0
    

    Will do some debugging to try and narrow it down.

  12. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    Also ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__ does the trick on OSX for the strerror_r issue.

  13. cjcliffe commented on Sep 26, 2015

    @cjcliffe
    ContributorAuthor

    gdb reveals:

    Found Rafael Micro R820T tuner
    [R82XX] PLL not locked!
    
    Program received signal EXC_ARITHMETIC, Arithmetic exception.
    [Switching to process 23811 thread 0xe07]
    0x0000000100030790 in SoapyStreamEndpoint::SoapyStreamEndpoint (this=0x1002012b0, streamSock=@0x100200214, statusSock=@0x100200218, isRecv=false, numChans=0, elemSize=8, mtu=1500, window=16384) at /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyStreamEndpoint.cpp:41
    41      _buffSize(((_xferSize-HEADER_SIZE)/numChans)/elemSize),
    

    looks like numChans is causing divide-by-zero?

  14. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    I think i see...

    1. The error is forwarded over the rpc and then not caught, the log acceptor should at least catch here
    2. The channels vector is empty, which we support, but its also used to set numChan, so that needs to bumped to a minimum of 1
  15. guruofquality commented on Sep 26, 2015

    @guruofquality
    Contributor

    Well, i pushed some fixes, I hope that does the trick

  16. 9 remaining items

  17. guruofquality commented on Sep 27, 2015

    @guruofquality
    Contributor

    More typos from when I added the CS8, works over here now...

    Is there an OSX sysctl equivalent setting to get the buffer larger than 2M?

    I'm curious about the role the flow control is playing in the stutters. We might need a different strategy when the buffer is small. This will disable waiting on flow control (ACK) packets.

    diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
    index c3bfa78..f26c320 100644
    --- a/common/SoapyStreamEndpoint.cpp
    +++ b/common/SoapyStreamEndpoint.cpp
    @@ -252,6 +252,7 @@ void SoapyStreamEndpoint::releaseRecv(const size_t handle)
      **********************************************************************/
     bool SoapyStreamEndpoint::waitSend(const long timeoutUs)
     {
    +    return true;
         //are we within the allowed number of sequences in flight?
         while (not _receiveInitial or
             uint32_t(_lastRecvSequence+_maxInFlightSeqs) < uint32_t(_lastSendSequence))
    

    Just for some rough maths here, if there is a 2MB window, then roughly 1 million samples (at CS8) can be held in that window. At 1 Msps (sample rate), the remote endpoint has about 1 second before it will block on flow control. And it should see a flow control update every eight of a window. It seems like plenty of time and that 2MB should not be an issue for lower speeds.

  18. cjcliffe commented on Sep 27, 2015

    @cjcliffe
    ContributorAuthor

    CS16 is fixed; seems a little bit smoother with the patch but still stuttering -- though it improves slightly when moving from CF32 to CS16 and from CS16 to CS8, but stutter seems to be of similar consistency whether it's running at 250KHz or 2.56Mhz -- the stuttering increases with rate (i.e. 2.048Mhz seems to have twice as much stuttering as 1.024Mhz)

    Window seems to have a larger margin of improvement now, default window vs. 2MB window is much better; but still stuttering quickly such that it sounds "grainy"

  19. cjcliffe commented on Sep 27, 2015

    @cjcliffe
    ContributorAuthor

    Doubt it's related; but I do get this warning several times from a fresh compile:

    In file included from /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyURLUtils.cpp:4:
    /Users/ccliffe/Documents/GitHub/SoapyRemote/build/common/SoapySocketDefs.hpp:71:13: warning: 'htonll' macro redefined [-Wmacro-redefined]
        #define htonll(x) __builtin_bswap64(x)
                ^
    /usr/include/sys/_endian.h:141:9: note: previous definition is here
    #define htonll(x)       __DARWIN_OSSwapInt64(x)
            ^
    
  20. guruofquality commented on Sep 27, 2015

    @guruofquality
    Contributor

    Oh, thats nice. I didn't get htonll on my platform. I fixed up the ifdefs, should be ok now.

  21. guruofquality commented on Sep 27, 2015

    @guruofquality
    Contributor

    Recap

    To recap, you are streaming from the Pi to the OSX around 2 Msps. Increasing the socket buffering helps, increasing the samples per packet helps. But you are still seeing some kind of overflow or loss through the link.

    Some thoughts

    The OSX side is probably keeping up just fine pulling samples out of the socket. So its probably an issue of the Pi keeping up on the send() side. I guess we are seeing overflow, but the RTL doesn't have an indicator when this happens. Some things worth investigating here

    • Is there a problem feeding the socket on the Pi at ~2 Msps?
      • Is it a intermittent issue vs an actual bottleneck?
    • Also consider packet loss itself in the UDP layer
    • If the Pi is keeping up, this could also be a overflow in the recv socket buffer, this would happen if the OS has overhead tracking in the socket buffer that the SoapyRemote didn't account for
      • or similarly, the Pi is keeping up, but its the one overflowing the socket buffer (SO_SNDBUF), I dont think this is the case because linux doubles this buffer just for overhead management, and ethernet stack should be able to drain this buffer faster than RTL @ 2 Msps can fill it
      • It may also be worthwhile to try a larger MTU over Ethernet, if the Pi and OSX support it: https://github.com/pothosware/SoapyRemote/wiki#remote-mtu

    Replicating the issue

    I don't have exactly the same hardware, but I think I will give RTL on a Novena to my linux desktop a shot and mess around with the window sizes and some of the ideas mentioned here to see what happens.

    Diff for printing S when there is a packet drop or overflow in socket buffer -- packet loss in network layer

    diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
    index c3bfa78..5e3ab59 100644
    --- a/common/SoapyStreamEndpoint.cpp
    +++ b/common/SoapyStreamEndpoint.cpp
    @@ -208,6 +208,10 @@ int SoapyStreamEndpoint::acquireRecv(size_t &handle, const void **buffs, int &fl
         const int numElemsOrErr = int(ntohl(header->elems));
    
         //TODO sequence out of order or skip failures
    +    if (uint32_t(_lastRecvSequence+1) != uint32_t(ntohl(header->sequence)))
    +    {
    +        SoapySDR::log(SOAPY_SDR_SSI, "S");
    +    }
    
         //update flow control
         _lastRecvSequence = ntohl(header->sequence);
    

    Maybe we are overflowing on the OSX side and the socket buffer has unaccounted for overhead:

    diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
    index c3bfa78..1efbf46 100644
    --- a/common/SoapyStreamEndpoint.cpp
    +++ b/common/SoapyStreamEndpoint.cpp
    @@ -100,6 +100,7 @@ SoapyStreamEndpoint::SoapyStreamEndpoint(
         {
             //calculate maximum in-flight sequences allowed
             _maxInFlightSeqs = actualWindow/mtu;
    +        _maxInFlightSeqs /= 2;
    
             //calculate the flow control ACK conditions
             _triggerAckWindow = _maxInFlightSeqs/_numBuffs;
    
    
  22. cjcliffe commented on Sep 27, 2015

    @cjcliffe
    ContributorAuthor

    Ok, I'll give these a go -- note that most of my testing since the arithmetic error fix has actually just been to localhost (running SoapySDRServer and CubicSDR on my macbook) since the Pi introduced it's own issues testing over Wifi.

    The MTU on lo0 I've been able to push up to 65535; but no actual network device seems to let me go over 1500 on the OSX side. And pushing lo0 to 65535 doesn't seem to affect the 178 element limit connecting to localhost anyways.

    Will get back to you soon with results.

  23. guruofquality commented on Sep 27, 2015

    @guruofquality
    Contributor

    I should add that for the mtu stuff, you need to set both "remote:mtu=4096" as well as ifconfig lo mtu 4096 -- for example.

  24. cjcliffe commented on Sep 28, 2015

    @cjcliffe
    ContributorAuthor

    Ok, so current status:

    First patch:
    Printing 'S' only really reveals it showing an 'S' when I move the center frequency or do other things that actually interrupt the stream.

    Second patch:
    Overflow patch seems to have no effect.

    But Removing all the patches and going back to master I have managed to get it reasonably stable using:

       streamArgs["remote:mtu"] = "8192";
       streamArgs["remote:format"] = "CS8";
       streamArgs["remote:window"] = "16384000";
    

    remote:mtu at 8192 seems to be the maximum; going beyond this it just connects successfully and then no data is sent at all. It looks like remote:window might be capable of going higher though.

    Though it complains in the log about "No buffer space available"; cranking the mtu and the window to these levels seems to actually work and changes the endpoint init values in the log.

    I had been stopping when I hit the error previously since I assumed they weren't being applied but it appears they are; and they work:

    [INFO] SoapyRemote::setupRxStream(remoteFormat=CS8, localFormat=CF32, scaleFactor=128, mtu=8192, window=16384000)
    [INFO] Client side stream bound to 127.0.0.1:56215
    [INFO] Client side status bound to 127.0.0.1:56215
    [INFO] Using format CS8.
    [INFO] Server side stream bound to [::ffff:127.0.0.1]:59978
    [INFO] Server side stream connected to [::ffff:127.0.0.1]:56215
    [INFO] Server side status connected to [::ffff:127.0.0.1]:50924
    �[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
    [INFO] Configured sender endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
    [INFO] Client side stream connected to 127.0.0.1:59978
    �[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
    [INFO] Configured receiver endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
    calculated optimal element count of 40448
    

    Perhaps OSX just doesn't let you tweak the socket buffer manually and is handling buffering behind the scenes in a dynamic way?

    At these settings I can also now use SoapyRemote with localhost at up to 3.2Msps somewhat stable -- which is strange since it stutters at 3.2M using just the SoapyRTLSDR module directly and has never really worked properly on my MacBook at that rate..

    I do still seem to notice the odd glitch but they're several seconds apart at minimum and sound more like just a skipped sample instead of a constant grain like before. CF32 format is almost usable as well but has more frequent stuttering.

    Edit:
    Tried again with the Raspberry Pi, with the above streamArgs I can almost use 250khz over the hard-wired network but any higher it gets choppy. remote:mtu seems to be Ok with 65535 and I've had remote:window up to 32MB with the Pi remotely, neither settings improves much upon the 8192/16384000.

  25. changed the title [-]Raspberry Pi issues[/-] [+]OSX / Raspberry Pi issues[/+] on Sep 28, 2015
  26. guruofquality commented on Sep 28, 2015

    @guruofquality
    Contributor

    I gave my FM demod demo a shot, and the SoapyRTLSDR is very choppy and causing audio underflows compared to the RTL support from SoapyOsmo. I'm going to take a closer look, but I think there is an issue in SoapyRTLSDR readStream and buffer length somewhere. Just wanted to report this first...

  27. guruofquality commented on Sep 28, 2015

    @guruofquality
    Contributor

    I switched to the RTL async API on this branch, and my issue goes away: https://github.com/pothosware/SoapyRTLSDR/tree/async_stream

    Just curious if it also fixes any of the stuttering mentioned here.

  28. cjcliffe commented on Sep 28, 2015

    @cjcliffe
    ContributorAuthor

    Yeah, the async version works 100% locally with SoapyRemote and any stream options I throw at it now; I'm going to update the Pi and see how it goes.

    I've been using the sync api for quite some time without issue so I'm surprised it's the apparent cause; perhaps because we were requesting buffer sizes that weren't in multiples of 512 or whatever the requirement is during SoapyRemote streaming?

  29. cjcliffe commented on Sep 28, 2015

    @cjcliffe
    ContributorAuthor

    OK, nice work; Pi is awesome now!

    I can use it stable at about 1Mhz at default settings; but with the stream args of:

       streamArgs["remote:mtu"] = "8192";
       streamArgs["remote:format"] = "CS8";
       streamArgs["remote:window"] = "16384000";
    

    3.2Mhz now appears to be quite stable! And that's over Wifi :)

    Edit: yeah this is much better than rtl_tcp 👍
    Edit2: been running it stable for an 1/2 hour now on the Pi, don't recall hearing a single glitch so far

  30. guruofquality commented on Sep 29, 2015

    @guruofquality
    Contributor

    The sync api was always using bulk transfers of 16384*16. I did try smaller sizes like 16384 as well. Actually, I wasnt even using SoapyRemote when I mentioned this issue. So it really may be a sync api bug in rtl. I guess we can close this one out at least.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions