You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Just trying out SoapyRTLSDR and SoapyRemote on a Raspberry Pi here, few items to note:
-std-c++11 in CMakeLists.txt isn't valid for GCC 4.6.3 on current Raspbian, -std-c++0x seems to work everywhere
SoapySDRServer --bind doesn't seem to work alone; just outputs the following with no reason after the 'FAIL: ':
pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################
upid://raspberrypi?pid=5198&hid=8323329
Launching the server... tcp://[::]:55132
Server socket bind FAIL:
Running it with a specific bind parameter seems to work:
pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################
upid://raspberrypi?pid=5202&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server
I'm aware of the bind issue, probably its on platforms which are missing ipv6 support in the kernel. You can also use --bind="0.0.0.0" to get the default port. I think it needs to test and fallback to ipv4 automatically... seems more graceful that way.
Regarding the flag, I think it need a test and fallback as well. I dont want a newer gcc to bomb out either just in case they finally take -std-c++0x away.
Aye, it looks IPV6 related for sure; not a huge issue in that regard. Having a fallback for c++0x would be a good idea but I'm guessing it'll be around for awhile to support the many projects that haven't updated yet.
On another note might as well keep this issue going with the following:
pi@raspberrypi ~/SoapyRemote/build $ SoapySDRServer --bind="0.0.0.0:55132"
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################
upid://raspberrypi?pid=5241&hid=8323329
Launching the server... tcp://0.0.0.0:55132
Server bound to 0.0.0.0:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept(192.168.1.101:63177)
Segmentation fault
Seems to crash right away once I connect from desktop; I'll do some more debugging on the Pi to figure out what's going on.
Another quick typo, looks like you didn't mean to return here, compiles when removed:
SoapyRemote/common/SoapyRPCSocket.cpp:214:12: error: cannot initialize return object of type 'const char *' with an rvalue of type 'int'
return strerror_r(errno, buff, sizeof(buff));
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Unfortunately there are two versions of strerror_r, and one of them returns a char pointer, which I have to return, the other doesn't. The ifdefs were supposed to be the way to figure it out: http://linux.die.net/man/3/strerror_r
It looks like (_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE should become ((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__
Seems to be working properly with SoapySDRUtil; probes, finds, makes remotely; but when I actually try to start a stream:
cjmacbook:build ccliffe$ SoapySDRServer --bind
######################################################
## Soapy Server -- Use any Soapy SDR remotely
######################################################
upid://cjmacbook.local?pid=23325&hid=0
Launching the server... tcp://[::]:55132
Server bound to [::]:55132
Press Ctrl+C to stop the server
SoapyServerListener::accept([::ffff:127.0.0.1]:53537)
SoapyServerListener::accept([::ffff:127.0.0.1]:53538)
Found Rafael Micro R820T tuner
SoapyServerListener::accept([::ffff:127.0.0.1]:53539)
SoapyServerListener::close()
SoapyServerListener::close()
SoapyServerListener::accept([::ffff:127.0.0.1]:53540)
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Floating point exception: 8
This isn't specific to the Pi; that's output from my OSX console connecting to localhost.
On the CubicSDR side:
[DEBUG] Found RTL-SDR Device using device index parameter 'rtl' = 0
[DEBUG] RTL-SDR opening device 0
[DEBUG] RTL-SDR Using buffer length 16384
[DEBUG] RTL-SDR Using 6 buffers
[DEBUG] RTL-SDR direct sampling mode 0
[DEBUG] RTL-SDR I/Q swap: No
[DEBUG] RTL-SDR offset_tune mode: No
[DEBUG] RTL-SDR PPM: 0
[DEBUG] RTL-SDR Tuner type: Rafael Micro R820T
[DEBUG] Setting sample rate: 2400000
[DEBUG] Setting center freq: 106072873
[DEBUG] Setting RTL-SDR AGC: Automatic
[INFO] SoapyRemote::setupRxStream(remoteFormat=CF32, localFormat=CF32, scaleFactor=32768, mtu=1500, window=16384)
[INFO] Client side stream bound to 127.0.0.1:60019
[INFO] Client side status bound to 127.0.0.1:60019
[DEBUG] Generating RTL-SDR lookup tables
[INFO] Using format CF32.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:63781
[INFO] Server side stream connected to [::ffff:127.0.0.1]:60019
[INFO] Server side status connected to [::ffff:127.0.0.1]:63736
Found Rafael Micro R820T tuner
[R82XX] PLL not locked!
Program received signal EXC_ARITHMETIC, Arithmetic exception.
[Switching to process 23811 thread 0xe07]
0x0000000100030790 in SoapyStreamEndpoint::SoapyStreamEndpoint (this=0x1002012b0, streamSock=@0x100200214, statusSock=@0x100200218, isRecv=false, numChans=0, elemSize=8, mtu=1500, window=16384) at /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyStreamEndpoint.cpp:41
41 _buffSize(((_xferSize-HEADER_SIZE)/numChans)/elemSize),
More typos from when I added the CS8, works over here now...
Is there an OSX sysctl equivalent setting to get the buffer larger than 2M?
I'm curious about the role the flow control is playing in the stutters. We might need a different strategy when the buffer is small. This will disable waiting on flow control (ACK) packets.
diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..f26c320 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -252,6 +252,7 @@ void SoapyStreamEndpoint::releaseRecv(const size_t handle)
**********************************************************************/
bool SoapyStreamEndpoint::waitSend(const long timeoutUs)
{
+ return true;
//are we within the allowed number of sequences in flight?
while (not _receiveInitial or
uint32_t(_lastRecvSequence+_maxInFlightSeqs) < uint32_t(_lastSendSequence))
Just for some rough maths here, if there is a 2MB window, then roughly 1 million samples (at CS8) can be held in that window. At 1 Msps (sample rate), the remote endpoint has about 1 second before it will block on flow control. And it should see a flow control update every eight of a window. It seems like plenty of time and that 2MB should not be an issue for lower speeds.
CS16 is fixed; seems a little bit smoother with the patch but still stuttering -- though it improves slightly when moving from CF32 to CS16 and from CS16 to CS8, but stutter seems to be of similar consistency whether it's running at 250KHz or 2.56Mhz -- the stuttering increases with rate (i.e. 2.048Mhz seems to have twice as much stuttering as 1.024Mhz)
Window seems to have a larger margin of improvement now, default window vs. 2MB window is much better; but still stuttering quickly such that it sounds "grainy"
Doubt it's related; but I do get this warning several times from a fresh compile:
In file included from /Users/ccliffe/Documents/GitHub/SoapyRemote/common/SoapyURLUtils.cpp:4:
/Users/ccliffe/Documents/GitHub/SoapyRemote/build/common/SoapySocketDefs.hpp:71:13: warning: 'htonll' macro redefined [-Wmacro-redefined]
#define htonll(x) __builtin_bswap64(x)
^
/usr/include/sys/_endian.h:141:9: note: previous definition is here
#define htonll(x) __DARWIN_OSSwapInt64(x)
^
To recap, you are streaming from the Pi to the OSX around 2 Msps. Increasing the socket buffering helps, increasing the samples per packet helps. But you are still seeing some kind of overflow or loss through the link.
Some thoughts
The OSX side is probably keeping up just fine pulling samples out of the socket. So its probably an issue of the Pi keeping up on the send() side. I guess we are seeing overflow, but the RTL doesn't have an indicator when this happens. Some things worth investigating here
Is there a problem feeding the socket on the Pi at ~2 Msps?
Is it a intermittent issue vs an actual bottleneck?
If the Pi is keeping up, this could also be a overflow in the recv socket buffer, this would happen if the OS has overhead tracking in the socket buffer that the SoapyRemote didn't account for
or similarly, the Pi is keeping up, but its the one overflowing the socket buffer (SO_SNDBUF), I dont think this is the case because linux doubles this buffer just for overhead management, and ethernet stack should be able to drain this buffer faster than RTL @ 2 Msps can fill it
I don't have exactly the same hardware, but I think I will give RTL on a Novena to my linux desktop a shot and mess around with the window sizes and some of the ideas mentioned here to see what happens.
Diff for printing S when there is a packet drop or overflow in socket buffer -- packet loss in network layer
diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..5e3ab59 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -208,6 +208,10 @@ int SoapyStreamEndpoint::acquireRecv(size_t &handle, const void **buffs, int &fl
const int numElemsOrErr = int(ntohl(header->elems));
//TODO sequence out of order or skip failures
+ if (uint32_t(_lastRecvSequence+1) != uint32_t(ntohl(header->sequence)))
+ {
+ SoapySDR::log(SOAPY_SDR_SSI, "S");
+ }
//update flow control
_lastRecvSequence = ntohl(header->sequence);
Maybe we are overflowing on the OSX side and the socket buffer has unaccounted for overhead:
diff --git a/common/SoapyStreamEndpoint.cpp b/common/SoapyStreamEndpoint.cpp
index c3bfa78..1efbf46 100644
--- a/common/SoapyStreamEndpoint.cpp
+++ b/common/SoapyStreamEndpoint.cpp
@@ -100,6 +100,7 @@ SoapyStreamEndpoint::SoapyStreamEndpoint(
{
//calculate maximum in-flight sequences allowed
_maxInFlightSeqs = actualWindow/mtu;
+ _maxInFlightSeqs /= 2;
//calculate the flow control ACK conditions
_triggerAckWindow = _maxInFlightSeqs/_numBuffs;
Ok, I'll give these a go -- note that most of my testing since the arithmetic error fix has actually just been to localhost (running SoapySDRServer and CubicSDR on my macbook) since the Pi introduced it's own issues testing over Wifi.
The MTU on lo0 I've been able to push up to 65535; but no actual network device seems to let me go over 1500 on the OSX side. And pushing lo0 to 65535 doesn't seem to affect the 178 element limit connecting to localhost anyways.
First patch:
Printing 'S' only really reveals it showing an 'S' when I move the center frequency or do other things that actually interrupt the stream.
Second patch:
Overflow patch seems to have no effect.
But Removing all the patches and going back to master I have managed to get it reasonably stable using:
remote:mtu at 8192 seems to be the maximum; going beyond this it just connects successfully and then no data is sent at all. It looks like remote:window might be capable of going higher though.
Though it complains in the log about "No buffer space available"; cranking the mtu and the window to these levels seems to actually work and changes the endpoint init values in the log.
I had been stopping when I hit the error previously since I assumed they weren't being applied but it appears they are; and they work:
[INFO] SoapyRemote::setupRxStream(remoteFormat=CS8, localFormat=CF32, scaleFactor=128, mtu=8192, window=16384000)
[INFO] Client side stream bound to 127.0.0.1:56215
[INFO] Client side status bound to 127.0.0.1:56215
[INFO] Using format CS8.
[INFO] Server side stream bound to [::ffff:127.0.0.1]:59978
[INFO] Server side stream connected to [::ffff:127.0.0.1]:56215
[INFO] Server side status connected to [::ffff:127.0.0.1]:50924
�[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
[INFO] Configured sender endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
[INFO] Client side stream connected to 127.0.0.1:59978
�[1m�[31m[ERROR] StreamEndpoint resize socket buffer FAIL: No buffer space available�[0m
[INFO] Configured receiver endpoint: dgram=8144 bytes, 4060 elements @ 2 bytes, window=16000 KiB
calculated optimal element count of 40448
Perhaps OSX just doesn't let you tweak the socket buffer manually and is handling buffering behind the scenes in a dynamic way?
At these settings I can also now use SoapyRemote with localhost at up to 3.2Msps somewhat stable -- which is strange since it stutters at 3.2M using just the SoapyRTLSDR module directly and has never really worked properly on my MacBook at that rate..
I do still seem to notice the odd glitch but they're several seconds apart at minimum and sound more like just a skipped sample instead of a constant grain like before. CF32 format is almost usable as well but has more frequent stuttering.
Edit:
Tried again with the Raspberry Pi, with the above streamArgs I can almost use 250khz over the hard-wired network but any higher it gets choppy. remote:mtu seems to be Ok with 65535 and I've had remote:window up to 32MB with the Pi remotely, neither settings improves much upon the 8192/16384000.
I gave my FM demod demo a shot, and the SoapyRTLSDR is very choppy and causing audio underflows compared to the RTL support from SoapyOsmo. I'm going to take a closer look, but I think there is an issue in SoapyRTLSDR readStream and buffer length somewhere. Just wanted to report this first...
Yeah, the async version works 100% locally with SoapyRemote and any stream options I throw at it now; I'm going to update the Pi and see how it goes.
I've been using the sync api for quite some time without issue so I'm surprised it's the apparent cause; perhaps because we were requesting buffer sizes that weren't in multiples of 512 or whatever the requirement is during SoapyRemote streaming?
The sync api was always using bulk transfers of 16384*16. I did try smaller sizes like 16384 as well. Actually, I wasnt even using SoapyRemote when I mentioned this issue. So it really may be a sync api bug in rtl. I guess we can close this one out at least.
Activity
guruofquality commentedon Sep 26, 2015
I'm aware of the bind issue, probably its on platforms which are missing ipv6 support in the kernel. You can also use --bind="0.0.0.0" to get the default port. I think it needs to test and fallback to ipv4 automatically... seems more graceful that way.
Regarding the flag, I think it need a test and fallback as well. I dont want a newer gcc to bomb out either just in case they finally take -std-c++0x away.
cjcliffe commentedon Sep 26, 2015
Aye, it looks IPV6 related for sure; not a huge issue in that regard. Having a fallback for c++0x would be a good idea but I'm guessing it'll be around for awhile to support the many projects that haven't updated yet.
On another note might as well keep this issue going with the following:
Seems to crash right away once I connect from desktop; I'll do some more debugging on the Pi to figure out what's going on.
guruofquality commentedon Sep 26, 2015
Might be the same problem, not sure since he said the module wasn't all filled out: https://groups.google.com/forum/#!topic/pothos-users/w_ABDy-xOU0
But arm to/from amd64 should be OK in principal. So I guess a backtrace will be very telling.
guruofquality commentedon Sep 26, 2015
Hopefully the first issues have been fixed now in master, fixed flags, strerror, and ipv6 bind
guruofquality commentedon Sep 26, 2015
This may be the fix for the crash: 2fbb9ac
cjcliffe commentedon Sep 26, 2015
Excellent, was just about to do some more testing on the Pi
cjcliffe commentedon Sep 26, 2015
Just a typo found so far:
should be
guruofquality commentedon Sep 26, 2015
doh! and i copied that around quite a bit -- fixed now
cjcliffe commentedon Sep 26, 2015
Another quick typo, looks like you didn't mean to return here, compiles when removed:
(when compiled on OSX)
guruofquality commentedon Sep 26, 2015
Unfortunately there are two versions of strerror_r, and one of them returns a char pointer, which I have to return, the other doesn't. The ifdefs were supposed to be the way to figure it out: http://linux.die.net/man/3/strerror_r
It looks like
(_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCEshould become((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__Thats a mess :-)
cjcliffe commentedon Sep 26, 2015
Seems to be working properly with SoapySDRUtil; probes, finds, makes remotely; but when I actually try to start a stream:
This isn't specific to the Pi; that's output from my OSX console connecting to localhost.
On the CubicSDR side:
Connect seems fine, but as soon as I call:
Immediately throws a runtime error:
Will do some debugging to try and narrow it down.
cjcliffe commentedon Sep 26, 2015
Also
((_POSIX_C_SOURCE >= 200112L || _XOPEN_SOURCE >= 600) && ! _GNU_SOURCE) || __APPLE__does the trick on OSX for the strerror_r issue.cjcliffe commentedon Sep 26, 2015
gdb reveals:
looks like numChans is causing divide-by-zero?
guruofquality commentedon Sep 26, 2015
I think i see...
guruofquality commentedon Sep 26, 2015
Well, i pushed some fixes, I hope that does the trick
9 remaining items
guruofquality commentedon Sep 27, 2015
More typos from when I added the CS8, works over here now...
Is there an OSX sysctl equivalent setting to get the buffer larger than 2M?
I'm curious about the role the flow control is playing in the stutters. We might need a different strategy when the buffer is small. This will disable waiting on flow control (ACK) packets.
Just for some rough maths here, if there is a 2MB window, then roughly 1 million samples (at CS8) can be held in that window. At 1 Msps (sample rate), the remote endpoint has about 1 second before it will block on flow control. And it should see a flow control update every eight of a window. It seems like plenty of time and that 2MB should not be an issue for lower speeds.
cjcliffe commentedon Sep 27, 2015
CS16 is fixed; seems a little bit smoother with the patch but still stuttering -- though it improves slightly when moving from CF32 to CS16 and from CS16 to CS8, but stutter seems to be of similar consistency whether it's running at 250KHz or 2.56Mhz -- the stuttering increases with rate (i.e. 2.048Mhz seems to have twice as much stuttering as 1.024Mhz)
Window seems to have a larger margin of improvement now, default window vs. 2MB window is much better; but still stuttering quickly such that it sounds "grainy"
cjcliffe commentedon Sep 27, 2015
Doubt it's related; but I do get this warning several times from a fresh compile:
guruofquality commentedon Sep 27, 2015
Oh, thats nice. I didn't get htonll on my platform. I fixed up the ifdefs, should be ok now.
guruofquality commentedon Sep 27, 2015
Recap
To recap, you are streaming from the Pi to the OSX around 2 Msps. Increasing the socket buffering helps, increasing the samples per packet helps. But you are still seeing some kind of overflow or loss through the link.
Some thoughts
The OSX side is probably keeping up just fine pulling samples out of the socket. So its probably an issue of the Pi keeping up on the send() side. I guess we are seeing overflow, but the RTL doesn't have an indicator when this happens. Some things worth investigating here
Replicating the issue
I don't have exactly the same hardware, but I think I will give RTL on a Novena to my linux desktop a shot and mess around with the window sizes and some of the ideas mentioned here to see what happens.
Diff for printing S when there is a packet drop or overflow in socket buffer -- packet loss in network layer
Maybe we are overflowing on the OSX side and the socket buffer has unaccounted for overhead:
cjcliffe commentedon Sep 27, 2015
Ok, I'll give these a go -- note that most of my testing since the arithmetic error fix has actually just been to localhost (running SoapySDRServer and CubicSDR on my macbook) since the Pi introduced it's own issues testing over Wifi.
The MTU on lo0 I've been able to push up to 65535; but no actual network device seems to let me go over 1500 on the OSX side. And pushing lo0 to 65535 doesn't seem to affect the 178 element limit connecting to localhost anyways.
Will get back to you soon with results.
guruofquality commentedon Sep 27, 2015
I should add that for the mtu stuff, you need to set both "remote:mtu=4096" as well as ifconfig lo mtu 4096 -- for example.
cjcliffe commentedon Sep 28, 2015
Ok, so current status:
First patch:
Printing 'S' only really reveals it showing an 'S' when I move the center frequency or do other things that actually interrupt the stream.
Second patch:
Overflow patch seems to have no effect.
But Removing all the patches and going back to master I have managed to get it reasonably stable using:
remote:mtu at 8192 seems to be the maximum; going beyond this it just connects successfully and then no data is sent at all. It looks like remote:window might be capable of going higher though.
Though it complains in the log about "No buffer space available"; cranking the mtu and the window to these levels seems to actually work and changes the endpoint init values in the log.
I had been stopping when I hit the error previously since I assumed they weren't being applied but it appears they are; and they work:
Perhaps OSX just doesn't let you tweak the socket buffer manually and is handling buffering behind the scenes in a dynamic way?
At these settings I can also now use SoapyRemote with localhost at up to 3.2Msps somewhat stable -- which is strange since it stutters at 3.2M using just the SoapyRTLSDR module directly and has never really worked properly on my MacBook at that rate..
I do still seem to notice the odd glitch but they're several seconds apart at minimum and sound more like just a skipped sample instead of a constant grain like before. CF32 format is almost usable as well but has more frequent stuttering.
Edit:
Tried again with the Raspberry Pi, with the above streamArgs I can almost use 250khz over the hard-wired network but any higher it gets choppy. remote:mtu seems to be Ok with 65535 and I've had remote:window up to 32MB with the Pi remotely, neither settings improves much upon the 8192/16384000.
[-]Raspberry Pi issues[/-][+]OSX / Raspberry Pi issues[/+]guruofquality commentedon Sep 28, 2015
I gave my FM demod demo a shot, and the SoapyRTLSDR is very choppy and causing audio underflows compared to the RTL support from SoapyOsmo. I'm going to take a closer look, but I think there is an issue in SoapyRTLSDR readStream and buffer length somewhere. Just wanted to report this first...
guruofquality commentedon Sep 28, 2015
I switched to the RTL async API on this branch, and my issue goes away: https://github.com/pothosware/SoapyRTLSDR/tree/async_stream
Just curious if it also fixes any of the stuttering mentioned here.
cjcliffe commentedon Sep 28, 2015
Yeah, the async version works 100% locally with SoapyRemote and any stream options I throw at it now; I'm going to update the Pi and see how it goes.
I've been using the sync api for quite some time without issue so I'm surprised it's the apparent cause; perhaps because we were requesting buffer sizes that weren't in multiples of 512 or whatever the requirement is during SoapyRemote streaming?
cjcliffe commentedon Sep 28, 2015
OK, nice work; Pi is awesome now!
I can use it stable at about 1Mhz at default settings; but with the stream args of:
3.2Mhz now appears to be quite stable! And that's over Wifi :)
Edit: yeah this is much better than rtl_tcp 👍
Edit2: been running it stable for an 1/2 hour now on the Pi, don't recall hearing a single glitch so far
guruofquality commentedon Sep 29, 2015
The sync api was always using bulk transfers of 16384*16. I did try smaller sizes like 16384 as well. Actually, I wasnt even using SoapyRemote when I mentioned this issue. So it really may be a sync api bug in rtl. I guess we can close this one out at least.