Re: [Discuss-gnuradio] fast parallel filtering

[Andy Walls]

Hi Dirk:

In the IQ data file you provided I can seem to find any pulses of the
nominal 10 msec length, no matter how I filter and rotate the
spectrum.
There is a lot of EMI, which looks like the intermodulation products
of a continuously on guy who is drifting/chirping down in frequency.

So could you please confirm or clarify the following:

1. The format of the binary data file.  I am assuming 32 bit float I
and 32 bit float Q pairs, as that is what the GNURadio file sink would
normally create.
2. The sample rate.  I am assuming 1 Msps.
3. The center freq.  I am assuming 150.22 MHz.
4. The pulse duration. I am assuming 10 milliseconds.
5. At what frequency offset, from the center frequency, should the
pulse be at?  I'm assuming somewhere withing +/- 5 kHz of the center
spike, but there are at least two EMI spikes in that range.

Thanks.

-Andy

On Wed, Mar 15, 2017 at 5:23 AM, Dirk Gorissen <address@hidden> wrote:
> Hi Andy, Marcus,
>
> Thanks very much for taking the time to think about this.
>
> Just to answer your questions:
>
>>Dirk's problem was the low SNR in his recording, so he'd like to take
>>the shape of his pulse into consideration, but he doesn't know the
>>spectral position of the same – Dirk, can you confirm?
>
> Yes, Im dealing with a very weak signal against quite a bit of
> background noise so the more prior knowledge I can leverage the
> better.
>
>>Dirk, thinking about this, the relation of the spectral uncertainty to
>>the bandwidth of the pulse would be interesting  – can you refresh our
>>memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>how it looked like.
>
> Yes, 10ms long (this is pretty exact), occurring every 1.5 seconds, in
> the 150MHz range. My input stream is coming from an SDR.
> As an aside I actually have multiple transmitters each pulsing at
> slightly different frequencies (e.g., 150.10, 150.22, ...) but Im
> happy to treat them independently so you can ignore that and assume
> there is just one.
>
> You can see pictures of the pulse (taken a while back, for a specific
> tx frequency) here: https://goo.gl/photos/Y3Ea6kJo1ewrcDYM8
>
> Note though that this is taken with the transmitter very close to the
> antenna so the signal is unrealistically strong. So its just to
> illustrate.
>
> I also quickly captured a file with raw IQ samples (File sink) in case
> anybody is interested. It starts with the transmitter very close to
> antenna, moves progressively further until out of range and then back.
> Its only about a minute or two tops but at 1msps it ended
> up as 813 MB though.
>
> https://drive.google.com/drive/folders/0B5dKo9igl8W4dWpiMmlEYWx3b0k?usp=sharing
>
>>What you really care about is
>>that there's a *tone* for about 8ms < t < 12ms where there is none else
>>(to rule out detection of spurs and other interference), is that right?
>
> Yes. The "tone" happens every 1.5 seconds and I want to catch as many
> of those occurrences as possible as the receiver moves through space.
> So for every 1.5 seconds I need to make a decision: was there one, or
> not.
>
> Note that the receiver is moving. So perhaps initially I may be well
> out of range of the signal and get nothing. But as I move closer as
> some point I will start picking up those pulses. The earlier and more
> reliably I can pick them up the better.
>
> I actually did spend some time looking at PLLs & the gnu radio block
> at some point. However I readily admit to being a software person not
> a DSP/Radio person. I have the day job to deal with today but I will
> have a study of the PLL block given Andy's tips and report back.
>
> Cheers
> Dirk
>
>
>
>
>
> On 14 March 2017 at 16:37, Andy Walls <address@hidden> wrote:
>> On Tue, 2017-03-14 at 12:00 -0400, address@hidden
>> wrote:
>>> Date: Tue, 14 Mar 2017 15:49:44 +0100
>>> From: Marcus M?ller <address@hidden>
>>> To: address@hidden
>>> Subject: Re: [Discuss-gnuradio] fast parallel filtering
>>
>>> Hi Andy,
>>>
>>> Dirk's problem was the low SNR in his recording, so he'd like to take
>>> the shape of his pulse into consideration, but he doesn't know the
>>> spectral position of the same ? Dirk, can you confirm?
>>
>> Ah, OK.
>>
>>
>>> Dirk, thinking about this, the relation of the spectral uncertainty to
>>> the bandwidth of the pulse would be interesting  ? can you refresh our
>>> memory? I think the pulse was 10ms long (so, pretty long) but I forgot
>>> how it looked like.
>>>
>>> Having slept about this a couple of times: What you really care about
>>> is
>>> that there's a *tone* for about 8ms < t < 12ms where there is none
>>> else
>>> (to rule out detection of spurs and other interference), is that
>>> right?
>>>
>>> Maybe we've been focussing too much on filter-based detection. What if
>>> we just *use* that feature of the signal being periodic for a duration
>>> of t, and not else? A PLL would be able to "lock" on the tone (much
>>> like
>>> an FM Radio with a PLL will lock on the station's carrier), and if we
>>> observe the phase error being limited for t, we can derive there was a
>>> pulse.
>>>
>>> Andy, does that sound like a feasible approach?
>>
>> Yes.  With the added benefit of GNURadio's carrier tracking PLL actually
>> performing part of the correlation operation for you, if it works out.
>>
>> So use the carrier_tracking_pll block.  If it locks onto a good signal,
>> it will mix it down to DC.  Then all you need is an averaging filter,
>> like the single pole IIR filter block or the moving average filter
>> block, operating on the real part of that output.  That will act as a
>> lock detector, and, I think, also completes a correlation operation, if
>> you use a non-normalized moving average filter (since the carrier
>> tracking PLL block gives you a point by point complex multiply with the
>> conjugate of the complex carrier tone).
>>
>> You'll want to set the PLL allowed min and max frequency bounds
>> properly; be careful since the input arguments have tricky units.
>>
>> You'll also want the loop bandwidth set pretty wide, since you want to
>> lock-in rapidly.
>>
>> Also, if I'm thinking about this properly:
>> To get faster lock in, you may want your frequency range of interest to
>> be somewhere in between Fs/4 and Fs/2; and not near DC.  If you're
>> within the lock-in range of the PLL, you'll lock within 1 cycle.  If
>> you're in the pull-in range of the PLL, it can take many cycles to get
>> into lock.
>>
>> Regards,
>> Andy
>>
>>> Cheers,
>>>
>>> Marcus
>>>
>>>
>>> On 14.03.2017 14:18, Andy Walls wrote:
>>> > If there is no information on the signal, why not just do a straight
>>> > energy detector:
>>> >
>>> > source -> complex bandpass filter -> complex to mag -> burst/energy
>>> detector -> QT Time sink (triggering on start of burst tag)
>>> >
>>> > The complex bandpass filter just has to catch all the frequencies of
>>> > interest.  (A complex bandpass filter does not have a symmetrical
>>> image
>>> > around DC.)
>>> >
>>> > The burst/energy detector has to detect some minimum number of time
>>> > domain samples contiguously above some noise/signal threshold that
>>> you
>>> > set, and tag the pulse.  It also has to detect when the time domain
>>> > samples fall below the threshold.  The burst/energy detector can
>>> work
>>> > with a preset noise/signal threshold, or you could periodically
>>> > re-estimate that threshold.
>>> >
>>> > GNURadio does not have a stock block that does burst/energy
>>> detection,
>>> > so you'll have to find one on the 'Net or write one yourself.
>>> >
>>> > -Andy
>>> >
>>> >
>>> > Dirk Gorissen wrote:
>>> >> Hi Martin,
>>> >>
>>> >> The aim is to check for the existence of a 10ms pulse in the
>>> incoming
>>> >> radio signal (from an sdr). The thing is I only know the frequency
>>> of
>>> >> the pulse to within ~1khz.
>>> >>
>>> >> So a single filter in my case is: generate a synthetic pulse
>>> (complex
>>> >> sin wave) of the same length and with a frequency f. Then take the
>>> >> reverse of the complex conjugate of this pulse to give me the taps
>>> for
>>> >> a FIR filter.
>>> >>
>>> >> Repeat the above for each frequency I want to check. E.g., 10
>>> filters
>>> >> each 100Hz apart.
>>> >> Then I just take the magnitude of each filter output and push
>>> through
>>> >> a Max to get my final correlations.
>>> >>
>>> >> I can come up with an algorithm that tries to be clever and with
>>> some
>>> >> accounting tries to find what frequency matters most but I wouldnt
>>> be
>>> >> surprised if there is some theory you can use to do this more
>>> >> efficiently or even in a single shot. But this is where Im unsure.
>>> >>
>>> >> Cheers
>>> >> Dirk
>>> >>
>>> >>
>>> >> On 14 March 2017 at 01:09, Martin Braun <address@hidden> wrote:
>>> >>> How related are those filters? Is this a candidate for polyphase
>>> DSP?
>>> >>>
>>> >>> -- M
>>> >>>
>>> >>> On 03/11/2017 02:01 PM, Dirk Gorissen wrote:
>>> >>>>  Hi Marcus,
>>> >>>>
>>> >>>>  Sorry, I should have clarified. You may recall an earlier thread
>>> from
>>> >>>>  mine where Im looking to pick out a short pulse from background
>>> noise
>>> >>>>  but I dont know the exact frequency of the pulse. Thought I
>>> would
>>> >>>>  start a new thread with a clear, specific question.
>>> >>>>
>>> >>>>  There is an uncertainty of +/- 1khz. So I can define multiple
>>> filters,
>>> >>>>  correlating for different pulse frequencies across the 1khz
>>> range. I
>>> >>>>  can implement this with different filters running in parallel
>>> but I
>>> >>>>  was looking for a more flexible / efficient way.
>>> >>>>
>>> >>>>  If you think this is out of scope for this list no problem at
>>> all,
>>> >>>>  just let me know.
>>> >>>>
>>> >>>>  Cheers
>>> >>>>  Dirk
>>> >>>>
>>> >>>>> On 11 March 2017 at 20:02, Marcus M?ller <address@hidden> wrote:
>>> >>>>>> Hi Dirk,
>>> >>>>>>
>>> >>>>>> this is more of a general DSP question than a GNU Radio
>>> question:
>>> >>>>>>
>>> >>>>>> How do these filters relate to each other?
>>> >>>>>>
>>> >>>>>> My gut feeling is that this gets a lot easier as soon as you
>>> tell us why
>>> >>>>>> you're doing this, i.e. for what purpose :)
>>> >>>>>>
>>> >>>>>> Best regards,
>>> >>>>>> Marcus
>>> >>>>>>
>>> >>>>>> On 11.03.2017 19:28, Dirk Gorissen wrote:
>>> >>>>>>> Hello all,
>>> >>>>>>>
>>> >>>>>>> Given a stream of samples I would like to apply n slightly
>>> different
>>> >>>>>>> filters to it with n being able to be chosen at runtime. Then
>>> combine
>>> >>>>>>> the results back to a single stream.
>>> >>>>>>>
>>> >>>>>>> As a test I built a flowgraph with the following chains in
>>> parallel for n
>>> >>>>>>> = 6
>>> >>>>>>>
>>> >>>>>>>                 | ->  decimating fir filter 1 -> complex to
>>> mag ->    |
>>> >>>>>>> stream -> | ->  decimating fir filter 2 -> complex to mag ->
>>> |  -> Max
>>> >>>>>>> -> ...
>>> >>>>>>>                 |                                  ....
>>> >>>>>>>                         |
>>> >>>>>>>                 | ->  decimating fir filter n -> complex to
>>> mag ->    |
>>> >>>>>>>
>>> >>>>>>> So the same stream is sent to each chain (decimation is 1) and
>>> the
>>> >>>>>>> output of each chain is pushed through a big Max block with 6
>>> inputs.
>>> >>>>>>>
>>> >>>>>>> This works but not particularly elegant and a bit annoying to
>>> change
>>> >>>>>>> if I suddenly decide I want to change n. In particular it also
>>> does
>>> >>>>>>> not seem computationally efficient.
>>> >>>>>>>
>>> >>>>>>> What I would like is to replace the above by a single block
>>> that
>>> >>>>>>>
>>> >>>>>>> - replicates the input n times
>>> >>>>>>> - applies each filter on each replica
>>> >>>>>>> - combines the output again to a single stream
>>> >>>>>>> - have a tunable n parameter
>>> >>>>>>> - is fast
>>> >>>>>>>
>>> >>>>>>> I did this with an Embedded python block doing essentially
>>> this:
>>> >>>>>>>
>>> >>>>>>> for i in range(n):
>>> >>>>>>>      out[i] = scipy.signal.lfilter(taps[i], 1, input)
>>> >>>>>>>
>>> >>>>>>> This is using exactly the same taps as in the chain case. This
>>> works
>>> >>>>>>> but the output is different and worse than what I get with the
>>> >>>>>>> separate chains. As a test, instead of lfilter I tried:
>>> >>>>>>>
>>> >>>>>>>
>>> gnuradio.filter.fir_filter_ccc(1,taps[i]).work(input[0],output)
>>> >>>>>>>
>>> >>>>>>> Thinking perhaps that is a closer replica. But couldnt get it
>>> to work..
>>> >>>>>>>
>>> >>>>>>> I suspect there should be an easy / natural way of doing this
>>> in
>>> >>>>>>> gnuradio. Looked at the filter bank / channelliser blocks but
>>> failed
>>> >>>>>>> to get anywhere.
>>> >>>>>>>
>>> >>>>>>> So what is the best way forward to do this?
>>> >>>>>>>
>>> >>>>>>> Many thanks
>>> >>>>>>> Dirk
>>> >>>>>>>
>>> >>>>>>> _______________________________________________
>>> >>>>>>> Discuss-gnuradio mailing list
>>> >>>>>>> address@hidden
>>> >>>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>> >>>>>>
>>> >>>>>> _______________________________________________
>>> >>>>>> Discuss-gnuradio mailing list
>>> >>>>>> address@hidden
>>> >>>>>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>> >>>>>
>>> >>>>>
>>> >>>>> --
>>> >>>>> _________________________________________
>>> >>>>> Dr. Dirk Gorissen
>>> >>>>> Mob: +44-7763-806-809
>>> >>>>> Web: dirkgorissen.com
>>> >>>>> Skype: dirk.gorissen
>>> >>>>> Twitter  : twitter.com/dirkgor
>>> >>>>> LinkedIn: linkedin.com/in/dirkgorissen
>>> >>>>
>>> >>>>
>>> >>>
>>> >>> _______________________________________________
>>> >>> Discuss-gnuradio mailing list
>>> >>> address@hidden
>>> >>> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>> >>
>>> >>
>>> >> --
>>> >> _________________________________________
>>> >> Dr. Dirk Gorissen
>>> >> Mob: +44-7763-806-809
>>> >> Web: dirkgorissen.com
>>> >> Skype: dirk.gorissen
>>> >> Twitter  : twitter.com/dirkgor
>>> >> LinkedIn: linkedin.com/in/dirkgorissen
>>> >
>>> >
>>> > _______________________________________________
>>> > Discuss-gnuradio mailing list
>>> > address@hidden
>>> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>>
>>
>
>
>
> --
> _________________________________________
> Dr. Dirk Gorissen
> Mob: +44-7763-806-809
> Web: dirkgorissen.com
> Skype: dirk.gorissen
> Twitter  : twitter.com/dirkgor
> LinkedIn: linkedin.com/in/dirkgorissen