GNU Radio, One Step at a Time: September 2019

Monday, September 30, 2019

Re: [Discuss-gnuradio] Determine index of max value in FFT

Hi Tom,

On 9/30/19 3:27 PM, tom sutherland via Discuss-gnuradio wrote:
> How do you determine the index of the maximum value of the output of a
> FFT? e.g. I have a FFT and then a "ComplexToMag^2" block and I want to
> know the max value and its corresponding index (i.e. for a 4096 point
> FFT the index occured at 63).

The argmax block will give you the index to the vector element with the
largest value. Set its number of inputs to 1, and the vector length as
appropriate. The 'max_vec' output gives the number you are looking for.

As there is only one vector, the 'max_inp' should be connected to a null
sink. (I would argue that the port in question should disappear when
there's only one input).

For the value itself, there's the 'max' block, set number of inputs and
length of output vector to 1.

Regards, Paul Boven.

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] Determine index of max value in FFT

Hi Tom,

My usual approach is to write an Embedded Python Block and use the numpy functions to find the max. You can add in any other operations that you want to do once you know the max value and index such as converting it to a frequency.

Regards,
Derek

On 30/09/2019 14:27, tom sutherland via Discuss-gnuradio wrote:

How do you determine the index of the maximum value of the output of a FFT? e.g. I have a FFT and then a "ComplexToMag^2" block and I want to know the max value and its corresponding index (i.e. for a 4096 point FFT the index occured at 63).

Thanks...Tom
_______________________________________________  Discuss-gnuradio mailing list  Discuss-gnuradio@gnu.org  https://lists.gnu.org/mailman/listinfo/discuss-gnuradio  

Re: [Discuss-gnuradio] Keeping an index into a file or data being read

Hi Tom,

Each block has counters for how many samples have passed through it.
https://www.gnuradio.org/doc/doxygen/classgr_1_1block.html#a2279d1eb421203bc5b0f100a6d5dc263

uint64_t gr::block::nitems_read(unsigned int which_input)

On 30/09/2019 14:32, tom sutherland via Discuss-gnuradio wrote:

How do I keep up with how many values in a file or streamed value (from RTL device) that have been read into my flow graph? Normally in C/C++, Matlab etc... I would just have a counter of some variety.
_______________________________________________  Discuss-gnuradio mailing list  Discuss-gnuradio@gnu.org  https://lists.gnu.org/mailman/listinfo/discuss-gnuradio  

Re: [Discuss-gnuradio] GNU-Radio Companion Python flowgraph - return signal that it is finished?

On Mon, Sep 30, 2019 at 8:02 AM Andrew Payne <wandrewp@gmail.com> wrote:

I always do a tb.stop() followed by a tb.wait(), because I assume that .stop() is not blocking but .wait() is blocking

Yes, this is correct.

If you call stop() then wait() you stop the flow graph and wait for the threads to actually shut down after they are commanded to. If you call wait() then stop() then you also wait for the flow graph to finish its work rather than stopping it early.

and this is the best practice way to prematurely stop a flowgraph. Or is the .wait() necessary?

You must call wait() if you want to call start() again, or if you want to wait for the threads to actually stop (which blocks may react to by releasing resources or other cleanup actions).

In my programming I make a habit of always calling stop() and wait() together, in whichever order suits the application.

Re: [Discuss-gnuradio] GNU-Radio Companion Python flowgraph - return signal that it is finished?

Thanks Kevin. I ended up calling the python method os.stat.st_size on
the IQ file that the flowgraph will transmit, converted that file size
to a time duration in seconds considering the sample size and rate,
start the flowgraph, then have a sleep function that checks for a flag
every 100 milliseconds to stop the flowgraph if necessary. I always
do a tb.stop() followed by a tb.wait(), because I assume that .stop()
is not blocking but .wait() is blocking and this is the best practice
way to prematurely stop a flowgraph. Or is the .wait() necessary?

Thanks.

On Sat, Sep 28, 2019 at 10:43 PM Kevin Reid <kpreid@switchb.org> wrote:
>
> On Sat, Sep 28, 2019 at 6:34 PM Andrew Payne <wandrewp@gmail.com> wrote:
>>
>> I have a short IQ file that I simply want to play once without repeating, and I want the calling Python code to "know" when it's completed after commanding the GR class instance to start via a call to tb.start(), if tb is the class instance variable.
>
>
> Call
> tb.start()
> tb.wait()
> tb.stop()
> in that order. The call to wait() will block until the flow graph finishes.
>
> If you need to do something else while waiting, you can create another thread from Python (threading.Thread) and either have it call tb.wait() or have it do the other work, whichever suits your application.

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

[Discuss-gnuradio] Keeping an index into a file or data being read

How do I keep up with how many values in a file or streamed value (from RTL device) that have been read into my flow graph? Normally in C/C++, Matlab etc... I would just have a counter of some variety.

[Discuss-gnuradio] Determine index of max value in FFT

How do you determine the index of the maximum value of the output of a FFT? e.g. I have a FFT and then a "ComplexToMag^2" block and I want to know the max value and its corresponding index (i.e. for a 4096 point FFT the index occured at 63).

Thanks...Tom

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Hi,

> I'm very certain that flatbuffers does handle this correctly, yes.
> Haven't tried it, though. But "serialization library that doesn't deal
> with cross-platform" sounds bad, doesn't it?

You would think so yes, but it's not like it's un-heard of, especially
if the primary goal was performance and games where "All our relevant
targets are little endian" is probably a valid trade-off.

But in any case, after digging in a little, they explicitly state that
the on-the-wire representation is little endian for performance of the
most common CPU today, but it works on big endian at the cost of the
accessor needing to do the byte swapping.

Cheers,

Sylvain

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Hi Sylvain,

I'm very certain that flatbuffers does handle this correctly, yes.
Haven't tried it, though. But "serialization library that doesn't deal
with cross-platform" sounds bad, doesn't it?

Best regards,
Marcus

On Mon, 2019-09-30 at 08:37 +0200, Sylvain Munaut wrote:
> Hi,
>
> > > Tbh, I'd just assume that in all these formats, being tight-packing by
> > > default, std::complex<float> can just be represented by the equivalent
> > > of struct {float re; float im;} complex;.
>
> I haven't delved into the code, but do you know if it handles properly
> architecture differences between sender and receiver ?
> Things like 32b long vs 64b long and little endian vs big endian ?
>
> Because the message end up send across network, this matters, there
> historically has been at least one instance of a bug because PMT was
> not resilient to that when exchanging data between a x86_64 PC and an
> E310 for instance.
>
> Cheers,
>
> Sylvain
>
> _______________________________________________
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Sunday, September 29, 2019

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Hi,

>> Tbh, I'd just assume that in all these formats, being tight-packing by
>> default, std::complex<float> can just be represented by the equivalent
>> of struct {float re; float im;} complex;.

I haven't delved into the code, but do you know if it handles properly
architecture differences between sender and receiver ?
Things like 32b long vs 64b long and little endian vs big endian ?

Because the message end up send across network, this matters, there
historically has been at least one instance of a bug because PMT was
not resilient to that when exchanging data between a x86_64 PC and an
E310 for instance.

Cheers,

Sylvain

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

[Discuss-gnuradio] UHD / -lboost_system / pkg-config / Debian patch

Hi all,

Ettus points at this mailing list as the official forum for raising UHD
related questions.

This mail is sent to seek input on a recent regression we are seeing when
the official Debian UHD package removed "-lboost_system" from uhd.pc 'Libs'
in the following patch:
https://sources.debian.org/patches/uhd/3.14.1.0-2/fix-pkg-config/

This results in (among probably other failures) the osmo-trx Debian package
failing to build [1], and it has actually been scheduled for removal from debian
due to this FTBFS (fail to build from source) [2].

The related Debian bug report for uhd is at
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=940135
where the Debian package maintainer A. Maitland Bottoms argues his case
for why the removal of "-lboost_system" is correct. I'm not sure if
that patch has ever been submitted upstream or has been discussed here?

I'm looking for some guidance from upstream UHD as to what is the correct
course of action.

From us as the application developers, it seems rather clear:
An application program, like osmo-trx, which doesn't use boost directly, shouldn't
have to manually add any boost related libraries to the linker command line.

We are using UHD, and we're using pkg-config to tell us what kind of
compiler flags and linker command line snippets we should use in order
to link against uhd. It's not our responsibility to know about
implementation details of UHD and what kind of libraries it may need
in terms of its upstream dependencies.

I'm not a pkg-config expert, but *if* Debian is right to remove -lboost_system from
Libs, then maybe it simply needs to be added to Requires? Both the 'Requires'
and the 'Requires.private' lines are empty in libuhd-dev:amd63 3.14.1.0-2

Thanks for any help in advance.

Regards,
Harald

[1] https://osmocom.org/issues/4182
[2] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=939974

--
- Harald Welte <laforge@osmocom.org> http://laforge.gnumonks.org/
============================================================================
"Privacy in residential applications is a desirable marketing option."
(ETSI EN 300 175-7 Ch. A6)

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

If you decide to go that route I'm happy to discuss possible re-use of libsigmf and/or lessons learned from the trenches offline (I happen to be using it a *lot*). It's not *really* specific to SigMF. It's really just about getting c++ struct types easily converted to/from python. External dependencies are flatbuffers and Modern Json (nlohmann/json on github). It doesn't care much about the binary serialization but does make working with json-namespaced objects and moving between the c++ struct types and json real nice. In a truly ideal world we actually wouldn't need libsigmf because flatbuffers would include this parsing + namespace handling out of the box and that may be possible if flatbuffers devs woulf find this functionality valuable.

Nathan

On Sun, Sep 29, 2019 at 10:04 AM Marcus Müller <mmueller@gnuradio.org> wrote:

Wow, thanks. Yeah, that clearly reinforces flatbuffers as tool of
choice.

Tbh, I'd just assume that in all these formats, being tight-packing by
default, std::complex<float> can just be represented by the equivalent
of struct {float re; float im;} complex;.

On Sun, 2019-09-29 at 09:32 -0500, West, Nathan wrote:
> While developing libsigmf (https://github.com/deepsig/libsigmf) I
> went through a similar experience. The objective was to have painless
> json file/string <-> real c++ type (this means the serialization
> thing had to generate a slimmed down type that is effectively a
> struct of vector, int, float, etc). I looked at cap'n proto,
> flatbuffers, protobuf, and some others that weren't as
> mature/supported. I actually started with cap'n proto and had to
> switch to flatbuffers.
>
> Other bonuses I encountered were the whole "zero-time serialization"
> (In my experience people love making fun of cap'n proto for this, but
> it's true for all of the popular architectures and then you pay for
> shuffling bytes on more exotic architectures) and the sort of random-
> access parsing. A big perk of cap'n proto was it actually came with
> json-parsing to flatbuffer type *and* had the ability to generate a
> sane struct definition.
>
> I eventually switched to flatbuffers. Here's a randomly-ordered list
> of why
>
> * cap'n proto's author is no doubt an incredible programmer. However,
> he made some really weird choices. Like wrote his own string class in
> libkj and made all of cap'n proto depend on it. And it's not all that
> friendly with c++ string. So you wind up having to convert every
> single string in to a kj::string just for serialization (and vice
> versa)
> * libkj also implements its own memory allocators, filesystem library
> (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm
> not capable of vetting those implementation but beg to be asked a big
> old "WHY?!"
> * cap'n proto is incredibly strict (in CS parlance I guess you can
> call it opinionated) on protocol description language. It enforces
> things such as FirstletterMustBeCapital
> * I also felt the syntax of moving around these objects was a bit
> awkward
>
> I think there may be some other less appalling things that I've
> struck from my memory as "writing your own string that's incompatible
> with the rest of c++" is a big enough grievance that I'm only just
> now starting to forgive gcc dual abi for (but at least even that is
> at the ABI level which is always hard to get right). Flatbuffers
> offered pretty much the same promises as cap'n proto but without the
> "reinvent the standard library first" mindset and let's you make your
> own choices about how to name your fields, etc. I will say cap'n
> proto had a nice-looking extension capability to their parsing which
> I haven't found a good alternative for.
>
> The downside is flatbuffers is released by google/android's
> benevolence and somewhat competes with protobufs which is clearly
> their darling. I can't tell how much anyone else cares about
> flatbuffers although the *major* advantage to flatbuffers that I
> couldn't find in protobuf (which made me choose flatbuffers) is the
> struct code generator so you get out pretty normal struct's from
> their code generator that look like something you would write if you
> did it by hand.
>
> The downside is I don't know what complex<T> support looks like in
> any of these things.
>
> Cheers,
> Nathan
>
> On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <mmueller@gnuradio.org>
> wrote:
> > Hi Nick,
> >
> >
> > On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> > > I think it's wise to avoid hitching our metaphorical horse to any
> > > software project we can't absolutely depend on; questionable
> > > dependency choices have bit us, as you mention, many times.
> >
> > Sad but true.
> >
> > > With that (somewhat) in mind, what about Flatbuffers as an
> > > alternative to Cap'n Proto? Seems much more actively maintained,
> > more
> > > widely used, and designed with the same goals in mind. No parsing
> > > step required, so performance is good. Memory use seems to be
> > > similarly efficient.
> >
> > See, that's the kind of input I was looking for!
> > So, what I don't like about flatbuffers was the lack of field IDs
> > in
> > the schema; relying on new fields only being added to the end of a
> > structure.
> >
> > However, thanks to your encouragement I've revisited that and:
> > Flatbuffers *does* do IDs, if you want (it's slightly awkward, but
> > all
> > formal structure definition languages are, aside from ASN.1 ;) ).
> >
> > Regarding efficiency: https://github.com/thekvs/cpp-serializers
> > seems
> > to agree with your assessment.
> >
> > What flatbuffers really got going for it compared to the Cap'n is
> > that
> > docs are structured instead of "cool feature-oriented". What Cap'n
> > Proto has is the more sympathetic code base, but maybe I shouldn't
> > be
> > applying too many human qualities to code bases. gnuradio-
> > runtime/lib/
> > gets jealous everytime I do that.
> >
> > Best regards,
> > Marcus
> >
> > >
> > > Now, I haven't actually used Flatbuffers before, so anyone who
> > has
> > > experience with them (or your other favorite serialization
> > library!),
> > > please chime in here.
> > >
> > > Nick
> > >
> > > On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <
> > mmueller@gnuradio.org>
> > > wrote:
> > > > To avoid the setting in of post-Conference downs:
> > > >
> > > > Dearest SDR Community with the Highest count of "Awesome" in
> > the
> > > > Universe,
> > > >
> > > > discussions at GRCon have made it abundantly clear that PMT
> > needs
> > > > to be
> > > > replaced¹.
> > > > Main issues with PMT are:
> > > >
> > > > * broken type system,
> > > > * low suitability as tool for internal packeted data,
> > > > * lack of interoperability with anything but the languages we
> > > > support
> > > > natively,
> > > > * platform-dependency of the serialization format,
> > > > * bugs.
> > > >
> > > > (to a far lesser degree: performance.)
> > > >
> > > > Having talked to multiple domain experts in networked systems
> > and
> > > > heterogenous cooperative platforms, I've come to the
> > conclusion,
> > > > furthermore, that in the interest of RPC'ing everything², the
> > > > structure
> > > > of messages exchanged between blocks should be predefined.
> > > >
> > > > In this sense, PMT's (limited) ability do describe itself is of
> > no
> > > > help
> > > > – instead, it introduces ambiguity.
> > > >
> > > > The dominant choice for binary serialization with known
> > schemata is
> > > > Proto Buffers; however, they require deserialization of a whole
> > > > message
> > > > if there's substructures before the field you want to access³.
> > > >
> > > > Cap'n Proto[1] (by one of the original authors of protobufs)
> > seems
> > > > to
> > > > be a suitable alternative, and comparing it to its competitors,
> > it
> > > > seems to have achieved a significant level of maturity and has
> > at
> > > > least
> > > > one large-scale adopter⁴.
> > > >
> > > > Among its advantages seems to be that operations on the message
> > > > happen
> > > > in-place (i.e. there's actually a compatible C++ struct
> > > > representation
> > > > of the memory layout), which rendered it my favourite library
> > for
> > > > this
> > > > job.
> > > >
> > > > Now, problem: I've never personally worked with Cap'n Proto, or
> > > > protobuf, or any modern, non-self-describing serialization
> > library;
> > > > I've got minimal experience with MessagePack, but as said,
> > that's
> > > > not
> > > > probably going to make it due to lack of schema.
> > > >
> > > > So, here I am, calling for your experience! Hurl ins(igh|ul)ts
> > at
> > > > me,
> > > > preferable here on this list.
> > > >
> > > > Best regards,
> > > > Marcus
> > > >
> > > >
> > > > [1] https://capnproto.org/index.html
> > > >
> > > > ¹This doesn't come as a surprise – it's just a good point in
> > time
> > > > to
> > > > point this out.
> > > > ²You've probably noticed that we're working towards a less
> > machine-
> > > > dependent, more portable, distributable computational model
> > > > ³This can become problematic for messages with large data blobs
> > in
> > > > between, or very complex protocol headers
> > > > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > > > example.
> > > > However, it's but at revision 0.7, which indeed implies a bit
> > of
> > > > headache – other tools that never reached 1.0 that we rely on
> > > > include
> > > > Thrift, and that's really not a club that you'd want to be in.
> > > >
> > > >
> > > > _______________________________________________
> > > > Discuss-gnuradio mailing list
> > > > Discuss-gnuradio@gnu.org
> > > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > Discuss-gnuradio@gnu.org
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
> _______________________________________________
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Wow, thanks. Yeah, that clearly reinforces flatbuffers as tool of
choice.

Tbh, I'd just assume that in all these formats, being tight-packing by
default, std::complex<float> can just be represented by the equivalent
of struct {float re; float im;} complex;.

On Sun, 2019-09-29 at 09:32 -0500, West, Nathan wrote:
> While developing libsigmf (https://github.com/deepsig/libsigmf) I
> went through a similar experience. The objective was to have painless
> json file/string <-> real c++ type (this means the serialization
> thing had to generate a slimmed down type that is effectively a
> struct of vector, int, float, etc). I looked at cap'n proto,
> flatbuffers, protobuf, and some others that weren't as
> mature/supported. I actually started with cap'n proto and had to
> switch to flatbuffers.
>
> Other bonuses I encountered were the whole "zero-time serialization"
> (In my experience people love making fun of cap'n proto for this, but
> it's true for all of the popular architectures and then you pay for
> shuffling bytes on more exotic architectures) and the sort of random-
> access parsing. A big perk of cap'n proto was it actually came with
> json-parsing to flatbuffer type *and* had the ability to generate a
> sane struct definition.
>
> I eventually switched to flatbuffers. Here's a randomly-ordered list
> of why
>
> * cap'n proto's author is no doubt an incredible programmer. However,
> he made some really weird choices. Like wrote his own string class in
> libkj and made all of cap'n proto depend on it. And it's not all that
> friendly with c++ string. So you wind up having to convert every
> single string in to a kj::string just for serialization (and vice
> versa)
> * libkj also implements its own memory allocators, filesystem library
> (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm
> not capable of vetting those implementation but beg to be asked a big
> old "WHY?!"
> * cap'n proto is incredibly strict (in CS parlance I guess you can
> call it opinionated) on protocol description language. It enforces
> things such as FirstletterMustBeCapital
> * I also felt the syntax of moving around these objects was a bit
> awkward
>
> I think there may be some other less appalling things that I've
> struck from my memory as "writing your own string that's incompatible
> with the rest of c++" is a big enough grievance that I'm only just
> now starting to forgive gcc dual abi for (but at least even that is
> at the ABI level which is always hard to get right). Flatbuffers
> offered pretty much the same promises as cap'n proto but without the
> "reinvent the standard library first" mindset and let's you make your
> own choices about how to name your fields, etc. I will say cap'n
> proto had a nice-looking extension capability to their parsing which
> I haven't found a good alternative for.
>
> The downside is flatbuffers is released by google/android's
> benevolence and somewhat competes with protobufs which is clearly
> their darling. I can't tell how much anyone else cares about
> flatbuffers although the *major* advantage to flatbuffers that I
> couldn't find in protobuf (which made me choose flatbuffers) is the
> struct code generator so you get out pretty normal struct's from
> their code generator that look like something you would write if you
> did it by hand.
>
> The downside is I don't know what complex<T> support looks like in
> any of these things.
>
> Cheers,
> Nathan
>
> On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <mmueller@gnuradio.org>
> wrote:
> > Hi Nick,
> >
> >
> > On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> > > I think it's wise to avoid hitching our metaphorical horse to any
> > > software project we can't absolutely depend on; questionable
> > > dependency choices have bit us, as you mention, many times.
> >
> > Sad but true.
> >
> > > With that (somewhat) in mind, what about Flatbuffers as an
> > > alternative to Cap'n Proto? Seems much more actively maintained,
> > more
> > > widely used, and designed with the same goals in mind. No parsing
> > > step required, so performance is good. Memory use seems to be
> > > similarly efficient.
> >
> > See, that's the kind of input I was looking for!
> > So, what I don't like about flatbuffers was the lack of field IDs
> > in
> > the schema; relying on new fields only being added to the end of a
> > structure.
> >
> > However, thanks to your encouragement I've revisited that and:
> > Flatbuffers *does* do IDs, if you want (it's slightly awkward, but
> > all
> > formal structure definition languages are, aside from ASN.1 ;) ).
> >
> > Regarding efficiency: https://github.com/thekvs/cpp-serializers
> > seems
> > to agree with your assessment.
> >
> > What flatbuffers really got going for it compared to the Cap'n is
> > that
> > docs are structured instead of "cool feature-oriented". What Cap'n
> > Proto has is the more sympathetic code base, but maybe I shouldn't
> > be
> > applying too many human qualities to code bases. gnuradio-
> > runtime/lib/
> > gets jealous everytime I do that.
> >
> > Best regards,
> > Marcus
> >
> > >
> > > Now, I haven't actually used Flatbuffers before, so anyone who
> > has
> > > experience with them (or your other favorite serialization
> > library!),
> > > please chime in here.
> > >
> > > Nick
> > >
> > > On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <
> > mmueller@gnuradio.org>
> > > wrote:
> > > > To avoid the setting in of post-Conference downs:
> > > >
> > > > Dearest SDR Community with the Highest count of "Awesome" in
> > the
> > > > Universe,
> > > >
> > > > discussions at GRCon have made it abundantly clear that PMT
> > needs
> > > > to be
> > > > replaced¹.
> > > > Main issues with PMT are:
> > > >
> > > > * broken type system,
> > > > * low suitability as tool for internal packeted data,
> > > > * lack of interoperability with anything but the languages we
> > > > support
> > > > natively,
> > > > * platform-dependency of the serialization format,
> > > > * bugs.
> > > >
> > > > (to a far lesser degree: performance.)
> > > >
> > > > Having talked to multiple domain experts in networked systems
> > and
> > > > heterogenous cooperative platforms, I've come to the
> > conclusion,
> > > > furthermore, that in the interest of RPC'ing everything², the
> > > > structure
> > > > of messages exchanged between blocks should be predefined.
> > > >
> > > > In this sense, PMT's (limited) ability do describe itself is of
> > no
> > > > help
> > > > – instead, it introduces ambiguity.
> > > >
> > > > The dominant choice for binary serialization with known
> > schemata is
> > > > Proto Buffers; however, they require deserialization of a whole
> > > > message
> > > > if there's substructures before the field you want to access³.
> > > >
> > > > Cap'n Proto[1] (by one of the original authors of protobufs)
> > seems
> > > > to
> > > > be a suitable alternative, and comparing it to its competitors,
> > it
> > > > seems to have achieved a significant level of maturity and has
> > at
> > > > least
> > > > one large-scale adopter⁴.
> > > >
> > > > Among its advantages seems to be that operations on the message
> > > > happen
> > > > in-place (i.e. there's actually a compatible C++ struct
> > > > representation
> > > > of the memory layout), which rendered it my favourite library
> > for
> > > > this
> > > > job.
> > > >
> > > > Now, problem: I've never personally worked with Cap'n Proto, or
> > > > protobuf, or any modern, non-self-describing serialization
> > library;
> > > > I've got minimal experience with MessagePack, but as said,
> > that's
> > > > not
> > > > probably going to make it due to lack of schema.
> > > >
> > > > So, here I am, calling for your experience! Hurl ins(igh|ul)ts
> > at
> > > > me,
> > > > preferable here on this list.
> > > >
> > > > Best regards,
> > > > Marcus
> > > >
> > > >
> > > > [1] https://capnproto.org/index.html
> > > >
> > > > ¹This doesn't come as a surprise – it's just a good point in
> > time
> > > > to
> > > > point this out.
> > > > ²You've probably noticed that we're working towards a less
> > machine-
> > > > dependent, more portable, distributable computational model
> > > > ³This can become problematic for messages with large data blobs
> > in
> > > > between, or very complex protocol headers
> > > > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > > > example.
> > > > However, it's but at revision 0.7, which indeed implies a bit
> > of
> > > > headache – other tools that never reached 1.0 that we rely on
> > > > include
> > > > Thrift, and that's really not a club that you'd want to be in.
> > > >
> > > >
> > > > _______________________________________________
> > > > Discuss-gnuradio mailing list
> > > > Discuss-gnuradio@gnu.org
> > > > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > Discuss-gnuradio@gnu.org
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio
>
> _______________________________________________
> Discuss-gnuradio mailing list
> Discuss-gnuradio@gnu.org
> https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

While developing libsigmf (https://github.com/deepsig/libsigmf) I went through a similar experience. The objective was to have painless json file/string <-> real c++ type (this means the serialization thing had to generate a slimmed down type that is effectively a struct of vector, int, float, etc). I looked at cap'n proto, flatbuffers, protobuf, and some others that weren't as mature/supported. I actually started with cap'n proto and had to switch to flatbuffers.

Other bonuses I encountered were the whole "zero-time serialization" (In my experience people love making fun of cap'n proto for this, but it's true for all of the popular architectures and then you pay for shuffling bytes on more exotic architectures) and the sort of random-access parsing. A big perk of cap'n proto was it actually came with json-parsing to flatbuffer type *and* had the ability to generate a sane struct definition.

I eventually switched to flatbuffers. Here's a randomly-ordered list of why

* cap'n proto's author is no doubt an incredible programmer. However, he made some really weird choices. Like wrote his own string class in libkj and made all of cap'n proto depend on it. And it's not all that friendly with c++ string. So you wind up having to convert every single string in to a kj::string just for serialization (and vice versa)

* libkj also implements its own memory allocators, filesystem library (ala boost filesystem / c++ filesystem), mutexes, and threads. I'm not capable of vetting those implementation but beg to be asked a big old "WHY?!"

* cap'n proto is incredibly strict (in CS parlance I guess you can call it opinionated) on protocol description language. It enforces things such as FirstletterMustBeCapital

* I also felt the syntax of moving around these objects was a bit awkward

I think there may be some other less appalling things that I've struck from my memory as "writing your own string that's incompatible with the rest of c++" is a big enough grievance that I'm only just now starting to forgive gcc dual abi for (but at least even that is at the ABI level which is always hard to get right). Flatbuffers offered pretty much the same promises as cap'n proto but without the "reinvent the standard library first" mindset and let's you make your own choices about how to name your fields, etc. I will say cap'n proto had a nice-looking extension capability to their parsing which I haven't found a good alternative for.

The downside is flatbuffers is released by google/android's benevolence and somewhat competes with protobufs which is clearly their darling. I can't tell how much anyone else cares about flatbuffers although the *major* advantage to flatbuffers that I couldn't find in protobuf (which made me choose flatbuffers) is the struct code generator so you get out pretty normal struct's from their code generator that look like something you would write if you did it by hand.

The downside is I don't know what complex<T> support looks like in any of these things.

Cheers,

Nathan

On Sun, Sep 29, 2019 at 5:21 AM Marcus Müller <mmueller@gnuradio.org> wrote:

Hi Nick,

On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> I think it's wise to avoid hitching our metaphorical horse to any
> software project we can't absolutely depend on; questionable
> dependency choices have bit us, as you mention, many times.

Sad but true.

> With that (somewhat) in mind, what about Flatbuffers as an
> alternative to Cap'n Proto? Seems much more actively maintained, more
> widely used, and designed with the same goals in mind. No parsing
> step required, so performance is good. Memory use seems to be
> similarly efficient.

See, that's the kind of input I was looking for!
So, what I don't like about flatbuffers was the lack of field IDs in
the schema; relying on new fields only being added to the end of a
structure.

However, thanks to your encouragement I've revisited that and:
Flatbuffers *does* do IDs, if you want (it's slightly awkward, but all
formal structure definition languages are, aside from ASN.1 ;) ).

Regarding efficiency: https://github.com/thekvs/cpp-serializers seems
to agree with your assessment.

What flatbuffers really got going for it compared to the Cap'n is that
docs are structured instead of "cool feature-oriented". What Cap'n
Proto has is the more sympathetic code base, but maybe I shouldn't be
applying too many human qualities to code bases. gnuradio-runtime/lib/
gets jealous everytime I do that.

Best regards,
Marcus

>
> Now, I haven't actually used Flatbuffers before, so anyone who has
> experience with them (or your other favorite serialization library!),
> please chime in here.
>
> Nick
>
> On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <mmueller@gnuradio.org>
> wrote:
> > To avoid the setting in of post-Conference downs:
> >
> > Dearest SDR Community with the Highest count of "Awesome" in the
> > Universe,
> >
> > discussions at GRCon have made it abundantly clear that PMT needs
> > to be
> > replaced¹.
> > Main issues with PMT are:
> >
> > * broken type system,
> > * low suitability as tool for internal packeted data,
> > * lack of interoperability with anything but the languages we
> > support
> > natively,
> > * platform-dependency of the serialization format,
> > * bugs.
> >
> > (to a far lesser degree: performance.)
> >
> > Having talked to multiple domain experts in networked systems and
> > heterogenous cooperative platforms, I've come to the conclusion,
> > furthermore, that in the interest of RPC'ing everything², the
> > structure
> > of messages exchanged between blocks should be predefined.
> >
> > In this sense, PMT's (limited) ability do describe itself is of no
> > help
> > – instead, it introduces ambiguity.
> >
> > The dominant choice for binary serialization with known schemata is
> > Proto Buffers; however, they require deserialization of a whole
> > message
> > if there's substructures before the field you want to access³.
> >
> > Cap'n Proto[1] (by one of the original authors of protobufs) seems
> > to
> > be a suitable alternative, and comparing it to its competitors, it
> > seems to have achieved a significant level of maturity and has at
> > least
> > one large-scale adopter⁴.
> >
> > Among its advantages seems to be that operations on the message
> > happen
> > in-place (i.e. there's actually a compatible C++ struct
> > representation
> > of the memory layout), which rendered it my favourite library for
> > this
> > job.
> >
> > Now, problem: I've never personally worked with Cap'n Proto, or
> > protobuf, or any modern, non-self-describing serialization library;
> > I've got minimal experience with MessagePack, but as said, that's
> > not
> > probably going to make it due to lack of schema.
> >
> > So, here I am, calling for your experience! Hurl ins(igh|ul)ts at
> > me,
> > preferable here on this list.
> >
> > Best regards,
> > Marcus
> >
> >
> > [1] https://capnproto.org/index.html
> >
> > ¹This doesn't come as a surprise – it's just a good point in time
> > to
> > point this out.
> > ²You've probably noticed that we're working towards a less machine-
> > dependent, more portable, distributable computational model
> > ³This can become problematic for messages with large data blobs in
> > between, or very complex protocol headers
> > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > example.
> > However, it's but at revision 0.7, which indeed implies a bit of
> > headache – other tools that never reached 1.0 that we rely on
> > include
> > Thrift, and that's really not a club that you'd want to be in.
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > Discuss-gnuradio@gnu.org
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Hi Barry,

thanks for bringing this up! JSON was one of the considered, but
scrapped ideas; I'll explain why:

On Sat, 2019-09-28 at 20:13 -0400, Barry Duggan wrote:
> Just thinking outside the box here, but what about JSON? It is well
> defined,

JSON is not really overly well-defined when it comes to how to
transport raw samples, and many of those.

There's one sorry-but-it-kills-that-option argument against JSON, by
the way: If the origins of modern chaos theory have taught us anything,
you don't want serialization to change your floating point numbers¹
(and that's the dominant data format, so that's kind of important
here). JSON's text-based float format does exactly that, round. We
could fiddle with the serializers in established JSON encoders, but
we'd risk producing non-standards compliant JSON, and there goes your
compatibility and well-definedness.

> self documenting,

That's actually kind of an anti-feature! We'd want data to fit into
predefined schemata, rather than allowing arbitrary fields in arbitrary
order.

> easy to exchange between platforms and OS's,

that's true

> and supported natively by Python.

That is a bonus, but it doesn't outweigh its downsides.

> Yes, it requires parsing, but I don't
> think that is a major drawback.

Sadly: You'd be wrong! JSON parsing is compared to what we want to do
with the data a pretty compute-intense job. Think of it like this:
People will want to build things where half a million messages per
second is not an implausible requirement.

> Also it would not require any
> documentation to define how it works!

I think the main workload would be documenting how the thing we're
building should be used, and that's pretty independent from how we
encode messages beneath the surface :)

Best regards,
Marcus

¹ https://en.wikipedia.org/wiki/Edward_Norton_Lorenz#Chaos_Theory

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Hi Nick,

On Sat, 2019-09-28 at 15:54 -0700, Nick Foster wrote:
> I think it's wise to avoid hitching our metaphorical horse to any
> software project we can't absolutely depend on; questionable
> dependency choices have bit us, as you mention, many times.

Sad but true.

> With that (somewhat) in mind, what about Flatbuffers as an
> alternative to Cap'n Proto? Seems much more actively maintained, more
> widely used, and designed with the same goals in mind. No parsing
> step required, so performance is good. Memory use seems to be
> similarly efficient.

See, that's the kind of input I was looking for!
So, what I don't like about flatbuffers was the lack of field IDs in
the schema; relying on new fields only being added to the end of a
structure.

However, thanks to your encouragement I've revisited that and:
Flatbuffers *does* do IDs, if you want (it's slightly awkward, but all
formal structure definition languages are, aside from ASN.1 ;) ).

Regarding efficiency: https://github.com/thekvs/cpp-serializers seems
to agree with your assessment.

What flatbuffers really got going for it compared to the Cap'n is that
docs are structured instead of "cool feature-oriented". What Cap'n
Proto has is the more sympathetic code base, but maybe I shouldn't be
applying too many human qualities to code bases. gnuradio-runtime/lib/
gets jealous everytime I do that.

Best regards,
Marcus

>
> Now, I haven't actually used Flatbuffers before, so anyone who has
> experience with them (or your other favorite serialization library!),
> please chime in here.
>
> Nick
>
> On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <mmueller@gnuradio.org>
> wrote:
> > To avoid the setting in of post-Conference downs:
> >
> > Dearest SDR Community with the Highest count of "Awesome" in the
> > Universe,
> >
> > discussions at GRCon have made it abundantly clear that PMT needs
> > to be
> > replaced¹.
> > Main issues with PMT are:
> >
> > * broken type system,
> > * low suitability as tool for internal packeted data,
> > * lack of interoperability with anything but the languages we
> > support
> > natively,
> > * platform-dependency of the serialization format,
> > * bugs.
> >
> > (to a far lesser degree: performance.)
> >
> > Having talked to multiple domain experts in networked systems and
> > heterogenous cooperative platforms, I've come to the conclusion,
> > furthermore, that in the interest of RPC'ing everything², the
> > structure
> > of messages exchanged between blocks should be predefined.
> >
> > In this sense, PMT's (limited) ability do describe itself is of no
> > help
> > – instead, it introduces ambiguity.
> >
> > The dominant choice for binary serialization with known schemata is
> > Proto Buffers; however, they require deserialization of a whole
> > message
> > if there's substructures before the field you want to access³.
> >
> > Cap'n Proto[1] (by one of the original authors of protobufs) seems
> > to
> > be a suitable alternative, and comparing it to its competitors, it
> > seems to have achieved a significant level of maturity and has at
> > least
> > one large-scale adopter⁴.
> >
> > Among its advantages seems to be that operations on the message
> > happen
> > in-place (i.e. there's actually a compatible C++ struct
> > representation
> > of the memory layout), which rendered it my favourite library for
> > this
> > job.
> >
> > Now, problem: I've never personally worked with Cap'n Proto, or
> > protobuf, or any modern, non-self-describing serialization library;
> > I've got minimal experience with MessagePack, but as said, that's
> > not
> > probably going to make it due to lack of schema.
> >
> > So, here I am, calling for your experience! Hurl ins(igh|ul)ts at
> > me,
> > preferable here on this list.
> >
> > Best regards,
> > Marcus
> >
> >
> > [1] https://capnproto.org/index.html
> >
> > ¹This doesn't come as a surprise – it's just a good point in time
> > to
> > point this out.
> > ²You've probably noticed that we're working towards a less machine-
> > dependent, more portable, distributable computational model
> > ³This can become problematic for messages with large data blobs in
> > between, or very complex protocol headers
> > ⁴CloudFlare seems to be using Cap'n Proto for Web workers, for
> > example.
> > However, it's but at revision 0.7, which indeed implies a bit of
> > headache – other tools that never reached 1.0 that we rely on
> > include
> > Thrift, and that's really not a club that you'd want to be in.
> >
> >
> > _______________________________________________
> > Discuss-gnuradio mailing list
> > Discuss-gnuradio@gnu.org
> > https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Saturday, September 28, 2019

Re: [Discuss-gnuradio] GNU-Radio Companion Python flowgraph - return signal that it is finished?

On Sat, Sep 28, 2019 at 7:46 PM Andrew Payne <wandrewp@gmail.com> wrote:

Understood. I do need to thread that tb.wait() because I have a callback to start the flow graph and a callback to stop the flow graph. Say I start then wait for it in an independent thread. Then say a different thread decides to stop it. How should the first thread that is waiting handle it? Would the results be unknown?

[Please make sure to always send your messages to the mailing list when replying.]

I don't know the answer to what would happen; what I would hope would happen is that the wait() returns as soon as it is stopped by either means. That would be consistent with other behavior (e.g. if you reuse a flowgraph, you must call wait() after stop() (to wait for cleanup, if any) before start()ing again.)

I encourage you to try it out.

Re: [Discuss-gnuradio] GNU-Radio Companion Python flowgraph - return signal that it is finished?

On Sat, Sep 28, 2019 at 6:34 PM Andrew Payne <wandrewp@gmail.com> wrote:

I have a short IQ file that I simply want to play once without repeating, and I want the calling Python code to "know" when it's completed after commanding the GR class instance to start via a call to tb.start(), if tb is the class instance variable.

Call

tb.start()

tb.wait()

tb.stop()

in that order. The call to wait() will block until the flow graph finishes.

If you need to do something else while waiting, you can create another thread from Python (threading.Thread) and either have it call tb.wait() or have it do the other work, whichever suits your application.

[Discuss-gnuradio] GNU-Radio Companion Python flowgraph - return signal that it is finished?

Hello,

I am writing a program in Python to interface with a simple file
source to USRP sink flowgraph.

I have a short IQ file that I simply want to play once without
repeating, and I want the calling Python code to "know" when it's
completed after commanding the GR class instance to start via a call
to tb.start(), if tb is the class instance variable. I came across
the C++ GR API here:
https://www.gnuradio.org/doc/doxygen/classgr_1_1block.html#a89242cb1ac9d2bf5e75f923745baa576

^ That shows there's a gr.finished() boolean return that tells you
when the flowgraph is done, but evidently .finished is not an
attribute of tb. The only methods that worked for me are
tb.start/tb.run, tb.wait, tb.stop.

I'd also be happy with polling the GR class instance for the
noutput_items from the work function of the USRP_sink block. I know
the file size so I can have a while loop with the noutput_items value
as a condition, but how do I do that in Python? I think I am getting
close with these 2 results from Google but still not getting it:
1) https://stackoverflow.com/questions/50938245/gnuradio-number-of-output-items/50943105
2) https://stackoverflow.com/questions/32305813/gnuradio-how-to-change-the-noutput-items-dynamically-when-writing-oot-block

Thanks,
Andrew

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

Just thinking outside the box here, but what about JSON? It is well
defined, self documenting, easy to exchange between platforms and OS's,
and supported natively by Python. Yes, it requires parsing, but I don't
think that is a major drawback. Also it would not require any
documentation to define how it works!

Cheers!
--
Barry Duggan KV4FV

On Sat, 28 Sep 2019 23:01:36 +0200, Marcus Mueller wrote:

Dearest SDR Community with the Highest count of "Awesome" in the
Universe,

discussions at GRCon have made it abundantly clear that PMT needs to be
replaced¹.
Main issues with PMT are:

* broken type system,
* low suitability as tool for internal packeted data,
* lack of interoperability with anything but the languages we support
natively,
* platform-dependency of the serialization format,
* bugs.

(to a far lesser degree: performance.)

<snip>

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

I think it's wise to avoid hitching our metaphorical horse to any software project we can't absolutely depend on; questionable dependency choices have bit us, as you mention, many times.

With that (somewhat) in mind, what about Flatbuffers as an alternative to Cap'n Proto? Seems much more actively maintained, more widely used, and designed with the same goals in mind. No parsing step required, so performance is good. Memory use seems to be similarly efficient.

Now, I haven't actually used Flatbuffers before, so anyone who has experience with them (or your other favorite serialization library!), please chime in here.

Nick

On Sat, Sep 28, 2019 at 2:02 PM Marcus Müller <mmueller@gnuradio.org> wrote:

To avoid the setting in of post-Conference downs:

Dearest SDR Community with the Highest count of "Awesome" in the
Universe,

discussions at GRCon have made it abundantly clear that PMT needs to be
replaced¹.
Main issues with PMT are:

* broken type system,
* low suitability as tool for internal packeted data,
* lack of interoperability with anything but the languages we support
natively,
* platform-dependency of the serialization format,
* bugs.

(to a far lesser degree: performance.)

Having talked to multiple domain experts in networked systems and
heterogenous cooperative platforms, I've come to the conclusion,
furthermore, that in the interest of RPC'ing everything², the structure
of messages exchanged between blocks should be predefined.

In this sense, PMT's (limited) ability do describe itself is of no help
– instead, it introduces ambiguity.

The dominant choice for binary serialization with known schemata is
Proto Buffers; however, they require deserialization of a whole message
if there's substructures before the field you want to access³.

Cap'n Proto[1] (by one of the original authors of protobufs) seems to
be a suitable alternative, and comparing it to its competitors, it
seems to have achieved a significant level of maturity and has at least
one large-scale adopter⁴.

Among its advantages seems to be that operations on the message happen
in-place (i.e. there's actually a compatible C++ struct representation
of the memory layout), which rendered it my favourite library for this
job.

Now, problem: I've never personally worked with Cap'n Proto, or
protobuf, or any modern, non-self-describing serialization library;
I've got minimal experience with MessagePack, but as said, that's not
probably going to make it due to lack of schema.

So, here I am, calling for your experience! Hurl ins(igh|ul)ts at me,
preferable here on this list.

Best regards,
Marcus

[1] https://capnproto.org/index.html

¹This doesn't come as a surprise – it's just a good point in time to
point this out.
²You've probably noticed that we're working towards a less machine-
dependent, more portable, distributable computational model
³This can become problematic for messages with large data blobs in
between, or very complex protocol headers
⁴CloudFlare seems to be using Cap'n Proto for Web workers, for example.
However, it's but at revision 0.7, which indeed implies a bit of
headache – other tools that never reached 1.0 that we rely on include
Thrift, and that's really not a club that you'd want to be in.

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

[Discuss-gnuradio] [RFC] PMT succession: experts' opinions on serialization libs

To avoid the setting in of post-Conference downs:

Dearest SDR Community with the Highest count of "Awesome" in the
Universe,

discussions at GRCon have made it abundantly clear that PMT needs to be
replaced¹.
Main issues with PMT are:

(to a far lesser degree: performance.)

Having talked to multiple domain experts in networked systems and
heterogenous cooperative platforms, I've come to the conclusion,
furthermore, that in the interest of RPC'ing everything², the structure
of messages exchanged between blocks should be predefined.

In this sense, PMT's (limited) ability do describe itself is of no help
– instead, it introduces ambiguity.

The dominant choice for binary serialization with known schemata is
Proto Buffers; however, they require deserialization of a whole message
if there's substructures before the field you want to access³.

Cap'n Proto[1] (by one of the original authors of protobufs) seems to
be a suitable alternative, and comparing it to its competitors, it
seems to have achieved a significant level of maturity and has at least
one large-scale adopter⁴.

Among its advantages seems to be that operations on the message happen
in-place (i.e. there's actually a compatible C++ struct representation
of the memory layout), which rendered it my favourite library for this
job.

Now, problem: I've never personally worked with Cap'n Proto, or
protobuf, or any modern, non-self-describing serialization library;
I've got minimal experience with MessagePack, but as said, that's not
probably going to make it due to lack of schema.

So, here I am, calling for your experience! Hurl ins(igh|ul)ts at me,
preferable here on this list.

Best regards,
Marcus

[1] https://capnproto.org/index.html

¹This doesn't come as a surprise – it's just a good point in time to
point this out.
²You've probably noticed that we're working towards a less machine-
dependent, more portable, distributable computational model
³This can become problematic for messages with large data blobs in
between, or very complex protocol headers
⁴CloudFlare seems to be using Cap'n Proto for Web workers, for example.
However, it's but at revision 0.7, which indeed implies a bit of
headache – other tools that never reached 1.0 that we rely on include
Thrift, and that's really not a club that you'd want to be in.

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Thursday, September 26, 2019

[Discuss-gnuradio] setup_env.sh on 3.8-maint for python3

Hi Folks!

I'm building gnuradio using pybombs on Ubuntu 18.04 (x86_64)

My intention is to have it use python3.

When I do this:

sudo pip3 install PyBOMBS
pybombs auto-config
pybombs recipes add-defaults
pybombs prefix init ~/gr/default -a default -R gnuradio-default

It builds, and generates python3 code, however setup_env.sh references 2.6 and 2.7.

Am I missing something?

The config seems reasonable:

default$ pybombs config
PyBOMBS.ConfigManager - INFO - Prefix Python version is: 3.6.8
PyBOMBS - INFO - PyBOMBS Version 2.3.3
PyBOMBS.config - INFO - Using config file: /home/eric/.pybombs/config.yml
python_ver: 3.6.8
- Python version for this prefix
default_prefix: default
- Default Prefix
git-cache: /home/eric/.pybombs/gitcache
- Path to git reference repository (git cache)
makewidth: 4
- Concurrent make threads [1,2,4,8...]
elevate_pre_args: ['sudo', '-H']
- For commands that need elevated privileges, prepend this
packagers: apt,pymod,pip,pkgconfig,cmd
- Priority of non-source package managers
keep_builddir:
- When rebuilding, default to keeping the build directory
builddocs: ON
- Build doxygen while compiling packages? options are: ON, OFF
cmakebuildtype: RelWithDebInfo
- CMAKE_BUILD_TYPE args to pass to cmake projects, options are: Debug, Release, RelWithDebInfo, MinSizeRel

I've worked around it by editing setup_env.sh. Is there a better fix? Where would it go?

Thanks for all of your efforts! Super happy to see 3.8.0.0 released!

Eric

Wednesday, September 25, 2019

Re: [Discuss-gnuradio] Using Qt5 widgets in an Embedded Python block

After several more tutorials, lots of searching, an lots of
trial-and-error, I think I am getting closer to a solution. However, I
get the following error with my new code:
```
QWidget: Must construct a QApplication before a QWidget
```
If I declare a new QApplication, I don't know how to tie it to anything.

Here is my revised code:
```
# Embedded Python Block

import numpy as np
from gnuradio import gr
from gnuradio import qtgui
import sip
import sys
import signal

from PyQt5 import Qt
from PyQt5 import QtGui
from PyQt5.QtWidgets import (QWidget, QLineEdit, QPushButton,
QTextEdit, QGridLayout, QApplication)

d_buffer = ""
textboxValue = ""

class cons_in (gr.sync_block):

def __init__(self):
gr.sync_block.__init__(self,
name='Console Input', # will show up in GRC
in_sig=None,
out_sig=[np.byte])
self.initUI()

def initUI(self):

# create text display area
self.textarea = QTextEdit(QWidget=None)
self.textarea.setReadOnly(True)

# create input text area
self.textbox = QLineEdit(QWidget=None)
self.textbox.editingFinished.connect(self.onEnter)
self.textbox.setFocus()

grid = QGridLayout()
grid.setSpacing(10)

grid.addWidget(self.textarea, 0, 0, 5, 1)
grid.addWidget(self.textbox, 5, 0)

self.setLayout(grid)
self.show()

def onEnter(self):
global d_buffer
global textboxValue

textboxValue = self.textbox.text()
d_buffer += (textboxValue + "\n")
self.textarea.setText(d_buffer)
self.textbox.setText("")

def work(self, input_items, output_items):
global textboxValue

# <send textboxValue to output port>
_num_elem = len(textboxValue)
for x in range (_num_elem):
output_items[0][x] = textboxValue[x]
return (_num_elem)
```

A stand-alone version of the code works properly.

Any help is appreciated!
Thanks,
--
Barry Duggan

On Tue, 24 Sep 2019 20:15:41 -0400, Barry Duggan wrote:

Hi,

My flowgraph has an Embedded Python block feeding a Throttle feeding a
Text sink (https://github.com/dl1ksv/gr-display). In my Embedded Python
block I have a QLineEdit widget and a QTextEdit widget. The user input
is fed to the output port.

<snip>

_______________________________________________
Discuss-gnuradio mailing list
Discuss-gnuradio@gnu.org
https://lists.gnu.org/mailman/listinfo/discuss-gnuradio

Re: [Discuss-gnuradio] undefined reference to `pthread_create' while pybombs installation

Ahh sorry, that was not an error. Looks like new version does not assigns a default id (previously it did) and user have to do by itself.

Sorry for spamming.

On Wed, 25 Sep 2019 at 21:52, sumit kumar <sumitstop@gmail.com> wrote:

Ok I managed to fix it but exporting the python path to dist-packages of python 3.6 (I never needed to do this when I used pybombs),
But now I see a new thing
When I open GNU Radio companion and try to make an example
I see "ID "default" is blacklisted." Figure attached.
So many new things today!

On Wed, 25 Sep 2019 at 21:41, sumit kumar <sumitstop@gmail.com> wrote:
Hi Vasil,
Thanks for detailed info. I did that straight away as you suggested. Installation was smooth, but now I am getting "Cannot import gnuradio" error! when I start gnuradio-companion.
I have attached the screenshot and logs

ubadmin@ub-exotic02:~$ cd prefix/
ubadmin@ub-exotic02:~/prefix$ source setup_env.sh
ubadmin@ub-exotic02:~/prefix$ gnuradio-companion
Gtk-Message: 21:39:40.832: GtkDialog mapped without a transient parent. This is discouraged.
ubadmin@ub-exotic02:~/prefix$ echo $LD_LIBRARY_PATH
/home/ubadmin/prefix/lib:/home/ubadmin/prefix/lib64/:
ubadmin@ub-exotic02:~/prefix$

On Wed, 25 Sep 2019 at 21:15, Vasil Velichkov <vvvelichkov@gmail.com> wrote:
Hi Sumit,

On 25/09/2019 21.25, sumit kumar wrote:
> Attaching output of Configuration (which failed)
>
> On Wed, 25 Sep 2019 at 20:12, sumit kumar <sumitstop@gmail.com> wrote:
>
>> Hi,
>> I am getting undefined reference to `pthread_create' when I am
>> installing GNU Radio using pybombs. The desktop has fresh install of ubuntu
>> 18.04.
>> Error and Output log files are attached.

The problem is that cmake cannot find numpy.

> -- Python checking for numpy - not found
> --
> -- Configuring gnuradio-companion support...
> -- Dependency ENABLE_GNURADIO_RUNTIME = ON
> -- Dependency ENABLE_PYTHON = ON
> -- Dependency PYTHON_MIN_VER_FOUND = TRUE
> -- Dependency PYYAML_FOUND = TRUE
> -- Dependency MAKO_FOUND = TRUE
> -- Dependency PYGI_FOUND = TRUE
> -- Dependency GTK_GI_FOUND = TRUE
> -- Dependency CAIRO_GI_FOUND = TRUE
> -- Dependency PANGOCAIRO_GI_FOUND = TRUE
> -- Dependency NUMPY_FOUND = FALSE
> CMake Error at cmake/Modules/GrComponent.cmake:75 (message):
> user force-enabled gnuradio-companion but configuration checked failed

It should have been installed by pybombs as the gnuradio recipe contains numpy as dependency [1] and numpy recipe contains deb packages for both python2 [2] and python3 [3].

> ubadmin@ub-exotic02:~$ pybombs prefix init -a default prefix/default/ -R gnuradio-default
> PyBOMBS.ConfigManager - INFO - Prefix Python version is: 2.7.15

You have used pip to install pybombs and currently in ubuntu 18.04 pip uses python2 while pip3 uses python3.

> -- User set python executable /usr/bin/python3.6
> -- Found PythonInterp: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "2.7.6")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable exact version "3.6.8")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable version "3.6.8", minimum required is "2.7.6")

When both python3 and python2 are available gnuradio's cmake will choose python3 and mixing python2 (the one used by pybombs) with python3 (the one used by cmake) could lead to many hard to debug problems.

My advice is to remove the pybombs that is using python2, then install it using pip3, remove your prefix directory and ~/.pybombs (or use another directory) and start from scratch. It should be something like:

pip uninstall pybombs
pip3 install pybombs
rm -rf prefix/default/
rm -rf ~/.pybombs
pybombs auto-config
pybombs add-defaults
pybombs prefix init -a default prefix/default/ -R gnuradio-default

An alternative approach could be to modify the gnuradio recipe (~/.pybombs/recipes/gr-recipes/gnuradio.lwr) and add "-DPYTHON_EXECUTABLE=/usr/bin/python2" at the end of config_opt string

config_opt: " -DENABLE_DOXYGEN=$builddocs -DENABLE_GR_AUDIO=ON -DENABLE_GR_BLOCKS=ON -DENABLE_GR_DIGITAL=ON -DENABLE_GR_FEC=ON -DENABLE_GR_FFT=ON -DENABLE_GR_FILTER=ON -DENABLE_GR_QTGUI=ON -DENABLE_GR_UHD=ON -DENABLE_PYTHON=ON -DENABLE_VOLK=ON -DENABLE_GRC=ON -DPYTHON_EXECUTABLE=/usr/bin/python2"

[1] https://github.com/gnuradio/gr-recipes/blob/deb170d/gnuradio.lwr#L30
[2] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L27
[3] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L34

Regards,
Vasil

--
Sumit Kumar

--
Sumit Kumar

Sumit Kumar

Re: [Discuss-gnuradio] undefined reference to `pthread_create' while pybombs installation

Ok I managed to fix it but exporting the python path to dist-packages of python 3.6 (I never needed to do this when I used pybombs),

But now I see a new thing

When I open GNU Radio companion and try to make an example

I see "ID "default" is blacklisted." Figure attached.

So many new things today!

On Wed, 25 Sep 2019 at 21:41, sumit kumar <sumitstop@gmail.com> wrote:

Hi Vasil,
Thanks for detailed info. I did that straight away as you suggested. Installation was smooth, but now I am getting "Cannot import gnuradio" error! when I start gnuradio-companion.
I have attached the screenshot and logs

ubadmin@ub-exotic02:~$ cd prefix/
ubadmin@ub-exotic02:~/prefix$ source setup_env.sh
ubadmin@ub-exotic02:~/prefix$ gnuradio-companion
Gtk-Message: 21:39:40.832: GtkDialog mapped without a transient parent. This is discouraged.
ubadmin@ub-exotic02:~/prefix$ echo $LD_LIBRARY_PATH
/home/ubadmin/prefix/lib:/home/ubadmin/prefix/lib64/:
ubadmin@ub-exotic02:~/prefix$

On Wed, 25 Sep 2019 at 21:15, Vasil Velichkov <vvvelichkov@gmail.com> wrote:
Hi Sumit,

On 25/09/2019 21.25, sumit kumar wrote:
> Attaching output of Configuration (which failed)
>
> On Wed, 25 Sep 2019 at 20:12, sumit kumar <sumitstop@gmail.com> wrote:
>
>> Hi,
>> I am getting undefined reference to `pthread_create' when I am
>> installing GNU Radio using pybombs. The desktop has fresh install of ubuntu
>> 18.04.
>> Error and Output log files are attached.

The problem is that cmake cannot find numpy.

> -- Python checking for numpy - not found
> --
> -- Configuring gnuradio-companion support...
> -- Dependency ENABLE_GNURADIO_RUNTIME = ON
> -- Dependency ENABLE_PYTHON = ON
> -- Dependency PYTHON_MIN_VER_FOUND = TRUE
> -- Dependency PYYAML_FOUND = TRUE
> -- Dependency MAKO_FOUND = TRUE
> -- Dependency PYGI_FOUND = TRUE
> -- Dependency GTK_GI_FOUND = TRUE
> -- Dependency CAIRO_GI_FOUND = TRUE
> -- Dependency PANGOCAIRO_GI_FOUND = TRUE
> -- Dependency NUMPY_FOUND = FALSE
> CMake Error at cmake/Modules/GrComponent.cmake:75 (message):
> user force-enabled gnuradio-companion but configuration checked failed

It should have been installed by pybombs as the gnuradio recipe contains numpy as dependency [1] and numpy recipe contains deb packages for both python2 [2] and python3 [3].

> ubadmin@ub-exotic02:~$ pybombs prefix init -a default prefix/default/ -R gnuradio-default
> PyBOMBS.ConfigManager - INFO - Prefix Python version is: 2.7.15

You have used pip to install pybombs and currently in ubuntu 18.04 pip uses python2 while pip3 uses python3.

> -- User set python executable /usr/bin/python3.6
> -- Found PythonInterp: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "2.7.6")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable exact version "3.6.8")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable version "3.6.8", minimum required is "2.7.6")

When both python3 and python2 are available gnuradio's cmake will choose python3 and mixing python2 (the one used by pybombs) with python3 (the one used by cmake) could lead to many hard to debug problems.

My advice is to remove the pybombs that is using python2, then install it using pip3, remove your prefix directory and ~/.pybombs (or use another directory) and start from scratch. It should be something like:

pip uninstall pybombs
pip3 install pybombs
rm -rf prefix/default/
rm -rf ~/.pybombs
pybombs auto-config
pybombs add-defaults
pybombs prefix init -a default prefix/default/ -R gnuradio-default

An alternative approach could be to modify the gnuradio recipe (~/.pybombs/recipes/gr-recipes/gnuradio.lwr) and add "-DPYTHON_EXECUTABLE=/usr/bin/python2" at the end of config_opt string

config_opt: " -DENABLE_DOXYGEN=$builddocs -DENABLE_GR_AUDIO=ON -DENABLE_GR_BLOCKS=ON -DENABLE_GR_DIGITAL=ON -DENABLE_GR_FEC=ON -DENABLE_GR_FFT=ON -DENABLE_GR_FILTER=ON -DENABLE_GR_QTGUI=ON -DENABLE_GR_UHD=ON -DENABLE_PYTHON=ON -DENABLE_VOLK=ON -DENABLE_GRC=ON -DPYTHON_EXECUTABLE=/usr/bin/python2"

[1] https://github.com/gnuradio/gr-recipes/blob/deb170d/gnuradio.lwr#L30
[2] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L27
[3] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L34

Regards,
Vasil

--
Sumit Kumar

Sumit Kumar

Re: [Discuss-gnuradio] undefined reference to `pthread_create' while pybombs installation

Hi Vasil,

Thanks for detailed info. I did that straight away as you suggested. Installation was smooth, but now I am getting "Cannot import gnuradio" error! when I start gnuradio-companion.

I have attached the screenshot and logs

ubadmin@ub-exotic02:~$ cd prefix/
ubadmin@ub-exotic02:~/prefix$ source setup_env.sh
ubadmin@ub-exotic02:~/prefix$ gnuradio-companion
Gtk-Message: 21:39:40.832: GtkDialog mapped without a transient parent. This is discouraged.
ubadmin@ub-exotic02:~/prefix$ echo $LD_LIBRARY_PATH
/home/ubadmin/prefix/lib:/home/ubadmin/prefix/lib64/:
ubadmin@ub-exotic02:~/prefix$

On Wed, 25 Sep 2019 at 21:15, Vasil Velichkov <vvvelichkov@gmail.com> wrote:

Hi Sumit,

On 25/09/2019 21.25, sumit kumar wrote:
> Attaching output of Configuration (which failed)
>
> On Wed, 25 Sep 2019 at 20:12, sumit kumar <sumitstop@gmail.com> wrote:
>
>> Hi,
>> I am getting undefined reference to `pthread_create' when I am
>> installing GNU Radio using pybombs. The desktop has fresh install of ubuntu
>> 18.04.
>> Error and Output log files are attached.

The problem is that cmake cannot find numpy.

> -- Python checking for numpy - not found
> --
> -- Configuring gnuradio-companion support...
> -- Dependency ENABLE_GNURADIO_RUNTIME = ON
> -- Dependency ENABLE_PYTHON = ON
> -- Dependency PYTHON_MIN_VER_FOUND = TRUE
> -- Dependency PYYAML_FOUND = TRUE
> -- Dependency MAKO_FOUND = TRUE
> -- Dependency PYGI_FOUND = TRUE
> -- Dependency GTK_GI_FOUND = TRUE
> -- Dependency CAIRO_GI_FOUND = TRUE
> -- Dependency PANGOCAIRO_GI_FOUND = TRUE
> -- Dependency NUMPY_FOUND = FALSE
> CMake Error at cmake/Modules/GrComponent.cmake:75 (message):
> user force-enabled gnuradio-companion but configuration checked failed

It should have been installed by pybombs as the gnuradio recipe contains numpy as dependency [1] and numpy recipe contains deb packages for both python2 [2] and python3 [3].

> ubadmin@ub-exotic02:~$ pybombs prefix init -a default prefix/default/ -R gnuradio-default
> PyBOMBS.ConfigManager - INFO - Prefix Python version is: 2.7.15

You have used pip to install pybombs and currently in ubuntu 18.04 pip uses python2 while pip3 uses python3.

> -- User set python executable /usr/bin/python3.6
> -- Found PythonInterp: /usr/bin/python3.6 (found suitable version "3.6.8", minimum required is "2.7.6")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable exact version "3.6.8")
> -- Found PythonLibs: /usr/lib/x86_64-linux-gnu/libpython3.6m.so (found suitable version "3.6.8", minimum required is "2.7.6")

When both python3 and python2 are available gnuradio's cmake will choose python3 and mixing python2 (the one used by pybombs) with python3 (the one used by cmake) could lead to many hard to debug problems.

My advice is to remove the pybombs that is using python2, then install it using pip3, remove your prefix directory and ~/.pybombs (or use another directory) and start from scratch. It should be something like:

pip uninstall pybombs
pip3 install pybombs
rm -rf prefix/default/
rm -rf ~/.pybombs
pybombs auto-config
pybombs add-defaults
pybombs prefix init -a default prefix/default/ -R gnuradio-default

An alternative approach could be to modify the gnuradio recipe (~/.pybombs/recipes/gr-recipes/gnuradio.lwr) and add "-DPYTHON_EXECUTABLE=/usr/bin/python2" at the end of config_opt string

config_opt: " -DENABLE_DOXYGEN=$builddocs -DENABLE_GR_AUDIO=ON -DENABLE_GR_BLOCKS=ON -DENABLE_GR_DIGITAL=ON -DENABLE_GR_FEC=ON -DENABLE_GR_FFT=ON -DENABLE_GR_FILTER=ON -DENABLE_GR_QTGUI=ON -DENABLE_GR_UHD=ON -DENABLE_PYTHON=ON -DENABLE_VOLK=ON -DENABLE_GRC=ON -DPYTHON_EXECUTABLE=/usr/bin/python2"

[1] https://github.com/gnuradio/gr-recipes/blob/deb170d/gnuradio.lwr#L30
[2] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L27
[3] https://github.com/gnuradio/gr-recipes/blob/deb170d/numpy.lwr#L34

Regards,
Vasil

Sumit Kumar