Re: [Paparazzi-devel] Support for multiple communication links

Another observation/question: How would you handle network topologies where links are shared by multiple aircraft? For example, a zigbee mesh or a link to a node that is relaying messages on behalf of multiple aircraft?

In link_combiner.py, the Link class assumes each link communicates with exactly one only aircraft (i.e. Link constructor takes an aircraft_id parameter). Would it be possible to make this parameter optional, and somehow let the link learn what aircraft(s) it is communicating with by message inspection?

Chris Gough

On Mon, Apr 22, 2013 at 7:04 AM, Cameron Lee <address@hidden> wrote:

Hello again everyone,

I've been working hard on this redundant link stuff for the past month and I've gotten the plane to ground working well. It reliably combines the links and shows information about each link in the GCS. For more details, pictures, and a video of it working, see this post about it: http://uaarg-news.blogspot.ca/2013/04/redundant-communication-improvement-to.html

Also, the code is here: https://github.com/uaarg/paparazzi/commits/redundant_comms

I have a few questions now though:

1. I want to make it so that the Link page only appears in the GCS when multiple links are used. I'm not sure how exactly to do this though. Does anybody know how? The two relevant files are pages.ml and live.ml. The Infrared page seems like it might have this functionality since since it's frame is defined as such: let infrared_frame = GBin.frame ~show:false ~shadow_type:`NONE () in

and it doesn't show up in the notebook, but I'm not sure how you would change it to be visible.

2. I'd like to contribute these changes back to the main Paparazzi repo if they are wanted. Should I submit a pull request now?

3. If there is any slight tweaks or additional functionality that people thing would be useful. let me know and I'll see if I can add it. For example, we can easily add more information for each link in the Link page of the notebook.

Thanks,

Cameron
On Fri, Mar 22, 2013 at 5:08 PM, Cameron Lee <address@hidden> wrote:
O man. It did it again. I'm sorry for the multiple e-mails. This is the last one. Feel free to ignore the two others.
Thanks for the information about OCaml versions. It does say in the documentation that String.map is for OCaml 4 only, but I overlooked that. Anyway, I got it working now. I still have to do some more extensive testing, but I'm pretty sure it'll work well. Feel free to review though: https://github.com/uaarg/paparazzi/commits/redundant_comms

My next step is to add support for monitoring multiple links from the GCS. There are two main areas in this and I have questions regarding both:

1. In the GCS GUI, where should I add information regarding the links? Also, why isn't there already more information (all that I'm aware of currently existing is the little green/red link indicator with the time since the last received message)? What I'm thinking of doing is two things: 1. Adding a tab in the notebook (is that the right terminology? I'm thinking of the area where the GPS and PFD tabs are). This tab could have detailed info about each link, maybe plots over time, and switches to change settings. 2. Modifying the green/red link indicator to become two indicators when there are two links, three when there are three, etc...

2. What information is there to send to and display in the GCS? I've found the DOWNLINK_STATUS which is generated in the Link agent and sent to the server where I think it's not used other than being logged. Then there's the TELEMETRY_STATUS message which is sent from the Server to the GCS to be used in the link indicator. I've also found the MODEM_STATUS message, but I'm not sure where it's used. Also, is there a message sent from the plane with the status of the the link as seen from it's end (so UPLINK_STATUS would be a suitable name)? Is there anything else that I should be aware of?

So, my plan is to modify both the DOWNLINK_STATUS and TELEMETRY_STATUS messages. DOWNLINK_STATUS would just have a link_name field added. TELEMETRY_STATUS would be modified extensively to include all of the information in DOWNLINK_STATUS and a link id or something. My questions regarding this are: can I go ahead and change these messages, or is that going to cause issues? In particular, if I change them, won't that mean that the logs will be different, making old logs not playable with the new softare? Also, should I modify the Server to keep track of the link statuses and send the new TELEMETRY_STATUS messages? The alternative would be to get my new Redundant Link program to do this. Maybe I could make a new message in the ground class that is sent from Redundant Link to the GCS with link statuses. Then the GCS would be modified to listen to this and add the additional information if it's present. Actually, I kind of like that idea.

I'm going to go ahead and get things working the best I can one way or another. But because of the complex nature of Paparazzi, any guidance, advice, suggestions, or comments are very welcome and helpful.

Thanks,

Cameron
On Fri, Mar 22, 2013 at 4:49 PM, Cameron Lee <address@hidden> wrote:
whoops. A keyboard short cut sent that e-mail before I finished it. Here's what it should be:
Thanks for the information about OCaml versions. It does say in the documentation that String.map is for OCaml 4 only, but I overlooked that. Anyway, I got it working now.

My next step is to add support for monitoring multiple links from the GCS. There are two main areas in this and I have questions regarding both:

1. In the GCS GUI, where should I add information regarding the links? Also, why isn't there already more information (all that I'm aware of currently existing is the little green/red link indicator with the time since the last received message)? What I'm thinking of doing is two things: 1. Adding a tab in the notebook (is that the right terminology? I'm thinking of the area where the GPS and PFD tabs are). This tab could have detailed info about each link, maybe plots over time, and switches to change settings. 2. Modifying the green/red link indicator to become two indicators when there are two links, three when there are three, etc...

2. What information is there to send to and display in the GCS? I've found the DOWNLINK_STATUS which is generated in the Link agent and sent to the server where I think it's not used other than being logged. Then there's the TELEMETRY_STATUS message which is sent from the Server to the GCS to be used in the link indicator. I've also found the MODEM_STATUS message, but I'm not sure where it's used. Also, is there a message sent from the plane with the status of the the link as seen from it's end (so UPLINK_STATUS
On Fri, Mar 22, 2013 at 4:42 PM, Cameron Lee <address@hidden> wrote:
Thanks for the information about OCaml versions. It does say in the documentation that String.map is for OCaml 4 only, but I overlooked that. Anyway, I got it working now.

My next step is to add support for monitoring multiple links from the GCS. There are two main areas in this and I have questions regarding both:

1. In the GCS GUI, where should I add information regarding the links? Also, why isn't there already more information (all that I'm aware of currently existing is the little green/red indicator with the time since the last received message)? What I'm thinking of doing is two things: 1. Adding a tab in the notebook (is that the right terminology? I'm thinking of area where the GPS and PFD tabs are). This tab could have detailed info about each link, maybe plots over time, and switches to change settings. 2. Modifying the green/red indicator to become two indicators when there are two links, three when there are three, etc...

2. What information is there to send to and display in the GCS? I've found the DOWNLINK_STATUS

Next Steps:

1. Modify the TELEMETRY_STATUS message so that it has a link id. The Server will
On Mon, Mar 18, 2013 at 8:29 AM, Gautier Hattenberger <address@hidden> wrote:
Hi,

String.map is only available with ocaml 4 (you should have ocaml 3.12). You can use the regular _expression_ module instead (std) to make replacement in strings.

Gautier

Le 16/03/2013 21:45, Cameron Lee a écrit :
Ok. So I think I've figured out what ";sv" means. It's colon-seperated-value, right? That makes sense since a value of a message can't have spaces in it, even if it's a string.

I have another question though: I'm trying to write the OCaml code to change the message into colon-seperated-value string. So I want to change this: "1 MESSAGE_NAME 1 2 3 4" into this: "1;MESSAGE_NAME;1;2;3;4". I think I've figured out how to do this, using String.map to change all spaces to semi-colons. However, when I try to do this I get an error: Error: Unbound value String.map

I can access other functions from the String module, such as String.length. So what's going on? Is this something specific to Paparazzi's installation of Ocaml? I'm referring to this site for reference: http://caml.inria.fr/pub/docs/manual-ocaml/libref/String.html

Thanks,

Cameron
On Fri, Mar 15, 2013 at 5:09 PM, Cameron Lee <address@hidden> wrote:
Thanks for your input. It's very much appreciated. Also, I've started pushing my commits, so you can see my progress here if you want to: https://github.com/uaarg/paparazzi/commits/redundant_comms

Regarding my message format, that is the biggest thing I am concerned about. I thought that my plan worked, but I just did some more testing and it doesn't quite. I know that there is the ac_id and optional timestamp before the message name. Since the ac_id and timestamp are both numbers, the link combiner can use appropriate regex to not mishandle anything. I've already got this working and have tested it successfully. What doesn't work though, and I overlooked this until now, is that the Server is picking up the new message format and taking the link_name to be the message name. So that's not good.

So, I'll change this to use a different system. I like your suggestion of using a message in a message, so I'll look into that. However, if we make a new message, are the other agents that are listening to messages going to listen to that one and not know what to do with it? Would we need to make a new message class to avoid this? If so, I don't see any issue with doing that. Also, I took a look in the messages.xml file at RAW_DATALINK, and I'm wondering what the format=";sv" means.

As for the uplink, I was going to take a closer look at that once I got the downlink working. My plan until then was to let the Link agent bind to uplink messages just like it currently does. This would mean that the Link Combiner is only handling data in one direction and that any message the GCS sends to the plane will be sent out over all of the links. Once I start working on the uplink redundancy, I would likely change this. The biggest reason is to have a serial number or timestamp be included in each message so that the aircraft can easily filter out duplicate messages with 100% accuracy (unlike the downlink). I'm not sure if I'd add the option to have different messages sent out over different links (or if this would be useful or not).

Cameron
On Fri, Mar 15, 2013 at 8:15 AM, Gautier Hattenberger <address@hidden> wrote:
Hello Cameron,

I'm not sure about the best choice. While your idea might work (with minor modification to the ocaml code), I'm not sure your message format is really nice. If you look at http://paparazzi.enac.fr/wiki/DevGuide/Server_GCS_com you'll see that the number of elements in front of the message name can be one or two, so you may have some misused messages.
Also, the link agents are bind to all uplink messages, so how will you prevent them to send the messages before link_combiner filters them ?
With the current design, I don't see any straightforward nor easy solution to this problem, sorry. If I have a good idea, I'll let you know.

Gautier

Le 12/03/2013 05:14, Cameron Lee a écrit :
Hello again everyone,

I've started working on some of the coding. I've changed my plan slightly though, to make it a simpler change. The new plan is this:

Just like before, to use multiple links you'll run multiple link processes, so that minimal changes to link.ml are necessary. I'll get each link process to send ivy messages that will be received by a new process (and only by this process), a so-called "link combiner". The link combiner process listens to the ivy messages from each link instance and outputs ivy messages to the server, etc... as if it was a link. This way, the Server, logging, and other processes don't need to be changed at all. They only see one link, the combined link, as output by the "link combiner".

Here's a simple diagram:

Old method: serial port → link → server → GCS

New method: serial port 1 → link →→→↓

serial port 2 → link → →link combiner → server → GCS

:

:

serial port n → link →→→↑

Then, in order to get the GCS to display data regarding the status of each link, I'll be changing the message with this info (TELEMETRY_STATUS I think) to support multiple links. So this will involve some changes to the GCS.

Some details:

The link agent will have an additional option, in order to set it's unique name. If this option is set (ex. link -d /dev/ttyUSB0 -s 57600 -name link_1) then the link will transmit over the ivy bus in the slightly different format, which is listed to by the link combiner only, not the server. If it's not, then link will act like it does right now.

The old ivy message format is (for example):

1 DOWNLINK 0 626 1212 (where "1" is the aircraft id)

The new ivy message format for going between the link and the link combiner will be (for example):

1 link_1 DOWNLINK 0 626 1212 (where "1" is the aircraft id and "link_1" is the link name).

This new message format isn't matched by the regex used by the GCS or messages agent. I think it's safe to use, but this is a week point of this plan.

I'll write the link combiner in Python (what I'm familiar with, although I'm starting to be able to read OCaml).

The link combiner will perform it's best guess at removing repeat messages, but it might not be perfect. I plan on filtering out messages that are identical to messages that came in through another link within the past x seconds. A typical x could probably be 2. For the ground to plan redundancy on the other hand, I'll be making sure it eliminates repeat messages 100% reliably. This will necessitate adding a serial number or time-stamp to outgoing messages.

The main advantages are:

1. Keep the power, flexibility, and features of the link agent intact (for example, you'll be able to have a link using UDP and another using a serial port at the same time).

2. Backwards compatible with the current single link system (I.e. you could run the slightly modified link, server, and GCS agents just like you used to without any issues).

3. Limited agents need changing, so it won't break everything.

4. You don't have to send the same messages over each link. You could send some data over just one link and not the other.

5. Most programming is in Python, so I'll actually be able to do it.

The main disadvantages are:

1. Lots more data going over the ivy bus. It could accidentally be listened to by processes that shouldn't listen to it.

2. When the link agents are talking to the redundant link agent, it'll be OCaml code talking to Python code. Not a big deal, but to make changes to how this works, you'll need to change two sets of code. So it's not following DRY (Don't Repeat Yourself), and requires knowledge of both languages.

Please let me know if you have any questions, suggestions, or concerns.

Cameron Lee

On Wed, Feb 27, 2013 at 9:12 PM, Cameron Lee <address@hidden> wrote:

So I've started my project to add support for multiple links in order to provide redundancy. So far I've mostly been looking at the code and trying to understand the system. I think I've got a good grasp of things and am going to start implementing my changes. I have a few questions though.

So my plan is to do pretty much what Chris suggested. The Link process will have a command line argument to specify it's id/name and so one Link process can be run for each link. The messages sent over the ivy bus to the server will contain this link id and the other ground segment processes will all need to be changed to support this change. This way, the GCS can display the status of each link, the messages process can display messages of both links, and many other advantages.

My questions for right now are:

1. The format of the messages sent over the ivy bus by the Link process seems to be (I'm using ivyprobe to look at them):

<Process_name> <Message>

where Message is:

<aircraft_id> <message_name> <value1> <value2> <value3> ...

for example:

Link sent '1 DOWNLINK_STATUS 98 96 6 610 0.0 0.0 -1362023310933.28'

What I'm wondering is, where in here should I include the link id? Should I change the process name? ex:

Link_primary sent '1 DOWNLINK_STATUS 98 96 6 610 0.0 0.0 -1362023310933.28'

or add a new field before the message name? ex:

Link sent '1 primary DOWNLINK_STATUS 98 96 6 610 0.0 0.0 -1362023310933.28'

I'm going to start looking into how the server reads these messages to try and answer my own question, but any advice would be great.

2. The processes that I know I need to change are: Link, GCS, Server, Messages, Messages (Python), Real Time Plotter. Are there others? I figure that the logging (in the Server process, right?) and Log File Player probably need to be changed as well. Anything else? Also, the above is the order I would start changing things.

Thanks for any help,

Cameron

On Mon, Oct 29, 2012 at 11:05 PM, Cameron Lee <address@hidden> wrote:

Thanks for your comments Chris. I haven't looked into this much yet so I'm not sure exactly what will need to be done. There will be a significant learning curve for me, which is partly why I'm asking for advice and suggestions.

I heard that there is a new messaging system being developed. Would this change things at all? Should I look at adding this functionality to the existing system, or the new system?

Cameron

On Sun, Oct 28, 2012 at 9:32 PM, Chris Gough <address@hidden> wrote:

Hi Cameron,

On Mon, Oct 29, 2012 at 11:52 AM, Cameron Lee <> wrote:
> I mean two links from the autopilot to the GCS but only one link from the
> GCS to the autopilot. New software at the GCS would listen to both incoming
> links and pass the data on to the GCS itself if either incoming link has
> valid data. The new software I would write for the autopilot wouldn't be
> used in this setup.

Ah, I was getting things mixed up.

Rather than making a "virtual link" that aggregates the modems behind
another process, have you looked into modifying the existing server,
GCS and link programs to support multiple links natively? Maybe there
are already features there I don't know about - the comments in
link.ml tantalisingly suggest that multiple concurrent links are
supported, but reading through the code it's not clear to me how
(link_id is always 0?).

server.ml sends TELEMETRY_STATUS and TELEMETRY_ERROR messages with an
aircraft_id but not any kind of link_id. Either these messages would
need to have a link_id added, or some new messages would be required.

The GCS would need to be modified to do something with the multiple
links described in TELEMETRY_* messages, and I think link.ml would
need to be modified to get/use a link_id, maybe assigned by a new
command line option

Maybe some new TELEMETRY_INFO messages ("link 2 down" etc). It would
be nice to know how many links there are to a given aircraft, and/or
how many aircraft are visible on a certain link.

Is this a sensible approach (anyone)?

> What I was wondering most is if the Lisa M could easily be set up to allow
> multiple links.

I suspect it could be, but am not the best person to help with that.

Chris Gough

> On Oct 28, 2012 6:34 PM, "Chris Gough" <address@hidden>
> wrote:
>>
>> > Whatever the case, for us, the primary use would be to have two links
>> > from
>> > the autopilot the the groups station but only one from the ground
>> > station to
>> > the autopilot. We can split the Tx of the TWOG in two and send one over
>> > 900
>> > MHz and the other over WiFi.
>>
>> Do you mean only one link (from GCS->autoplot) at a time, but with the
>> ability to change? Otherwise, like you said, you can just splice Tx
>> from the autopilot to both modems (and connect autopilot Rx from the
>> one you chose).
>>
>> > Also, future hardware could have more UARTs for this, couldn't it?
>>
>> I guess so.
>>
>> Chris Gough
>>
>> > Cameron
>> >
>> > On Oct 28, 2012 5:20 PM, "Chris Gough" <address@hidden>
>> > wrote:
>> >>
>> >> Hi Cameron,
>> >>
>> >> If you going to add multiple serial modems directly to an existing
>> >> autopilot, you might need to find a way to free up some serial IO (i2c
>> >> GPS?).
>> >>
>> >> We did this by cheating; inserting a linux computer between the
>> >> autopilot and modems. This was required because our ubiquity link
>> >> (bullet/rocket) is only accessible via ethernet, but it would allow an
>> >> arbitrary number of serial modems to be added (we use one or two).
>> >>
>> >> It's not a very elegant solution though (or cheap, or light, or
>> >> compact), ethernet/usb connectors don't tolerate vibrations very well
>> >> so the whole thing requires a lot of hot glue to make it reliable.
>> >>
>> >> Chris Gough
>> >>
>> >>
>> >> On Mon, Oct 29, 2012 at 6:58 AM, Cameron Lee <address@hidden>
>> >> wrote:
>> >> > Hello everyone,
>> >> >
>> >> > I'm a fourth year Electrical Engineering student interested in
>> >> > working
>> >> > on an
>> >> > aspect of Paparazzi for a class of mine. I'm planning on adding
>> >> > support
>> >> > for
>> >> > multiple communication links at both the GCS and on the autopilot.
>> >> > Similar
>> >> > to item 6 in the wishlist
>> >> > (http://paparazzi.enac.fr/wiki/Software_Wish_List):
>> >> >
>> >> >
>> >> > The possibility to use multiple ground modem connected to a single
>> >> > ground
>> >> > station. The RSSI could be use to dynamically choose which currently
>> >> > has
>> >> > the
>> >> > best signal. This would allow the use of different antennas on each
>> >> > of
>> >> > the
>> >> > modems or have antenna pointing in different directions(?Possibly
>> >> > more
>> >> > hardware related)
>> >> >
>> >> >
>> >> >
>> >> > Here's a description I've written up:
>> >> >
>> >> > The goal is to improve the open source Paparazzi autopilot by adding
>> >> > support
>> >> > for multiple communication links to provide redundancy and increased
>> >> > flexibility. Currently, a single bi-directional serial data link
>> >> > enables
>> >> > the
>> >> > autopilot to provide telemetry to the ground station and the ground
>> >> > station
>> >> > to provide commands to the autopilot. This serial data link is
>> >> > usually
>> >> > created using RF radios and if it’s lost for any reason, all
>> >> > communication
>> >> > with the autopilot is lost. If two data links can be created, then
>> >> > communication can be maintained even if one of the links is lost.
>> >> > Typically,
>> >> > the two links would be of different varieties: two different
>> >> > frequencies, or
>> >> > a short-range high-throughput link and a long-range low-throughput
>> >> > link,
>> >> > or
>> >> > the same type of radio, but with different types of antennas, other
>> >> > combination.
>> >> > This project will involve enabling the autopilot and the ground
>> >> > station
>> >> > to
>> >> > each transmit their data through multiple links as well as receive
>> >> > data
>> >> > through multiple links. Receiving data through multiple links will
>> >> > involve
>> >> > deciding which link to take as correct based on the data coming in
>> >> > through
>> >> > all available links.
>> >> >
>> >> >
>> >> >
>> >> > Before I start working on this, I figured that I should ask for
>> >> > feedback
>> >> > -
>> >> > is this functionality indeed useful? Is this the best way to go about
>> >> > it? Is
>> >> > this project achievable (in particular on the autopilot side - will
>> >> > it
>> >> > take
>> >> > too much system resources)? Any tips or advice would be appreciated.
>> >> > Also,
>> >> > how difficult would it be to have different telemetry sent out over
>> >> > each
>> >> > link?
>> >> >
>> >> > Thanks,
>> >> >
>> >> > Cameron
>> >> >
>> >> > _______________________________________________
>> >> > Paparazzi-devel mailing list
>> >> > address@hidden
>> >> > https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> .
>> >>
>> >> _______________________________________________
>> >> Paparazzi-devel mailing list
>> >> address@hidden
>> >> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>> >>
>> >
>> > _______________________________________________
>> > Paparazzi-devel mailing list
>> > address@hidden
>> > https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>> >
>>
>>
>>
>> --
>> .
>>
>> _______________________________________________
>> Paparazzi-devel mailing list
>> address@hidden
>> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>>
>
> _______________________________________________
> Paparazzi-devel mailing list
> address@hidden
> https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
>

--
.

_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel
_______________________________________________
Paparazzi-devel mailing list
address@hidden
https://lists.nongnu.org/mailman/listinfo/paparazzi-devel

--
.

From:	Chris Gough
Subject:	Re: [Paparazzi-devel] Support for multiple communication links
Date:	Mon, 22 Apr 2013 12:18:07 +1000