Skip to content

Add support for Pluggable Discoveries #1333

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 74 commits into from
Aug 23, 2021
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
74 commits
Select commit Hold shift + click to select a range
386740b
[skip changelog] Add DiscoveryManager to PackageManager
silvanocerza Aug 9, 2021
ebca55f
Add loading of PluggableDiscoveries when loading a platform release
silvanocerza Jun 23, 2021
7cf3215
Added compatibility layer for non-pluggable platforms
cmaglie Jun 29, 2021
f442b1a
Implemented board list with discoveries
silvanocerza Aug 9, 2021
dd6a8fc
Implemented discovery loading after initialization
cmaglie Jun 29, 2021
f07775a
Implemented board watch with discoveries
silvanocerza Aug 9, 2021
e80869b
Fix load discoveries tests
silvanocerza Jun 29, 2021
23bab36
Fix some issues with board list watcher
silvanocerza Jun 30, 2021
3b4bd38
Fix FindToolsRequiredFromPlatformRelease not returning discoveries
silvanocerza Jun 30, 2021
47b84f5
Enhanced handling of some discoveries states
silvanocerza Jul 2, 2021
b32749f
Fix PackageManager reset
silvanocerza Jul 6, 2021
0d2add4
Add function to convert discovery.Port to rpc.Port
silvanocerza Aug 9, 2021
8bb84b2
Moved reference argument parsing to new package
silvanocerza Jul 6, 2021
3ff8287
Fix functions docstrings
silvanocerza Jul 8, 2021
ec3dfbc
Remove duplicated code to initialize Sketch path
silvanocerza Aug 9, 2021
fb934e5
Add property conversion for platform not supporting pluggable discovery
silvanocerza Jul 15, 2021
38c4325
Fix board list watch not working
silvanocerza Jul 15, 2021
a3007fe
Fix crash when converting Port to rpc struct
silvanocerza Jul 20, 2021
2d69063
Add generic Port argument
silvanocerza Jul 15, 2021
7e6dfca
Change gRPC upload functions to use new Port message
silvanocerza Aug 9, 2021
e6c33ea
Add support for upload user fields
silvanocerza Aug 9, 2021
4a586d9
Fix upload unit tests
silvanocerza Jul 20, 2021
67ae164
Fix code naming issues
silvanocerza Jul 20, 2021
ddf1603
Added builtin:mdns-discovery
cmaglie Jul 21, 2021
ea61dca
Do not panic if discovery tool is not installed
cmaglie Jul 21, 2021
d255c3d
Implemented port/protocol detection at CLI startup time
silvanocerza Aug 9, 2021
18777b2
Perform 1200bps-touch only on serial ports
cmaglie Jul 21, 2021
709db6f
Added missing properties for pluggable upload
cmaglie Jul 21, 2021
322ce23
Correctly implemented 'board list' timeout option
silvanocerza Aug 9, 2021
4fc7857
Updated mdns-discovery to 0.9.2
cmaglie Jul 22, 2021
77a40fd
Add documentation
silvanocerza Aug 10, 2021
f8ac21d
Add board properties to board list command and gRPC function
silvanocerza Aug 9, 2021
5ecf4d8
Fix documentation and code comments
silvanocerza Aug 10, 2021
c8fb6e4
Fix crash when attempting upload without specifying port address
silvanocerza Jul 23, 2021
2a7d3d2
Fix unit tests
silvanocerza Jul 23, 2021
015aea3
Update go-properties-orderedmap to fix discovery properties issues
silvanocerza Aug 11, 2021
33f8848
Fix more documentation
silvanocerza Jul 26, 2021
bd68d5e
Clarify pluggable discovery specification
silvanocerza Jul 26, 2021
69c6812
More documentation fixes
silvanocerza Jul 27, 2021
aec9695
Add upload_port properties docs in platform specification
silvanocerza Jul 27, 2021
ed318af
Change links from pluggable discovery RFC to official docs
silvanocerza Jul 27, 2021
95b69fa
Add more upload mock integration tests
silvanocerza Jul 27, 2021
3e3fcd2
Fix integration tests
silvanocerza Jul 28, 2021
b7d594f
Change property to declare pluggable discoveries
silvanocerza Aug 10, 2021
763ce6f
Change property to declare pluggable discoveries
silvanocerza Jul 28, 2021
ae3a497
Fix documentation
silvanocerza Jul 28, 2021
3f484ed
Fix loading of platform not supporting pluggable discovery
silvanocerza Jul 28, 2021
be9e879
Fix more documentation
silvanocerza Jul 29, 2021
225c123
Add pluggable discovery states documentation
silvanocerza Aug 2, 2021
569429d
Enhanced handling of pluggable discoveries states
silvanocerza Aug 9, 2021
a0f2aee
Discoveries processes are now killed if the HELLO command fails
silvanocerza Aug 3, 2021
2624785
Add pluggable discovery logging
silvanocerza Aug 4, 2021
2a555ff
Enhanced handling of failing pluggable discoveries
silvanocerza Aug 4, 2021
71dcc0e
Fix pluggable discoveries parallelization
silvanocerza Aug 9, 2021
a0a6371
Discoveries event channels are now created when start sync is called
silvanocerza Aug 9, 2021
cfc7aa4
Cached ports are now reset on discovery stop
silvanocerza Aug 9, 2021
2a34f7d
Renamed ListSync methods to ListCachedPorts
silvanocerza Aug 9, 2021
85607d6
Pluggable discovery upload user fields are now limited to 50 chars
silvanocerza Aug 9, 2021
a5be354
Fix i18n strings
silvanocerza Aug 9, 2021
e68e4e0
Fix failing integration tests
silvanocerza Aug 10, 2021
eef60aa
Fix i18n data
silvanocerza Aug 10, 2021
f8814bc
Fix integration tests again
silvanocerza Aug 10, 2021
5aa1f1d
[skip changelog] Internationalize strings added for pluggable discove…
per1234 Aug 10, 2021
9941402
Update docs/pluggable-discovery-specification.md
silvanocerza Aug 10, 2021
53e1d25
Fix failing workflows
silvanocerza Aug 11, 2021
9a355b7
Updated upload-mock tests for generation
cmaglie Aug 12, 2021
41db8b5
Added a lot of mock upload test (also with programmer option)
cmaglie Aug 16, 2021
b322874
test_upload_mock: Handle '{' and '}' in recipes
cmaglie Aug 16, 2021
d7a576c
network ota: autoconvert network_patter from legacy
cmaglie Aug 17, 2021
3c10109
Automatically add port detection properties for network discovery
cmaglie Aug 19, 2021
e6ee163
Slightly improved 'board list' text output
cmaglie Aug 20, 2021
75c7fef
Default 'board list' timeout to 1s
cmaglie Aug 20, 2021
c100e14
Added some code review fixes
cmaglie Aug 23, 2021
40943b1
Added unit test for legacy-package conversion to pluggable discovery
cmaglie Aug 23, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Fix more documentation
Co-authored-by: per1234 <accounts@perglass.com>
  • Loading branch information
2 people authored and cmaglie committed Aug 19, 2021
commit 33f8848966a33363a1f720606f8648f27d817bcd
54 changes: 26 additions & 28 deletions docs/pluggable-discovery-specification.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
These tools must be in the form of executables that can be launched as a subprocess using a `platform.txt` command line
recipe. They will communicate to the parent process via stdin/stdout, in particular a discovery will accept commands as
plain text strings from stdin and will send answers back in JSON format on stdout. Each tool will implement the commands
to list and enumerate ports for a specific protocol as specified in this document.
recipe. They communicate to the parent process via stdin/stdout, accepting commands as plain text strings from stdin and
sending answers back in JSON format on stdout. Each tool will implement the commands to list and enumerate ports for a
specific protocol as specified in this document.

### Pluggable discovery API via stdin/stdout

Expand All @@ -10,7 +10,7 @@ All the commands listed in this specification must be implemented in the discove
After startup, the tool will just stay idle waiting for commands. The available commands are: `HELLO`, `START`, `STOP`,
`QUIT`, `LIST` and `START_SYNC`.

After each command the client always expect a response from the discovery. The discovery must not introduce any delay
After each command the client always expects a response from the discovery. The discovery must not introduce any delay
and must respond to all commands as fast as possible.

#### HELLO command
Expand All @@ -21,8 +21,7 @@ pluggable discovery protocol that the client/IDE supports. The syntax of the com
`HELLO <PROTOCOL_VERSION> "<USER_AGENT>"`

- `<PROTOCOL_VERSION>` is the maximum protocol version supported by the client/IDE (currently `1`)

- `<USER_AGENT>` is the name and version of the client (double-quotes `"` are not allowed)
- `<USER_AGENT>` is the name and version of the client. It must not contain double-quotes (`"`).

some examples:

Expand All @@ -48,13 +47,13 @@ are three possible cases:
- if the discovery supports a more recent version of the protocol than the client/IDE: the discovery should downgrade
itself into compatibility mode and report a `protocolVersion` that is less than or equal to the one supported by the
client/IDE.
- if the discovery cannot go into compatibility mode, it will report the protocol version supported (even if greater
- if the discovery cannot go into compatibility mode, it must report the protocol version supported (even if greater
than the version supported by the client/IDE) and the client/IDE may decide to terminate the discovery or produce an
error/warning.

#### START command

The `START` command initializes and start the discovery internal subroutines. This command must be called before `LIST`
The `START` command initializes and starts the discovery internal subroutines. This command must be called before `LIST`
or `START_SYNC`. The response to the start command is:

```JSON
Expand All @@ -78,7 +77,7 @@ The `error` field must be set to `true` and the `message` field should contain a

#### STOP command

The `STOP` command stops the discovery internal subroutines and possibly free the internally used resources. This
The `STOP` command stops the discovery internal subroutines and possibly frees the internally used resources. This
command should be called if the client wants to pause the discovery for a while. The response to the command is:

```JSON
Expand Down Expand Up @@ -130,7 +129,8 @@ call. The format of the response is the following:
"properties": {
<-- A LIST OF PROPERTIES OF THE PORT
}
}, {
},
{
... <-- OTHER PORTS...
}
]
Expand All @@ -142,17 +142,13 @@ The `ports` field contains a list of the available ports.
Each port has:

- an `address` (for example `/dev/ttyACM0` for serial ports or `192.168.10.100` for network ports)

- a `label` that is the human readable form of the `address` (it may be for example `ttyACM0` or
`SSH on 192.168.10.100`)

- `protocol` is the protocol identifier (such as `serial` or `dfu` or `ssh`)

- `protocolLabel` is the `protocol` in human readable form (for example `Serial port` or `DFU USB` or `Network (ssh)`)

- `properties` is a list of key/value pairs that represent information relative to the specific port

To make the above more clear lets show an example with the `serial_discovery`:
To make the above more clear let's show an example with the `serial_discovery`:

```JSON
{
Expand All @@ -174,18 +170,18 @@ To make the above more clear let’s show an example with the `serial_discovery`
}
```

In this case the serial port metadata comes from an USB serial converter. Inside the `properties` we have all the
In this case the serial port metadata comes from a USB serial converter. Inside the `properties` we have all the
properties of the port, and some of them may be useful for product identification (in this case only USB VID/PID is
useful to identify the board model).

The `LIST` command performs a one-shot polling of the ports. The discovery should answer as soon as reasonably possible,
without any additional delay.

Some discoveries may require some time to discover a new port (for example network protocols like MDNS, Bluetooth, etc.
requires some seconds to receive the broadcasts from all available clients) in that case is fine to answer with an empty
or incomplete list.
require some seconds to receive the broadcasts from all available clients) in that case it is fine to answer with an
empty or incomplete list.

If an error occurs and the discovery can't complete the enumeration is must report the error with:
If an error occurs and the discovery can't complete the enumeration, it must report the error with:

```JSON
{
Expand All @@ -210,7 +206,7 @@ command is:
}
```

After this message the discoery will send `add` and `remove` event asyncronoushly (more on that later). If an error
After this message the discovery will send `add` and `remove` events asynchronously (more on that later). If an error
occurs and the discovery can't go in "events" mode the error must be reported as:

```JSON
Expand Down Expand Up @@ -283,29 +279,31 @@ If the client sends an invalid or malformed command, the discovery should answer

The `properties` associated to a port can be used to identify the board attached to that port. The algorithm is simple:

- each board listed in the platform file `boards.txt` may declare a set of `upload_port.*` properties
- each board listed in the platform file [`boards.txt`](platform-specification.md#boardstxt) may declare a set of
`upload_port.*` properties
- if each `upload_port.*` property has a match in the `properties` set coming from the discovery then the board is a
candidate board attached to that port.
"candidate" board attached to that port.

Some port `properties` may not be precise enough to uniquely identify a board, in that case more boards may match the
same set of `properties`, thats why we called it candidate.
same set of `properties`, that's why we called it "candidate".

Lets see an example to clarify things a bit, let's suppose that we have the following `properties` coming from the
Let's see an example to clarify things a bit, let's suppose that we have the following `properties` coming from the
serial discovery:

```JSON
```
"port": {
"address": "/dev/ttyACM0",
"properties": {
"pid": "0x804e",
"vid": "0x2341",
"serialNumber": "EBEABFD6514D32364E202020FF10181E",
"name": "ttyACM0"
}
},
...
```

in this case we can use `vid` and `pid` to identify the board. The `serialNumber`, instead, is unique for that specific
instance of the board so it can't be used to identify the board model. Lets suppose we have the following `boards.txt`:
instance of the board so it can't be used to identify the board model. Let's suppose we have the following `boards.txt`:

```
# Arduino Zero (Programming Port)
Expand Down Expand Up @@ -365,4 +363,4 @@ myboard.upload_port.1.apples=40
```

will match on both `pears=20, apples=30` and `pears=30, apples=40` but not `pears=20, apples=40`, in that sense each
"set" of identification properties is indepentent from each other and cannot be mixed for port matching.
"set" of identification properties is independent from each other and cannot be mixed for port matching.
2 changes: 1 addition & 1 deletion mkdocs.yml
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@ nav:
- sketch-specification.md
- library-specification.md
- platform-specification.md
- Pluggable Discovery specification: pluggable-discovery-specification.md
- Pluggable discovery specification: pluggable-discovery-specification.md
- Package index specification: package_index_json-specification.md

extra:
Expand Down
2 changes: 1 addition & 1 deletion rpc/cc/arduino/cli/commands/v1/upload.proto
Original file line number Diff line number Diff line change
Expand Up @@ -167,7 +167,7 @@ message UserField {
string tool_id = 1;
// Name used internally to store and retrieve this field
string name = 2;
// Label is the text shown to the user when needs to input this field
// Label is the text shown to the user when they need to input this field
string label = 3;
// True if the value of the field must not be shown when typing, for example
// when the user inputs a network password
Expand Down