update esp-idf; allow start/stop channels in wifi scanning

[dhalbert]

• Fixes Espressif boards wifi scanning instability upon reload (soft reset) #6866.

---

• The snapshot of ESP-IDF we were using was out of date. Updating it fixed the wifi scanning issues in Espressif boards wifi scanning instability upon reload (soft reset) #6866. Added yet another new branch to https://github.com/adafruit/esp-idf : https://github.com/adafruit/esp-idf/tree/release/v4.4-circuitpython. That branch is based on the current tip of https://github.com/espressif/esp-idf/tree/release/v4.4. Cherry-picked fixes from the https://github.com/adafruit/esp-idf/tree/circuitpython8 branch, which we were using previously. Some fixes from circuitpython8 were already in upstream, so did not cherry-pick those.
• Added start_channel and stop_channel arguments to wifi.Radio.start_scanning_networks(). The documentation signature said the args were there, but they were not actually implemented.
• Did a bit of renaming and code restructuring in the scanning code so I could understand it better.
• Made sure that unused scanning results were discarded properly.

It's possible the ESP-IDF update is going to fix some other bugs as well.

[anecdata]

Turned off web workflow, and loaded this on a few QT Py ESP32-S2s, and S2 and S3 Feather TFTs. Scans and connections are looking good across resets and reloads. Vast improvement!

(Still getting some hard faults on -S3 somewhere after a control-C, but that's a separate issue; I haven't seen any hard faults on -S2)

[microdev1]

Looks good, one suggestion.

[microdev1]

maybe raise not implemented error when provided channel range doesn't match the default range

[dhalbert]

I don't actually know which channels are scanned by default. I looked for documentation, but maybe I am looking in the wrong place. These config params are set to zero:

    int32_t channel_num;
    uint16_t channel_list[1];

@jepler Do you know, or know where to look?

Maybe can figure this out later and add support.

[jepler]

As far as I can tell there's no way to set the channels to scan in picow. some internal default is always used. It may be controlled by the country setting, which we never change.

[bill88t]

Imma test on S2 too very soon.
400 reloads should be enough.

[bill88t]

Ran my test on it:

{
    "Pass": 232,
    "Pass_env": 92, 
    "PartialFail": 68,
    "PartialFail_env": 8,
    "CompleteFail": 0,
    "CompleteFail_env": 0,
    "og_networks": 17,
    "version": "8.0.0-beta.1-22-g12c05d027",
    "want_env": false,
    "iterations": 402,
    "board": "Waveshare ESP32-S2-Pico",
    "cpu": "ESP32S2",
    "espmem_total": 148176,
    "espmem_before_env": [51692, 51488, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51376, 51372, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560],
    "espmem_before": [58944, 58628, 58592, 58520, 58468, 58448, 58392, 58372, 58340, 58308, 58292, 58272, 58240, 58208, 58176, 58144, 58112, 58080, 58048, 58016, 57984, 57952, 57920, 57888, 57856, 57820, 57784, 57756, 57728, 57692, 57664, 57628, 57592, 57564, 57532, 57500, 57472, 57456, 57420, 57384, 57348, 57320, 57300, 57248, 57220, 57184, 57160, 57120, 57108, 57076, 57044, 57004, 56972, 56924, 56876, 56868, 56832, 56804, 56752, 56716, 56672, 56672, 56640, 56608, 56588, 56556, 56524, 56492, 56460, 56428, 56396, 56364, 56332, 56300, 56268, 56236, 56204, 56172, 56140, 56108, 56076, 56032, 56012, 55980, 55948, 55916, 55880, 55852, 55816, 55780, 55744, 55724, 55692, 55660, 55628, 55592, 55564, 55532, 55484, 55464, 55420, 55412, 55380, 55344, 55308, 55268, 55232, 55216, 55184, 55148, 55112, 55076, 55056, 55024, 54972, 54960, 54908, 54872, 54840, 54808, 54772, 54740, 54708, 54676, 54644, 54612, 54580, 54548, 54516, 54484, 54452, 54412, 54376, 54356, 54324, 54292, 54260, 54228, 54196, 54164, 54132, 54100, 54068, 54036, 54004, 53968, 53940, 53908, 53876, 53840, 53804, 53784, 53748, 53712, 53676, 53656, 53624, 53584, 53548, 53512, 53492, 53456, 53428, 53392, 53356, 53336, 53300, 53268, 53240, 53212, 53180, 53152, 53116, 53084, 53048, 53012, 52992, 52960, 52916, 52892, 52864, 52820, 52800, 52740, 52724, 52692, 52660, 52628, 52596, 52564, 52532, 52500, 52468, 52436, 52404, 52360, 52340, 52308, 52268, 52232, 52204, 52168, 52148, 52120, 52072, 52052, 52016, 51980, 51948, 51912, 51892, 51864, 51828, 51796, 51760, 51724, 51704, 51668, 51632, 51600, 51580, 51548, 51512, 51484, 51456, 51412, 51392, 51360, 51324, 51288, 51264, 51216, 51196, 51168, 51132, 51104, 51060, 51040, 50996, 50964, 50928, 50896, 50864, 50828, 50796, 50752, 50732, 50700, 50660, 50624, 50604, 50568, 50540, 50504, 50468, 50444, 50408, 50372, 50348, 50312, 50284, 50248, 50212, 50176, 50156, 50124, 50088, 50060, 50024, 49996, 49960, 49924, 49900, 49864, 49828, 49804, 49772, 49740, 49708, 49676, 49644, 49604, 49580, 49548, 49512, 49476, 49448, 49420, 49388, 49356, 49324, 49292, 49260, 49228, 49196, 49164, 49132, 49100, 49068, 49036],
    "espmem_after_env": [50364, 51040, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51376, 51372, 51376, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556, 51560, 51556],
    "espmem_after": [58900, 58592, 58552, 58480, 58420, 58412, 58356, 58336, 58304, 58272, 58256, 58236, 58204, 58172, 58140, 58108, 58076, 58044, 58012, 57980, 57948, 57916, 57884, 57852, 57820, 57784, 57740, 57720, 57692, 57648, 57628, 57592, 57556, 57528, 57496, 57464, 57436, 57420, 57384, 57348, 57312, 57284, 57264, 57212, 57184, 57148, 57124, 57084, 57072, 57040, 57008, 56968, 56936, 56888, 56840, 56832, 56796, 56768, 56716, 56680, 56636, 56636, 56604, 56572, 56552, 56520, 56488, 56456, 56424, 56392, 56360, 56316, 56296, 56264, 56232, 56200, 56156, 56132, 56104, 56064, 56040, 55996, 55976, 55944, 55912, 55880, 55844, 55816, 55780, 55744, 55708, 55688, 55652, 55624, 55592, 55556, 55528, 55484, 55448, 55420, 55384, 55376, 55344, 55308, 55272, 55232, 55196, 55180, 55148, 55112, 55076, 55040, 55020, 54988, 54936, 54924, 54872, 54836, 54804, 54772, 54736, 54704, 54672, 54640, 54608, 54576, 54544, 54512, 54480, 54448, 54416, 54376, 54340, 54320, 54288, 54256, 54224, 54192, 54160, 54128, 54096, 54064, 54024, 54000, 53968, 53932, 53904, 53872, 53840, 53804, 53768, 53748, 53712, 53676, 53640, 53620, 53588, 53548, 53512, 53476, 53456, 53420, 53392, 53356, 53320, 53300, 53264, 53232, 53204, 53176, 53144, 53116, 53080, 53048, 53012, 52972, 52956, 52916, 52880, 52856, 52824, 52784, 52756, 52704, 52688, 52656, 52624, 52592, 52560, 52528, 52496, 52464, 52432, 52400, 52360, 52324, 52304, 52268, 52232, 52196, 52168, 52132, 52112, 52084, 52036, 52016, 51980, 51944, 51912, 51876, 51856, 51828, 51792, 51760, 51724, 51688, 51668, 51632, 51596, 51564, 51544, 51512, 51476, 51448, 51420, 51364, 51356, 51324, 51288, 51252, 51216, 51180, 51160, 51132, 51096, 51068, 51024, 51004, 50960, 50928, 50892, 50860, 50828, 50792, 50752, 50716, 50696, 50660, 50624, 50588, 50568, 50532, 50504, 50468, 50432, 50408, 50372, 50336, 50312, 50276, 50248, 50212, 50176, 50140, 50120, 50088, 50052, 50024, 49988, 49960, 49924, 49888, 49864, 49828, 49792, 49768, 49736, 49704, 49660, 49640, 49604, 49568, 49544, 49512, 49476, 49440, 49412, 49384, 49352, 49320, 49288, 49256, 49224, 49192, 49160, 49128, 49096, 49064, 49032, 49000]
}

Success!

Still there is some signal loss wo/ .env.
Did monitor it, about 2-3 networks more than average were lost.

But hey it works!

[anecdata]

Do we have any thoughts about what got fixed in the esp-idf (or why it worked with Web Workflow enabled)? Since this worked with Web Workflow on, but didn't with it off, it seems like there is some different use of the esp-idf API between those two flows that could still come back to bite us later.

[dhalbert]

When the web workflow is in use, the wifi stays initialized: it does not get deinited and reinited This is the same scenario as doing a scan repeatedly in a single startup of the program, which worked fine previously.

I saw several ESP-IDF commits that sound like they might be related to this. GIven that it did work in 4.4.1, I think it got broken and then repaired again.

[jepler]

I have no way of assessing the esp-idf changes but the rest seems fine.

I don't know what to do about the picow scanning limitation except to document, as you have.

[jepler]

As far as I can tell there's no way to set the channels to scan in picow. some internal default is always used. It may be controlled by the country setting, which we never change.