Error at generating PlatformIO manifest

[HamzaHajeir]

Hi There

The ./build.sh script ends with:

fatal: ref HEAD is not a symbolic ref
usage: gen_platformio_manifest.py [-h] -o DST_DIR -s VERSION_STRING -c COMMIT_HASH
gen_platformio_manifest.py: error: argument -s/--version-string: expected one argument

It seems to me, however, related to the patches done to esp_eth component.

$ git symbolic-ref --short HEAD
fatal: ref HEAD is not a symbolic ref

and:

$ git status
HEAD detached at cb174b0fe1
Changes not staged for commit:
(use "git add ..." to update what will be committed)
(use "git restore ..." to discard changes in working directory)
modified: components/esp_eth/include/esp_eth_mac.h
modified: components/esp_eth/src/esp_eth_mac_dm9051.c
modified: components/esp_eth/src/esp_eth_mac_ksz8851snl.c
modified: components/esp_eth/src/esp_eth_mac_w5500.c

no changes added to commit (use "git add" and/or "git commit -a")

I've temporarily override it by a manual generation.

[krdarrah]

Same issue here - this works:
./build.sh -t esp32c3 -b build

this does not:
./build.sh -t esp32c3

Get this:
usage: gen_platformio_manifest.py [-h] -o DST_DIR -s VERSION_STRING -c COMMIT_HASH gen_platformio_manifest.py: error: argument -s/--version-string: expected one argument

[HamzaHajeir]

Hi @krdarrah

It's the issue of this command:

esp32-arduino-lib-builder/build.sh

Line 234 in c39e942

python3 ./tools/gen_platformio_manifest.py -o "$TOOLS_JSON_OUT/" -s $(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD) -c $(git -C "$IDF_PATH" rev-parse --short HEAD)

Wherein git in esp-idf repository failed to symbolic-ref, so I've manually called the script providing mockup argument.

python3 ./tools/gen_platformio_manifest.py -o ./out/tools/esp32-arduino-libs -s release/v5.1 -c $(git -C esp-idf rev-parse --short HEAD)

[me-no-dev]

how did you guys setup ESP-IDF for this command to fail?

[HamzaHajeir]

It was the responsibility of ./build.sh script, with provided esp-idf branch `release/v5.1`

…

On Fri, Oct 13, 2023, 10:19 Me No Dev ***@***.***> wrote: how did you guys setup ESP-IDF for this command to fail? — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3O7J43HABVPFXADU6MND3X7DTQPANCNFSM6AAAAAA5XXYEPU> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[me-no-dev]

that is rather strange, given that CI runs that same script and it works fine. if you run git -C $IDF_PATH symbolic-ref manually, what does it return?

[HamzaHajeir]

It does return this line:

fatal: ref HEAD is not a symbolic ref

..

…

On Fri, Oct 13, 2023, 11:17 Me No Dev ***@***.***> wrote: that is rather strange, given that CI runs that same script and it works fine. if you run git -C $IDF_PATH symbolic-ref manually, what does it return? — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3O7J4MLHTTBVY6RXDPSSLX7D2KXAVCNFSM6AAAAAA5XXYEPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTONRRGEYDEOBWGE> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[Jason2866]

Can confirm this issue. It works locally the first run. All runs after do fail here.
The issue never happens with Github Actions, since it is always "the first run"

[me-no-dev]

interesting... what would be different? I am using the same scripts here. @Jason2866 what does git status say?

[brentru]

@me-no-dev Having a similar issue as discussed above, except running ./build.sh fails at:

-- Generating done (1.2s)
-- Build files have been written to: /Users/brentrubell/Desktop/github_brentru/esp32-arduino-lib-builder/build
make: *** No rule to make target `idf_libs'.  Stop.
command "idf_libs" is not known to idf.py and is not a Unix Makefiles target

Edit: this is from not having ninja installed and doesn't error since installing.

I also think it may be due to the patches within esp_eth as they require a manual confirmation prompt the second time you run the script.

I've included the script output on this gist and highlighted the line with the symbolic ref issue: https://gist.github.com/brentru/aa9acf05b808575b753f6e461509bf35#file-gistfile1-txt-L281

Everything is configured on a local machine running macOS and bash shell.

[me-no-dev]

@brentru you must be running the script from the wrong folder. Rules for idf_libs are defined in the Arduino component

[Jason2866]

@me-no-dev

/Volumes/T7-Mac/esp32-arduino-lib-builder/out/tools/esp32-arduino-libs/tools.json generated
Schwerwiegend: ref HEAD is not a symbolic ref
usage: gen_platformio_manifest.py [-h] -o DST_DIR -s VERSION_STRING -c COMMIT_HASH
gen_platformio_manifest.py: error: argument -s/--version-string: expected one argument

hans@MacBook-Air-von-Johann esp32-arduino-lib-builder % git status
Auf Branch v5.1_no_SECP256R1
Ihr Branch ist auf demselben Stand wie 'origin/v5.1_no_SECP256R1'.

Unversionierte Dateien:
  (benutzen Sie "git add <Datei>...", um die Änderungen zum Commit vorzumerken)
	release-info.txt

nichts zum Commit vorgemerkt, aber es gibt unversionierte Dateien
(benutzen Sie "git add" zum Versionieren)

[me-no-dev]

I can not get myself in this situation... everything works. I do have an idea though... it will build but the PIO manifest might need editing.
Around line 232 of build.sh edit to look like this:

# Generate PlatformIO manifest file
if [ "$BUILD_TYPE" = "all" ]; then
    ibr=$(git -C "$IDF_PATH" symbolic-ref --short HEAD || git -C "$IDF_PATH" tag --points-at HEAD)
    ic=$(git -C "$IDF_PATH" rev-parse --short HEAD)
    python3 ./tools/gen_platformio_manifest.py -o "$TOOLS_JSON_OUT/" -s "$ibr" -c "$ic"
    if [ $? -ne 0 ]; then exit 1; fi
fi

[Jason2866]

@me-no-dev Not yet

/Volumes/T7-Mac/esp32-arduino-lib-builder/out/tools/esp32-arduino-libs/tools.json generated
Schwerwiegend: ref HEAD is not a symbolic ref
Failed to find a regex match for 'v(?P<MAJOR>0|[1-9]\d*)\.(?P<MINOR>0|[1-9]\d*)\.*(?P<PATCH>0|[1-9]\d*)*' in ''
Failed to convert version ''

[me-no-dev]

well... for some reason IDF is not on a properly named branch. So PIO can not extract the correct version. Maybe just comment the call... I can test if it's empty string and skip it... but why is it even happening? What did you guys do to your IDFs?

[Jason2866]

What did you guys do to your IDFs?

Happens with orig. espressif repo and branch for me too. Anyways not a big deal for me. In CI it is working (since first run). Building local i just do what you suggested -> commenting out that part.
For building local it is more anyoing that the Ethernet patch part needs human interaction what to do.

[me-no-dev]

it's not the patches, because I run the same thing here (not the first run also), so still clueless... the patch only causes a suffix -dirty to be added to the commit hash

[Jason2866]

I didn't thought the patches break the json generating. After a run the patch is applied and it stops asking the next run what to do, applying the Ethernet patch again or not.

[me-no-dev]

patch should be applied only once. I already have it here. If it tries to apply it again, that would/should fail. Logic is not perfect, but TBH it's meant to run in CI. I really do not understand in what state has ESP-IDF ended for commands to fail so badly.

[brentru]

After a run the patch is applied and it stops asking the next run what to do, applying the patch again or not.

I run into the same issue as well.

@brentru you must be running the script from the wrong folder. Rules for idf_libs are defined in the Arduino component

Seems like the cause of this error was by not having ninja-dev installed prior to running IDF.

[jefflongo]

Also cannot figure out how to build this locally more than once.

[doncogit]

Guys, please, can anyone help me? iam trying for 3 days and 40hours to make this work, iam so desperate. iam constantly getting the same errors like discussed here

fatal: "ref HEAD is not a symbolic ref"

i just setup a fresh system of ubuntu 23.10.1 on a virtual machine. No espressif IDF or anything pre-installed.

i made every step of the documentation, installed everything.
i mean is there anything which i can do wrong?

but still iam getting errors and it seems the script loops forever?
the only thing what i want to do is to make a some edits with menuconfig so i can have custom settings for my esp32s3 in my arduino ide.

This lib builder does that right?

kind regards!

[Jason2866]

@doncogit What error(s) do you get exactly. Logs? Maybe easiest way. Fork this repo and do the changes in the config files you want. The GH Actions https://github.com/espressif/esp32-arduino-lib-builder/blob/master/.github/workflows/push.yml
will build the framework and uploads it as artifact. You can download the artifact and use it :-)

[doncogit]

@doncogit What error(s) do you get exactly. Logs? Maybe easiest way. Fork this repo and do the changes in the config files you want. The GH Actions https://github.com/espressif/esp32-arduino-lib-builder/blob/master/.github/workflows/push.yml will build the framework and uploads it as artifact. You can download the artifact and use it :-)

thanks for your fast reply. i dont even know how to do a logfile. i made a screenshot of the beginning where already some errors exist. if you tell me how to do it i can provide more information. Sorry iam a beginner.

also i never heard about your other suggestion but it sounds interesting :-D

[doncogit]

when i use your suggestion through forking and using GH. How can i edit the sdkconfig file? As i cant find it in this repo?

EDIT: ok settings should be set in the defconfig file of the chip i think

[Jason2866]

Yes, look through the defconfig* files in this folder https://github.com/espressif/esp32-arduino-lib-builder/tree/master/configs
and do the changes you need. When valid for every MCU in defconfig.common

Btw. fatal: ref HEAD is not a symbolic ref is misleading it is not an error just an info.
The screenshot does show no error. As long it does not stop, it is working!

[doncogit]

thanks a lot. i downloaded the artefacts now containing folders for several chips. I unpacked the tar.gz and the containting .tar.

so now i will copy the folder content of my chip to Arduino15\packages\esp32\hardware\esp32\2.0.11\tools\sdk\esp32s3

and overwrite everything. then it has my new settings? is that correct?

[Jason2866]

@doncogit You should overwrite everything. Not just the path Arduino15\packages\esp32\hardware\esp32\2.0.11\tools\sdk\esp32s3

[doncogit]

this is done all by hand?
unpacking everything and searching the matching folders in arduino?

[Jason2866]

Just delete all old and copy the new. The build structure does not differ.
So done in 2 minutes.

[doncogit]

guys appreciated your work and everything but the workflow seems not clear to me, iam an idiot and iam desperate.

i forked it and did the GA thing, now for testing without my modifications.
The GA got me the 3 files as posted in the screenshot before.
Jason you told me i should download the artefacts file and i did. (i only downloaded artefacts and not the other two)
You told me the folder structure does not differ but it does.
the most matching folders i found:

now on windows:
C:\Users\user\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.11\tools\

there is a folder sdk containing

the artefacts folder has the folder tools\esp32-arduino-libs

so i copied and overwrote all files from the artefacts\tools\esp32-arduino-libs\ to C:\Users\user\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.11\tools\sdk\

(didnt delete the old folders. i tried that too and got even more errors like not finding freertos.h etc.)

then i restartet arduino ide 2.1.1, tried to compile and i got errors.

C:\Users\user\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.11/tools/sdk/esp32s3/include/driver/include/driver/gpio.h:71:29: error: 'gpio_config_t' does not name a type; did you mean 'gpio_mode_t'?
 esp_err_t gpio_config(const gpio_config_t *pGPIOConfig);

C:\Users\user\AppData\Local\Arduino15\packages\esp32\hardware\esp32\2.0.11/tools/sdk/esp32s3/include/driver/include/driver/gpio.h:326:53: error: 'gpio_isr_t' has not been declared
 esp_err_t gpio_isr_handler_add(gpio_num_t gpio_num, gpio_isr_t isr_handler, void *args);

am i missing a simple step which is clear for you pro's but not for me?

kind regards

[jefflongo]

I also struggled initially to create a custom build of arduino-esp32, I found the process pretty unintuitive. I was eventually able to get things building locally, I'll share how I got it working. Note these instructions are on linux, but the instructions are pretty similar for Windows, you can figure it out.

1. Install dependencies:

sudo apt-get install git wget curl libssl-dev libncurses-dev flex bison gperf cmake ninja-build ccache jq python3
pip install --upgrade pip
pip install --user setuptools pyserial click cryptography future pyparsing pyelftools

2. Clone the repo and cd into it:

git clone https://github.com/espressif/esp32-arduino-lib-builder
cd esp32-arduino-lib-builder

3. Checkout the appropriate tag. This is probably where you're running into problems. You must know which version of esp-idf your version of anduino-esp32 is targeting. In my case, I'm using arduino-esp32 2.x which targets esp-idf 4.4.x.

git checkout release/v4.4

4. Now make whatever changes to configs/defconfig.<YOUR-ESP32>

5. Run the build, specifying the ESP32 you're targeting, the esp32-arduino-lib-builder branch, and the esp-idf version

./build.sh -t <YOUR-ESP32> -I release/v4.4 -i v4.4.6

6. Replace the files from your local install

cp -rf out/tools/. $HOME/.arduino15/packages/esp32/hardware/esp32/<YOUR-ARDUINO-ESP32-VERSION>/tools

[doncogit]

thanks jeff but i gave up on linux, i tried with a virtual machine and the setup was horrible. at the end i didnt even manage to install arduino IDE 2 on linux other than the weird portable version.

[jefflongo]

Regardless, the instructions should mostly apply to Windows, besides changing some paths. Your issue is probably having the builder use the wrong combination of arduino-esp32 and esp-idf as I outlined in my previous post.

[doncogit]

shouldn't any version work fine, as iam building the whole core from ground up?

Iam able to build something with the Github actions and forked repo but dont know where to put the files as mentioned. I tried several possibilities and nothing worked. Maybe i understand it wrong and should fork 4.4 instead...

[HamzaHajeir]

Not really, major version releases usually holds API changes and sometimes incompatibility with the previous version.

…

On Fri, Feb 2, 2024, 04:00 doncogit ***@***.***> wrote: shouldn't any version work fine, as iam building the whole core from ground up? Iam able to build something with the Github actions and forked repo but dont know where to put the files as mentioned. I tried several possibilities and nothing worked. Maybe i understand it wrong and should fork 4.4 instead... — Reply to this email directly, view it on GitHub <#139 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AH3O7JYR2X455KJFCQNMUKDYRQ3BNAVCNFSM6AAAAAA5XXYEPWVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMRSGU4TKNBWGM> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[doncogit]

workflow doesnt even work for v4.4 branch so jason couldnt mean that.

@Jason2866 may you help with my previous question in the post above about the folder etc.? i have the feeling iam so close to the goal but missing the last step.

[Jason2866]

@doncogit Ahh, forgot that i do it different. Nothing needs to be replaced.
Download the artifact extract and upload to your new github repo like espressif does here https://github.com/espressif/esp32-arduino-libs -> branch idf-release/v5.1
Fork espressif Arduino and change file https://github.com/espressif/arduino-esp32/tree/master/package/package_esp32_index.template.json in your!! fork with the changed values for URLs, Checksum and size. Maybe you can delete the Checksum and size entrys. To save work to be done.
The long hex value in the URL (example "url": "https://codeload.github.com/espressif/esp32-arduino-libs/zip/1a373b86ea98023ebebbfb3b23ad8100376ba946") is the corresponding commit sha in your repo esp32-arduino-libs branch idf-release/v5.1
When all this is done install espressif32 Arduino from your fork. Now it should load and install the precompiled libs from your fork too.

I build my framework only for Platformio and integrate the compiled libs there. Don't like the hassle to upload and download the libs from somewhere.

[StevenCellist]

Hi folks, I am struggling with apparently the same issue. The arduino-esp32 repo now includes Zigbee examples that I want to check out - for that, I need to build an Arduino release as far as I understand. So I installed the lib builder here and got to the same error with the PIO manifest.
Given that I want Zigbee for ESP32-C6 I assume I also need IDF 5.1, but I couldn't follow the above guide, as I am not using a forked repo; my intention was to just build locally once now and be able to get going with my students.
@Jason2866 could you please explain if how I want to do this is correct, and/or how I could solve the error? Thanks in advance!

Here's the last part of the output for reference:

[1874/1874] Generating mem_variant
fatal: ref HEAD is not a symbolic ref
IDF_TARGET: esp32c6, MEMCONF: dio_qspi
* Skipping Target: esp32c3
* Skipping Target: esp32
* Skipping Target: esp32s2
* Skipping Target: esp32s3
fatal: ref HEAD is not a symbolic ref
Source /home/sb-laptop/esp32-arduino-lib-builder/components/arduino/package/package_esp32_index.template.json.
Found xtensa-esp-elf-gdb, version: 12.1_20231023
Updating dependency version of xtensa-esp-elf-gdb from 12.1_20221002 to 12.1_20231023
Updating binaries of xtensa-esp-elf-gdb to version 12.1_20231023
Found riscv32-esp-elf-gdb, version: 12.1_20231023
Updating dependency version of riscv32-esp-elf-gdb from 12.1_20221002 to 12.1_20231023
Updating binaries of riscv32-esp-elf-gdb to version 12.1_20231023
Found xtensa-esp32-elf-gcc, version: esp-12.2.0_20230208
Skipping xtensa-esp32-elf-gcc. Same version esp-12.2.0_20230208
Found xtensa-esp32s2-elf-gcc, version: esp-12.2.0_20230208
Skipping xtensa-esp32s2-elf-gcc. Same version esp-12.2.0_20230208
Found xtensa-esp32s3-elf-gcc, version: esp-12.2.0_20230208
Skipping xtensa-esp32s3-elf-gcc. Same version esp-12.2.0_20230208
Found riscv32-esp-elf-gcc, version: esp-12.2.0_20230208
Skipping riscv32-esp-elf-gcc. Same version esp-12.2.0_20230208
Found openocd-esp32, version: v0.12.0-esp32-20230921
Skipping openocd-esp32. Same version v0.12.0-esp32-20230921
/home/sb-laptop/esp32-arduino-lib-builder/out/package_esp32_index.template.json generated
Source was not selected
Found xtensa-esp-elf-gdb, version: 12.1_20231023
Adding dependency: xtensa-esp-elf-gdb version 12.1_20231023
Adding tool: xtensa-esp-elf-gdb version 12.1_20231023
Found riscv32-esp-elf-gdb, version: 12.1_20231023
Adding dependency: riscv32-esp-elf-gdb version 12.1_20231023
Adding tool: riscv32-esp-elf-gdb version 12.1_20231023
Found xtensa-esp32-elf-gcc, version: esp-12.2.0_20230208
Adding dependency: xtensa-esp32-elf-gcc version esp-12.2.0_20230208
Adding tool: xtensa-esp32-elf-gcc version esp-12.2.0_20230208
Found xtensa-esp32s2-elf-gcc, version: esp-12.2.0_20230208
Adding dependency: xtensa-esp32s2-elf-gcc version esp-12.2.0_20230208
Adding tool: xtensa-esp32s2-elf-gcc version esp-12.2.0_20230208
Found xtensa-esp32s3-elf-gcc, version: esp-12.2.0_20230208
Adding dependency: xtensa-esp32s3-elf-gcc version esp-12.2.0_20230208
Adding tool: xtensa-esp32s3-elf-gcc version esp-12.2.0_20230208
Found riscv32-esp-elf-gcc, version: esp-12.2.0_20230208
Adding dependency: riscv32-esp-elf-gcc version esp-12.2.0_20230208
Adding tool: riscv32-esp-elf-gcc version esp-12.2.0_20230208
Found openocd-esp32, version: v0.12.0-esp32-20230921
Adding dependency: openocd-esp32 version v0.12.0-esp32-20230921
Adding tool: openocd-esp32 version v0.12.0-esp32-20230921
/home/sb-laptop/esp32-arduino-lib-builder/out/tools/esp32-arduino-libs/tools.json generated
fatal: ref HEAD is not a symbolic ref
usage: gen_platformio_manifest.py [-h] -o DST_DIR -s VERSION_STRING -c COMMIT_HASH
gen_platformio_manifest.py: error: argument -s/--version-string: expected one argument

[Jason2866]

@StevenCellist Which IDE do you want to use for your built framework?
If it is Arduino IDE just comment the part where the Platformio manifest is build.
You don't need for ArduinoIDE use. Comment the Line 277 until L281
https://github.com/espressif/esp32-arduino-lib-builder/blob/master/build.sh#L277-L281

Didn't search the reason what is causing the error to build the Platformio manifest in second run. Do my builds with Github Actions, since it is always the first run there, not encountering the error.

[StevenCellist]

Thanks, I can live with the ArduinoIDE for now - it seems it's working with those lines commented. The compile now ends with:

/home/sb-laptop/esp32-arduino-lib-builder/out/tools/esp32-arduino-libs/tools.json generated

I will try and see if it is usable!