www.php.net blocks "fetch libfetch/2.0" user-agent

[CyberCr33p]

Description

This is what I get when I try to upgrade PHP from 7.4.26 to 7.4.27 using FreeBSD ports:

=> php-7.4.27.tar.xz doesn't seem to exist in /usr/ports/distfiles//.
=> Attempting to fetch https://www.php.net/distributions/php-7.4.27.tar.xz
fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable
=> Attempting to fetch http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz
fetch: http://distcache.FreeBSD.org/ports-distfiles/php-7.4.27.tar.xz: Not Found
=> Couldn't fetch it - please try to retrieve this
=> port manually into /usr/ports/distfiles// and try again.

---

FreeBSD ports uses "fetch" to download https://www.php.net/distributions/php-7.4.27.tar.xz but looks like www.php.net web-server blocks user agent "fetch libfetch/2.0" and returns "Service Unavailable".

Can you whitelist this user agent?

PHP Version

PHP 7.4

Operating System

FreeBSD

[cmb69]

curl -A "fetch libfetch/2.0" --output php-7.4.27.tar.xz https://www.php.net/distributions/php-7.4.27.tar.xz works for me. What am I missing?

[cmb69]

Is this possibly certificate related?

[dsh2dsh]

I can confirm:

❯ fetch https://www.php.net/distributions/php-7.4.27.tar.xz
fetch: https://www.php.net/distributions/php-7.4.27.tar.xz: Service Unavailable

[olegsidorkin]

wget https://www.php.net/distributions/php-7.4.27.tar.xz - does work
fetch -vvvv https://www.php.net/distributions/php-7.4.27.tar.xz - doesn't. It successfully establishes connection and then fails:

Security CheckOur systems have detected unusual traffic from your computer network. This page checks to see if it is really you sending the requests and not a bot.Please click here to verify that you are not a bot. Browser Working Myra Working Host WorkingWhat happened?You ran into a security check to verify the validity of your request.What can I do?If you are a visitor of this website:You must confirm that you are human.If you are the owner of this website:Please check your security settings.

Headers are:
`resolving server address: www.php.net:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.php.net
Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
requesting https://www.php.net/distributions/php-7.4.27.tar.xz

GET /distributions/php-7.4.27.tar.xz HTTP/1.1
Host: www.php.net
Accept: /
User-Agent: fetch libfetch/2.0
Connection: close

<<< HTTP/1.1 503 Service Temporarily Unavailable
<<< Server: myracloud
<<< Date: Sun, 26 Dec 2021 16:05:06 GMT
<<< Content-Type: text/html
<<< Transfer-Encoding: chunked
<<< Connection: close
<<< cache-control: no-cache, no-store, max-age=0
<<< X-Frame-Options: SAMEORIGIN
<<< X-Content-Type-Options: nosniff
<<< X-XSS-Protection: 1; mode=block`

[cmb69]

Ah, so this would be a systems issue. Maybe @saschaschumann can have a look?

[5u623l20]

I have tried changing user agent of fetch too with no results:
fetch -vvvvv --user-agent="curl/7.80.0" https://www.php.net/distributions/php-8.1.1.tar.xz
scheme: "https"
user: ""
password: ""
host: "www.php.net"
port: "0"
document: "/distributions/php-8.1.1.tar.xz"
---> www.php.net:443
resolving server address: www.php.net:443
SSL options: 82004854
Peer verification enabled
Using CA cert file: /usr/local/etc/ssl/cert.pem
Verify hostname
TLSv1.2 connection established using ECDHE-RSA-AES128-GCM-SHA256
Certificate subject: /CN=*.php.net
Certificate issuer: /C=PL/O=Unizeto Technologies S.A./OU=Certum Certification Authority/CN=Certum Domain Validation CA SHA2
requesting https://www.php.net/distributions/php-8.1.1.tar.xz

GET /distributions/php-8.1.1.tar.xz HTTP/1.1
Host: www.php.net
Accept: /
User-Agent: curl/7.80.0
Connection: close

<<< HTTP/1.1 503 Service Temporarily Unavailable
<<< Server: myracloud
<<< Date: Sun, 26 Dec 2021 16:30:08 GMT
<<< Content-Type: text/html
<<< Transfer-Encoding: chunked
<<< Connection: close
<<< cache-control: no-cache, no-store, max-age=0
<<< X-Frame-Options: SAMEORIGIN
<<< X-Content-Type-Options: nosniff
<<< X-XSS-Protection: 1; mode=block
<<<
offset 0, length -1, size -1, clength -1
<<< 7738

[5u623l20]

In case someone is already investigating I believe the problem is with http1.1 vs http2. In case we try curl with --http1.1 it also fails with the same errors.

[amezin]

It's not only libfetch

Sometimes, node.js also gets blocked nodejs/help#4516 - seemingly by TLS fingerprints. It gets blocked, then, after a few days, unblocked, then some day blocked again.

If blocking by TLS fingerprint, switching to a different HTTP version likely won't help (although I can't recall if I tried it last time).

Is it really not possible to tweak the configuration not to block PHP downloads (both tarballs and the https://www.php.net/downloads.php)?

What's the point of having things like https://www.php.net/releases/index.php?json if automatic fetches get blocked anyway

[derickr]

Our CDN sometimes realises big spikes from networks, and then puts this up. I can turn this off, but only for all urls, not specific ones. And I don't really want to do that, so I'm going to have to close this ticket.