v2.9.5 Breaks my Setup and constantly get HHTP 502 BadGateway when I try to login

[tucker19]

Overnight Watchtower upgraded me from 2.9.4 to 2.9.5 and starting seeing container marked as Unhealthy within Portainarr. I see under health checks this error parse error: Invalid numeric literal at line 1, column 7 NOT OK and within my logs I just see these errors

ENV:
OS - Ubuntu 20.4 LTS
Docker - 2.4
DB - ghcr.io/linuxserver/mariadb

[7/20/2021] [12:04:18 PM] [Global   ] › ✖  error     Command failed: logrotate /etc/logrotate.d/nginx-proxy-manager
error: skipping "/data/logs/fallback_access.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.
error: skipping "/data/logs/fallback_error.log" because parent directory has insecure permissions (It's world writable or writable by group which is not "root") Set "su" directive in config file to tell logrotate which user/group should be used for rotation.

[chaptergy]

See #1249. Could you check what the owner of your files is?

[tucker19]

The directory for nginx-proxy-manager when I ls -al shows drwxrwxrwx and is owned by the user I am running Docker within. Same userID and groupID as I pass in within my compose

[chaptergy]

I mean the owner of the /data/logs directory inside the container.

But I am confused about your errors. From what I can see you are talking about three very separate issues. In the title you mention a 502 error when trying to log in, which sounds like a database issue where the database is not available.
Then there is the unhealty information, which I can't exactly say what causes the issue, if the database is unavailable it could be because of that.
The last thing is the logrotate not working, but this is not a breaking error at all, just a minor inconvenience. I was just talking about that last issue when asking about the owner of the folders.

Here's what you could check about the first two errors:

1. The healthcheck just checks what the api returns. So open your NPM frontend and change the url path to /api, e.g. http://domain.com:81/api and see what that returns.
2. Check the log of your database container, to see if it might have any errors which prevents the database from starting
3. If none of the above seem to contain any relevant information, please post the full log of your npm container, as logrotate is not the culprit here.

[PlasmatikSteak]

When I navigate to http://domain.com:81/api i am greeted with a 502 Bad Gateway message

Only the npm_app_1 is marked as unhealthy.

CONTAINER ID   IMAGE          COMMAND             CREATED          STATUS                      PORTS                                                                                  NAMES
81f22a38d81a   f4f8dac2255c   "/init"             22 minutes ago   Up 22 minutes (unhealthy)   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   npm_app_1
848b2d4772ea   174fbfd81bac   "/scripts/run.sh"   24 minutes ago   Up 24 minutes               3306/tcp                                                                               npm_db_1

[chaptergy]

Hm, maybe it is related to the logrotate. Could one of you please try the following:

1. Open a shell on the docker container through docker exec -ti <CONTAINER_ID> bash or portainer
2. Install the nano text editor by running apt update && apt install nano -y
3. Open the setup.js file by running nano /app/setup.js
4. Scroll to the very bottom and you should find

   const runLogrotate = async () => {
       await utils.exec('logrotate /etc/logrotate.d/nginx-proxy-manager');
       logger.info('Logrotate completed.');
   };

5. Replace it with

   const runLogrotate = async () => {
       try {
           await utils.exec('logrotate /etc/logrotate.d/nginx-proxy-manager');
           logger.info('Logrotate completed.');
       } catch (e) { logger.warn(e) }
   };

See if that solves the problem.

[PlasmatikSteak]

That did it!

The api does not return 502 Bad Gateway and the container is now marked as healthy :D

CONTAINER ID   IMAGE          COMMAND             CREATED       STATUS                        PORTS                                                                                  NAMES
81f22a38d81a   f4f8dac2255c   "/init"             6 hours ago   Up About a minute (healthy)   0.0.0.0:80-81->80-81/tcp, :::80-81->80-81/tcp, 0.0.0.0:443->443/tcp, :::443->443/tcp   npm_app_1
848b2d4772ea   174fbfd81bac   "/scripts/run.sh"   6 hours ago   Up 6 hours                    3306/tcp                                                                               npm_db_1

Thank you so much!

[tucker19]

I will try this as well when I get some downtime from work. Crazy few days.

[tucker19]

Hm, maybe it is related to the logrotate. Could one of you please try the following:

1. Open a shell on the docker container through `docker exec -ti <CONTAINER_ID> bash` or portainer

2. Install the nano text editor by running `apt update && apt install nano -y`

3. Open the `setup.js` file by running `nano /app/setup.js`

4. Scroll to the very bottom and you should find
   ```js
   const runLogrotate = async () => {
       await utils.exec('logrotate /etc/logrotate.d/nginx-proxy-manager');
       logger.info('Logrotate completed.');
   };
   ```

5. Replace it with
   ```js
   const runLogrotate = async () => {
       try {
           await utils.exec('logrotate /etc/logrotate.d/nginx-proxy-manager');
           logger.info('Logrotate completed.');
       } catch (e) { logger.warn(e) }
   };
   ```

See if that solves the problem.

I can confirm this fixes it on my Docker as well but I am seeing these messages now in Portainer

❯ Enabling IPV6 in hosts: /etc/nginx/conf.d
  ❯ /etc/nginx/conf.d/include/ssl-ciphers.conf
  ❯ /etc/nginx/conf.d/include/assets.conf
  ❯ /etc/nginx/conf.d/include/force-ssl.conf
  ❯ /etc/nginx/conf.d/include/block-exploits.conf
  ❯ /etc/nginx/conf.d/include/letsencrypt-acme-challenge.conf
  ❯ /etc/nginx/conf.d/include/ip_ranges.conf
  ❯ /etc/nginx/conf.d/include/proxy.conf
  ❯ /etc/nginx/conf.d/include/resolvers.conf
  ❯ /etc/nginx/conf.d/production.conf
  ❯ /etc/nginx/conf.d/default.conf
❯ Enabling IPV6 in hosts: /data/nginx
[7/22/2021] [10:14:16 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:17 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:18 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:19 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:20 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:21 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:22 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307
[7/22/2021] [10:14:23 AM] [Global   ] › ✖  error     connect ECONNREFUSED 172.30.0.19:3307

I do have my ports mapped to 981:81 and then I try and do a <IP>:981/api it will redirect to <IP>:81/api and give me a HTTP 302. I still get the Bad Gateway when I just try and log in with the default admin@example.com and this is the initially setup of a new container.

[chaptergy]

Your remaining issue is definitely a database issue. The database does not respond. You need to check why your database is not running.

[tucker19]

Okay, thanks for that info. I suspected that.

[tucker19]

Your remaining issue is definitely a database issue. The database does not respond. You need to check why your database is not running.

Found my errors in db. I have confirmed all my stuff is working now. Thank you for all the help.