Intermittent error 504

[george1421]

@Kagashe First on the errors you see about the bandwidth.php, while its a problem its just an annoyance message at the moment.

The fpm message is a bit troubling and I need to look into it.

Your configuration is a bit abnormal (based on what we typically see) with the number of mirrors. That is fine, it might mean that we need to tweak our configuration a bit.

There is a timeout setting I would like you to make/change to see if we can address this issue. Specifically to tell apache to wait a bit longer for php-fpm to respond than its default settings. I can see the potential if you have your mirrors behind a slow network connection that it may take a little longer to get a response.

    <Proxy "fcgi://127.0.0.1:9000">
        ProxySet timeout=300
    </Proxy>

https://forums.fogproject.org/topic/11713/503-service-unavailable-error/40
and
https://forums.fogproject.org/topic/12057/fog-unresponsive-under-heavy-load/17

[Kagashe]

@george1421 Sorry haven’t answered been a hectic few days, have updated my config with the script you posted, will test it and feedback. Thanks for all the help so far, it’s much appreciated.

[Kagashe]

@george1421 This seems so have solved the issue so far, I’ve added to more mirrors to the mix and still haven’t had the 504 error again.

[george1421]

@Kagashe so what solved it. The timeout value being added to the apache config file?

[Kagashe]

@george1421 Yes that was the last change I made to the config.

[Kagashe]

@george1421 Aaaaaaand it just happened again

[Sebastian Roth]

@Kagashe Trying to reach you on chat. See the speech bubble in the top right corner.

[Sebastian Roth]

Marking as unsolved for now as the issue was worked around by George’s good advice to adjust proxy timeout but I am working on fixing the code to not have those very long timeouts at all.

[Sebastian Roth]

@Kagashe Ok here we go. Can you please upgrade all your nodes to the latest dev-branch version and see if you still get timeouts? I’d even suggest reverting the proxy timeout change (if not reverted by upgrage anyway) to see if the code improvements really do make a difference.

[Kagashe]

@Sebastian-Roth Have upgraded all the sites to 1.5.5.1 except one where I’m getting the following error

Adding Needed Repository…Failed!

Apart from that, all seems to be going well.

[Sebastian Roth]

@Kagashe Please check the install logs which reside in the bin directory from which you run the installer.

Are all your servers Ubuntu 16.04?

[Kagashe]

@Sebastian-Roth All servers are Ubuntu 16.04, below are the logs

/usr/bin/lsb_release
/bin/systemctl
Reading package lists...
Building dependency tree...
Reading state information...
ntpdate is already the newest version (1:4.2.8p4+dfsg-3ubuntu5.9).
software-properties-common is already the newest version (0.96.20.7).
python-software-properties is already the newest version (0.96.20.7).
The following packages were automatically installed and are no longer required:
  libapr1 libaprutil1 libaprutil1-dbd-sqlite3 libaprutil1-ldap libbrotli1
  libgd3 libjansson4 libjbig0 liblua5.2-0 libnghttp2-14 libtiff5 libwebp6
  php-common
Use 'sudo apt autoremove' to remove them.
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
29 Nov 13:40:51 ntpdate[8213]: adjust time server 46.17.63.196 offset -0.019258 sec
Generating locales (this might take a while)...
  en_US.UTF-8... done
Generation complete.
gpg: keyring `/tmp/tmpr79_9sh8/secring.gpg' created
gpg: keyring `/tmp/tmpr79_9sh8/pubring.gpg' created
gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
Error: retrieving gpg key timed out.
gpg: /tmp/tmpr79_9sh8/trustdb.gpg: trustdb created
gpg: key E5267A6C: public key "Launchpad PPA for Ondřej Surý" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK
gpg: keyring `/tmp/tmp0z4tf8uy/secring.gpg' created
gpg: keyring `/tmp/tmp0z4tf8uy/pubring.gpg' created
gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
Error: retrieving gpg key timed out.
gpg: /tmp/tmp0z4tf8uy/trustdb.gpg: trustdb created
gpg: key E5267A6C: public key "Launchpad PPA for Ondřej Surý" imported
gpg: Total number processed: 1
gpg:               imported: 1  (RSA: 1)
OK

[Sebastian Roth]

@Kagashe said in Intermittent error 504:

gpg: requesting key E5267A6C from hkp server keyserver.ubuntu.com
Error: retrieving gpg key timed out.

There is nothing we can do. It’s not able to retrieve the key from the server. Probably a firewall is blocking that request?

To get around this you can edit the script code fogproject/lib/common/functions.sh, jump to line 601 and comment that so it looks like this.

...
            esac
            ;;
    esac
#    errorStat $?
    dots "Preparing Package Manager"
    $packmanUpdate >>$workingdir/error_logs/fog_error_${version}.log 2>&1
...

Now rerun the installer and you should get past the repo stuff.

[Kagashe]

@Sebastian-Roth Weird I figured that was the issue, so I whitelisted the machine on my firewall just to test but still didn’t work and there’s nothin in the firewall logs to suggest that it’s blocking anything, it’s a strange one…
Will give your suggestion a go now.

[Kagashe]

@Kagashe Editing the script solved it, thanks Sebastian.