Updating from SVN 3731 to GIT 4250 HTTP500

[sudburr]

Running Debian 8.1.0, I updated my SVN 3731 to GIT 4250.

At the Press Enter key when database is updated/installed. prompt all I see is an HTTP 500 Internal Server Error.
Pressing forwards with a Y response, the installation proceeds but web access remains HTTP 500 Internal Server Error. Checking the Apache error.log I find these entries of note:

:error pid 48093 client 172.19.252.91:52645 PHP Warning:  http_build_query(): Parameter 1 expected to be Array or Object.  Incorrect value given in /var/www/fog/lib/fog/FOGURLRequests.class.php on line 60, referer: http://devfog/fog/management/index.php?node=about&sub=kernel&install=1&file=aHR0cDovL2Rvd25sb2Fkcy5zb3VyY2Vmb3JnZS5uZXQvcHJvamVjdC9mcmVlZ2hvc3QvS2VybmVsTGlzdC9LZXJuZWwuVG9tRWxsaW90dC4zLjE4LjUuNjQ=
ssl:warn pid 644 AH01909: 172.19.244.13:443:0 server certificate does NOT include an ID which matches the server name
:error pid 13817 client 172.19.252.91:53620 PHP Fatal error:  Call to undefined method FOGCore::cleanInvalidEntries() in /var/www/html/fog/commons/init.php on line 179

For giggles I rolled back the installation by re-installing SVN 3731 and the problem persists.

Re-installation also generated this message immediately prior to the Press Enter key when database is updated/installed.

Downloading New FOG Client file.............................Backgrounded
ln: failed to create symbolic link ‘/var/www/html/fog/fog’: File exists

[Wayne Workman]

I think this is the same problem as here: https://forums.fogproject.org/topic/5548/internal-500-error-in-fog-client

[sudburr]

Getting this in 4252, still working on top of munged system.

Setting up and starting TFTP and PXE Servers................Failed!

[sudburr]

@Wayne-Workman said:

I think this is the same problem as here: https://forums.fogproject.org/topic/5548/internal-500-error-in-fog-client

Checking this out.

[sudburr]

Well the other thread is also unresolved, and my problems start with the server.

[sudburr]

Fooling around with the results …

I tried

systemctl restart tftp* mysql* FOG* apache*

and got

Job for tftpd-hpa.service failed. See 'systemctl status tftpd-hpa.service' and 'journalctl -xn' for details.

… which got me

 tftpd-hpa.service - LSB: HPA's tftp server
   Loaded: loaded (/etc/init.d/tftpd-hpa)
   Active: failed (Result: exit-code) since Wed 2015-07-29 16:28:05 EDT; 3min 6s ago
  Process: 8289 ExecStop=/etc/init.d/tftpd-hpa stop (code=exited, status=0/SUCCESS)
  Process: 8335 ExecStart=/etc/init.d/tftpd-hpa start (code=exited, status=71)

Jul 29 16:28:04 devfog systemd[1]: tftpd-hpa.service: control process exited, code=exited status=71
Jul 29 16:28:05 devfog systemd[1]: Failed to start LSB: HPA's tftp server.
Jul 29 16:28:05 devfog systemd[1]: Unit tftpd-hpa.service entered failed state.
Jul 29 16:28:12 devfog tftpd-hpa[8335]: Starting HPA's tftpd: in.tftpd

and

-- Logs begin at Wed 2015-07-29 13:01:02 EDT, end at Wed 2015-07-29 16:28:43 EDT. --
Jul 29 16:28:39 devfog mysql[8424]: Starting MySQL database server: mysqld . ..
Jul 29 16:28:41 devfog mysql[8424]: Checking for tables which need an upgrade, are corrupt or were
Jul 29 16:28:41 devfog mysql[8424]: not closed cleanly..
Jul 29 16:28:39 devfog /etc/mysql/debian-start[8893]: Upgrading MySQL tables if necessary.
Jul 29 16:28:42 devfog /etc/mysql/debian-start[8896]: /usr/bin/mysql_upgrade: the '--basedir' option is always ignored
Jul 29 16:28:42 devfog /etc/mysql/debian-start[8896]: Looking for 'mysql' as: /usr/bin/mysql
Jul 29 16:28:42 devfog /etc/mysql/debian-start[8896]: Looking for 'mysqlcheck' as: /usr/bin/mysqlcheck
Jul 29 16:28:42 devfog /etc/mysql/debian-start[8896]: This installation of MySQL is already upgraded to 5.5.44, use --force if you still need to run mysql_upgrade
Jul 29 16:28:42 devfog /etc/mysql/debian-start[8911]: Checking for insecure root accounts.
Jul 29 16:28:43 devfog /etc/mysql/debian-start[8918]: Triggering myisam-recover for all MyISAM tables

[Wayne Workman]

Can you check how much free space you have on your HDD?

[sudburr]

About 120 GiB.

[sudburr]

I’m still hacking away at this problem machine.

I’ve rolled it up and back and forth and sideways, and I’m still hitting Internal 500.

I did notice that you’re now adding to /opt/fog/.fogsettings these lines

docroot="/var/www/html/";
webroot="fog/";

Removing them doesn’t seem to make a diff, but older SVN like 3731 throws up the previously mentioned

ln: failed to create symbolic link ‘/var/www/html/fog/fog’: File exists

when installed over 4250+.

While I’m here, I’m looking for the FOG-approved method for repairing php5, apache and tftpd-hpa .

[sudburr]

I just finished doing

apt-get install --reinstall apache2 php5 php5-json php5-gd php5-cli php5-curl mysql-server mysql-client tftpd-hpa tftp-hpa nfs-kernel-server vsftpd net-tools wget xinetd sysv-rc-conf tar gzip build-essential cpp gcc g++ m4 htmldoc lftp php-gettext php5-mcrypt php5-mysqlnd curl libc6 libcurl3 zlib1g php5-fpm

Restarted and

svn co https://svn.code.sf.net/p/freeghost/code/trunk /opt/trunk &&
cd /opt/trunk/bin &&
./installfog.sh

Which gave me 4266, and I’m still hitting the 500 error on the web interface for database update. Then

Setting up storage..........................................OK
 * Setting up and starting DHCP Server.........................Skipped
 * Setting up and starting TFTP and PXE Servers................Failed!

Client’s are iPXE booting to this

iPXE 1.0.0 (2bcf1) -- Open Source Network Boot Firmware -- http://ipxe.org
Features: DNS FTP HTTP HTTPS iSCSI NFS TFTP VLAN AoE EFL MBOOT PXE bzImage Menu PXEXT
tftp://172.19.244.13/default.ipxe.... ok
http://172.19.244.13/fog/service/ipxe/boot.php... Input/output error (http://ipxe.org/1d0c6139)
Could not boot: Input/output error (http://ipxe.org/1d0c6139)

Oy …

[Wayne Workman]

@sudburr Have you considered a rebuild?

[sudburr]

I’m trying to avoid a rebuild, or a rollback. I want to solve this because I expect it to happen again when I again update from SVN3731 to GIT4266 .

[sudburr]

So these three lines repeat themselves in apache’s error.log

[:error] [pid 9651] [client 172.19.252.91:57574] PHP Fatal error:  Call to undefined method FOGCore::cleanInvalidEntries() in /var/www/html/fog/commons/init.php on line 179
[core:notice] [pid 1066] AH00094: Command line: '/usr/sbin/apache2'
[ssl:warn] [pid 1046] AH01909: 172.19.244.13:443:0 server certificate does NOT include an ID which matches the server name

Following the error for /var/www/html/fog/commons/init.php on line 179, we commented out

 /** $FOGCore->cleanInvalidEntries(); */

And all appears well again with the web interface with 3731. Still digging though.

[sudburr]

Relating to my top post here, the recursive symbolic link that is created allows me to do this

http://devfog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/fog/

and get to the management page.

Spammity spam, spammity spam…

[sudburr]

Hmmm, nfs volume mount error now.

So next I tried a GIT 4266 install over the existing. At the prompt to visit the management page I connected with another session and commented out line 179 /var/www/html/fog/commons/init.php.
I was able to reach the web site. Strangely, my little cloud still says 3731 .

The rest of the install complete seemingly without error. The web interface is working, but now I’m stuck with the NFS volume mount error on clients attempting to quick image.

Okay, try updating using SVN 3821 which checks out to 4266 . The install fails at

Setting up and starting MySQL...............................Failed!

This is fun!

[Wayne Workman]

@sudburr said:

I’m trying to avoid a rebuild, or a rollback. I want to solve this because I expect it to happen again when I again update from SVN3731 to GIT4266 .

I’ve been trying to move people to Fedora lately (and have moved several) simply because there are less problems. Just throwing that out there. Here’s a link to an article I’ve been working on: https://wiki.fogproject.org/wiki/index.php/Fedora_21_Server

The Developers will naturally want to solve the problem with Debian and that’s understandable and fine. But I don’t know what your imaging workload looks like, and if it’s quite demanding and pressing, you could get a fedora box built before lunch, with images and hosts ported over too.

To move or to stay all depends on your needs and how much time you have. With Debian, you have an unknown amount of time till it’s fixed. With Fedora, it’ll be before lunch. It’s up to you though.

[Tom Elliott]

I’ve updated code and hopefully this problem will stop.happening now. Though I don’t know 100% for sure if it will work.

I’m making some pretty rigid assumptions, and will change it accordingly but need to know if it will work.

It should also update the packages properly. This way you don’t get blank screens, I hope.

[sudburr]

@Wayne-Workman said:

I’ve been trying to move people to Fedora lately (and have moved several) simply because there are less problems. Just throwing that out there. Here’s a link to an article I’ve been working on: https://wiki.fogproject.org/wiki/index.php/Fedora_21_Server

That’s funny. Until this incident I hadn’t had a single problem with Debian. I was sold on the conversion from Ubuntu. We started on Ubuntu. I moved to Debian (in development and future planned updates at least) at I think what was one of the developers suggestion, and now you’re telling me that Fedora is the cat’s meow instead?

I’m not complaining. I like trying out new stuff. I’ll give it a whirl for sure, but what is the horse’s mouth opinion on this? I want to be as compliant as possible for my setups. what is THE distro recommended as the least problematic for FOG now ?

[Wayne Workman]

@sudburr said:

I moved to Debian (in development and future planned updates at least) at I think what was one of the developers suggestion,

And whoever told you that made a good suggestion.

I’m only suggesting Fedora because you’ve had no luck all day.

And, if your priority is a working fog box in the least amount of time…

Try the latest revision though, see if that fixes it.

[sudburr]

GIT 4270 made it through the web interface, but the installer is still failing at:

Setting up and starting TFTP and PXE Servers................Failed!

Also

 Adding needed repository....................................OK
 Preparing Package Manager...................................

Were both lightning fast on the first install attempt of GIT 4270, but took for-ever (ie: several minutes) on the next attempt.

And no, I’m not in a bind or a hurry. Not until tomorrow evening, and then I’ll just revert if things aren’t working right yet.