While I think this is a kernel/root filesystem issue lets start with the basics.
What version of FOG are you using? Are you using 1.2.0 or a SVN trunk (beta) version?
What OS is FOG running under?
What is the target hardware you are trying to clone (make and manufacturer)
There are a few things that you might do to clear this up (proxy server related). I run FOG on centos 6.7 behind a proxy server and this is what I needed to do.
In the bashrc file I placed the following commands.
export http_proxy=http://192.168.1.10:3128
export https_proxy=http://192.168.1.10:3128
export ftp_proxy=http://192.168.1.10:3128
export no_proxy=“192.168.1.88”
In your wgetrc you can have the proxy settings too. The key is the no_proxy variable. That needs to be set to the IP address of the FOG server. Once these are set log out and back in. These new env variables will be set. From there FOG will install and backup correctly. If you look at the code for the backup, the script uses wget to pull the backup image. Well, wget isn’t that smart. Its trying to contact the proxy server instead of just connecting locally. The no_proxy statement tells wget to not proxy any requests for the FOG server.
As for the second question, that to find out if this was a new install that never worked, or a working install that broke after the upgrade.
First, what is the host OS that FOG is running under?
Was this system running under the 1.2.0 prior to you upgrading to the latest SVN?
I think we need to divide your issue into two parts
I guess I have two questions here
Actually we would like to see you generate that error message again, then view the apache error log in /var/log/httpd/error_log. The last few lines should indicate the current error. The access_log should only show use details when things go right.
FWIW: I have seen if you stop the fog image replicator while a transfer is underway the lftp process will continue. If you stop and restart the fog image replicator multiple times, you might end up with 3 or more lftp processes running at the same time each with their own bandwidth limitation and ignorant of the other running processes.
If you really want to go there, using the current svn trunk (if things go sideways, just rerun the fog installer to correct the files. No harm done then)
Edit this file:
/var/www/html/fog/lib/service/FOGService.class.php
Search for the first occurrence of the word fragment: limit.
You should see something like this:
$limitsend = $this->byteconvert($StorageNodeToSend->get('bandwidth'));
if ($limitmain > 0) $limitset = "set net:limit-total-rate 0:$limitmain;";
if ($limitsend > 0) $limitset .= "set net:limit-rate 0:$limitsend;";
$limit[] = $limitset;
}
}
unset($StorageNodeToSend);
$this->outall(_(' * Starting Sync Actions'));
foreach ((array)$nodename AS $i => &$name) {
$process[$name] = popen("lftp -e 'set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; ".$limit[$i]." mirror -c -R --ignore-time ".$includeFile[$i]." -vvv --exclude 'dev/' --exclude 'ssl/' --exclude 'CA/' --delete-first ".$myItem[$i].' '.$remItem[$i]."; exit' -u ".$username[$i].','.$password[$i].' '.$ip[$i]." 2>&1","r");
Insert the following line after: $this->outall(_(’ * Starting Sync Actions’));
$this->outall(_(' * Speed limiter settings: $limitset'));
That should make it like this:
unset($StorageNodeToSend);
$this->outall(_(' * Starting Sync Actions'));
$this->outall(_(' * Speed limiter settings: $limitset'));
foreach ((array)$nodename AS $i => &$name) {
Stop and restart the FOGImageReplicator. You may need to delete a file on the storage node for it to see a change. When the replication runs it should output the speed limits to /opt/fog/log/FOGImageReplicator
I’d do this in my test environment but its only partially rebuilt.
First I’m not a programmer. But I know the dot ( . ) string concatenation and .= may be short hand for “string = string + newstring”. Then that might make the statement accurate.
Right now I can’t think of a way to capture the environment that lftp is running in to see what is actually being set. It may be possible to hack the FOGImageReplicator to insert a few extra log statements to see what is actually being set by logging the variables.
@Jbob
Actually I share the same concerns about disabling selinux and the firewall. But this seems to be the standard practice for installing FOG so I didn’t want to push a different agenda. I also consider that FOG appears to be targeting the smaller SMB realm from a security stance and most have FOG installed inside a properly protected network so that does mitigate some of the risks.
Just thinking out loud, if we have a list of all services used for FOG then constructing firwall rules shouldn’t be difficult at all. The application of the rules may be difficult since each linux distro handles the iptables rules a bit differently.
The selinux part will be a bit harder since FOG is writing to files all over the place. We would have to get the selinux tagging just right (we could get there if we used the permissive setting then scanned the log for the required rights. Its been a while since I did that type of activity). But from a security standpoint I like where you are going with this.
@Wayne-Workman I tied to find your monster command but didn’t have much luck. I did see when you posted it the other day.
Just for clarity this is what I keyed in to setup the trunk.
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/sysconfig/selinux
sed -i 's/SELINUX=enforcing/SELINUX=disabled/' /etc/selinux/config
systemctl disable firewalld
reboot
yum install -y wget subversion
mkdir /opt/fog_trunk
cd /opt/fog_trunk
svn co https://svn.code.sf.net/p/freeghost/code/trunk
Its pretty much what you keyed in. But you have to do a reboot after changing the selinux stuff or it won’t be in active and the install will fail. But great job on the monster command.
As long as you are up and running on 6.7 stay there for a bit and working on getting your system just the way you need it. 6.7 won’t go EOS until 2020. By then other factors will force you to upgrade.
For what its worth. I just spun on a centos 7 fog server. With the exception of a few commands that are different between 6.7 and 7. The current FOG trunk installs and operates on Centos 7 without issue. Understand at this point I haven’t uploaded or downloaded an image yet. But basic functions on Centos 7 are working.
Right now you appear to be in a good place with Centos 6.7 so hang there for a bit and then focus on make your fog configurations perfect for your tasks.
Looking in the code this is what I see for the replicator. It uses lftp to move the files.
"lftp -e 'set ftp:list-options -a;set net:max-retries 10;set net:timeout 30; ".$limit[$i]." mirror -c -R --ignore-time ".$includeFile[$i]." -vvv --exclude 'dev/' --exclude 'ssl/' --exclude 'CA/' --delete-first ".$myItem[$i].' '.$remItem[$i]."; exit' -u ".$username[$i].','.$password[$i].' '.$ip[$i]."
The proper limit parameter for lftp is this: net:limit-rate 0:512000 where the first number is the download rate and the second number is the upload rate (in Kb/s).
Looking a bit deeper into the code I see this:
if ($limitmain > 0) $limitset = "set net:limit-total-rate 0:$limitmain;";
if ($limitsend > 0) $limitset .= "set net:limit-rate 0:$limitsend;";
$limit[] = $limitset;
I’m questioning the extra period in the second line and the format just seems a bit off.
I understand that a bit, but I come from the redhat world so it is not as foreign to me than when I use ubuntu (fwiw: I use zorin on my home computer which is ubuntu based). The ifconfig vs ip … does throw me every time I use ubuntu also.
While the worlds are a bit different FOG did install ok on Centos 7. I can say I used this wiki page [https://wiki.fogproject.org/wiki/index.php/Installation_on_CentOS_7] for the syntax to turn off selinux and to disable the firewall. All other things install the same between 6.7 and 7.
You didn’t mention if your fog server is behind a proxy server or does it have direct internet access? If it has direct internet access then there should be nothing else you need to do to make it work.
It sounds like you gave up on it because it was too different more than FOG flat didn’t work on Centos 7. We just want to ensure that there isn’t a problem with FOG running on Centos 7, because then the devs will need to dig into why.
OK, I was able to install FOG SVN 4344/Git 5235 on a newly created centos 7 server. I did forget to disable selinux at first, but I disabled it, rebooted and reran the install.sh script again. The install script completed without issue. As far as I can tell the server is up and running. I haven’t captured an image yet with it, but all of the menus and plugins respond as I expect (the authorization plugin is still broken, but that is a topic of another thread).
Where did the install fail for you?
Is your cento 7 box behind a proxy server, because there are some tweaks that need to be done first? (but same tweaks are needed for centos 6.7)
I haven’t tried the nucs yet with the svn trunk yet, but I did have an issue with them using 1.2.0. They were getting stuck at the ipxe initializing devices. It was having a problem configuring the realtek nic.
I was tried to boot legacy mode (since uefi did not work well with 1.2.0). If I remember correctly I had to update the bios to the latest version. Then on the NUCs I was using in the advanced bios setting there was a compatibility mode. On the one I was using there was linux, win7, win8. I had to set the bios to legacy boot, I think enable legacy roms, and select win7 mode. Save and exit the bios, then POWER IT OFF. If I did not power it off after making the changes it will continue to get stuck at the ipxe initializing devices. By accident I powered it off and then after that it started working. I should have another one in about 2 weeks where I can try them again to see if they are any better with the svn trunk version.
No promises, but I’m spinning up a centos 7 server right now for FOG. I’ve been meaning to do this anyway. My preferred server is Centos 6.7, because it just works well with FOG.
I’ll let you know how it goes.
Right now what the OP is saying is a bit conflicting in my mind. So we need to get to a known good place with the SVN trunk. I can say, absolutely that FOG SVN trunk works on Centos 6.7 with 0 alternations or messing around. Follow the documented install instructions (even on the svn trunk) and it will work, period. Understand that I’m not saying that Centos 7 (rhel7) is not a viable candidate. Personally I don’t have quite enough experience with Centos 7 to say absolutely there will be no issue. It appears that the OP is having issues that could or could not be related to the OS, or the way the FOG interacts with the OS. To go to a known good state I recommended Centos 6.7 on alternate hardware. If it works there and not on the main system then we can start to pick the OS apart. If FOG acts exactly the same between the production system and this test system on a known stable OS then we have more justification to point to something in the FOG SVN trunk is amiss.
Also for 6.7 its still a viable option since LTS ends in 2020 with full system updates ending in summer 2017 (it kind of parallels the question of Win7 vs Win10, what do you install today?). The answer will be different if you ask me the same question next July. FWIW: LTS for Centos 7 ends in 2024.
While this recommendation isn’t fog specific, I think I would take this approach.
You have 1.2.0 working right? If so leave that system in place for a short time. It is working and the school can deploy.
Take a new machine (desktop with a dual core and 4GB of ram). Install Centos 6.7 on that system. Its in the redhat family so you should not have much of a learning curve switching from fedora to centos. Do a minimal centos install. Just be aware that there is no gui with the minimal install so you will have to use putty and/or the console for configuration. Ensure you update the computer to the latest patches using yum. Then download and install the latest svn trunk directly onto the system. Do no upgrade from 1.2.0 go directly to the svn trunk. This will ensure there are no remaining older bits on the system. Then just follow the installation instructions for centos. Document your steps, step by step so you can create an installation manual and have a history of changes. Once you get this second “clean” system built are you are sure it is configured correctly, change the dhcp options to point to this new server. If something goes wrong with the new server then just change the dhcp setting back to the working 1.2.0 build.
Once you have the svn trunk version perfected then resetup the main server. I can tell you that installing fog on Centos works and works well. Centos isn’t leading edge like fedora, but for servers you want stability over a fancy gui or latest hardware support.
Again this is only my opinion and how I would approach the problem.
I found this bug report for 14.04 of ubuntu.
https://bugs.launchpad.net/ubuntu/+source/php-apcu/+bug/1422484
In a nutshell the fix seems to be:
sudo apt-get purge php5-apcu
sudo service apache2 restart
Not a FOG issue more about ubuntu.