Install hangs at "Ensuring node username and passwords match"

[mpmackenna]

We were running two separate FOG servers at two locations on our internal network each on its own subnet. We have a central DHCP server running Ubuntu 16.04 and isc-dhcp-server. In an attempt to better support UEFI PXE boot we decided to centralize our FOG installation and replace one of the installs with a storage node. I converted the secondary installation to a storage node and we started having issues with the web interface not being responsive. I took down the storage node and removed it from the configuration on the Normal Server installation. The web interface was still not responding well so I decided to run the installer again on that installation to see if that would correct issues with the web interface. Now the web interface loads but hangs on login and the installer is hung at “Ensuring node username and passwords match”. I found another post that said add the “no_proxy = <ip of fog server>” to /etc/wgetrc I tried adding that and restarting the installer but the installation still hangs in the same spot. Any ideas on how I can further troubleshoot this issue would be greatly appreciated! Thanks.

[george1421]

Please don’t be offended, but I’m not sure what direction to give you since you seem to be a bit all over the place on your design.

What end design do you really want? There are advantages and disadvantages and advantages of have 2 independent fog servers as well as having a main fog server with remote storage nodes. What configuration do you need in the end. Forget about the rest of the troubles at the moment.

[mpmackenna]

@george1421 I would like a central pxe server and storage nodes at the different locations.

[george1421]

@mpmackenna OK great,

So your main fog server, is that the one with slow web interface? If so what version of FOG are you running. There are some post install “fixes” that need to be made to make fog happy again (even before we talk storage nodes).

[mpmackenna]

@george1421 1.5.4, yes, that is the one with the unresponsive web interface. Thank you for your help!

[george1421]

@mpmackenna There are a few steps that you need to do (these will be addressed when fog 1.5.5 is released.

For these changes you will need console access to the master fog server linux command prompt:

1. Change to the /etc directory from the fog server linux command prompt.
   Search for www.conf file. It can be in a number of locations depending on what version of php is installed. Use this command.
   find /etc -name www.conf (hopefully you will only find one)
2. Edit that file file and ensure these settings are accurate. Don’t just add them since all should be there except php_admin_value[memory_limit] = 256M you will need to add that entry.

php_admin_value[memory_limit] = 256M
pm.max_requests = 2000
pm.max_children = 35
pm.min_spare_servers = 5
pm.start_servers = 5

3. Save and exit your text editor.
4. Reboot the fog server.

Now you need to roll back the version of the FOG kernel installed on the master node. The deployed version with FOG 1.5.4 (kernel 4.17.0) has something not right with it. Under certain circumstances it take between 3-5 minutes to create the disk structure. The rest of imaging is fine, its just related to creating the disk structure. You can roll back the kernel from the FOG Settings->FOG Kernel page.

Also while you are in the fog settings area, install the Location plugin. You will need that a bit later in the setup.

[george1421]

@mpmackenna said in Install hangs at "Ensuring node username and passwords match":

@george1421 1.5.4, yes, that is the one with the unresponsive web interface.

If the last post doesn’t solve the problem, then we need to ensure that php-fpm is running correctly. The FOG developers switched from the native php engine in apache to php-fpm in version 1.5.3 to address the slow response they were seeing from the new UI in the 1.5.x version. The settings below were discovered after 1.5.4 was released. These settings help with tweaking php-fpm under heavy load. We have also seen depending on the linux distro that php-fpm some times is not hooked in correctly with apache, which still gives the user an unhappy experience with the UI.

To see if php-fpm is hooked into apache correctly, launch top from the linux console. Then sort by CPU usage by keying in P. The top 2 or 3 processes should be php-fpm. If it is apache then something isn’t hooked correctly.

[mpmackenna]

@george1421 Got this when trying to roll back the kernel “Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 463, Message: ftp_login(): Login incorrect., Host: 10.10.2.8, Username: fog”. I know I’ve changed the FOG user password when trying to setup the storage node. Now I am unable to ssh to the box with the fog account using the passwords that I thought it was set to a while back. I also tried every other password I could think it may be including what used to the default of “password”. I still have ssh access with other accounts and root access. Should I set the FOG account password to something? Thank you!
Also, the changes made to www.conf made all the difference with the web interface. It is working well at this point.

[george1421]

@mpmackenna Tell me if you fiddled with the linux user called fog (such as changed/or reset the password). That’s not the default webui admin user fog this is the linux user fog.

If you did, shame on you. You will need to go through the process of resyncing the linux user (and may be part of your issue with the storage node): https://forums.fogproject.org/topic/11203/resyncing-fog-s-service-account-password

[mpmackenna]

@george1421 Oh I fiddled with it. Charlie Daniels has got nothing on me. I am completing the steps you outlined to repair. Thank you!
Update: I ran your repair and the installer finished properly. I then was able to roll back the kernel without issue.

[george1421]

@mpmackenna Yeah, that account is a fog service account used by the fog application and should only be managed by the fog installer script. That point should be better documented on the wiki. You are not he first, or the last to have this issue.

[mpmackenna]

@george1421 Things seem to be running well now. Thank you so much for your help! I am going to go back read the docs for adding a storage node and attempt to attach a clean install of a storage node to my working primary installation. Should I mark this thread resolved and start a new one if I have an issue with the storage node? Or, perhaps you/moderator mark as resolved? I don’t see that option. Thanks again!

[george1421]

@mpmackenna Lets keep working through on this one.

Now when/if you have a storage node. If you are going to change modes on that Normal Node -> Storage Node. You need to delete the /opt/fog/.fogsettings file. Understand when you do that you may have issues with the local linux fog user accont. The fog installer script should keep everything sane, but if things go wrong then you have a place to look.

Also you will need to make the edits to the www.conf file on every storage node too. Also you will need to downgrade the FOS kernel. I know its a pain, but it is what we have until 1.5.5 is released.

You can copy from your master node to the storage nodes. The files are in /var/www/html/fog/service/ipxe You can use scp to copy and the files you need to copy are bzImage and bzImage32 to all of your storage nodes in the same location.

[mpmackenna]

@george1421 said in Install hangs at "Ensuring node username and passwords match":

To see if php-fpm is hooked into apache correctly, launch top from the linux console. Then sort by CPU usage by keying in P. The top 2 or 3 processes should be php-fpm. If it is apache then something isn’t hooked correctly.

I checked this and it seems to be correct. Top processes are php-fpm7.1. Thanks!

[mpmackenna]

@george1421 said in Install hangs at "Ensuring node username and passwords match":

@mpmackenna Lets keep working through on this one.

Now when/if you have a storage node. If you are going to change modes on that Normal Node -> Storage Node. You need to delete the /opt/fog/.fogsettings file. Understand when you do that you may have issues with the local linux fog user accont. The fog installer script should keep everything sane, but if things go wrong then you have a place to look.

Also you will need to make the edits to the www.conf file on every storage node too. Also you will need to downgrade the FOS kernel. I know its a pain, but it is what we have until 1.5.5 is released.

You can copy from your master node to the storage nodes. The files are in /var/www/html/fog/service/ipxe You can use scp to copy and the files you need to copy are bzImage and bzImage32 to all of your storage nodes in the same location.

That sounds great! I am going to run the installer on my Storage Node. This server will be the Normal/Master install. I will make the edits listed and let you know how it goes. Thank you!

[mpmackenna]

@george1421 I followed your instructions and the new Storage Node seems to be working. I put them in the same storage group and I am replicating the images to the new node. Thank you for your help!

[mpmackenna]

@george1421 I checked my storage node this morning and it appears some of my images did not sync. I have a number of them that did but there are definitely some that are missing. Syncing has stopped so it’s not matter of not enough time to sync. Can you point me to a document or offer assistance as to how I can troubleshoot this issue? My first thought was to just use rsync but I am concerned that perhaps there are files in the folder that are supposed to differ between a master node in a storage group and a slave node in the same group? Thank you!

[george1421]

@mpmackenna If there is an image definition for the image and its set to replicate, then it should replicate for you. You CAN use rsync if you want to seed the remote repository.

There are replication log files in /opt/fog/log that might give you a clue to why certain images were skipped.

IMO rsync might be a better tool for FOG to use in the future for moving files than its current replication method. But that’s not up to me to decide.

[mpmackenna]

@george1421 Use to be all the FOG services were listed with a “service --status-all” command with previous versions of FOG, but now I don’t see them. If I want to stop the replication service while I run rsync how would I do that? Thank you!

[george1421]

@mpmackenna If you have a systemd based system then its.
sudo systemctl stop FOGImageReplicator

If you have a systemv based system its
sudo system FOGImageReplicator stop