Storage ID 0 is not valid / A valid database connection could not be made

[mtanigawa]

Server

• FOG Version: 1.3.0 RC15
• OS: Ubuntu 16.04

Client

• Service Version: 0.11.5
• OS: Win 7

Description

I have recently upgraded to RC15 from RC11.
I have three storage nodes with within two groups / locations.
All fogs have been updated to RC15 and running the same SVN.

After the update, I have noticed that my nodes are reporting “A valid database connection could not be made”.

When removing and re-adding the storage node, I see the message “Storage ID 0 is not valid”. The node is added to the list and looks as it should aside from the messages.

• the dashboard shows each node with a listing of “(undefined)” below the name in the grid
• storage group activity does show and register the nodes

The TFTP boot from our node is working - I have not yet tried to image or perform other tasks from the node as of yet.

• I have re-installed the RC on the main fog with no success.
• I have re-installed the RC on one of the nodes with no success.

I have also triple checked and confirmed that the node username and password is correct.

As a note, when updating the main fog, when asked to update the DB scheme, I logged in and it went straight to the home page and not the database update page. I understand that not all updates will require a DB update, but am just curious if one was needed.

I am unsure of which logs you would like to see. Please let me know and I will get them posted to assist with figuring out this issue.

Thank you for your time.

[Tom Elliott]

Remoted in and found that the server was in fairly funky state.

It is ubuntu 12.04, and while we typically recommend to update, this particular issue was simply due to ondrej attempting to install php 7, but the packages variable on fogsettings was pre-labelled as php 5. Removed the packages variable and php/apache on the node, updated the php version stored in the fogsettings, reinstalled ondrej, and re-ran installer .

[Wayne Workman]

It’s one of three things.

Replication is happening - and there’s just no bandwidth for anything else. This is not likely. If this is the case, just wait it out until the traffic settles down - then it should be working.

The fogstorage user/pass on all the nodes changed, or the actual credentials for the fogstorage account changed on the main server changed. This is not likely.

Remote access was somehow turned off for the main server. This is more likely.

So, to get the credentials for the fogstorage user, go here:
FOG Web Interface -> FOG Configuration -> FOG Settings -> FOG Storage Nodes
Let’s say the password there is MyAwesomePassword for example.

Now, log in/ssh into one of your storage nodes. Issue the below command to try to connect to the main server’s database using the fogstorage credentials. x.x.x.x is obviously your main server’s IPv4 address. Note that there is no space between -p and the password.

mysql -h x.x.x.x -pMyAwesomePassword -D fog

If this connects, Your password is wrong on all your storage nodes. Change snmysqlhost and snmysqluser and snmysqlpass inside of /opt/fog/.fogsettings and then re-run the fog installer on each storage node.

If it does not connect, the error matters. Was it a bad password or flat-out access-denied or is it just no response at all?

To Re-enable remote access on the main server’s fog database, and maybe even reset the password for the fogstorage account too, do the following.

mysql -D fog
SET PASSWORD FOR 'fogstorage'@'%' = PASSWORD('MyAwesomePassword');
GRANT ALL PRIVILEGES ON fog.* TO 'fogstorage'@'%' IDENTIFIED BY 'MyAwesomePassword' WITH GRANT OPTION;
quit

Then try to connect again from the remote storage node using the same command as before and see if it works.

Let us know how it goes.

[mtanigawa]

@Wayne-Workman
Thank you for the quick reply.

From the node, I have entered mysql -h x.x.x.x -pMyAwesomePassword -D fog and received the following error message:

mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'user'@'x.x.x.x' (using password: YES)

I have proceeded to enter mysql to enter the command set as the connection was denied.

mysql> SET PASSWORD FOR 'fogstorage'@'%' = PASSWORD('MyAwesomePassword');
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON fog.* TO 'fogstorage'@'%' IDENTIFIED BY 'MyAwesomePassword' WITH GRANT OPTION;
Query OK, 0 rows affected, 1 warning (0.00 sec)

mysql> quit

I have again tried to connect from the node back to main fog and have received the same error message.

user@SHO-FOG-N1:~$ mysql -h x.x.x.x -pMyAwesomePassword -D fog
mysql: [Warning] Using a password on the command line interface can be insecure.
ERROR 1045 (28000): Access denied for user 'user'@'x.x.x.x' (using password: YES)

I have also double checked the three listings for .fogsettings and confirm that the password info listed there is correct.

After the double checks, I have checked back into the GUI and found that all nodes are now registering and providing information as I am accustomed to.

Fingers are crossed that it continues to behave nicely. I would assume that this message is now able to be marked as solved?

I am unsure if I am not able to connect from the node to the main by default, but it seems to have corrected the issue… Will know more as I do more.

Please let me know if there are any logs or other I am able to provide to assist with anything on your end.

Thank you again for your time and quick response. It is immensely appreciated.

[Tom Elliott]

@mtanigawa Please take a look at the “Main” server’s mysql configuration. There’s supposed to be some automation in the installer that, from what I can tell here, didn’t take place for whatever reason. I’d start by cd-ing to /etc and running grep -rl 'bind-address' ..

This should return any configuration filename that contains the text bind-address.

Please open the mysql configuration and find where this is setup. It should be commented by prepending the specific line with a # symbol.

Once you make this change, restart mysql with systemctl restart mysql

This should help allow connections. Essentially, right now it appears the nodes are unable to communicate with the main server. Your main server (the one returning the kernel information (–I think this was called SHO-FOG–) is working because it is the local system.

[mtanigawa]

@Tom-Elliott

Thank you for the reply.

I will check on this when I return to the office in the morning.

Thank you again.

[mtanigawa]

@Tom-Elliott

I have checked through and found the configuration file in /etc with the ‘bind-address’.

As you have mentioned, it is commented. The bind-address gives the value of 127.0.0.1.

My apologies, however, if you would again advise what I need to do with this… should I un-comment this line? Or change to the IPv4 address instead of the loopback? Or is this to confirm that the line is commented?

Please let me know.

As of now, with the steps recommended by @Wayne-Workman, fog is running and able to see and provide information on the nodes - will be trying a deploy and upload later today to confirm all is working nicely.

Thank you for your time.

[Tom Elliott]

@mtanigawa then fog worked properly. Is there a firewall on the main fog server?

[Wayne Workman]

@mtanigawa These are simple questions but I have to ask. Did you chang the x.x.x.x part of the commands below or not, and if you changed the example password to the real one.

[mtanigawa]

@Tom-Elliott

Sorry, didn’t see this earlier.

No firewalls or filters are running on the machine. I will reconfirm tomorrow when back at work.

I have been able to image a machine via the node and all worked as expected - though having an issue with Snapins. Will check this later – one thing at a time.

Thank you again for your time.

[mtanigawa]

@Tom-Elliott

I have confirmed that the firewall is disabled on the main fog.

Noticed that RC16 has been released, will try to update to that release tomorrow and see if any of these issues have returned.

Thank you for your time and help!

[Tom Elliott]

@mtanigawa What do you mean returned? Are you having issues currently?

[mtanigawa]

@Tom-Elliott

Apologies for not being specific. ‘Return’ meaning the error experienced from this thread. I am also experiencing ongoing inconsistency with snapins deploying, but is something I am looking into separately.

Thank you for your follow up, quick replies and great work!

[entr0py]

@Tom-Elliott

This problem seems to exist in RC20 as well. I just did an upgrade of our fog system to RC20 from a much older SVN (RC8 I think?) and have the exact same error. I’ve dropped users, recreated them, I can mount nfs, ftp, ssh and mysql between node and the full Fog server with the passwords and users as specified in the configs with no issues but the existing storage nodes are dead. Trying to add another storage node results in the Storage ID 0 is not valid error as described.

No firewalls, nothing new. Did a straight svn up from the fogproject folder and an installfog.sh as usual. No hardware or software changes on either machine and this is (was) a running production system a few hours ago that had been deploying and capturing from the nodes just fine. I’ve done basically everything possible to make this as insecure as possible, granting all with %'s and everything to no avail.

SERVER

FOG Version: 1.3.0 RC20
OS: Ubuntu 12.04
IP: 172.16.0.13

I’ve done the following on the server:

mysql> SET PASSWORD FOR ‘fogstorage’@‘%’ = PASSWORD(‘password’);
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON fog.* TO ‘fogstorage’@‘%’ IDENTIFIED BY ‘password’ WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

mysql> SET PASSWORD FOR ‘fog’@‘%’ = PASSWORD(‘password’);
Query OK, 0 rows affected (0.00 sec)

mysql> GRANT ALL PRIVILEGES ON fog.* TO ‘fog’@‘%’ IDENTIFIED BY ‘password’ WITH GRANT OPTION;
Query OK, 0 rows affected (0.00 sec)

NODE0

FOG Version: 1.3.0 RC20
OS: Ubuntu 12.04
IP: 172.16.0.16

Results on the Node:

root@node0-MicroServer-Gen8:/images# mysql -u fogstorage -h 172.16.0.13 -ppassword -D fog
Reading table information for completion of table and column names
You can turn off this feature to get a quicker startup with -A

Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 1617
Server version: 5.5.53-0ubuntu0.12.04.1 (Ubuntu)

root@node0-MicroServer-Gen8:/images# lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 12.04.5 LTS
Release: 12.04
Codename: precise

[Wayne Workman]

@entr0py can we begin with your apache error log please?

[entr0py]

@Wayne-Workman

0_1478476447405_error.log.txt

Attached.

Nothing in the last 3ish hours…

root@FOG-Virtual-Machine:/var/log/apache2# date
Sun Nov 6 16:55:33 MST 2016

[Tom Elliott]

I don’t know what is wrong, but if I had to guess, looking earlier down the road.

You set the fog user for DB connections.

The fog user may be created, but probably doesn’t have permissions to communicate with the FOG database. I think you intended to use fogstorage for db connections?

I’m not sure why you had shown us the values in the storage node. If I had to guess, you changed “management user/password” to match what you set for the fog database connection? If that is the case, this means your FTP information is likely broken at this point. To fix this, (after fixing the fogstorage user/password to the db) edit the /opt/fog/.fogsettings file and ensure the username and password is correct for the “local” linux user. Ensure the snmysqluser portion is set to snmysqluser='fogstorage'. Ensure the snmysqlpass portion is set to the password as set in FOG Configuration Page->FOG Settings->FOG Storage Nodes->FOG_STORAGENODE_MYSQLPASS

Please see if making this fixes will work.

[entr0py]

@Tom-Elliott

Sorry for it being so confusing what I posted. I was just playing around with a ton of options trying to find something that might work before I started to complain at the overworked and underpaid developers. My snmysqluser=‘fogstorage’ on the node already. Passwords all match. I can ftp between the boxes using the management user of “fog” and password of “password” and mysql works with “fogstorage” and “password” as well. The user works as shown below:

root@node0-MicroServer-Gen8:/opt/fog# mysql -h 172.16.0.13 -u fogstorage -ppassword -A fog
Welcome to the MySQL monitor. Commands end with ; or \g.
Your MySQL connection id is 2984
Server version: 5.5.53-0ubuntu0.12.04.1 (Ubuntu)

Copyright 2000, 2016, Oracle and/or its affiliates. All rights reserved.

Oracle is a registered trademark of Oracle Corporation and/or its
affiliates. Other names may be trademarks of their respective
owners.

Type ‘help;’ or ‘\h’ for help. Type ‘\c’ to clear the current input statement.

mysql> SELECT * FROM hostMAC WHERE hmHostID=0;
Empty set (0.00 sec)

It also matches the web interface settings.

Attached are both .fogsettings files.

0_1478478272991_dotfogsettings-host.txt
0_1478478279431_dotfogsettings-node.txt

[entr0py]

Also, either user should work given the grants in MySQL.

mysql> show grants for fog;
±---------------------------------------------------------------------------------------------------+
| Grants for fog@% |
±---------------------------------------------------------------------------------------------------+
| GRANT USAGE ON . TO ‘fog’@‘%’ IDENTIFIED BY PASSWORD ‘2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19’ |
| GRANT ALL PRIVILEGES ON fog. TO ‘fog’@’%’ WITH GRANT OPTION |
±---------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql> show grants for fogstorage;
±----------------------------------------------------------------------------------------------------------+
| Grants for fogstorage@% |
±----------------------------------------------------------------------------------------------------------+
| GRANT USAGE ON . TO ‘fogstorage’@‘%’ IDENTIFIED BY PASSWORD ‘2470C0C06DEE42FD1618BB99005ADCA2EC9D1E19’ |
| GRANT ALL PRIVILEGES ON fog. TO ‘fogstorage’@’%’ WITH GRANT OPTION |
±----------------------------------------------------------------------------------------------------------+
2 rows in set (0.00 sec)

mysql>

[Tom Elliott]

@entr0py I confirmed the storage id 0 is not valid issue. Mind you, this was literally just a message. The node was indeed created and functional even though this message was there.

When you create a new node, the page redirects to the “edit” page. This edit page needs to have the id passed into it. I missed passing it the “edit” portion and had the id being incorrectly set.

While this was a very minor bug, I have republished as RC-20. Sorry for not noticing it sooner.

[entr0py]

@Tom-Elliott

Any thoughts on why the storage node still doesn’t work? If I try to create a task it says the node doesn’t exist. If I try to create a task I get the following:

If I try to show the node on the dashboard it isn’t even listed:

If I try to image a computer using “Deploy Image” on GRUB menu: