Failed to Update Database and Host

[maxcarpone]

I may have found something related to my problem. When I rebooted the Fog Server, it tried to stop 2 instances of MariaDB. Could it be possible that a part of Fog is trying to join the wrong database?

[Tom Elliott]

@maxcarpone I don’t know for sure if 2 instances running would cause a problem specifically that you’re seeing, but I could see there being a mass confusion in “what instance am I connecting to” definitely causing headaches.

[Fog_Newb]

@maxcarpone I had the same problem with 2390 and reverted to an older back up. That version was broke AF

[maxcarpone]

@Fog_Newb : Good to know I’m not the only one having this problem. I’ll wait for a patch.

[Fog_Newb]

@maxcarpone 1.5.10.2402 is out. It works. I was able to capture from both my PC and laptop successfully. But it has a weird issue with null tasks.

https://forums.fogproject.org/topic/18230/1-5-10-2402-additional-null-tasks-created-but-this-time-manages-to-start-a-transfer

[maxcarpone]

@Fog_Newb : Well it doesn’t work for me. I still have problems with this version.

[maxcarpone]

I tried new things : I created a new user to see if I have the same problems as user “Fog”.

Well, when I perform a registration with immediate deployment : it works!

But it failed to update the database at the end of the deployment.

It looks like it has something to do with active tasking.

So, it’s still broken somewhere…

[Tom Elliott]

@maxcarpone

I’m suspecting you’re not entering the correct password?

This is your FOG Web login user/password.

[maxcarpone]

@Tom-Elliott : Unfortunately, it is the right password. I was wondering myself when the problem appeared but the prompt says something when it’s not the good one and you have 3 tries before giving up.

And by the way, it works using “deploy image” from PXE menu.

I’m available if you want me to do some special moves to find out where is the culprit ^^

[Tom Elliott]

I’ve found and fixed several ways these empty tasks get created — a group deploy of two or more hosts was failing outright, and deleting an image was leaving its queued tasks pointing at nothing — but before I ship a cleanup for the rows you already have, I need to know whether the bad tasks hold a 0 or the id of something that got deleted, because that changes what the cleanup has to match on. Could you run this against your FOG database and paste the output?

SELECT  t.taskID,
        t.taskStateID,
        t.taskTypeID,   IF(tt.ttID   IS NULL, 'MISSING', tt.ttName)   AS type_row,
        t.taskHostID,   IF(h.hostID  IS NULL, 'MISSING', h.hostName)  AS host_row,
        t.taskImageID,  IF(i.imageID IS NULL, 'MISSING', i.imageName) AS image_row,
        t.taskName,
        t.taskCreateBy,
        t.taskCreateTime
FROM       tasks      t
LEFT JOIN  hosts      h  ON h.hostID   = t.taskHostID
LEFT JOIN  images     i  ON i.imageID  = t.taskImageID
LEFT JOIN  taskTypes  tt ON tt.ttID    = t.taskTypeID
WHERE      t.taskStateID IN (0,1,2,3)
ORDER BY   t.taskID;

SELECT  t.taskStateID,
        COUNT(*)                AS rows_total,
        SUM(h.hostID  IS NULL)  AS host_missing,
        SUM(i.imageID IS NULL)  AS image_missing,
        SUM(tt.ttID   IS NULL)  AS type_missing,
        SUM(t.taskHostID  = 0)  AS host_zero,
        SUM(t.taskImageID = 0)  AS image_zero,
        SUM(t.taskTypeID  = 0)  AS type_zero,
        MIN(t.taskCreateTime)   AS oldest,
        MAX(t.taskCreateTime)   AS newest
FROM       tasks      t
LEFT JOIN  hosts      h  ON h.hostID   = t.taskHostID
LEFT JOIN  images     i  ON i.imageID  = t.taskImageID
LEFT JOIN  taskTypes  tt ON tt.ttID    = t.taskTypeID
GROUP BY   t.taskStateID;

SELECT * FROM schemaVersion;

[maxcarpone]

@Tom-Elliott : I can do that if you tell me how to do it!

Update : I figured it out.

[Tom Elliott]

@maxcarpone I don’t know if this is directed specifically at you @maxcarpone but if you want the gist:

sudo mariadb -u root fog

Then enter the select statements provided earlier.

[maxcarpone]

@Tom-Elliott : I found a way to do it :

mysql --user=fogstorage --password=MYPASSWORDHERE -D fog

Here are the results :

[Tom Elliott]

@maxcarpone That output was exactly what I needed — thank you for running it, and for the screenshot of the failing deployment, which turned out to be the most useful thing in the thread.

Your database is fine and your schema is current. schemaVersion 286 is the right number for 1.5.10.2402; 287 only landed this morning. And your first query returning an empty set means you have no stuck tasks at all — nothing in Queued or In Progress. The blank entries you’ve been seeing are all in Complete and Cancelled, and they’re just history: 300 of those rows point at hosts and 459 at images that were deleted at some point over the nine years that table has been filling up. Nothing is wrong with them, they simply have nothing left to show.

The real bug is the one in your deployment screenshot, and it is not your FTP password. Here is what was happening:

* Task Complete
* Updating Database.....................Failed
* Error returned: Failed to update imaging log
* Reattempting to update database.......Failed
* Error returned: No Active Task found for Host: VM-W11-TEST

Only that first error is real. FOG looks up the open imaging-log row to close it by asking for the row whose finish time is empty, and a change earlier in the 1.5.10 line moved “empty” from a zero date to a real NULL. The query FOG was building for that could never match a NULL — so on every single deployment, on every install, it failed to find the row it had just created minutes earlier. It then reported “Failed to update imaging log” after it had already marked the task Complete, which is why every retry after that came back “No Active Task found for Host” and the machine sat there rebooting into an error.

That also explains your very first post: “the host is well imaged but it is not updated in FOG, information about the last image isn’t written in the database.” It wasn’t. The imaging log never closed, the task was already marked Complete, and nothing recorded that the image had landed.

I’ve reproduced it end to end and fixed it, along with three other ways a task could end up pointing at nothing: https://github.com/FOGProject/fogproject/pull/1391

Two things still outstanding on your side:

1. The two MariaDB instances are worth sorting out on their own. I don’t think they caused the above, but two servers on one box is a real problem waiting to happen and it makes everything harder to diagnose. systemctl list-units 'maria*' 'mysql*' and ss -lntp | grep -E '3306|mysql' will show what’s actually listening, and sudo mariadb -u root fog -e "SELECT COUNT(*) FROM tasks" compared against the count in the web UI will tell you whether FOG and your shell are looking at the same database.

2. “Invalid Login” on register-with-immediate-deploy I have not explained yet, and I’d like to. It’s suspicious that a brand new user worked and fog didn’t, because that rules out the typing. Two questions: does the fog user’s password contain any accented or punctuation characters, and can you log into the web UI as fog right now with that same password? If the answer to the second is yes, the problem is in how the password survives the trip from the boot menu, and that’s a different fix from the one above.

[maxcarpone]

@Tom-Elliott : Good explanation and I understood the most of it. Many thanks.

I will try to help resolving these remaining bugs.

About the MariaDB Instances. I don’t understand why there are 2 instances but I’m not the only person using this server. Are they both related to Fog? I used your codelines :

I don’t know where to find the count in the web UI?

I remember I had problems with phpmysql for several years and I had to modify a file to be able to update our FOG servers. I noticed I hadn’t have to modify that file (functions.sh) anymore. Could it be related to those MariaDB instances?

Now, speaking about invalid login led me to think the password could be the culprit because it’s started with ‘&’. I’ve just modified it and will try again and finally let you know what happened.

[maxcarpone]

Bingo!

(I tried the older password just to be sure before typing the good one)

This one was related to our password which started with ‘&’
I modified it and now, I can image I soon as I register a new host. I must say that password has always worked right from the beginning till the previous version 1734 (prior to update to 2390).

So now, everything should be sorted out after updating our server to the last commit?

(Still those instances of MariaDB but I’m no good to debug it without help)