Huge database entries number

siarkowski

Hello All,

Happy New Year!

Since I upgraded FOG to 1.5.10.1734 I noticed Apache is returning 503. I started deployment for over 200 PCs and after few hours 503 occurred.

I could SSH the server, and I found it is run out of disk space. It was occupied by database files, to be precise tasks table.

There are no entry in images table with ImageID equal to 0.

I deleted all data in task table with taskStateID equal to 1 and taskImageID equal to 0.
Tasks resumed, but I had to delete data in task table with taskImageID equal to 0 to avoid disk run out of space.

After all task were finished I cleaned task table, there were entries with taskStateID equal to 4 or 5 only.

OK, I starter a single PC deploy task and watched task table. A single entry with taskStateID equal to 1 apeared, after client started the task it was altered to taskStateID equal to 3, after task was finished it was altered to taskStatedID equal to 4. That’s how it should wark.

I started a 31 PCs deploy task, at the beging 31 new entries apeard in task table with taskStateID equal to 1, 10 PCs started to deploy and 10 entries were altered to taskStateID equal to 3. However I noticed a huge number of enries apeared, all of them with taskStateID equal to 1 and taskImageID equal to 0. After all 31 PCs deployments were finished I checked tasks table. In the task table there were over 500000 entries with taskStateId equal to 1 and taskImageID equal to 0!

I cleaned task table again and started deployment for 11 PCs.
It started nice with 11 new entries in task table with taskStateID equal to 1. 10 PCs started to deploy and 10 entries were altered to taskStateID equal to 3. Again new entries with taskStateID equal to 1 and taskImageID equal to 0 sarted to apear. A single new entry appeared about every second. It corresponds with task slot check at client side, I think so. I forced paused deploy to start and new entries stopped.

Did You have similar situation.
I’ll be grateful for any advice.

Best
Marcin

Tom Elliott

@siarkowski I’m not aware of anything that might be causing this.

The fact that your taskTypeID == 0 in these cases is interesting.

FOG doesn’t have a “create a new task” from Checking in or anything which makes me think what you’re seeing is a symptom, not a cause.

The way tasks get created is either via UI, API, or (similar to UI/API pair) by having a scheduled task.

We have (although it may need more refinement) checks to make sure that type of task that does get created are valid (of which taskTypeID 0 != valid task)

Can you check your scheduled tasks table and see if maybe something there is doing this at some time?

The fact the task was created at 2026-01-02 09:52:02 looks like it could be a schedule task that somehow has a now invalid task type id that’s just managing to get through.

siarkowski

@Tom-Elliott
Thanks for replay.

In the tasks menu, scheduled Tasks there are no entries. However but in database table scheduledTasks the are two entries:

±-----±------------±-------±-------±-------------±---------±-------±------±--------±------±----------±--------------±----------±-----------±---------±---------±---------±---------±---------±-----------±---------+
| stID | stName | stDesc | stType | stTaskTypeID | stMinute | stHour | stDOM | stMonth | stDOW | stIsGroup | stGroupHostID | stImageID | stShutDown | stOther1 | stOther2 | stOther3 | stOther4 | stOther5 | stDateTime | stActive |
±-----±------------±-------±-------±-------------±---------±-------±------±--------±------±----------±--------------±----------±-----------±---------±---------±---------±---------±---------±-----------±---------+
| 1 | Deploy Task | | S | 1 | | | | | | 1 | 6 | 0 | | | -1 | fog | 1 | | 1681921800 | 0 |
| 2 | Deploy Task | | S | 1 | | | | | | 1 | 8 | 0 | | | -1 | fog | 1 | | 1741105800 | 0 |
±-----±------------±-------±-------±-------------±---------±-------±------±--------±------±----------±--------------±----------±-----------±---------±---------±---------±---------±---------±-----------±---------+

Tom Elliott

@siarkowski Thanks, the scheduledTasks you’re showing aren’t showing anything out of the ordinary other than these were “Delayed” tasks

First was delayed to run at 2023-04-19 16:30:00 (UTC)
Second was delayed to run at 2025-03-04 16:30:00 (UTC)

I am pretty sure, this isn’t the issue you’re seeing at least.

So I wonder if there’s some corruption due to running out of disk space that’s causing this?

siarkowski

@Tom-Elliott
I finished all running deploy tasks, cleaned task table.
At this moment partition was over 5GiB free disk space.
I started 37PCs deploy task and the story is same again. Database tasks table files were 40kB before I started the task and after 30 minutes database tasks table files are over 4MB.
Services work fine, tasks are processed, but tasks table is over 26000 records long and it’s increasing.
I’m pretty sure if I start task for 200 PCs disk will run out of space after few hours

Tom Elliott

@siarkowski I understand what you’re saying, what I don’t understand is why this is happening at all. If this was a normal issue, we’d have been notified of this issue probably many many times.

I’m not saying this couldn’t be a bug, I just have no way to replicate the issue in a consistent manner.

Might you be able to pull a backup of your database using FOG Configuration -> Configuration Save -> Export

(After cleaning up the Task issue) and see if you can use a secondary system to import that DB and (as a test) and ensure all the data you require is there (not necessarily the physical images)).

I don’t know what to suggest or say here other than I suspect the database is corrupted and it’s that corruption adding and exploding task records. This (again) isn’t to say there isn’t a bug, but I want try to start with a clean baseline and see if things worked properly under “sanitized” environment.

If a fresh server (if you can share your images that much better) works without adding/exploding the task table, that at least proves it’s a DB Corruption issue. If it, too, starts exploding in task table records, then it shows there’s something in the code that’s doing this unexpectedly but at least we narrowed the scope to exactly where to being looking (somewhere in the task checkin process it seems?)

siarkowski

@siarkowski
Corrupeted DB or schema is highly possible.
I’ll backup tables: users, hosts, groups and images data and make a fresh FOG install.
I"ll let you know as soon as I’ll do it. It will be no sooner then next week I’m afraid.

Tom Elliott

@siarkowski That’s perfectly understandable and much appreciated. Sorry I don’t have a simple “here’s what to do” right now, and maybe a fresh install will fix the issue altogether, but right now it we have limited information. Thank you again for your understanding a patience with this as well.

siarkowski

@Tom-Elliott
As a curiosity, after two hours of 37 PCs deployment , 10 are finished, 10 are rolling, 17 are waiting. Tasks table is 815000 records long and tasks db files are 138MiB total.
Don’t get me wrong, FOG is awesome tool - seriously.

Huge database entries number

99

12.5k

17.5k

156.1k