fog creates thousands of new empty images - after one deletion

[Handtuch]

Hi community,
at this very moment, fog is producing new database-entries for images - more than 5 of them every second, each one of them completely blank.

Step 0: My fog-server together with one fog-node have been configured more than a year ago and they were doing a good job - until now.

Step 1: Today I was going to delete some old images to free some diskspace. So I logged into the WebUI and went to “Images”->“list of images”. I marked one entry and chose to delete them: the database-entry including the corresponding files. I repeated this step for two more images. The database-entries vanished from the list of images and the files vanished from the filesystem, as expected.

Step 2: But beginning with the first deletion, the list of images got filled with additional entries for fog-images, each of them completely blank (lacking any data). Within this process, no files have been created in the images-directory of the filesystem.
In this very moment, more than 5 new entries are appearing every second (To state it clearer: Log into the WebUI, choose “images”->“list of images”. A few minutes after Step 1, I needed to scroll down several thousand empty entries to see the “real” ones.)

Even after a reboot, this process went on. This is very disturbing and I really don’t want to re-install an re-configure fog from the very beginning.

Could you help me to
1.) Stop the creation of new empty images (the database entries)
2.) Delete these empty images from the database without starting that nightmare again
3.) Delete any valid images from the database without starting that nightmare again

I really appreciate your support!
Kindly,
Handtuch

[Wayne Workman]

This sounds like a fun one…
Turn off apache first. Ubuntu is service apache2 stop centos/fedora/RHEL is systemctl stop httpd or for older centos iit’s the same as ubuntu.

From here, we must wait for the bug fix, and clean up images manually in MySQL.

[Tom Elliott]

What version of fog are you running?

THis sounds like a bug that was in 1.2 (albeit very rarely did it happen).

[Handtuch]

@Tom-Elliott this if fog 1.2.0. Install date 28 Jul 2014

[Tom Elliott]

@Handtuch Yours is only the second case of this I’ve heard of. I’m pretty sure this is fixed in trunk.

I’m almost going to guess you’re on an imaging spree right now? I’ll even further guess you have a bunch of systems waiting to checkin, and some of them are likely reporting node failures.

[Handtuch]

@Wayne-Workman thanks. apache2 is now stopped. We’re going to deploy images to hundreds of computers this week…

[Handtuch]

@Tom-Elliott We’re indeed on an imaging-spree. Up- and download continued to work normally - at least within the first hour after the start of this nightmare. A bunch of systems were waiting to checkin, but that might be the normal consequence of out host limit.

[Tom Elliott]

@Handtuch It kind of is, and it’s the only situation in which I can see this issue even being potentially possible of happening. Please try upgrading to trunk. All should be well, of course feel free to take a snapshot or make a backup before. Trunk, while not necessarily perfect, is MUCH faster and overall more stable than 1.2.0 was. Yes 1.2.0 was released, but there’s a reason development continues moving forward.

[Handtuch]

@Tom-Elliott thanks for this idea. Do you assume upgrading will fix this problem - or will it “only” prevent it to re-appear again?

[Tom Elliott]

@Handtuch It won’t fix what went strange, but it should prevent it from re-appearing.

[Wayne Workman]

@Handtuch it will fix it in the sense that fake images will stop being made. You will still need to play the cleanup game in MySQL. When I’m at a real computer I can provide commands for this.

[Handtuch]

@Wayne-Workman @Tom-Elliott thanks for your advice. I’m in the process of upgrading right now.

[Handtuch]

@Wayne-Workman @Tom-Elliott
You were wrong After upgrading to trunk, the empty images disappeared automagically. Let’s hope everything will work like expected on tomorrow’s next imaging-spree. At least, this specific problem has been solved.
Thanks a lot. I really mean it.

[Tom Elliott]

@Handtuch I have been working hard and I don’t remember all the work I’ve done.

[Wayne Workman]

@Handtuch cool. Glad to be wrong in this case lol.

[Handtuch]

@Tom-Elliott And you can’t calculate all the hours you’ve saved for other admins.