"Failed to create deployment tasking..."
-
I created a new storage group called “MB” and added all the images to that group. I then assigned MB to the default storage node, and I deleted the default storage group. Still no luck.
Also, it would seem that it’s not just my host, it’s that particular image. The other images create tasks just fine. It’s just this one particular image that is troubling. It’s weird though, because, it was working fine two weeks ago, and I haven’t changed anything (before it broke).
I don’t get it.
-
Ok I just figured out the problem… it’s scary, like, really scary…
I went to look at the actual images folder on the FOG server, and this problematic image isn’t there… it’s just… gone…
It was there before… I used it two weeks ago on an entire lab! And it’s still showing up in the Web GUI ?!?
This is crazy. What happened to my image?
So, I made a snapshot of the FOG server as it currently is, so I can troubleshoot later if possible.
I’m going to just press forward and re-make the image. Good thing I had a backup copy in GHOST format.
-
glad you found the problem. I’ve never heard of anyone’s images disappearing without reason, so i hope you can track down what happened.
-
I’m going to copy my images from the /images directory to somewhere else from now on. Just in case this happens again; I can simply copy the image back to where it should be.
It’s going to use up space, but I don’t know what else to do to mitigate the problem in the future. I suppose this will use less space than snap shots.
-
Did you do a fresh upload already?? Hope you didn’t. One idea I just had is that possibly something went wrong when renaming the image from ‘/images/dev/<mac>/’ to ‘/images/<imagename>/’ on your last upload. So the image was still sitting in ‘/images/dev’… (maybe?!?).
I am new to this part of FOG and tried to find out about this in the source code. Would anyone with more insight please comment on this if I got this wrong. From what I can see in fog.upload script and Post_State2.php it seams like the old image gets deleted BEFORE the new one is moved in place. In case anything goes wrong when moving the new image you end up with the new image in ‘/images/dev’ and no old image in ‘/images’… is that correct??? I’d suggest to rename the old image first, then move the new one from dev and only then delete the old one!
Post_Stage2.php
[CODE]…
// Try to delete the file. Doesn’t hurt anything if it doesn’t delete anything.
$ftp->delete($dest);
if ($ftp->rename($dest,$src)||$ftp->put($dest,$src))
($_REQUEST[‘osid’] == ‘1’ || $REQUEST[‘osid’] == ‘2’ ? $ftp->delete($StorageNode->get(‘path’).‘/dev/’.$macftp) : null);
else
throw new Exception((‘Move/rename failed.’));
$ftp->close();
…[/CODE]
And another (unrelated) question arises, why check for OSID 1/2 here??? -
I did check /images/dev
The image is not in there.The image that disappeared is not a new one. It’s one that’s been on the FOG server for probably 3 weeks now. I’ve used that image many times already, including imaging an entire lab of 29. All went well.
I have a suspicion of what caused it… Won’t let off on it yet but, if I am correct, I’ll post here.
-
[quote=“Uncle Frank, post: 42182, member: 28116”]Did you do a fresh upload already?? Hope you didn’t. One idea I just had is that possibly something went wrong when renaming the image from ‘/images/dev/<mac>/’ to ‘/images/<imagename>/’ on your last upload. So the image was still sitting in ‘/images/dev’… (maybe?!?).
I am new to this part of FOG and tried to find out about this in the source code. Would anyone with more insight please comment on this if I got this wrong. From what I can see in fog.upload script and Post_State2.php it seams like the old image gets deleted BEFORE the new one is moved in place. In case anything goes wrong when moving the new image you end up with the new image in ‘/images/dev’ and no old image in ‘/images’… is that correct??? I’d suggest to rename the old image first, then move the new one from dev and only then delete the old one!
Post_Stage2.php
[CODE]…
// Try to delete the file. Doesn’t hurt anything if it doesn’t delete anything.
$ftp->delete($dest);
if ($ftp->rename($dest,$src)||$ftp->put($dest,$src))
($_REQUEST[‘osid’] == ‘1’ || $REQUEST[‘osid’] == ‘2’ ? $ftp->delete($StorageNode->get(‘path’).‘/dev/’.$macftp) : null);
else
throw new Exception((‘Move/rename failed.’));
$ftp->close();
…[/CODE]
And another (unrelated) question arises, why check for OSID 1/2 here???[/quote]The OSID 1/2 is from the old methods of how filenames were stored. Windows XP/Vista was a single file and that file was the “imagepath”
Because of how the image used to be created the folder would remain in the directory, because the file was named something like:
/images/dev/<macFolderName>/<macFileName>
The file was moved as the imagepath name:
/images/dev/<macFolderName>/<macFileName> /images/<imagePathName>
The Folder /images/dev/<macFolderName/ would not be removed. That’s why the check is there.
-
Ok I get what you’re saying about OSID 1/2 check and moving but I still wonder why ‘$ftp->delete($dest)’ is called before moving the new image to that destination. If things go wrong with moving you end up with no image in ‘/images’. What about renaming ‘/images/<imagePathName>’ e.g. to ‘/images/<imagePathName><random seed>’ and deleting it after moving from dev was successful??
-
dump
-
dump or bump?
-
The reason for the delete is two fold.
While moving the “random-seed” might work, it also means you have to have space to do such a thing. I know not all have the luxury of tons of extra space.
I do understand where your thoughts lie.
-
[quote=“Uncle Frank, post: 42228, member: 28116”]Ok I get what you’re saying about OSID 1/2 check and moving but I still wonder why ‘$ftp->delete($dest)’ is called before moving the new image to that destination. If things go wrong with moving you end up with no image in ‘/images’. What about renaming ‘/images/<imagePathName>’ e.g. to ‘/images/<imagePathName><random seed>’ and deleting it after moving from dev was successful??[/quote]
I thought i would add, the only time i’ve really seen anything “go wrong” with moving the image to /images is when there’s an issue with ftp working. In which case, the delete operation will have failed too.
-
[quote=“Uncle Frank, post: 42829, member: 28116”]dump[/quote]
I am dump… sorry for this. I was in a hurry today and just remembered that this “issue” is still not solved from my point of view. I quite often see newly uploaded images not being moved BUT the old image being deleted! I am getting used to it but it’s definitely a bit scary if you encounter this at first.
Probably a problem with how my vsftpd is setup?? So far I didn’t have the time to look into this any further…
Depending on the scenario you either don’t need extra space (FOG is on the same server as storage node is!) or you do need a lot of it (storage is a different node). Would it be a good idea to check if FOG and storage are the same server, move things locally in the filesystem and delete only if rename went well?? What do you think?
Maybe it’s just me but I think I’ve read about other people also wondering about “missing uploaded images”. -
I can’t put a finger on when the image went missing… that was a while back. I’ve simply shrugged, rebuilt a new image, WRITE PROTECTED all the images, and moved on.
But, my image DID go missing that one time, and I didn’t know what the hell happened.
-
It’s been a long time since I looked into this but I didn’t have enough time until now…
I am now able to reproduce this on our machines and then captured the FTP traffic to see exactly what’s happening when things go wrong.
[LIST]
[]Case 1: Create a complete new image in the web GUI and upload. Works like a charm!
[]Case 2: Upload a second (third …) time. Bam! Image does not get moved…
[*]Case 3: Delete /image/<name> before uploading. And again, rename works like a charm…
[/LIST]
The problem is that our image is very big (150GB+). But also smaller (50GB) images fail on out servers. [B]FTP timeout is set to 10 seconds [/B]but deleting such hugh files takes longer than 10 seconds on our system…I cranked it up to 90 seconds (PHP default value) and the 50GB images work now. 150GB image is still an issue (testing…). Maybe it’s just our storage system that takes ages to delete big files…??
-
Did you experience an image going missing?
-
Nope, never ever so far!
-
If added two new capabilities to the general settings of the fog configuration. FOG_FTP_TIMEOUT and FOG_FTP_PORT. The default timeout is now set to 90 but should reflect whatever is put in this field though I still need to add some error checking.
-
Thanks Tom! That’s great! Although I think we have a real issue with our virtualization software (proxmox). Still trying to find out why deleting big files takes MINUTES… definitely shouldn’t be the case!!
-
Learned something new … (pretty much every day!)
ext3 does some kind of defragmentation/reallocation foooo when files are being deleted!
[QUOTE]ext3, when freeing disk space is doing some kind of defragmentation, which moves some data around to maximize continuous free disk space.
So, when we had 3 large files (blocks A, B and C), and some free space (.):
AAAAAAAABBBBBBBBCCCCCCCC…
and I removed whole A:
…BBBBBBBBCCCCCCCC…
ext3 relocated some blocks (most likely from C):
CCCCCCCCBBBBBBBB…[/QUOTE]
[url]http://www.depesz.com/2010/04/04/how-to-remove-backups/[/url]I (re)moved all my images, re-formated with ext4, put it all back together and now everything seems fine. Deleting of 150GB image takes less than 10 seconds. Awesome!!! I really wonder why very little people have issues with this. Is ext3 not in use in most recent distributions??
[B]Could anyone please add [SOLVED] to the title, thanks![/B]
