"Failed to create deployment tasking..."


  • Moderator

    So, when I tried to image my workstation today (in order to update my image), I get this error…

    “Failed to create deployment tasking for the following hosts
    You must first upload an image to create a download task”

    But, I have about three images created already on the fog server. They show up under images. And I have the correct image assigned to my workstation.

    I can also create tasks for other hosts… just not MY host lol.

    I’ve not actually logged into the Fog machine in quite some time, and haven’t changed any settings.
    Today, I did some unicast imaging on some laptops earlier and it worked fine.

    I tried rebooting the Fog machine, twice.
    I’ve tried deleting my host and re-creating it… thought it was just a web GUI issue, so I did the full-registration when booting to the network, and set it up with an image that way, and told it to image. It goes through the paces, and then says “Image not found” or something.

    I ran some systemctl status commands, and found only one thing that looked weird.

    systemctl status FOGImageReplicator.service -l

    it gave about 10 errors, all identical.

    PHP Warning: Invalid argument supplied for foreach() in /var/www/html/fog/lib/fog/ImageReplicator.class.php on line 48

    I’ve checked the Log Viewer in the Web GUI, and nothing seems out of place.

    Using SVN 2920 (code named the “Wayne” version) :-)

    Any help is appreciated.


  • Developer

    [quote=“Wayne Workman, post: 45339, member: 28155”]So the moral of the story is to use ext4 ?[/quote]
    Well, there are a lot of other nice filesystems out there too! ext4 is working for us right now and I am sure it will for others having an issue with deleting hugh files on ext3. But I am not saying that ext4 is the only fs that can do this…


  • Moderator

    So the moral of the story is to use ext4 ?


  • Developer

    Learned something new … (pretty much every day!)

    ext3 does some kind of defragmentation/reallocation foooo when files are being deleted!

    [QUOTE]ext3, when freeing disk space is doing some kind of defragmentation, which moves some data around to maximize continuous free disk space.
    So, when we had 3 large files (blocks A, B and C), and some free space (.):
    AAAAAAAABBBBBBBBCCCCCCCC…
    and I removed whole A:
    …BBBBBBBBCCCCCCCC…
    ext3 relocated some blocks (most likely from C):
    CCCCCCCCBBBBBBBB…[/QUOTE]
    [url]http://www.depesz.com/2010/04/04/how-to-remove-backups/[/url]

    I (re)moved all my images, re-formated with ext4, put it all back together and now everything seems fine. Deleting of 150GB image takes less than 10 seconds. Awesome!!! I really wonder why very little people have issues with this. Is ext3 not in use in most recent distributions??

    [B]Could anyone please add [SOLVED] to the title, thanks![/B]


  • Developer

    Thanks Tom! That’s great! Although I think we have a real issue with our virtualization software (proxmox). Still trying to find out why deleting big files takes MINUTES… definitely shouldn’t be the case!!


  • Senior Developer

    If added two new capabilities to the general settings of the fog configuration. FOG_FTP_TIMEOUT and FOG_FTP_PORT. The default timeout is now set to 90 but should reflect whatever is put in this field though I still need to add some error checking.


  • Developer

    Nope, never ever so far!


  • Moderator

    Did you experience an image going missing?


  • Developer

    It’s been a long time since I looked into this but I didn’t have enough time until now…

    I am now able to reproduce this on our machines and then captured the FTP traffic to see exactly what’s happening when things go wrong.
    [LIST]
    []Case 1: Create a complete new image in the web GUI and upload. Works like a charm!
    [
    ]Case 2: Upload a second (third …) time. Bam! Image does not get moved…
    [*]Case 3: Delete /image/<name> before uploading. And again, rename works like a charm…
    [/LIST]
    The problem is that our image is very big (150GB+). But also smaller (50GB) images fail on out servers. [B]FTP timeout is set to 10 seconds [/B]but deleting such hugh files takes longer than 10 seconds on our system…

    I cranked it up to 90 seconds (PHP default value) and the 50GB images work now. 150GB image is still an issue (testing…). Maybe it’s just our storage system that takes ages to delete big files…??


  • Moderator

    I can’t put a finger on when the image went missing… that was a while back. I’ve simply shrugged, rebuilt a new image, WRITE PROTECTED all the images, and moved on.

    But, my image DID go missing that one time, and I didn’t know what the hell happened.


  • Developer

    [quote=“Uncle Frank, post: 42829, member: 28116”]dump[/quote]
    I am dump… sorry for this. I was in a hurry today and just remembered that this “issue” is still not solved from my point of view. I quite often see newly uploaded images not being moved BUT the old image being deleted! I am getting used to it but it’s definitely a bit scary if you encounter this at first.
    Probably a problem with how my vsftpd is setup?? So far I didn’t have the time to look into this any further…
    Depending on the scenario you either don’t need extra space (FOG is on the same server as storage node is!) or you do need a lot of it (storage is a different node). Would it be a good idea to check if FOG and storage are the same server, move things locally in the filesystem and delete only if rename went well?? What do you think?
    Maybe it’s just me but I think I’ve read about other people also wondering about “missing uploaded images”.


  • Developer

    [quote=“Uncle Frank, post: 42228, member: 28116”]Ok I get what you’re saying about OSID 1/2 check and moving but I still wonder why ‘$ftp->delete($dest)’ is called before moving the new image to that destination. If things go wrong with moving you end up with no image in ‘/images’. What about renaming ‘/images/<imagePathName>’ e.g. to ‘/images/<imagePathName><random seed>’ and deleting it after moving from dev was successful??[/quote]

    I thought i would add, the only time i’ve really seen anything “go wrong” with moving the image to /images is when there’s an issue with ftp working. In which case, the delete operation will have failed too.


  • Senior Developer

    The reason for the delete is two fold.

    While moving the “random-seed” might work, it also means you have to have space to do such a thing. I know not all have the luxury of tons of extra space.

    I do understand where your thoughts lie.


  • Senior Developer

    dump or bump?


  • Developer

    dump


  • Developer

    Ok I get what you’re saying about OSID 1/2 check and moving but I still wonder why ‘$ftp->delete($dest)’ is called before moving the new image to that destination. If things go wrong with moving you end up with no image in ‘/images’. What about renaming ‘/images/<imagePathName>’ e.g. to ‘/images/<imagePathName><random seed>’ and deleting it after moving from dev was successful??


  • Senior Developer

    [quote=“Uncle Frank, post: 42182, member: 28116”]Did you do a fresh upload already?? Hope you didn’t. One idea I just had is that possibly something went wrong when renaming the image from ‘/images/dev/<mac>/’ to ‘/images/<imagename>/’ on your last upload. So the image was still sitting in ‘/images/dev’… (maybe?!?).

    I am new to this part of FOG and tried to find out about this in the source code. Would anyone with more insight please comment on this if I got this wrong. From what I can see in fog.upload script and Post_State2.php it seams like the old image gets deleted BEFORE the new one is moved in place. In case anything goes wrong when moving the new image you end up with the new image in ‘/images/dev’ and no old image in ‘/images’… is that correct??? I’d suggest to rename the old image first, then move the new one from dev and only then delete the old one!
    Post_Stage2.php
    [CODE]…
    // Try to delete the file. Doesn’t hurt anything if it doesn’t delete anything.
    $ftp->delete($dest);
    if ($ftp->rename($dest,$src)||$ftp->put($dest,$src))
    ($_REQUEST[‘osid’] == ‘1’ || $REQUEST[‘osid’] == ‘2’ ? $ftp->delete($StorageNode->get(‘path’).’/dev/’.$macftp) : null);
    else
    throw new Exception(
    (‘Move/rename failed.’));
    $ftp->close();
    …[/CODE]
    And another (unrelated) question arises, why check for OSID 1/2 here???[/quote]

    The OSID 1/2 is from the old methods of how filenames were stored. Windows XP/Vista was a single file and that file was the “imagepath”

    Because of how the image used to be created the folder would remain in the directory, because the file was named something like:

    /images/dev/<macFolderName>/<macFileName>

    The file was moved as the imagepath name:

    /images/dev/<macFolderName>/<macFileName> /images/<imagePathName>

    The Folder /images/dev/<macFolderName/ would not be removed. That’s why the check is there.


  • Moderator

    I did check /images/dev
    The image is not in there.

    The image that disappeared is not a new one. It’s one that’s been on the FOG server for probably 3 weeks now. I’ve used that image many times already, including imaging an entire lab of 29. All went well.

    I have a suspicion of what caused it… Won’t let off on it yet but, if I am correct, I’ll post here.


  • Developer

    Did you do a fresh upload already?? Hope you didn’t. One idea I just had is that possibly something went wrong when renaming the image from ‘/images/dev/<mac>/’ to ‘/images/<imagename>/’ on your last upload. So the image was still sitting in ‘/images/dev’… (maybe?!?).

    I am new to this part of FOG and tried to find out about this in the source code. Would anyone with more insight please comment on this if I got this wrong. From what I can see in fog.upload script and Post_State2.php it seams like the old image gets deleted BEFORE the new one is moved in place. In case anything goes wrong when moving the new image you end up with the new image in ‘/images/dev’ and no old image in ‘/images’… is that correct??? I’d suggest to rename the old image first, then move the new one from dev and only then delete the old one!
    Post_Stage2.php
    [CODE]…
    // Try to delete the file. Doesn’t hurt anything if it doesn’t delete anything.
    $ftp->delete($dest);
    if ($ftp->rename($dest,$src)||$ftp->put($dest,$src))
    ($_REQUEST[‘osid’] == ‘1’ || $REQUEST[‘osid’] == ‘2’ ? $ftp->delete($StorageNode->get(‘path’).’/dev/’.$macftp) : null);
    else
    throw new Exception(
    (‘Move/rename failed.’));
    $ftp->close();
    …[/CODE]
    And another (unrelated) question arises, why check for OSID 1/2 here???


  • Moderator

    I’m going to copy my images from the /images directory to somewhere else from now on. Just in case this happens again; I can simply copy the image back to where it should be.

    It’s going to use up space, but I don’t know what else to do to mitigate the problem in the future. I suppose this will use less space than snap shots.


Log in to reply
 

422
Online

39.3k
Users

11.0k
Topics

104.6k
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.