Upload Task: Updating database... failed. FTP problems



  • Hi!

    This might be duplicate or related of Image Capture/Upload gets to 100%, then Updating Database task fails and FTP Delete an image - failing but I couldn’t figure it out.

    I’m having problems with fresh install of Fog. Running Version 1.3.0-RC-2, SVN Revision: 5937 on Ubuntu 14.04.4.

    I’ve registered 3 hosts and added 3 Windows-7 images. Nothing more. I’m trying to upload and image to the fog server, but get the “Updating database… Failed” error.
    Error after uploading

    My apache2 error.log is flooded with PHP warnings ftp_delete() and ftp_rmdir() failing, with occasionally the apache2 process segfaulting.

    $ head -n 4 /var/log/apache2/error.log
    [Tue Jul 26 21:17:21.586806 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
    [Tue Jul 26 21:17:21.586899 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
    [Tue Jul 26 21:17:21.588240 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
    [Tue Jul 26 21:17:21.588307 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
    ...
    ...
    ...
    [Tue Jul 26 21:21:34.467151 2016] [core:notice] [pid 24528] AH00051: child pid 28846 exit signal Segmentation fault (11), possible coredump in /etc/apache2
    

    I think the permissions should allow deleting, since they are all 777 permissioned, just in case.

    $ ls -laR /images/
    /images/:
    total 28
    drwxrwxrwx  7 fog  root 4096 Jul 26 19:42 .
    drwxr-xr-x 25 root root 4096 Jul 26 15:39 ..
    drwxrwxrwx  3 fog  root 4096 Jul 26 15:59 dev
    drwxrwxrwx  2 fog  root 4096 Jul 26 18:16 Gizmok4
    drwxrwxrwx  3 fog  root 4096 Jul 25 16:52 Gizmok5
    drwxrwxrwx  3 fog  root 4096 Jul 25 16:56 Gizmok6
    -rwxrwxrwx  1 fog  root    0 Jul 26 15:39 .mntcheck
    drwxrwxrwx  2 fog  root 4096 Jul 26 15:39 postdownloadscripts
    
    /images/dev:
    total 12
    drwxrwxrwx 3 fog root 4096 Jul 26 15:59 .
    drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
    drwxrwxrwx 2 fog root 4096 Jul 26 21:17 bc5ff4473427
    -rwxrwxrwx 1 fog root    0 Jul 26 15:39 .mntcheck
    
    /images/dev/bc5ff4473427:
    total 450560628
    drwxrwxrwx 2 fog root         4096 Jul 26 21:17 .
    drwxrwxrwx 3 fog root         4096 Jul 26 15:59 ..
    -rwxrwxrwx 1 fog root      1048576 Jul 26 18:54 d1.mbr
    -rwxrwxrwx 1 fog root      8912823 Jul 26 18:54 d1p1.img
    -rwxrwxrwx 1 fog root 461364057634 Jul 26 21:17 d1p2.img
    -rwxrwxrwx 1 fog root          190 Jul 26 18:54 d1.partitions
    
    /images/Gizmok4:
    total 8
    drwxrwxrwx 2 fog root 4096 Jul 26 18:16 .
    drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
    
    /images/Gizmok5:
    total 424530580
    drwxrwxrwx 3 fog root         4096 Jul 25 16:52 .
    drwxrwxrwx 7 fog root         4096 Jul 26 19:42 ..
    -rwxrwxrwx 1 fog root      1048576 Jul 22 20:02 d1.mbr
    -rwxrwxrwx 1 fog root      8570406 Jul 22 20:52 d1p1.img
    -rwxrwxrwx 1 fog root 434709616470 Jul 22 20:52 d1p2.img
    -rwxrwxrwx 1 fog root          190 Jul 22 20:02 d1.partitions
    drwxrwxrwx 2 fog root        16384 Jul 22 19:57 lost+found
    
    /images/Gizmok5/lost+found:
    total 20
    drwxrwxrwx 2 fog root 16384 Jul 22 19:57 .
    drwxrwxrwx 3 fog root  4096 Jul 25 16:52 ..
    
    /images/Gizmok6:
    total 24
    drwxrwxrwx 3 fog root  4096 Jul 25 16:56 .
    drwxrwxrwx 7 fog root  4096 Jul 26 19:42 ..
    drwxrwxrwx 2 fog root 16384 Jul 25 16:56 lost+found
    
    /images/Gizmok6/lost+found:
    total 20
    drwxrwxrwx 2 fog root 16384 Jul 25 16:56 .
    drwxrwxrwx 3 fog root  4096 Jul 25 16:56 ..
    
    /images/postdownloadscripts:
    total 12
    drwxrwxrwx 2 fog root 4096 Jul 26 15:39 .
    drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
    -rwxrwxrwx 1 fog root  233 Jul 26 15:39 fog.postdownload
    

    I do have every image on their own HDD, if that might be the reason?

    $ tail -n3 /etc/fstab
    UUID=59ea75b7-e91e-4965-814c-4f54e2c15fe4       /images/Gizmok6 ext4    defaults        0       2
    UUID=9fb7e4b0-6237-4c8b-b746-22212d89ad10       /images/Gizmok5 ext4    defaults        0       2
    UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba       /images/Gizmok4 ext4    defaults        0       2
    

    As I checked, the FTP credentials seemed to be correct in all the places shown in here, as they were set on the install process. I haven’t touched them. Also I tested the FTP as shown here and all the steps went through with no problem.

    I would appreciate a poke to the right direction :) Thanks!



  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    @zachu THe reason the path is specified (not allowing slashes) is because it can give access to other parts of the filesystem. This doesn’t mean it WILL happen, but from a security perspective what’s to ensure you don’t write a specific file or overwrite the root folder? I know there’s some other obfuscated methods of protection here, but let’s just imagine, you decide to put the image path be be as /images/../../../../bin/init or /images/../../../../usr/bin/init or some other binary name.
    While technically it shouldn’t be possible, with the wrong setup it is a possibility. Now your fog server will likely work for a while. Then start having some funny issues.

    I perfectly understand why it’s like this. I was more like toying with different ways to go around the problem I had. I’m ok with the way it works now.

    The Wiki is perfectly accurate. You would, essentially, change the images location to go from /images to /images/newdisk. No where in the wiki article, from what I’ve skimmed through, would state that you are placing the image on the disk in the way you originally described. Changing the way move_upload is not quite that simple.

    Oh right, I think I then read too far there between the lines. With the guide on the wiki article’s last lines, it’s meant to move all your images under the “newdisk”, not just one or part of them. I get it now :)

    Changing the way move_upload is not quite that simple.

    Yeah, I glanced just quickly at the codebase there and realized that it isn’t just a minor change, but I think it is doable still. All the needed actions are moving (renaming) and deleting files and folders, just as it is done now. Now it’s really simple and pretty when you can just move the whole directory instead of doing multiple actions to the files.

    Would you accept a pull request of changing the move_upload process, if I find time to put it together? I’m not making any promises, but just thinking about it. I’ve never really contributed to an open source project but I might have the skills to put that one together. But if it fights the way you want it to work, then I wont bother :)

    I really thank you for your time!

    Btw, I settled for a process that I just move the image files to the dedicated HDD always after upload. It’ll do now, since it’s only temporary :)


  • Senior Developer

    @zachu THe reason the path is specified (not allowing slashes) is because it can give access to other parts of the filesystem. This doesn’t mean it WILL happen, but from a security perspective what’s to ensure you don’t write a specific file or overwrite the root folder? I know there’s some other obfuscated methods of protection here, but let’s just imagine, you decide to put the image path be be as /images/../../../../bin/init or /images/../../../../usr/bin/init or some other binary name.
    While technically it shouldn’t be possible, with the wrong setup it is a possibility. Now your fog server will likely work for a while. Then start having some funny issues.

    I mean if you want to allow it, by all means, please, but I’m not going to make that a possibility for ALL people’s insitalls.

    The Wiki is perfectly accurate. You would, essentially, change the images location to go from /images to /images/newdisk. No where in the wiki article, from what I’ve skimmed through, would state that you are placing the image on the disk in the way you originally described. Changing the way move_upload is not quite that simple.

    Again we’re using NFS to perform the fastest possibly mounting and distributing the image. Moving the files are done through the FTP interface, not from the NFS. Because of this, you can only put or rename the path. This is not going to change.



  • Okay so imo this topic can be marked as solved. I think @Tom-Elliott’s approach is currently the closest one to go for my needs. Really big thanks to others too that helped me with this. For others with similar needs, @george1421’s approach may be good way to go too.

    So a wrap-up:
    My goal was to mount a device /dev/sdb1 to /images/Gizmok4 so that when I upload a image Gizmok4, it goes to a dedicated HDD. I assumed this was a valid approach since it is hinted so in Fog wiki

    Fog wiki in Moving_your_images_directory/Adding_Storage_to_the_Images_directory states:

    If you want to just add capacity to your images directory, you could instead mount your new device as a subfolder of the /images directory. In this case your wouldn’t need to move the original directory to /images1, and your fstab line would look like:

    UUID=XXXXXXXXXXXXXXXXXXXXXX    /images/newdisk      ext3 defaults 0 0
    

    As it turned out, you can’t do this, since after an upload task fog deletes the image path and renames the uploaded image directory in place of the old image directory the image path and renames the uploaded image directory in place of the old image directory. The task fails, because the mountpoint isn’t deletable. One approach at fogprojects end to fix or allow this, is instead of deleting and renaming, just move the content of /images/dev/<macaddr> to /images/dev/<imagename> (<imagename> being Gizmok4 in my case).

    I figured this limitation could be circled around by putting one directory before the image directory as a mountpoint. Mounting /dev/sdb1 to something like /images/sdb and have <imagename> folder under that (again Gizmok4 in my case). Then I thought to just change the Gizmok4 image path to /images/sdb/Gizmok4 and everything would be fixed. Now the image directory is removable and the process of deleting old files and renaming new files on the same place would work. In other words, I tried something like this:

    $ tree /images
    /images
    ├── dev
    ├── postdownloadscripts
    │   └── fog.postdownload
    └── sdb
        └── Gizmok4
            ├── d1.mbr
            ├── d1p1.img
            ├── d1p2.img
            └── d1.partitions
    

    The problem in this approach comes from Fog WebUI, more specifically when validating the image path input in fog.image.js. Only alphanumeric input and a plus (+) character is allowed there.

    Out of curiosity, I did disable that part of the javascript and uploaded an image to a path sdb/Gizmok4 (full path being /images/sdb/Gizmok4). Now for some reason I ended up having a file /images/sdb/Gizmok4 instead of a directory. All the real image files were kept in the dev/<macaddr> :)

    Conclusion… and a workaround request

    All in all, I think the wiki article (or the last part of it) is misleading, since it isn’t possible to add storage to /images/ by just mounting another device as a subfolder, as it suggests. Well, you can do it, but you can’t use it :) My opinion is that this could be very handy addition though and should be implemented. I think the best way to do that would be reworking the move_upload() function in taskqueue.class.php to work as following:

    1. Empty the image directory. (Currently the directory gets deleted)
    2. Move the contents of /images/dev/<macaddr> to the image directory. (Currently the whole directory gets moved/renamed)
    3. Remove the directory /images/dev/<macaddr>. (Currently it gets renamed, so no removing needed)

  • Senior Developer

    @zachu I’m only using Gizmok4 as the basis of information I need to better help you.

    Then what you are actually looking to do is make your disks the new “images” location.

    The “fix” for this would be to mount your disk AS /images. From there, you would need to create the dev folder and the requisite .mntcheck files. Mount your disk, and upload your image to the appropriate disk.

    The way to do:

    mount /dev/sdb1 /images
    mkdir /images/dev
    touch /images/.mntcheck
    touch /images/dev/.mntcheck
    chmod -R 777 /images
    

    Perform your upload operation for gizmok4.

    umount /images
    mount /dev/sdc1 /images
    mkdir /images/dev
    touch /images/.mntcheck
    touch /images/dev/.mntcheck
    chmod -R 777 /images
    

    Perform your upload operation for gizmok5

    umount /images
    mount /dev/sdd1 /images
    mkdir /images/dev
    touch /images/.mntcheck
    touch /images/dev/.mntcheck
    chmod -R 777 /images
    

    Perform your upload operation for gizmok6

    umount /images
    

    You will then have what you’ve been intending to do this whole time.



  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    So now I need further understanding.

    If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

    No. I have an image called Gizmok4 which locates in /images/Gizmok4 as it does on default settings. Picture of image Gizmok4

    I also have an images named Gizmok5 and Gizmok6, which locates in /images/Gizmok5 and /images/Gizmok6 respectively. They are on same settings as the Gizmok4-image with the exception of Gizmok6 being multiple disk image. Although that’s irrelevant here :)

    Why not mount /dev/sdb1 (or your relevant items) to /images?

    I need the images to be on different physical disks (one disk for one image) for reasons explained before. Mainly because of portability (need to physically carry the image to another location) and because of disk space (all the images don’t fit on one disk).

    To understand further, Gizmok4 is intended to be the <imagename> placeholder?

    Yes, Gizmok4 is the name of one of my images.



  • @george1421 said in Upload Task: Updating database... failed. FTP problems:

    @zachu make sure the fog (linux) user has full read/write access to those attached disks.

    The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

    I tried again the FTP on the fog user. I surely can create, write and delete on /images/dev/ and on /images/ and on /images/Gizmok4/. What I can’t do through ftp, is delete /images/Gizmok4 since it is a mountpoint. So I still guess that’s what the fog is trying to do on the “Updating database…” part. It deletes the /images/<imagename> and renames /images/dev/<macaddr> to /images/<imagename>. It fails because it cant delete the mountpoint.


  • Senior Developer

    So now I need further understanding.

    If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

    Why not mount /dev/sdb1 (or your relevant items) to /images?

    To understand further, Gizmok4 is intended to be the <imagename> placeholder?


  • Moderator

    @zachu make sure the fog (linux) user has full read/write access to those attached disks.

    The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.



  • @george1421 said in Upload Task: Updating database... failed. FTP problems:

    @zachu If it was me I’d go about it a little differently.

    What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

    This is actually about how we normally do things, and then transfer the images on network, but because of temporary special circumstances, I have to be on a slow net connection :)

    Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

    Yeah indeed this sounds easiest way to go, but since this is a temporary situation, I can’t get my hands on a usb drive easily :( Not until it’s not needed again. But good advices anyway. I think I’ll head that way if I can’t get this work some other way.

    Maybe this whole thing could work also if I’d put one directory after the mountpoint. Like this: mount /dev/sdb1 /images/sdb, and then I’d change the image path to /images/sdb/Gizmok4. If the problem is that fog cannot remove the mountpoint as I’ve guessed it is, then that would probably solve the problem. I’ll test this.



  • In other words, the only thing that diffrences here on just default out-of-box fog usage, is this:
    I would like that /images/<imagename> would be contained on a single HDD device that I can take away from the server and carry to another place.

    • /images/Gizmok4 => /dev/sdb
    • /images/Gizmok5 => /dev/sdc
    • /images/Gizmok6 => /dev/sdd


  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    @zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

    Umm… no I don’t need the mounted drive to be a clone on the system. I need it to contain the image files so I can deploy it elsewhere :)

    I have 3 completely different Windows7 hosts registered to my fog server. They all have different images. I’m trying to clone one of them and have the image files on the fog server /images/Gizmok4 folder. Another one would be on /images/Gizmok5 and the third on /images/Gizmok6. This is all regular fog usage?

    Now it would help me tremendously if that /images/Gizmok4/ (as well as the other image locations) would be a mountpoint for a single HDD that I coud take with me to another site when I’m deploying the image to a bunch of computers. There I also have a fog storage node that I can use to deploy the image. The transfering part is just a problem now, since I’m temporarily sitting on slow connection.

    Clarified anything? :)


  • Senior Developer

    @zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?



  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    @zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

    If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

    Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

    The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk :) No virtualization or anything.


  • Moderator

    @zachu If it was me I’d go about it a little differently.

    What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

    Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

    [edit] I did write a kb article that covers what I just mentioned above. https://forums.fogproject.org/topic/6642/moving-fog-s-images-files-off-the-root-partition [/edit]



  • @george1421 said in Upload Task: Updating database... failed. FTP problems:

    @zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

    So again, why do you have these disks mounted in this configuration?

    I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

    Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.


  • Senior Developer

    @zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

    If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

    Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.


  • Moderator

    @zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

    So again, why do you have these disks mounted in this configuration? I want to understand the business logic behind your setup?



  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    Where is /dev/sdb1?

    Now I don’t know what you mean :) If you mean now that the upload task worked, it’s still unmounted and not in use.


  • Senior Developer

    Where is /dev/sdb1?



  • @Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

    @zachu I don’t know what you mean.

    Please explain.

    Well wrapping up this whole thread:

    • I had mounted /dev/sdb1 to /images/Gizmok4
    • Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
    • /var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
    • I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

Log in to reply
 

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.