Upload Task: Updating database... failed. FTP problems

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I don’t know what you mean.

Please explain.

Well wrapping up this whole thread:

• I had mounted /dev/sdb1 to /images/Gizmok4
• Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
• /var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
• I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

[Tom Elliott]

Where is /dev/sdb1?

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

Where is /dev/sdb1?

Now I don’t know what you mean If you mean now that the upload task worked, it’s still unmounted and not in use.

[george1421]

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration? I want to understand the business logic behind your setup?

[Tom Elliott]

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration?

I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.

[george1421]

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

[edit] I did write a kb article that covers what I just mentioned above. https://forums.fogproject.org/topic/6642/moving-fog-s-images-files-off-the-root-partition [/edit]

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk No virtualization or anything.

[Tom Elliott]

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

Umm… no I don’t need the mounted drive to be a clone on the system. I need it to contain the image files so I can deploy it elsewhere

I have 3 completely different Windows7 hosts registered to my fog server. They all have different images. I’m trying to clone one of them and have the image files on the fog server /images/Gizmok4 folder. Another one would be on /images/Gizmok5 and the third on /images/Gizmok6. This is all regular fog usage?

Now it would help me tremendously if that /images/Gizmok4/ (as well as the other image locations) would be a mountpoint for a single HDD that I coud take with me to another site when I’m deploying the image to a bunch of computers. There I also have a fog storage node that I can use to deploy the image. The transfering part is just a problem now, since I’m temporarily sitting on slow connection.

Clarified anything?

[zachu]

In other words, the only thing that diffrences here on just default out-of-box fog usage, is this:
I would like that /images/<imagename> would be contained on a single HDD device that I can take away from the server and carry to another place.

• /images/Gizmok4 => /dev/sdb
• /images/Gizmok5 => /dev/sdc
• /images/Gizmok6 => /dev/sdd

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

This is actually about how we normally do things, and then transfer the images on network, but because of temporary special circumstances, I have to be on a slow net connection

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

Yeah indeed this sounds easiest way to go, but since this is a temporary situation, I can’t get my hands on a usb drive easily Not until it’s not needed again. But good advices anyway. I think I’ll head that way if I can’t get this work some other way.

Maybe this whole thing could work also if I’d put one directory after the mountpoint. Like this: mount /dev/sdb1 /images/sdb, and then I’d change the image path to /images/sdb/Gizmok4. If the problem is that fog cannot remove the mountpoint as I’ve guessed it is, then that would probably solve the problem. I’ll test this.

[george1421]

@zachu make sure the fog (linux) user has full read/write access to those attached disks.

The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

[Tom Elliott]

So now I need further understanding.

If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

Why not mount /dev/sdb1 (or your relevant items) to /images?

To understand further, Gizmok4 is intended to be the <imagename> placeholder?

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu make sure the fog (linux) user has full read/write access to those attached disks.

The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

I tried again the FTP on the fog user. I surely can create, write and delete on /images/dev/ and on /images/ and on /images/Gizmok4/. What I can’t do through ftp, is delete /images/Gizmok4 since it is a mountpoint. So I still guess that’s what the fog is trying to do on the “Updating database…” part. It deletes the /images/<imagename> and renames /images/dev/<macaddr> to /images/<imagename>. It fails because it cant delete the mountpoint.

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

So now I need further understanding.

If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

No. I have an image called Gizmok4 which locates in /images/Gizmok4 as it does on default settings.

I also have an images named Gizmok5 and Gizmok6, which locates in /images/Gizmok5 and /images/Gizmok6 respectively. They are on same settings as the Gizmok4-image with the exception of Gizmok6 being multiple disk image. Although that’s irrelevant here

Why not mount /dev/sdb1 (or your relevant items) to /images?

I need the images to be on different physical disks (one disk for one image) for reasons explained before. Mainly because of portability (need to physically carry the image to another location) and because of disk space (all the images don’t fit on one disk).

To understand further, Gizmok4 is intended to be the <imagename> placeholder?

Yes, Gizmok4 is the name of one of my images.

[Tom Elliott]

@zachu I’m only using Gizmok4 as the basis of information I need to better help you.

Then what you are actually looking to do is make your disks the new “images” location.

The “fix” for this would be to mount your disk AS /images. From there, you would need to create the dev folder and the requisite .mntcheck files. Mount your disk, and upload your image to the appropriate disk.

The way to do:

mount /dev/sdb1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok4.

umount /images
mount /dev/sdc1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok5

umount /images
mount /dev/sdd1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok6

umount /images

You will then have what you’ve been intending to do this whole time.

[zachu]

Okay so imo this topic can be marked as solved. I think @Tom-Elliott’s approach is currently the closest one to go for my needs. Really big thanks to others too that helped me with this. For others with similar needs, @george1421’s approach may be good way to go too.

So a wrap-up:
My goal was to mount a device /dev/sdb1 to /images/Gizmok4 so that when I upload a image Gizmok4, it goes to a dedicated HDD. I assumed this was a valid approach since it is hinted so in Fog wiki

Fog wiki in Moving_your_images_directory/Adding_Storage_to_the_Images_directory states:

If you want to just add capacity to your images directory, you could instead mount your new device as a subfolder of the /images directory. In this case your wouldn’t need to move the original directory to /images1, and your fstab line would look like:

UUID=XXXXXXXXXXXXXXXXXXXXXX    /images/newdisk      ext3 defaults 0 0

As it turned out, you can’t do this, since after an upload task fog deletes the image path and renames the uploaded image directory in place of the old image directory the image path and renames the uploaded image directory in place of the old image directory. The task fails, because the mountpoint isn’t deletable. One approach at fogprojects end to fix or allow this, is instead of deleting and renaming, just move the content of /images/dev/<macaddr> to /images/dev/<imagename> (<imagename> being Gizmok4 in my case).

I figured this limitation could be circled around by putting one directory before the image directory as a mountpoint. Mounting /dev/sdb1 to something like /images/sdb and have <imagename> folder under that (again Gizmok4 in my case). Then I thought to just change the Gizmok4 image path to /images/sdb/Gizmok4 and everything would be fixed. Now the image directory is removable and the process of deleting old files and renaming new files on the same place would work. In other words, I tried something like this:

$ tree /images
/images
├── dev
├── postdownloadscripts
│   └── fog.postdownload
└── sdb
    └── Gizmok4
        ├── d1.mbr
        ├── d1p1.img
        ├── d1p2.img
        └── d1.partitions

The problem in this approach comes from Fog WebUI, more specifically when validating the image path input in fog.image.js. Only alphanumeric input and a plus (+) character is allowed there.

Out of curiosity, I did disable that part of the javascript and uploaded an image to a path sdb/Gizmok4 (full path being /images/sdb/Gizmok4). Now for some reason I ended up having a file /images/sdb/Gizmok4 instead of a directory. All the real image files were kept in the dev/<macaddr>

Conclusion… and a workaround request

All in all, I think the wiki article (or the last part of it) is misleading, since it isn’t possible to add storage to /images/ by just mounting another device as a subfolder, as it suggests. Well, you can do it, but you can’t use it My opinion is that this could be very handy addition though and should be implemented. I think the best way to do that would be reworking the move_upload() function in taskqueue.class.php to work as following:

1. Empty the image directory. (Currently the directory gets deleted)
2. Move the contents of /images/dev/<macaddr> to the image directory. (Currently the whole directory gets moved/renamed)
3. Remove the directory /images/dev/<macaddr>. (Currently it gets renamed, so no removing needed)

[Tom Elliott]

@zachu THe reason the path is specified (not allowing slashes) is because it can give access to other parts of the filesystem. This doesn’t mean it WILL happen, but from a security perspective what’s to ensure you don’t write a specific file or overwrite the root folder? I know there’s some other obfuscated methods of protection here, but let’s just imagine, you decide to put the image path be be as /images/../../../../bin/init or /images/../../../../usr/bin/init or some other binary name.
While technically it shouldn’t be possible, with the wrong setup it is a possibility. Now your fog server will likely work for a while. Then start having some funny issues.

I mean if you want to allow it, by all means, please, but I’m not going to make that a possibility for ALL people’s insitalls.

The Wiki is perfectly accurate. You would, essentially, change the images location to go from /images to /images/newdisk. No where in the wiki article, from what I’ve skimmed through, would state that you are placing the image on the disk in the way you originally described. Changing the way move_upload is not quite that simple.

Again we’re using NFS to perform the fastest possibly mounting and distributing the image. Moving the files are done through the FTP interface, not from the NFS. Because of this, you can only put or rename the path. This is not going to change.

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu THe reason the path is specified (not allowing slashes) is because it can give access to other parts of the filesystem. This doesn’t mean it WILL happen, but from a security perspective what’s to ensure you don’t write a specific file or overwrite the root folder? I know there’s some other obfuscated methods of protection here, but let’s just imagine, you decide to put the image path be be as /images/../../../../bin/init or /images/../../../../usr/bin/init or some other binary name.
While technically it shouldn’t be possible, with the wrong setup it is a possibility. Now your fog server will likely work for a while. Then start having some funny issues.

I perfectly understand why it’s like this. I was more like toying with different ways to go around the problem I had. I’m ok with the way it works now.

The Wiki is perfectly accurate. You would, essentially, change the images location to go from /images to /images/newdisk. No where in the wiki article, from what I’ve skimmed through, would state that you are placing the image on the disk in the way you originally described. Changing the way move_upload is not quite that simple.

Oh right, I think I then read too far there between the lines. With the guide on the wiki article’s last lines, it’s meant to move all your images under the “newdisk”, not just one or part of them. I get it now

Changing the way move_upload is not quite that simple.

Yeah, I glanced just quickly at the codebase there and realized that it isn’t just a minor change, but I think it is doable still. All the needed actions are moving (renaming) and deleting files and folders, just as it is done now. Now it’s really simple and pretty when you can just move the whole directory instead of doing multiple actions to the files.

Would you accept a pull request of changing the move_upload process, if I find time to put it together? I’m not making any promises, but just thinking about it. I’ve never really contributed to an open source project but I might have the skills to put that one together. But if it fights the way you want it to work, then I wont bother

I really thank you for your time!

Btw, I settled for a process that I just move the image files to the dedicated HDD always after upload. It’ll do now, since it’s only temporary