Upload Task: Updating database... failed. FTP problems

[zachu]

@zachu said in Upload Task: Updating database... failed. FTP problems:

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

This seemed to work. So something goes wrong when i have an mounted device dedicated to one image. I wonder if fog tries to ftp_remove and ftp_rmdir that mountpoint and afterwards rename /images/dev/<macaddr> to /images/<imagename>. That could cause the behaviour we see. Instead imo fog should move the contents of dev to the image folder.

Wiki page Adding Storage to the Images directory does state that mounting /images/<imagename> should be possible, so do we concider this as a bug?

[Tom Elliott]

@zachu I don’t know what you mean.

Please explain.

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I don’t know what you mean.

Please explain.

Well wrapping up this whole thread:

• I had mounted /dev/sdb1 to /images/Gizmok4
• Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
• /var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
• I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

[Tom Elliott]

Where is /dev/sdb1?

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

Where is /dev/sdb1?

Now I don’t know what you mean If you mean now that the upload task worked, it’s still unmounted and not in use.

[george1421]

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration? I want to understand the business logic behind your setup?

[Tom Elliott]

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration?

I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.

[george1421]

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

[edit] I did write a kb article that covers what I just mentioned above. https://forums.fogproject.org/topic/6642/moving-fog-s-images-files-off-the-root-partition [/edit]

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk No virtualization or anything.

[Tom Elliott]

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

Umm… no I don’t need the mounted drive to be a clone on the system. I need it to contain the image files so I can deploy it elsewhere

I have 3 completely different Windows7 hosts registered to my fog server. They all have different images. I’m trying to clone one of them and have the image files on the fog server /images/Gizmok4 folder. Another one would be on /images/Gizmok5 and the third on /images/Gizmok6. This is all regular fog usage?

Now it would help me tremendously if that /images/Gizmok4/ (as well as the other image locations) would be a mountpoint for a single HDD that I coud take with me to another site when I’m deploying the image to a bunch of computers. There I also have a fog storage node that I can use to deploy the image. The transfering part is just a problem now, since I’m temporarily sitting on slow connection.

Clarified anything?

[zachu]

In other words, the only thing that diffrences here on just default out-of-box fog usage, is this:
I would like that /images/<imagename> would be contained on a single HDD device that I can take away from the server and carry to another place.

• /images/Gizmok4 => /dev/sdb
• /images/Gizmok5 => /dev/sdc
• /images/Gizmok6 => /dev/sdd

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

This is actually about how we normally do things, and then transfer the images on network, but because of temporary special circumstances, I have to be on a slow net connection

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

Yeah indeed this sounds easiest way to go, but since this is a temporary situation, I can’t get my hands on a usb drive easily Not until it’s not needed again. But good advices anyway. I think I’ll head that way if I can’t get this work some other way.

Maybe this whole thing could work also if I’d put one directory after the mountpoint. Like this: mount /dev/sdb1 /images/sdb, and then I’d change the image path to /images/sdb/Gizmok4. If the problem is that fog cannot remove the mountpoint as I’ve guessed it is, then that would probably solve the problem. I’ll test this.

[george1421]

@zachu make sure the fog (linux) user has full read/write access to those attached disks.

The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

[Tom Elliott]

So now I need further understanding.

If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

Why not mount /dev/sdb1 (or your relevant items) to /images?

To understand further, Gizmok4 is intended to be the <imagename> placeholder?

[zachu]

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu make sure the fog (linux) user has full read/write access to those attached disks.

The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

I tried again the FTP on the fog user. I surely can create, write and delete on /images/dev/ and on /images/ and on /images/Gizmok4/. What I can’t do through ftp, is delete /images/Gizmok4 since it is a mountpoint. So I still guess that’s what the fog is trying to do on the “Updating database…” part. It deletes the /images/<imagename> and renames /images/dev/<macaddr> to /images/<imagename>. It fails because it cant delete the mountpoint.

[zachu]

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

So now I need further understanding.

If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

No. I have an image called Gizmok4 which locates in /images/Gizmok4 as it does on default settings.

I also have an images named Gizmok5 and Gizmok6, which locates in /images/Gizmok5 and /images/Gizmok6 respectively. They are on same settings as the Gizmok4-image with the exception of Gizmok6 being multiple disk image. Although that’s irrelevant here

Why not mount /dev/sdb1 (or your relevant items) to /images?

I need the images to be on different physical disks (one disk for one image) for reasons explained before. Mainly because of portability (need to physically carry the image to another location) and because of disk space (all the images don’t fit on one disk).

To understand further, Gizmok4 is intended to be the <imagename> placeholder?

Yes, Gizmok4 is the name of one of my images.

[Tom Elliott]

@zachu I’m only using Gizmok4 as the basis of information I need to better help you.

Then what you are actually looking to do is make your disks the new “images” location.

The “fix” for this would be to mount your disk AS /images. From there, you would need to create the dev folder and the requisite .mntcheck files. Mount your disk, and upload your image to the appropriate disk.

The way to do:

mount /dev/sdb1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok4.

umount /images
mount /dev/sdc1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok5

umount /images
mount /dev/sdd1 /images
mkdir /images/dev
touch /images/.mntcheck
touch /images/dev/.mntcheck
chmod -R 777 /images

Perform your upload operation for gizmok6

umount /images

You will then have what you’ve been intending to do this whole time.

[zachu]

Okay so imo this topic can be marked as solved. I think @Tom-Elliott’s approach is currently the closest one to go for my needs. Really big thanks to others too that helped me with this. For others with similar needs, @george1421’s approach may be good way to go too.

So a wrap-up:
My goal was to mount a device /dev/sdb1 to /images/Gizmok4 so that when I upload a image Gizmok4, it goes to a dedicated HDD. I assumed this was a valid approach since it is hinted so in Fog wiki

Fog wiki in Moving_your_images_directory/Adding_Storage_to_the_Images_directory states:

If you want to just add capacity to your images directory, you could instead mount your new device as a subfolder of the /images directory. In this case your wouldn’t need to move the original directory to /images1, and your fstab line would look like:

UUID=XXXXXXXXXXXXXXXXXXXXXX    /images/newdisk      ext3 defaults 0 0

As it turned out, you can’t do this, since after an upload task fog deletes the image path and renames the uploaded image directory in place of the old image directory the image path and renames the uploaded image directory in place of the old image directory. The task fails, because the mountpoint isn’t deletable. One approach at fogprojects end to fix or allow this, is instead of deleting and renaming, just move the content of /images/dev/<macaddr> to /images/dev/<imagename> (<imagename> being Gizmok4 in my case).

I figured this limitation could be circled around by putting one directory before the image directory as a mountpoint. Mounting /dev/sdb1 to something like /images/sdb and have <imagename> folder under that (again Gizmok4 in my case). Then I thought to just change the Gizmok4 image path to /images/sdb/Gizmok4 and everything would be fixed. Now the image directory is removable and the process of deleting old files and renaming new files on the same place would work. In other words, I tried something like this:

$ tree /images
/images
├── dev
├── postdownloadscripts
│   └── fog.postdownload
└── sdb
    └── Gizmok4
        ├── d1.mbr
        ├── d1p1.img
        ├── d1p2.img
        └── d1.partitions

The problem in this approach comes from Fog WebUI, more specifically when validating the image path input in fog.image.js. Only alphanumeric input and a plus (+) character is allowed there.

Out of curiosity, I did disable that part of the javascript and uploaded an image to a path sdb/Gizmok4 (full path being /images/sdb/Gizmok4). Now for some reason I ended up having a file /images/sdb/Gizmok4 instead of a directory. All the real image files were kept in the dev/<macaddr>

Conclusion… and a workaround request

All in all, I think the wiki article (or the last part of it) is misleading, since it isn’t possible to add storage to /images/ by just mounting another device as a subfolder, as it suggests. Well, you can do it, but you can’t use it My opinion is that this could be very handy addition though and should be implemented. I think the best way to do that would be reworking the move_upload() function in taskqueue.class.php to work as following:

1. Empty the image directory. (Currently the directory gets deleted)
2. Move the contents of /images/dev/<macaddr> to the image directory. (Currently the whole directory gets moved/renamed)
3. Remove the directory /images/dev/<macaddr>. (Currently it gets renamed, so no removing needed)