Upload Task: Updating database... failed. FTP problems

zachu

@Wayne-Workman said in Upload Task: Updating database... failed. FTP problems:

How did you manually check?
Sometimes an invalid login error is thrown, sometimes not. Sometimes the problem lies deeper.
We have a troubleshooting article that addresses about >95% of what could go wrong:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

Well I manually checked the fields that the wiki article here https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP#Credentials_.2F_Passwords points out. I also logged in on the local shell with the password shown in there, and it worked. Also I tried ftp’ing with that username and password inside, and it worked too. I tried to point out on my initial post that I think I’ve done everything the troubleshooting article tells me to.

I’ll do installfog.sh -y now and retry the upload as you suggested.

Quazz

So from my understanding of this process, you login remotely to the server and then FTP tries to do its thing locally on the server.

As such, login credentials should be fine and seeing as permissions are fine too…

This will take some time to get to the bottom, but I’d like to note that a segmentation fault is generally caused by running out of RAM. How much is in your system?

Can you also post the output of

df -h

It might have something to do with disk space running out.

zachu

@Quazz

running out of RAM. How much is in your system?

8 GB

$ cat /proc/meminfo
MemTotal:        8175644 kB

Can you also post the output of df -h

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           799M  792K  798M   1% /run
/dev/sda1       909G  4.4G  859G   1% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdd1       917G   72M  871G   1% /images/Gizmok6
/dev/sdc1       917G  405G  466G  47% /images/Gizmok5
/dev/sdb1       917G   72M  871G   1% /images/Gizmok4

As for the segfaulting, I think it has something to do with the fact that fog spams the ftp_delete and ftp_rmdir REALLY MUCH. I mean the first error entry on the error.log has timestamp 21:17:21.586806 and the last 22:35:10.321097. In that timespan of ~~less than 20 minutes~~ 1 hour and 20 minutes, I have 996M of error.log just spamming the ftp_delete and ftp_rmdir warnings. There is 5351072 lines in my error.log on that timespan

edit: To clarify more, I think the segfaulting isn’t the root cause here. The ftp_rmdir and ftp_delete failing is. Or that’s what I think is happening

edit2: Time flies… Corrected the timespan of the error messages

Quazz

@zachu I notice you have some lost+found folders in some of your images folders.

It would be best to remove those. (and you might want to ask yourself why they’re there in the first place, did you have a lot of file system issues on your linux server?)

It might even resolve the problem at hand. (easiest way to test would be to try and capture Gizmok4)

Additionally, you should change your /etc/fstab, make your 2s into 0s for those disks. They’re being checked by fsck because of that option at every boot if I’m not mistaken, which creates those lost+found folders most likely.

https://en.wikipedia.org/wiki/Fstab

zachu

@Quazz said in Upload Task: Updating database... failed. FTP problems:

@zachu I notice you have some lost+found folders in some of your images folders.

It would be best to remove those. (and you might want to ask yourself why they’re there in the first place, did you have a lot of file system issues on your linux server?)

The lost+found directories are created when a partition is formatted as ext4. I think I can delete them and remove the fsck checks so they shouldn’t be recreated.

I already have an upload task going on, after doing fog install over again to fix the possibly-wrong-credentials. It shouldn’t matter that I deleted the lost+founds now during the upload task, since the problem occurs at the end with ftp, right?

Lets see if that fixes things

zachu

Removed lost+founds Didn’t help Also running foginstall again to make sure that ftp credentials are correct didn’t help either

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

zachu

@zachu said in Upload Task: Updating database... failed. FTP problems:

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

This seemed to work. So something goes wrong when i have an mounted device dedicated to one image. I wonder if fog tries to ftp_remove and ftp_rmdir that mountpoint and afterwards rename /images/dev/<macaddr> to /images/<imagename>. That could cause the behaviour we see. Instead imo fog should move the contents of dev to the image folder.

Wiki page Adding Storage to the Images directory does state that mounting /images/<imagename> should be possible, so do we concider this as a bug?

Tom Elliott

@zachu I don’t know what you mean.

Please explain.

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I don’t know what you mean.

Please explain.

Well wrapping up this whole thread:

I had mounted /dev/sdb1 to /images/Gizmok4
Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
/var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

Tom Elliott

Where is /dev/sdb1?

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

Where is /dev/sdb1?

Now I don’t know what you mean If you mean now that the upload task worked, it’s still unmounted and not in use.

george1421

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration? I want to understand the business logic behind your setup?

Tom Elliott

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

zachu

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration?

I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.

george1421

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

[edit] I did write a kb article that covers what I just mentioned above. https://forums.fogproject.org/topic/6642/moving-fog-s-images-files-off-the-root-partition [/edit]

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk No virtualization or anything.

Tom Elliott

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

Umm… no I don’t need the mounted drive to be a clone on the system. I need it to contain the image files so I can deploy it elsewhere

I have 3 completely different Windows7 hosts registered to my fog server. They all have different images. I’m trying to clone one of them and have the image files on the fog server /images/Gizmok4 folder. Another one would be on /images/Gizmok5 and the third on /images/Gizmok6. This is all regular fog usage?

Now it would help me tremendously if that /images/Gizmok4/ (as well as the other image locations) would be a mountpoint for a single HDD that I coud take with me to another site when I’m deploying the image to a bunch of computers. There I also have a fog storage node that I can use to deploy the image. The transfering part is just a problem now, since I’m temporarily sitting on slow connection.

Clarified anything?

zachu

In other words, the only thing that diffrences here on just default out-of-box fog usage, is this:
I would like that /images/<imagename> would be contained on a single HDD device that I can take away from the server and carry to another place.

/images/Gizmok4 => /dev/sdb
/images/Gizmok5 => /dev/sdc
/images/Gizmok6 => /dev/sdd

zachu

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

This is actually about how we normally do things, and then transfer the images on network, but because of temporary special circumstances, I have to be on a slow net connection

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

Yeah indeed this sounds easiest way to go, but since this is a temporary situation, I can’t get my hands on a usb drive easily Not until it’s not needed again. But good advices anyway. I think I’ll head that way if I can’t get this work some other way.

Maybe this whole thing could work also if I’d put one directory after the mountpoint. Like this: mount /dev/sdb1 /images/sdb, and then I’d change the image path to /images/sdb/Gizmok4. If the problem is that fog cannot remove the mountpoint as I’ve guessed it is, then that would probably solve the problem. I’ll test this.

Upload Task: Updating database... failed. FTP problems

141

11.7k

17.2k

154.7k