Upload Task: Updating database... failed. FTP problems

zachu

Hi!

This might be duplicate or related of Image Capture/Upload gets to 100%, then Updating Database task fails and FTP Delete an image - failing but I couldn’t figure it out.

I’m having problems with fresh install of Fog. Running Version 1.3.0-RC-2, SVN Revision: 5937 on Ubuntu 14.04.4.

I’ve registered 3 hosts and added 3 Windows-7 images. Nothing more. I’m trying to upload and image to the fog server, but get the “Updating database… Failed” error.
Error after uploading

My apache2 error.log is flooded with PHP warnings ftp_delete() and ftp_rmdir() failing, with occasionally the apache2 process segfaulting.

$ head -n 4 /var/log/apache2/error.log
[Tue Jul 26 21:17:21.586806 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
[Tue Jul 26 21:17:21.586899 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
[Tue Jul 26 21:17:21.588240 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
[Tue Jul 26 21:17:21.588307 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
...
...
...
[Tue Jul 26 21:21:34.467151 2016] [core:notice] [pid 24528] AH00051: child pid 28846 exit signal Segmentation fault (11), possible coredump in /etc/apache2

I think the permissions should allow deleting, since they are all 777 permissioned, just in case.

$ ls -laR /images/
/images/:
total 28
drwxrwxrwx  7 fog  root 4096 Jul 26 19:42 .
drwxr-xr-x 25 root root 4096 Jul 26 15:39 ..
drwxrwxrwx  3 fog  root 4096 Jul 26 15:59 dev
drwxrwxrwx  2 fog  root 4096 Jul 26 18:16 Gizmok4
drwxrwxrwx  3 fog  root 4096 Jul 25 16:52 Gizmok5
drwxrwxrwx  3 fog  root 4096 Jul 25 16:56 Gizmok6
-rwxrwxrwx  1 fog  root    0 Jul 26 15:39 .mntcheck
drwxrwxrwx  2 fog  root 4096 Jul 26 15:39 postdownloadscripts

/images/dev:
total 12
drwxrwxrwx 3 fog root 4096 Jul 26 15:59 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
drwxrwxrwx 2 fog root 4096 Jul 26 21:17 bc5ff4473427
-rwxrwxrwx 1 fog root    0 Jul 26 15:39 .mntcheck

/images/dev/bc5ff4473427:
total 450560628
drwxrwxrwx 2 fog root         4096 Jul 26 21:17 .
drwxrwxrwx 3 fog root         4096 Jul 26 15:59 ..
-rwxrwxrwx 1 fog root      1048576 Jul 26 18:54 d1.mbr
-rwxrwxrwx 1 fog root      8912823 Jul 26 18:54 d1p1.img
-rwxrwxrwx 1 fog root 461364057634 Jul 26 21:17 d1p2.img
-rwxrwxrwx 1 fog root          190 Jul 26 18:54 d1.partitions

/images/Gizmok4:
total 8
drwxrwxrwx 2 fog root 4096 Jul 26 18:16 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..

/images/Gizmok5:
total 424530580
drwxrwxrwx 3 fog root         4096 Jul 25 16:52 .
drwxrwxrwx 7 fog root         4096 Jul 26 19:42 ..
-rwxrwxrwx 1 fog root      1048576 Jul 22 20:02 d1.mbr
-rwxrwxrwx 1 fog root      8570406 Jul 22 20:52 d1p1.img
-rwxrwxrwx 1 fog root 434709616470 Jul 22 20:52 d1p2.img
-rwxrwxrwx 1 fog root          190 Jul 22 20:02 d1.partitions
drwxrwxrwx 2 fog root        16384 Jul 22 19:57 lost+found

/images/Gizmok5/lost+found:
total 20
drwxrwxrwx 2 fog root 16384 Jul 22 19:57 .
drwxrwxrwx 3 fog root  4096 Jul 25 16:52 ..

/images/Gizmok6:
total 24
drwxrwxrwx 3 fog root  4096 Jul 25 16:56 .
drwxrwxrwx 7 fog root  4096 Jul 26 19:42 ..
drwxrwxrwx 2 fog root 16384 Jul 25 16:56 lost+found

/images/Gizmok6/lost+found:
total 20
drwxrwxrwx 2 fog root 16384 Jul 25 16:56 .
drwxrwxrwx 3 fog root  4096 Jul 25 16:56 ..

/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fog root 4096 Jul 26 15:39 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
-rwxrwxrwx 1 fog root  233 Jul 26 15:39 fog.postdownload

I do have every image on their own HDD, if that might be the reason?

$ tail -n3 /etc/fstab
UUID=59ea75b7-e91e-4965-814c-4f54e2c15fe4       /images/Gizmok6 ext4    defaults        0       2
UUID=9fb7e4b0-6237-4c8b-b746-22212d89ad10       /images/Gizmok5 ext4    defaults        0       2
UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba       /images/Gizmok4 ext4    defaults        0       2

As I checked, the FTP credentials seemed to be correct in all the places shown in here, as they were set on the install process. I haven’t touched them. Also I tested the FTP as shown here and all the steps went through with no problem.

I would appreciate a poke to the right direction Thanks!

Wayne Workman

Do you have email notifications configured?

Also, you can re-run the fog installer to correct any FTP credentials mismatches, like this:

./installfog.sh -y

zachu

@Wayne-Workman No I don’t have any email notifications configured, and as far as I know, the credentials are correct since I haven’t changed anything after the install. And by manually checking, they seemed correct. Wouldn’t the error message come from ~~ftp_connect~~ ftp_login rather than ftp_delete or ftp_rmdir with invalid credentials anyway?

Wayne Workman

@zachu said in Upload Task: Updating database... failed. FTP problems:

And by manually checking, they seemed correct.

How did you manually check?

Sometimes an invalid login error is thrown, sometimes not. Sometimes the problem lies deeper.
We have a troubleshooting article that addresses about >95% of what could go wrong:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

zachu

@Wayne-Workman said in Upload Task: Updating database... failed. FTP problems:

How did you manually check?
Sometimes an invalid login error is thrown, sometimes not. Sometimes the problem lies deeper.
We have a troubleshooting article that addresses about >95% of what could go wrong:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

Well I manually checked the fields that the wiki article here https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP#Credentials_.2F_Passwords points out. I also logged in on the local shell with the password shown in there, and it worked. Also I tried ftp’ing with that username and password inside, and it worked too. I tried to point out on my initial post that I think I’ve done everything the troubleshooting article tells me to.

I’ll do installfog.sh -y now and retry the upload as you suggested.

Quazz

So from my understanding of this process, you login remotely to the server and then FTP tries to do its thing locally on the server.

As such, login credentials should be fine and seeing as permissions are fine too…

This will take some time to get to the bottom, but I’d like to note that a segmentation fault is generally caused by running out of RAM. How much is in your system?

Can you also post the output of

df -h

It might have something to do with disk space running out.

zachu

@Quazz

running out of RAM. How much is in your system?

8 GB

$ cat /proc/meminfo
MemTotal:        8175644 kB

Can you also post the output of df -h

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           799M  792K  798M   1% /run
/dev/sda1       909G  4.4G  859G   1% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdd1       917G   72M  871G   1% /images/Gizmok6
/dev/sdc1       917G  405G  466G  47% /images/Gizmok5
/dev/sdb1       917G   72M  871G   1% /images/Gizmok4

As for the segfaulting, I think it has something to do with the fact that fog spams the ftp_delete and ftp_rmdir REALLY MUCH. I mean the first error entry on the error.log has timestamp 21:17:21.586806 and the last 22:35:10.321097. In that timespan of ~~less than 20 minutes~~ 1 hour and 20 minutes, I have 996M of error.log just spamming the ftp_delete and ftp_rmdir warnings. There is 5351072 lines in my error.log on that timespan

edit: To clarify more, I think the segfaulting isn’t the root cause here. The ftp_rmdir and ftp_delete failing is. Or that’s what I think is happening

edit2: Time flies… Corrected the timespan of the error messages

Quazz

@zachu I notice you have some lost+found folders in some of your images folders.

It would be best to remove those. (and you might want to ask yourself why they’re there in the first place, did you have a lot of file system issues on your linux server?)

It might even resolve the problem at hand. (easiest way to test would be to try and capture Gizmok4)

Additionally, you should change your /etc/fstab, make your 2s into 0s for those disks. They’re being checked by fsck because of that option at every boot if I’m not mistaken, which creates those lost+found folders most likely.

https://en.wikipedia.org/wiki/Fstab

zachu

@Quazz said in Upload Task: Updating database... failed. FTP problems:

@zachu I notice you have some lost+found folders in some of your images folders.

It would be best to remove those. (and you might want to ask yourself why they’re there in the first place, did you have a lot of file system issues on your linux server?)

The lost+found directories are created when a partition is formatted as ext4. I think I can delete them and remove the fsck checks so they shouldn’t be recreated.

I already have an upload task going on, after doing fog install over again to fix the possibly-wrong-credentials. It shouldn’t matter that I deleted the lost+founds now during the upload task, since the problem occurs at the end with ftp, right?

Lets see if that fixes things

zachu

Removed lost+founds Didn’t help Also running foginstall again to make sure that ftp credentials are correct didn’t help either

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

zachu

@zachu said in Upload Task: Updating database... failed. FTP problems:

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

This seemed to work. So something goes wrong when i have an mounted device dedicated to one image. I wonder if fog tries to ftp_remove and ftp_rmdir that mountpoint and afterwards rename /images/dev/<macaddr> to /images/<imagename>. That could cause the behaviour we see. Instead imo fog should move the contents of dev to the image folder.

Wiki page Adding Storage to the Images directory does state that mounting /images/<imagename> should be possible, so do we concider this as a bug?

Tom Elliott

@zachu I don’t know what you mean.

Please explain.

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I don’t know what you mean.

Please explain.

Well wrapping up this whole thread:

I had mounted /dev/sdb1 to /images/Gizmok4
Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
/var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

Tom Elliott

Where is /dev/sdb1?

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

Where is /dev/sdb1?

Now I don’t know what you mean If you mean now that the upload task worked, it’s still unmounted and not in use.

george1421

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration? I want to understand the business logic behind your setup?

Tom Elliott

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

zachu

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration?

I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.

george1421

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

[edit] I did write a kb article that covers what I just mentioned above. https://forums.fogproject.org/topic/6642/moving-fog-s-images-files-off-the-root-partition [/edit]

zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk No virtualization or anything.

Upload Task: Updating database... failed. FTP problems

153

12.0k

17.3k

155.2k