FOG won't move images

[Tom Elliott]

@Quazz The table, I think, is nfsGroupMembers

[Quazz]

@Tom-Elliott The password is correct in the database.

[Tom Elliott]

@Quazz Is there anything in the vsftpd logs?

[Quazz]

@Tom-Elliott There is no vsftpd logfile! Most strange, since it’s definitely enabled in /etc/vsftp.conf

However, there is this when I check the status of the service: (I’ve been trying to upload images the past few hours)

apr 11 10:51:38 foghost systemd[1]: Started vsftpd FTP server.
apr 11 10:59:04 foghost vsftpd[10206]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=fog rhost=192.168.1.154  user=fog
apr 11 11:55:12 foghost vsftpd[11194]: pam_unix(vsftpd:auth): check pass; user unknown
apr 11 11:55:12 foghost vsftpd[11194]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=anonymous rhost=192.168.1.31
apr 11 11:55:32 foghost vsftpd[11196]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=fog rhost=192.168.1.31  user=fog
apr 11 11:55:40 foghost vsftpd[11199]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=fog rhost=192.168.1.31  user=fog
apr 11 11:56:23 foghost vsftpd[11212]: pam_unix(vsftpd:auth): check pass; user unknown
apr 11 11:56:23 foghost vsftpd[11212]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=anonymous rhost=192.168.1.31
apr 11 11:56:26 foghost vsftpd[11214]: pam_unix(vsftpd:auth): authentication failure; logname= uid=0 euid=0 tty=ftp ruser=fog rhost=192.168.1.31  user=fog

So clearly it passes along the wrong password for some reason, but I can’t figure out why.

[Tom Elliott]

@Quazz Hit me on chat, we should do a “join.me” session I think.

[Quazz]

@Tom-Elliott The log thing is definitely fixed, but it seems it still doesn’t move images. I noticed that it by default assigns root:root as ownership of the folders in /images/dev, could this be the problem?

[Tom Elliott]

@Quazz What’s the modifiable permissions?

root:root shouldn’t matter much, but in the move process, itself, it should be set to fog:fog or fog:root. How’s it maintained in dev? Here you want to just output the listed output of the directory.

Try:

chmod -R 777 /images

It should correct the issues for you, though test would be needed to validate.

[Quazz]

@Tom-Elliott

root@foghost:/var/log/fog# ls -la /images/dev
totaal 16
drwxrwxrwx  4 fog  root 4096 apr 11 14:47 .
drwxrwxrwx 24 fog  root 4096 mrt 15 15:00 ..
drwxrwxrwx  2 root root 4096 apr 11 14:46 08002710d395
drwxrwxrwx  2 root root 4096 apr 11 14:48 0800276fc5ee
-rwxrwxrwx  1 fog  root    0 mrt  2 17:05 .mntcheck

So, I guess it’s not a permission issue then. Same permissions for /images

Also I found the vsftpd log, it was named xferlog (after some googling this is merely a transfer log?)

Mon Apr 11 14:46:45 2016 1 192.168.1.154 0 /images/W7PRO64FR b _ i r fog ftp 0 * i
Mon Apr 11 15:07:29 2016 1 192.168.1.154 0 /images/W7HPR64NL b _ i r fog ftp 0 * i

This is the log of the images of just now.

[Wayne Workman]

@Quazz what OS do you use? Have you checked selinux, firewall?

[Quazz]

@Wayne-Workman Ubuntu, and yes all disabled. Don’t worry though, Tom nearly has it all sorted, should be ok tomorrow.

[Sebastian Roth]

@Quazz If you still cannot make it work show us the output of mount here in the forums. Possibly the filesystem is ext3?

[Quazz]

@Sebastian-Roth

/dev/mapper/ubuntu--vg-root on / type ext4 (rw,relatime,errors=remount-ro,data=ordered)

I haven’t had the time to test it properly since yesterday, Tom was pretty busy yesterday as well, but from a quick test it did seem to be able to move the images (if the target directory didn’t already exist)

[Sebastian Roth]

@Quazz said:

but from a quick test it did seem to be able to move the images (if the target directory didn’t already exist)

This sounds exactly like the issue I had some time ago with images being stored on a ext3 partition (see below). Possibly you are not running into the same but a similar issue - FTP timing out because deleting the old (huge?) image takes some time. Maybe play with FOG Setting FOG_FTP_TIMEOUT…

@Sebastian-Roth said in Failed to create deployment tasking…:

Learned something new … (pretty much every day!)
ext3 does some kind of defragmentation/reallocation foooo when files are being deleted!
http://www.depesz.com/2010/04/04/how-to-remove-backups/
I (re)moved all my images, re-formated with ext4, put it all back together and now everything seems fine. Deleting of 150GB image takes less than 10 seconds. Awesome!!! I really wonder why very little people have issues with this. Is ext3 not in use in most recent distributions??

[Quazz]

@Sebastian-Roth Well, it’s definitely in ext4, so I’ll try to up the FTP timeout and see if that helps.

[Wayne Workman]

@Quazz Does /home/fog exist ?

[Quazz]

@Wayne-Workman Yes.

I did notice that in the prep stages of the capture it sets permissions for /images/macaddress rather than /images/nameofimage ,not sure if that’s relevant or not though.

On latest trunk I get the same error as https://forums.fogproject.org/topic/7153/unable-to-upload-image-error-returned-type-2-file-var-www-fog-lib-fog-fogftp-class-php/2

Obviously, deleting the folders prior to capture each time would get tedious after a while, so I’m hoping for a better solution.

[Quazz]

This is what I get with latest update, not sure if this happens before the FTP move (since the images aren’t moved yet)

[Wayne Workman]

@Quazz I’ve had this error before, it turned out to be an FTP thing every time for me - but with different causes. a bad password set for the fog account, or bad permissions in the /images directory, or selinux turned on.

My suggestion would be to comb through all the FTP stuff again - but slower this time.

[Quazz]

@Wayne-Workman I can manually connect to the FTP by copy pasting the password from any location.

Permissions are 777 (set by chmod -R 777), ownership is fog:fog (chown -R fog:fog) and have been for a while now.

Running Ubuntu, so Selinux is not even installed at all.

I believe Tom checked those things as well when we had join.me session.

I double checked everything multiple times to make sure the issue wasn’t on my side, but I’ll explore some more out there ideas now. Will report back if I have some progress.

[Sebastian Roth]

@Quazz At this stage of upload the client is kind of reporting back to the FOG server - triggering the image file move via FTP. Please see if you have any errors in the apache log (/var/log/apache/error.log) when you see this messages on screen!

As well you can try sending that request from a different (linux) machine to see what’s going on.

wget --post-data="mac=<mac>&type=up&osid=<osid>" -qO - http://${web}service/Post_Stage2.php

Put in <mac> and <osid> to match your client/image information…

The (empty) Error returned: message on screen seems to tell us that there was no error returned. That’s why I guess you should see something in the apache error log…