Permission Denied when trying to image after update to 1.1.1/1.1.2

[ianabc]

It is worth checking your firewall and restarting the nfs service(s). Restarting the NFS service(s) depends on your system, but on mine looks like.
[CODE]
$ service nfs restart
[/CODE]
Also, could you send the output of exportfs -va, e.g.
[CODE]
$ exportfs -ua
$ exportfs -va
[/CODE]

Also, another good test is to see if you can mount the NFS share on the server itself under a temporary location, e.g.
[CODE]
$ mount IP.OF.YOUR.FOG:/images /mnt
$ ls /mnt
dev Mint17x64PIMS1 U1404GAH87NWIFIIAM W7x32Stat55GBDeploy-XXX
fog-math RHL6x64PIMS50GB u1404PIMS

 lost+found  RHL7x64BetaPIMS1  W7x32Stat55GBDeploy

*** If it worked, remember to unmount it ****
$ umount /mnt
[/CODE]

[Jamie Rozek]

[quote=“ianabc, post: 31328, member: 24548”]It is worth checking your firewall and restarting the nfs service(s). Restarting the NFS service(s) depends on your system, but on mine looks like.
[CODE]
$ service nfs restart
[/CODE]
Also, could you send the output of exportfs -va, e.g.
[CODE]
$ exportfs -ua
$ exportfs -va
[/CODE]

Also, another good test is to see if you can mount the NFS share on the server itself under a temporary location, e.g.
[CODE]
$ mount IP.OF.YOUR.FOG:/images /mnt
$ ls /mnt
dev Mint17x64PIMS1 U1404GAH87NWIFIIAM W7x32Stat55GBDeploy-XXX
fog-math RHL6x64PIMS50GB u1404PIMS

lost+found  RHL7x64BetaPIMS1  W7x32Stat55GBDeploy

*** If it worked, remember to unmount it ****
$ umount /mnt
[/CODE][/quote]

Firewall definitely not the issue. We haven’t changed anything, and Fog was working perfectly before the update yesterday.

Here is the output of exportfs -va:

exportfs: /etc/exports [1]: Neither ‘subtree_check’ or ‘no_subtree_check’ specified for export “*:/images”.
Assuming default behaviour (‘no_subtree_check’).
NOTE: this default has changed since nfs-utils version 1.0.x

exportfs: /etc/exports [2]: Neither ‘subtree_check’ or ‘no_subtree_check’ specified for export “*:/images/dev”.
Assuming default behaviour (‘no_subtree_check’).
NOTE: this default has changed since nfs-utils version 1.0.x

exporting *:/images/dev
exporting *:/images

I am able to mount IP.Address:/images.

[ianabc]

It looks like the problem is between the clients and your fog server then, first could you check that you aren’t using selinux, e.g.
[CODE]
$ sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: permissive
Policy version: 24

Policy from config file:        targeted

[/CODE]

Next, make sure that you aren’t using tcp_wrappers or configure it to allow rpcbind, portmap, statd etc. if you are.

Do you have any other linux clients on your network that you could try the NFS mount from? If not you can schedule a debug task on one of the clients and run the mount commands there, it might give you a better idea of where the problem is. e.g.
[CODE]
$ mount -vvv -o vers=3,rw IP.OF.YOUR.FOG:/images /mnt
[/CODE]

[Jamie Rozek]

Yes, I’m pretty sure there is a setting that is different on the Fog server between versions 1.1.0 and 1.1.1/1.1.2, because I did not have this problem until upgrading to 1.1.1.
I just re-installed Fog, minus the database—still having the same issue.

I did not change a single thing. I don’t touch the server unless it is for Fog. We have no other linux servers.
Selinux is disabled.

[Jamie Rozek]

I would really rather not uninstall the entire thing and start from scratch. I have some good “Golden images” on there.

By the way, when I do:

service nfs restart

I get “unknown service.”

[ianabc]

You shouldn’t need to reinstall, it sounds like just an NFS problem. NFS(3) is pretty simple so we should be able to debug the problem here.

Which distribution are you running? “service nfs restart” should work for RHEL based distributions but on ubuntu you might do something like “service nfs-kernel-server restart”. Alternatively you can just quickly reboot the fog server and let that restart the nfs service if you’re not sure. When it comes back up could you include the output from rpcinfo, e.g.
[CODE]
$ rpcbind -p
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 48324 status
100024 1 tcp 60634 status
100011 1 udp 875 rquotad
100011 2 udp 875 rquotad
100011 1 tcp 875 rquotad
100011 2 tcp 875 rquotad
100005 1 udp 51151 mountd
100005 1 tcp 56945 mountd
100005 2 udp 38168 mountd
100005 2 tcp 56201 mountd
100005 3 udp 49959 mountd
100005 3 tcp 41228 mountd
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 2 tcp 2049 nfs_acl
100227 3 tcp 2049 nfs_acl
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 2 udp 2049 nfs_acl
100227 3 udp 2049 nfs_acl
100021 1 udp 57018 nlockmgr
100021 3 udp 57018 nlockmgr
100021 4 udp 57018 nlockmgr
100021 1 tcp 48912 nlockmgr
100021 3 tcp 48912 nlockmgr

100021    4   tcp  48912  nlockmgr

[/CODE]

Once you are sure that the services are running and listening on the correct ports, that really only leaves things that can terminate inbound connections like tcpwrappers.

[Jamie Rozek]

It is not allowing me to post my RPCinfo. States that it is “spam.”

I’ve attached it as a text document.

I’m seriously thinking about just doing a complete uninstall/reinstall.

I’ve noticed that while this is happening, some of my images are doing this: (Every image I try to use, because “No size available” on the client.)

[SIZE=5][B][FONT=Ubuntu][CENTER][SIZE=18px][COLOR=#4c68a0]All Images[/COLOR][/SIZE][/CENTER][/FONT][/B][/SIZE]

[SIZE=13px][FONT=Ubuntu][COLOR=#555555][COLOR=#ededed]Image Name [CENTER]
Storage Group[/CENTER]
O/S [CENTER]
Image Size: ON CLIENT[/CENTER] [CENTER]
Image Size: ON SERVER[/CENTER] [CENTER]
Uploaded[/CENTER] [CENTER] [/CENTER][/COLOR]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=15’][COLOR=#7baa0f]2540P-Sales[/COLOR][/URL] [CENTER]default[/CENTER] Windows 7 [CENTER]223.47 GiB[/CENTER] [CENTER]80.54 GiB[/CENTER] [CENTER]10th, 3:19pm[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=17’][COLOR=#7baa0f]2570P-Win7ProBase[/COLOR][/URL] [CENTER]default[/CENTER] Windows 7 [CENTER]110.57 GiB[/CENTER] [CENTER]10.38 GiB[/CENTER] [CENTER]16th, 3:55pm[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=16’][COLOR=#7baa0f]6005-Base[/COLOR][/URL] [CENTER]default[/CENTER] Windows 7 [CENTER]No size available.[/CENTER] [CENTER]13.74 GiB[/CENTER] [CENTER]6th, 10:19pm[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=12’][COLOR=#7baa0f]e5540[/COLOR][/URL] [CENTER]default[/CENTER] Windows 8 [CENTER]223.57 GiB[/CENTER] [CENTER]102.12 GiB[/CENTER] [CENTER]20th, 12:25am[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=18’][COLOR=#7baa0f]E5540-MODPart[/COLOR][/URL] [CENTER]default[/CENTER] Windows 8 [CENTER]No size available.[/CENTER] [CENTER]10.60 GiB[/CENTER] [CENTER]17th, 6:30pm[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=19’][COLOR=#7baa0f]ModifiedImageE5540[/COLOR][/URL] [CENTER]default[/CENTER] Windows 8 [CENTER]No size available.[/CENTER] [CENTER]0.00 GiB[/CENTER] [CENTER]No Data[/CENTER] [CENTER] [/CENTER]
[URL=‘http://192.168.3.8/fog/management/index.php?node=images&sub=edit&id=14’][COLOR=#7baa0f]Win7Pro[/COLOR][/URL] [CENTER]default[/CENTER] Windows 7 [CENTER]No size available.[/CENTER] [CENTER]5.34 GiB[/CENTER] [CENTER]6th, 6:18pm[/CENTER] [CENTER] [/CENTER][/COLOR][/FONT][/SIZE]

[url=“/_imported_xf_attachments/1/1075_rpcinfo.txt?:”]rpcinfo.txt[/url]

[Jamie Rozek]

Okay. I just tried to delete one of the images off of the server.

I received:

[CENTER][FONT=Ubuntu][COLOR=#555555]FOGFTP: Login failed. Host: 192.168.3.8, [/COLOR][/FONT][/CENTER]
[CENTER][FONT=Ubuntu][COLOR=#555555]Username: fog, Password: xxxxxx, Error:[/COLOR][/FONT][/CENTER]
[CENTER][FONT=Ubuntu][COLOR=#555555] ftp_login(): Login incorrect.[/COLOR][/FONT][/CENTER]

[CENTER][FONT=Ubuntu][COLOR=#555555]So, we have an FTP login problem somewhere. [/COLOR][/FONT][/CENTER]

So, I check my GUI config:

FOG_TFTP_FTP_PASSWORD is the same as the password listed in the above error message.

I check Storage Management:

[CENTER][FONT=Ubuntu][COLOR=#555555]Image path: /fog/images[/COLOR][/FONT][/CENTER]

[CENTER][FONT=Ubuntu][COLOR=#555555]Management Password: Same as the password listed in the above error message.[/COLOR][/FONT][/CENTER]

[CENTER][FONT=Ubuntu][COLOR=#555555]I check the Config.class.php file:[/COLOR][/FONT][/CENTER]

TFTP_FTP_PASSWORD – matches.
STORAGE_FTP_PASSWORD - matches.

So, am I forgetting a config file somewhere or something? What is going on?
[CENTER][FONT=Ubuntu][COLOR=#555555][/COLOR][/FONT][/CENTER]

[CENTER][FONT=Ubuntu][COLOR=#555555][/COLOR][/FONT][/CENTER]

[Jamie Rozek]

I’ve reset the fog user password on the server, thinking that could be the issue. I changed it to match the password in the error in my previous post.

I am still getting the error:

mount: mounting 192.168.3.8:/fog/images/ on /images failed: Permission denied.

[Jamie Rozek]

[quote=“Jamie Rozek, post: 31440, member: 24394”]I’ve reset the fog user password on the server, thinking that could be the issue. I changed it to match the password in the error in my previous post.

I am still getting the error:

mount: mounting 192.168.3.8:/fog/images/ on /images failed: Permission denied.[/quote]

I tried to FTP into the server from a command prompt on my Windows box. I was able to get in using that username/password.

[ianabc]

The message [CODE]mount: mounting 192.168.3.8:/fog/images/ on /images failed: Permission denied.[/CODE] is from NFS so it shouldn’t care about FTP or FTP passwords. I still suspect it is a network problem somewhere - your NFS configuration should be able to function with all of the fog services turned off. If you boot one of your clients with a debug task, once it gives you a shell, can you try
[CODE]
$ mkdir /images
$ mount -o nolock,vers=3 IP.OF.YOUR.FOG:/images /images
[/CODE]
This should be the NFS mount that your client is doing during an image. When you run that command you should be able to see the server logging the mount, e.g. in /var/log/messages on RHEL
[CODE]

Jun 26 10:00:56 fog rpc.mountd[23650]: authenticated mount request from IP.OF.YOUR.CLIENT:720 for /images (/images)
[/CODE]
The log file location (and message) might be slightly different (depending on your distribution), but running “ls -lhrt /var/log” should tell you which log file was most recently changed.

For the disk image sizes, it looks like those are partimage (fog < 1.0) images. This is a known problem, see[URL=‘http://fogproject.org/forum/threads/disk-information-and-image-size-on-server-not-showing-any-data.10953/’] this thread[/URL] for more details on it.

[Jamie Rozek]

[QUOTE]For the disk image sizes, it looks like those are partimage (fog < 1.0) images. This is a known problem, see[URL=‘http://fogproject.org/forum/threads/disk-information-and-image-size-on-server-not-showing-any-data.10953/’] this thread[/URL] for more details on it.[/QUOTE]

Nope. They have all been made on 1.1.0 over the last few weeks.

As for the rest being a connection problem—I don’t see how. We haven’t changed a thing on our network. The ONLY thing that has changed is that we updated the Fog server. That’s it. It was upgraded to 1.1.1, and when that didn’t work – 1.1.2.
It was working just fine on 1.1.0.

[Jamie Rozek]

Output of ls -lhrt /var/log:

drwxr-xr-x 2 root root 4.0K Jun 26 07:03 unattended-upgrades
-rw-r–r-- 1 root root 235K Jun 26 07:08 dpkg.log
-rw-r----- 1 mysql adm 0 Jun 26 07:08 mysql.log
-rw-r----- 1 root adm 0 Jun 26 07:08 apport.log
drwxr-s— 2 mysql adm 4.0K Jun 26 07:08 mysql
-rw-r----- 1 syslog adm 1.3M Jun 26 07:08 syslog.1
drwxr-xr-x 2 root root 4.0K Jun 26 07:08 upstart
-rw-r----- 1 syslog adm 410K Jun 26 09:30 kern.log
-rw-r----- 1 syslog adm 5.3K Jun 26 13:22 syslog
-rw-r----- 1 syslog adm 72K Jun 26 13:23 auth.log
-rw-rw-r-- 1 root utmp 113K Jun 26 13:23 wtmp
-rw-rw-r-- 1 root utmp 429K Jun 26 13:23 lastlog

[Jamie Rozek]

Output of the SYSLOG: (It is refusing the mount request)

(Attached----it was reporting back as Spam when trying to post)

[url=“/_imported_xf_attachments/1/1078_syslog.txt?:”]syslog.txt[/url]

[ianabc]

OK, so it looks like syslog would be the where NFS mounts are logged on your system, is it Ubuntu? I have an ubuntu machine image and I’m deploying it now. Once it is installed I’ll take a look at the NFS setup so we can compare. I’ll post back when it is up and running.

When I said I thought the network was the problem I meant the networking on the fog server you are running: the installer script runs yum updates and my guess is that a new version of one of these packages broke something.

[ianabc]

[CODE]
Jun 26 13:22:56 troi rpc.mountd[1691]: authenticated mount request from 192.168.3.33:930 for /images (/images)
[/CODE]
Does this correspond with the fog debug task and mount command? did you get any output from that command? Could you turn on debugging for NFS, edit the /etc/default/nfs-kernel-server file to have the line
[CODE]
RPCMOUNTDOPTS=“–manage-gids --debug all”
[/CODE]
Then “service nfs-kernel-server restart” and retry the mount - syslog should get more information this time.

[Tom Elliott]

What are the permissions of /images?

Can you try these steps:
[code]touch /images/{,dev/}.mntcheck
chmod -R 777 /images
service nfs-kernel-server restart[/code]

[ianabc]

Hi Jamie, I just noticed in your syslog
[CODE]
refused mount request from 192.168.3.33 for /fog/images (/): not exported
[/CODE]
The fog clients are trying to mount the wrong location, that is almost certainly the problem. Check the image path in your storage configuration, you would either need to update that or change your export list, once they match your NFS problem should go away (and probably the FTP one as well).

[Jamie Rozek]

Thanks, guys. I finally got it working. It was a mess…I must have just started changing things willy-nilly and stopped paying attention to what I was doing.
The export file, as ianabc stated was incorrect. I did:
nano /etc/exports
[SIZE=2]pointed the first line to [/SIZE]fog/images *(ro,sync,no_wdelay,insecure_locks,no_root_sq$
[SIZE=2]and my machines immediately started imaging.[/SIZE]

The funny thing is that yesterday I’m almost certain I had a line in that file pointing to that path. Maybe I didn’t comment out the other one.

Either way, it appears to be working. Hopefully I didn’t change something else somewhere that’s going to drive me nuts!

Thanks!!!