• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

NFS problems after upgrade to trunk

Scheduled Pinned Locked Moved Solved
FOG Problems
4
42
12.6k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • G
    george1421 Moderator
    last edited by george1421 Feb 8, 2016, 2:08 PM Feb 8, 2016, 8:08 PM

    I didn’t read all of the posts here, but could you do a showmount -e 127.0.0.1 This will show us what you have NFS shared on your FOG server. You could then do the same command but use your FOG servers external IP address instead of the loopback interface. It could be a firewall issue.

    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

    J 1 Reply Last reply Feb 8, 2016, 8:10 PM Reply Quote 0
    • J
      John Sartoris @george1421
      last edited by Feb 8, 2016, 8:10 PM

      @george1421 said:

      I didn’t read all of the posts here, but could you do a showmount -e 127.0.0.1 This will show us what you have NFS shared on your FOG server.

      That’s shows that the exports are proper and they do actually work from the storage node cross site, but a client on the local site can’t connect.

      1 Reply Last reply Reply Quote 0
      • J
        John Sartoris @Wayne Workman
        last edited by Feb 8, 2016, 8:15 PM

        @Wayne-Workman said:

        @John-Sartoris refused is different than denied… Have you checked for IP conflicts?

        I haven’t specifically checked, but this server is configured in the same way as the rest in our environment. DHCP with a reservation. This is the only server that should get this address. Access ports are not configured in this vlan. I have not had any issues connecting to the server expect via NFS. Ping and SSH work from debug client.

        W 1 Reply Last reply Feb 8, 2016, 8:18 PM Reply Quote 0
        • W
          Wayne Workman @John Sartoris
          last edited by Feb 8, 2016, 8:18 PM

          @John-Sartoris I’m thinking it is probably something network related.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
          Daily Clean Installation Results:
          https://fogtesting.fogproject.us/
          FOG Reporting:
          https://fog-external-reporting-results.fogproject.us/

          J 1 Reply Last reply Feb 8, 2016, 8:28 PM Reply Quote 0
          • J
            John Sartoris @Wayne Workman
            last edited by Feb 8, 2016, 8:28 PM

            @Wayne-Workman

            I understand and agree with the assessment, but nothing has changed on the LAN in weeks other that the fog server updates. Firewalls are disabled and allowed for good measure. I even just tried added iptables rules without success.

            W 1 Reply Last reply Feb 8, 2016, 8:28 PM Reply Quote 0
            • W
              Wayne Workman @John Sartoris
              last edited by Feb 8, 2016, 8:28 PM

              @John-Sartoris can you quickly throw together a CentOS 7 VM and install fog trunk to test?

              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
              Daily Clean Installation Results:
              https://fogtesting.fogproject.us/
              FOG Reporting:
              https://fog-external-reporting-results.fogproject.us/

              G 1 Reply Last reply Feb 8, 2016, 8:34 PM Reply Quote 0
              • G
                george1421 Moderator @Wayne Workman
                last edited by george1421 Feb 8, 2016, 2:41 PM Feb 8, 2016, 8:34 PM

                @Wayne-Workman I’m in a meeting right now (yeah its a bit boring) so I can’t test. But from a debug (boot) session, is the showmount command installed in FOS? It would be interesting to know from the client perspective if the FOG server is showing its mount information.

                [edit] The other thing would be to try to do a manual nfs mount from the FOS client to the FOG server. If it maps then there is a parameter setup incorrectly (somewhere) in the FOG GUI. The mount command would be something like mount <fog_server_ip>:/images /img (or what ever the local directory is called on the FOG client) [/edit]

                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                J 1 Reply Last reply Feb 8, 2016, 8:59 PM Reply Quote 0
                • J
                  John Sartoris @george1421
                  last edited by Feb 8, 2016, 8:59 PM

                  @george1421

                  I’m trying to manually mount from the FOG Client now, and I’m receiving a connection refused.

                  @Wayne-Workman
                  Configuring another VM would be possible, but quite a heavy bit of work for what sure seems to be a firewall config or NFS ACL issue that happen during OS or Fog upgrade.

                  W 1 Reply Last reply Feb 8, 2016, 9:01 PM Reply Quote 0
                  • W
                    Wayne Workman @John Sartoris
                    last edited by Wayne Workman Feb 8, 2016, 3:09 PM Feb 8, 2016, 9:01 PM

                    @John-Sartoris said:

                    Configuring another VM would be possible, but quite a heavy bit of work for what sure seems to be a firewall config or NFS ACL issue that happen during OS or Fog upgrade.

                    That’s just the thing though. you’ve disabled UFW, and NFS has no protections. Ubuntu does not come pre-loaded with Security Enhanced Linux, either, like other distributions do https://wiki.ubuntu.com/SELinux

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                    Daily Clean Installation Results:
                    https://fogtesting.fogproject.us/
                    FOG Reporting:
                    https://fog-external-reporting-results.fogproject.us/

                    J 1 Reply Last reply Feb 8, 2016, 9:15 PM Reply Quote 0
                    • J
                      John Sartoris @Wayne Workman
                      last edited by Feb 8, 2016, 9:15 PM

                      @Wayne-Workman

                      Do NFS connections get logged on the server? I’ve tried several suggestions I’ve found but I don’t see any messages for successful or failed connections. I’m sure I’ve seen connection attempts in a log before when testing VMWare ESXi at home, just can’t recall where.

                      Looks like the export option “no_acl” deals with the file system ACLs not anything network related. And the export is to * so in theory anyone should be able to connect.

                      1 Reply Last reply Reply Quote 0
                      • G
                        george1421 Moderator
                        last edited by Feb 8, 2016, 9:30 PM

                        Just out of curiosity, do the /etc/export files match between the master fog server and the storage node? Can you post the export files here?

                        Other random thoughts:

                        Are you running NFSv4 on either of your server? NFSv4 has additional security requirements.

                        Does the FOG user ID have the same group and user IDs on all fog servers? (this may not be mandatory)

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                        J 1 Reply Last reply Feb 8, 2016, 9:43 PM Reply Quote 0
                        • J
                          John Sartoris @george1421
                          last edited by John Sartoris Feb 8, 2016, 3:47 PM Feb 8, 2016, 9:43 PM

                          @george1421

                          As posted earlier, config matches on both servers.

                          /etc/exports

                          /images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
                          /images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
                          

                          nfs-kernel-server 1.2.5-3ubuntu3.2 does appear to be NFSv4, however this is the version running on both servers. It may have been original with the working storage node, while the master may have been upgraded to 3 to 4.

                          http://packages.ubuntu.com/precise-updates/nfs-kernel-server

                          also NFS does appear to be reaching the server from the client.

                          # netstat -t -u -c | grep 10.2.ccc.bbb
                          tcp        0      0 fog-01:ssh  10.2.ccc.bbb:41650       ESTABLISHED
                          tcp        0      0 fog-01:nfs  10.2.ccc.bbb:692         ESTABLISHED
                          
                          
                          1 Reply Last reply Reply Quote 0
                          • G
                            george1421 Moderator
                            last edited by george1421 Feb 8, 2016, 3:54 PM Feb 8, 2016, 9:49 PM

                            Thank you for posting the export file. Understand from my perspective this is a system that was setup by hand (no offense intended). So we must look in every corner.

                            OK so this is a NFSv4 setup. I know the access controls are greater for NFSv4 over v3. Let me do a little google–fu and see what I can find.

                            [edit] Just for clarity I’m from the rhel world, so ubuntu is just enough different to be maddening at times. There appears to be two files that should be compared between the working server and the master fog server. /etc/default/nfs-common and /etc/default/nfs-kernel-server these hold the settings for the nfs server. Also ensure that the rpc.idmapd process is running [/edit]

                            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                            J 1 Reply Last reply Feb 8, 2016, 9:55 PM Reply Quote 0
                            • J
                              John Sartoris @george1421
                              last edited by John Sartoris Feb 8, 2016, 3:59 PM Feb 8, 2016, 9:55 PM

                              @george1421

                              I’ve tried to disable NFSv4 as per http://andy.delcambre.com/2007/06/25/disabling-nfsv4-on-ubuntu.html and the comments in “/etc/default/nfs-kernel-server” however the problem still exists.

                              @Wayne-Workman

                              Just wanted to say I really appreciate all the help you have both been.

                              I’m out for the day. I’ll pick this up again in the morning.

                              W 1 Reply Last reply Feb 9, 2016, 12:57 AM Reply Quote 1
                              • W
                                Wayne Workman @John Sartoris
                                last edited by Feb 9, 2016, 12:57 AM

                                @John-Sartoris I exhausted what I know about it.

                                If I were in your shoes, I’d rebuild the server. This thread has been open for 8+ hours when a server rebuild would fix it in under 4. It’s just my opinion, but again if it were me, I’d cut my losses and rebuild.

                                Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                Daily Clean Installation Results:
                                https://fogtesting.fogproject.us/
                                FOG Reporting:
                                https://fog-external-reporting-results.fogproject.us/

                                1 Reply Last reply Reply Quote 0
                                • G
                                  george1421 Moderator
                                  last edited by george1421 Feb 8, 2016, 8:31 PM Feb 9, 2016, 1:55 AM

                                  While I’m almost where Wayne is, I think you should compare these files /etc/default/nfs-common and /etc/default/nfs-kernel-server between your working storage node and your troublesome master server. Since you think the storage node was already running nfsv4 someone may have already adjusted this server properly. You are so close now I’d hate for you to give up.

                                  Right now you have the luxury of having one system that does work and one that doesn’t. You just need to find the differences in the setup.

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                  J 1 Reply Last reply Feb 9, 2016, 7:56 PM Reply Quote 0
                                  • J
                                    John Sartoris @george1421
                                    last edited by Feb 9, 2016, 7:56 PM

                                    @george1421 @Wayne-Workman

                                    I understand what you are telling me about rebuilding. Any reason I should switch from Ubuntu to CentOS like you suggest? Ever since I switched from Gentoo, probably 10 years ago, I’ve been using Ubuntu. It’s what I know, I could probably pick up the the intricacies of CentOS vs Ubuntu easily enough, but if there is no reason then why?

                                    As for my current state, the storage node stopped working, and I haven’t been able to get it back. I’ve upgraded both to Ubuntu 14.04, and after a Grub related issues on the master node both are back to the same state for NFS. I have however verified that I have NFSv4 disable and am working only in v3. I found the log file showing connections to rpc.mountd.

                                    /var/log/syslog shows a successful connection being made.

                                    Feb  9 13:30:42 lk-fog-01 rpc.mountd[1360]: authenticated mount request from 10.2.ccc.bbb:911 for /images (/images)
                                    

                                    reports just the same as the storage node which completes and can browse without issue.

                                    Feb  9 13:35:27 lk-fog-01 rpc.mountd[1360]: authenticated unmount request from 10.1.yyy.xxx:895 for /images (/images)
                                    
                                    G J 2 Replies Last reply Feb 9, 2016, 8:16 PM Reply Quote 0
                                    • G
                                      george1421 Moderator @John Sartoris
                                      last edited by Feb 9, 2016, 8:16 PM

                                      @John-Sartoris There is nothing wrong with ubuntu. Working with it is sometimes harder and then sometimes easier then RHEL. Its all perspective. I would not change unless you have to choice.

                                      Have you confirmed that the rpc.idmapd process is running on your FOG master server?

                                      Thinking a bit more, root is not able to nfs mount between the servers (I would assume this is the same if you tried to nfs mount from the fog master server to the images folder on the storage node (i.e. just remove the FOS client out of the picture).

                                      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!

                                      1 Reply Last reply Reply Quote 0
                                      • J
                                        John Sartoris @John Sartoris
                                        last edited by John Sartoris Feb 9, 2016, 2:41 PM Feb 9, 2016, 8:24 PM

                                        @george1421 @Wayne-Workman

                                        Well, it’s working. I’m not entirely sure what happened, but I can tell you what I noticed that made me try the full deploy again was the “fog.mount” command in debug mode. I was looking at all the commands available looking for ideas to try and I saw that. I tried it and it completed back to a prompt, then “mount” listed /images as connected. I still was unable to mount manually but…

                                        One thing I did do was reset the fog user password on the master server. I tried to download a kernel ( i’m not even using it) and received a complaint that the ftp password was wrong and it should be a long crypted string. Then when trying to create a deploy task the ftp password was again wrong, and it should be the normal short password I started with. I haven’t tried to download another kernel to test for the error.

                                        Edit - Correction, the master node is working, but the storage node still doesn’t mount…

                                        From debug mode mount is returning “operation not supported” to storage node:/Images

                                        W 1 Reply Last reply Feb 9, 2016, 8:42 PM Reply Quote 0
                                        • W
                                          Wayne Workman @John Sartoris
                                          last edited by Feb 9, 2016, 8:42 PM

                                          @John-Sartoris said:

                                          Edit - Correction, the master node is working, but the storage node still doesn’t mount…

                                          Am I mistaken, or was the problem exactly the opposite before?

                                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
                                          Daily Clean Installation Results:
                                          https://fogtesting.fogproject.us/
                                          FOG Reporting:
                                          https://fog-external-reporting-results.fogproject.us/

                                          J 1 Reply Last reply Feb 9, 2016, 8:45 PM Reply Quote 0
                                          • 1
                                          • 2
                                          • 3
                                          • 2 / 3
                                          2 / 3
                                          • First post
                                            29/42
                                            Last post

                                          218

                                          Online

                                          12.0k

                                          Users

                                          17.3k

                                          Topics

                                          155.2k

                                          Posts
                                          Copyright © 2012-2024 FOG Project