Problems after adding storage



  • Server OS: CentOS 7
    FOG Version: 1.5.9-RC1

    I recently installed a new hard drive to expand storage for images on the master FOG node. I followed the instructions here: Adding additional image storage space to FOG server. Everything seems fine with the install so I did a test image.

    d1f4fc99-970e-4dea-b59e-161500ece298-image.png

    FOG captured the image to the new storage node and the target machine restarted and booted normal. One problem is in the Tasks menu, the machine stayed “In-Progress” (No screen shot of that.) I cancel it and check Images and it shows…

    f38a99f3-5019-443a-8355-d2441e64a5e2-image.png

    I figured it is a bug so we’ll see when the image is deployed. I set the task to target machine, machine picks up task, but I get this error…

    d56d9d1f-e9a1-46ac-8a9a-738d618d30c6-image.png

    Tasks shows that the task is stuck in progress…

    719c41ac-4090-4b56-8040-50d9df5e48ce-image.png

    This is where I am stuck.

    Deployment of another image that is stored on the original storage space deployed with no problems.

    Thank you.



  • Thank you for this information. I will have to come back to this at a later date though. I really do want to continue with this but our school district is in the middle of preparing machines and we’re relying on FOG heavily right now. I am not a Linux user so everything is new to me right now, so if we lose FOG, it will set us back a bit. Once projects are done I will come back to this. Thanks again.


  • Developer

    @xburnerx00 Ok, here is what I would do in this situation. I am going to explain it and you can decide weather you want to go that route or not. Before you proceed I need to say that those kind of operations can have caveats that I might not be able to foresee not sitting in front of the machine myself but giving you the commands as this. Please be cautious when doing this and take a backup of it all first as in the worst case of the scenario data can go lost if there is something going wrong. I will triple check all the commands I post here but can’t be absolutely sure as I don’t have the same setup here to test with first. Although I don’t think it will go wrong I just wanted to make sure you are aware.

    So please make sure you save all the commands you run and resulting outputs for us to be able to diagnose and help you if anything goes wrong. We won’t be able to help you if we don’t have detailed information on what has been done and what the outcomes were!

    • Backup
      Take a backup of you FOG server. Either a full image backup if you want/need to be able to get back to that state quickly or a full dump of the FOG database along with the two images directories.

    • Move date from /images2
      To be able to add this extra space to your FOG server a different way we will need to empty that disk first. Make sure non of the images is being used by someone else for imaging at the moment as we will be moving some of those around. Go to the FOG web UI -> images view -> pay attention to the “Storage Group” column here and edit one image definition after the other that is being set to storage group ImageStoreGroup2 at the moment -> go to “Storage Group” tab, click the checkbox “Check here to see what storage groups can be added” to see the storage group called default, add that to the image, reload the page to see it, make it the default by setting the checkbox in the second column of the table and remove ImageStoreGroup2 at the end. Check you have all the images in the list set to Storage Group default and then run the following commands as root to move the images on disk (do not say yes if asked to confirm a folder to be overwritten - say no and investigate first):

    mkdir /images/tmp
    mv /images2/dev /images/tmp/
    mv /images2/postdownloadscripts /images/tmp/
    mv -i /images2/* /images/
    ls -al /images2
    
    • Prepare disk
      If you see anything left in /images2 other than a file called .mntcheck when running the final ls command above you need to stop and take a closer look. If in doubt you can post commands and outputs here in the forums so we can help. Now I would think there is no trouble and we can proceed. The following commands (run as root) will remove the extra NFS share and fstab entry (vi commands - just put a # sign in front of the /images2/ definitions and save the files), unmount the disk and prepare it for usage as a logical volume (the last command will wipe anything from that disk and you will probably need to confirm a warning here):
    systemctl stop nfs-server
    vi /etc/fstab
    vi /etc/exports
    umount /images2
    pvs
    pvcreate /dev/sdb1
    pvs
    
    • Adding space to the LVM
      Now as a last step we want to add this volume/space to the existing logical volume group to add to your /images directory - run as root and always pay a close look at the pvs/vgs/lvs commands output to validate the operations done:
    vgs
    vgextend centos /dev/sdb1
    vgs
    df -h | grep images
    umount /images
    lvs
    lvextend -l+100%FREE /dev/mapper/centos-images
    lvs
    xfs_growfs /dev/mapper/centos-images
    mount /dev/mapper/centos-images /images 
    df -h | grep images
    
    • Reboot and check
      After you’ve confirmed the images directory is now extended, I would reboot the server instead of just starting nfs-server again just to make sure it comes up all fine again. If everything seems fine (output of df -h also after reboot and free space in the FOG web UI, doing a test deploy and capture) you may want to review the things that we moved to /images/tmp/ directory. I would hope that there is nothing much of value in there but please run ls -alR /images/tmp/ and post output here.


  • [root@fogserver ~]# pvs
      PV         VG     Fmt  Attr PSize   PFree
      /dev/sda3  centos lvm2 a--  464.56g 4.00m
    
    
    [root@fogserver ~]# vgs
      VG     #PV #LV #SN Attr   VSize   VFree
      centos   1   4   0 wz--n- 464.56g 4.00m
    
    
    [root@fogserver ~]# lvs
      LV     VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
      home   centos -wi-ao----   10.00g                                                    
      images centos -wi-ao---- <426.81g                                                    
      root   centos -wi-ao----   20.00g                                                    
      swap   centos -wi-ao----    7.75g
    

  • Developer

    @xburnerx00 That looks promising as the system is setup to use LVM already! Please run the following commands and post output here as well: pvs, vgs and lvs



  • Thank you for the reply. Here are the requested commands.

    [centos@fogserver ~]$ mount
    sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
    proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
    devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3929676k,nr_inodes=982419,mode=755)
    securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
    tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
    devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
    tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
    tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
    cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
    pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
    efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
    cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
    cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
    cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
    cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
    cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
    cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
    cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
    cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
    cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
    cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
    configfs on /sys/kernel/config type configfs (rw,relatime)
    /dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
    systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14665)
    debugfs on /sys/kernel/debug type debugfs (rw,relatime)
    hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
    mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
    nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
    /dev/sdb1 on /images2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    /dev/sda2 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    /dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
    /dev/mapper/centos-home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    /dev/mapper/centos-images on /images type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
    sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
    tmpfs on /run/user/1002 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=789420k,mode=700,uid=1002,gid=1002)
    tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=789420k,mode=700,uid=42,gid=42)
    gvfsd-fuse on /run/user/1002/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1002,group_id=1002)
    fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)
    
    
    [centos@fogserver ~]$ df -h 
    Filesystem                 Size  Used Avail Use% Mounted on
    devtmpfs                   3.8G     0  3.8G   0% /dev
    tmpfs                      3.8G   26M  3.8G   1% /dev/shm
    tmpfs                      3.8G   10M  3.8G   1% /run
    tmpfs                      3.8G     0  3.8G   0% /sys/fs/cgroup
    /dev/mapper/centos-root     20G  8.3G   12G  42% /
    /dev/sdb1                  466G   35G  432G   8% /images2
    /dev/sda2                 1014M  386M  629M  39% /boot
    /dev/sda1                  200M   12M  189M   6% /boot/efi
    /dev/mapper/centos-home     10G  597M  9.5G   6% /home
    /dev/mapper/centos-images  427G  356G   72G  84% /images
    tmpfs                      771M   36K  771M   1% /run/user/1002
    tmpfs                      771M   12K  771M   1% /run/user/42
    
    
    [centos@fogserver ~]$ lsblk
    NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
    sda                 8:0    0 465.8G  0 disk 
    ├─sda1              8:1    0   200M  0 part /boot/efi
    ├─sda2              8:2    0     1G  0 part /boot
    └─sda3              8:3    0 464.6G  0 part 
      ├─centos-root   253:0    0    20G  0 lvm  /
      ├─centos-swap   253:1    0   7.8G  0 lvm  [SWAP]
      ├─centos-home   253:2    0    10G  0 lvm  /home
      └─centos-images 253:3    0 426.8G  0 lvm  /images
    sdb                 8:16   0 465.8G  0 disk 
    └─sdb1              8:17   0 465.8G  0 part /images2
    
    

  • Developer

    @xburnerx00 You cannot address both storages dynamically using the “static” USB boot method. You’d need to add a second boot entry to your GRUB config file to be able to select one or the other storage.

    But there is another way around this. The way George described adding more storage to your FOG server in the forum post is one way. But you can put things together differently and have all the space in your one DefaultMember storage node. But to be able to help you with that we need to know more about your setup. First run the commands mount, df -h and lsblk on your FOG server and post all the outputs here.



  • @xburnerx00 said in Problems after adding storage:

    storage=192.168.10.200:/images2/ storageip=192.168.10.200

    Thank you for the reply.

    Yes, this line change to the GRUB file seems to have worked, but it causes another issue. Now when I try to deploy an image that is stored in the original storage node, FOG says that the original storage node cannot be found, because I think it is only looking in the second storage node. Is there a way to change the GRUB on the usb where it would look through both storages for images?

    I’m thinking I change the line to this, just not sure.

    storage=192.168.10.200:/images storage=192.168.10.200:/images2/ storageip=192.168.10.200
    

    Thank you.


  • Moderator

    @xburnerx00 said in Problems after adding storage:

    linux $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4

    ^^^
    This line tells FOS Linux what to do when it boots. Normally the FOG server sends these settings to FOS Linux, since you are using the USB method, we rely on GRUB to tell FOS Linux what to do.

    While I haven’t been really following this thread, Sebastian’s instructions are to paste those two commands at the end of what I quoted above for menu selection 1.



  • This is the grub.cfg on the USB. Where do I put the line? Near the top with the lines that say set? Do I add ‘set’? before the lines you asked me to try? Are the lines two separate lines? Sorry, for all the questions.

    set myfogip=192.168.10.200
    set myimage=/boot/bzImage
    set myinits=/boot/init.xz
    set myloglevel=4
    set timeout=-1
    insmod all_video
    
    menuentry "1. FOG Image Deploy/Capture" {
     echo loading the kernel
     linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4
     echo loading the virtual hard drive
     initrd $myinits
     echo booting kernel...
    }
    
    menuentry "2. Perform Full Host Registration and Inventory" {
     echo loading the kernel
     linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=manreg
     echo loading the virtual hard drive
     initrd $myinits
     echo booting kernel...
    }
    
    menuentry "3. Quick Registration and Inventory" {
     echo loading the kernel
     linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=autoreg
     echo loading the virtual hard drive
     initrd $myinits
     echo booting kernel...
    }
    
    menuentry "4. Client System Information (Compatibility)" {
     echo loading the kernel
     linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=sysinfo
     echo loading the virtual hard drive
     initrd $myinits
     echo booting kernel...
    }
    
    menuentry "5. Run Memtest86+" {
     linux /boot/memdisk iso raw
     initrd /boot/memtest.bin
    }
    
    menuentry "6. FOG Debug Kernel" {
     echo loading the kernel
     linux  $myimage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=275000 keymap= boottype=usb consoleblank=0 rootfstype=ext4 isdebug=yes
     echo loading the virtual hard drive
     initrd $myinits
     echo booting kernel...
    }
    
    menuentry "7. FOG iPXE Jumpstart BIOS" {
     echo loading the kernel
     linux16  /boot/ipxe.krn
     echo booting iPXE...
    }
    
    menuentry "8. FOG iPXE Jumpstart EFI" {
     echo chain loading the kernel
     insmod chain 
     chainloader /boot/ipxe.efi
     echo booting iPXE-efi...
    }
    

  • Developer

    @xburnerx00 said in Problems after adding storage:

    Where exactly do I add this line? On the grub.cfg file on my USB boot?

    Yes exactly.



  • @Sebastian-Roth said in Problems after adding storage:

    storage=192.168.10.200:/images2/ storageip=192.168.10.200

    Thank you for the reply.

    Where exactly do I add this line? On the grub.cfg file on my USB boot? That is the only place I remember making changes on the drive when I followed the instructions from here.


  • Developer

    @xburnerx00 You are using USB boot method here which is not fully supported I have to say. See if you can still make it work adding those two variables to your kernel command line:

    storage=192.168.10.200:/images2/ storageip=192.168.10.200
    

    Make sure IP address and path are correct. Can’t promise you anything though!


Log in to reply
 

249
Online

7.0k
Users

14.2k
Topics

134.4k
Posts