Problems after adding storage

[xburnerx00]

Server OS: CentOS 7
FOG Version: 1.5.9-RC1

I recently installed a new hard drive to expand storage for images on the master FOG node. I followed the instructions here: Adding additional image storage space to FOG server. Everything seems fine with the install so I did a test image.

FOG captured the image to the new storage node and the target machine restarted and booted normal. One problem is in the Tasks menu, the machine stayed “In-Progress” (No screen shot of that.) I cancel it and check Images and it shows…

I figured it is a bug so we’ll see when the image is deployed. I set the task to target machine, machine picks up task, but I get this error…

Tasks shows that the task is stuck in progress…

This is where I am stuck.

Deployment of another image that is stored on the original storage space deployed with no problems.

Thank you.

[Sebastian Roth]

@xburnerx00 You are using USB boot method here which is not fully supported I have to say. See if you can still make it work adding those two variables to your kernel command line:

storage=192.168.10.200:/images2/ storageip=192.168.10.200

Make sure IP address and path are correct. Can’t promise you anything though!

[xburnerx00]

@Sebastian-Roth said in Problems after adding storage:

storage=192.168.10.200:/images2/ storageip=192.168.10.200

Thank you for the reply.

Where exactly do I add this line? On the grub.cfg file on my USB boot? That is the only place I remember making changes on the drive when I followed the instructions from here.

[Sebastian Roth]

@xburnerx00 said in Problems after adding storage:

Where exactly do I add this line? On the grub.cfg file on my USB boot?

Yes exactly.

[xburnerx00]

This is the grub.cfg on the USB. Where do I put the line? Near the top with the lines that say set? Do I add ‘set’? before the lines you asked me to try? Are the lines two separate lines? Sorry, for all the questions.

set myfogip=192.168.10.200
set myimage=/boot/bzImage
set myinits=/boot/init.xz
set myloglevel=4
set timeout=-1
insmod all_video

menuentry "1. FOG Image Deploy/Capture" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "2. Perform Full Host Registration and Inventory" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=manreg
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "3. Quick Registration and Inventory" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=autoreg
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "4. Client System Information (Compatibility)" {
 echo loading the kernel
 linux  $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4 mode=sysinfo
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "5. Run Memtest86+" {
 linux /boot/memdisk iso raw
 initrd /boot/memtest.bin
}

menuentry "6. FOG Debug Kernel" {
 echo loading the kernel
 linux  $myimage loglevel=7 init=/sbin/init root=/dev/ram0 rw ramdisk_size=275000 keymap= boottype=usb consoleblank=0 rootfstype=ext4 isdebug=yes
 echo loading the virtual hard drive
 initrd $myinits
 echo booting kernel...
}

menuentry "7. FOG iPXE Jumpstart BIOS" {
 echo loading the kernel
 linux16  /boot/ipxe.krn
 echo booting iPXE...
}

menuentry "8. FOG iPXE Jumpstart EFI" {
 echo chain loading the kernel
 insmod chain 
 chainloader /boot/ipxe.efi
 echo booting iPXE-efi...
}

[george1421]

@xburnerx00 said in Problems after adding storage:

linux $myimage loglevel=$myloglevel initrd=init.xz root=/dev/ram0 rw ramdisk_size=275000 keymap= web=$myfogip/fog/ boottype=usb consoleblank=0 rootfstype=ext4

^^^
This line tells FOS Linux what to do when it boots. Normally the FOG server sends these settings to FOS Linux, since you are using the USB method, we rely on GRUB to tell FOS Linux what to do.

While I haven’t been really following this thread, Sebastian’s instructions are to paste those two commands at the end of what I quoted above for menu selection 1.

[xburnerx00]

@xburnerx00 said in Problems after adding storage:

storage=192.168.10.200:/images2/ storageip=192.168.10.200

Thank you for the reply.

Yes, this line change to the GRUB file seems to have worked, but it causes another issue. Now when I try to deploy an image that is stored in the original storage node, FOG says that the original storage node cannot be found, because I think it is only looking in the second storage node. Is there a way to change the GRUB on the usb where it would look through both storages for images?

I’m thinking I change the line to this, just not sure.

storage=192.168.10.200:/images storage=192.168.10.200:/images2/ storageip=192.168.10.200

Thank you.

[Sebastian Roth]

@xburnerx00 You cannot address both storages dynamically using the “static” USB boot method. You’d need to add a second boot entry to your GRUB config file to be able to select one or the other storage.

But there is another way around this. The way George described adding more storage to your FOG server in the forum post is one way. But you can put things together differently and have all the space in your one DefaultMember storage node. But to be able to help you with that we need to know more about your setup. First run the commands mount, df -h and lsblk on your FOG server and post all the outputs here.

[xburnerx00]

Thank you for the reply. Here are the requested commands.

[centos@fogserver ~]$ mount
sysfs on /sys type sysfs (rw,nosuid,nodev,noexec,relatime,seclabel)
proc on /proc type proc (rw,nosuid,nodev,noexec,relatime)
devtmpfs on /dev type devtmpfs (rw,nosuid,seclabel,size=3929676k,nr_inodes=982419,mode=755)
securityfs on /sys/kernel/security type securityfs (rw,nosuid,nodev,noexec,relatime)
tmpfs on /dev/shm type tmpfs (rw,nosuid,nodev,seclabel)
devpts on /dev/pts type devpts (rw,nosuid,noexec,relatime,seclabel,gid=5,mode=620,ptmxmode=000)
tmpfs on /run type tmpfs (rw,nosuid,nodev,seclabel,mode=755)
tmpfs on /sys/fs/cgroup type tmpfs (ro,nosuid,nodev,noexec,seclabel,mode=755)
cgroup on /sys/fs/cgroup/systemd type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,xattr,release_agent=/usr/lib/systemd/systemd-cgroups-agent,name=systemd)
pstore on /sys/fs/pstore type pstore (rw,nosuid,nodev,noexec,relatime)
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
cgroup on /sys/fs/cgroup/blkio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,blkio)
cgroup on /sys/fs/cgroup/perf_event type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,perf_event)
cgroup on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,freezer)
cgroup on /sys/fs/cgroup/cpu,cpuacct type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuacct,cpu)
cgroup on /sys/fs/cgroup/pids type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,pids)
cgroup on /sys/fs/cgroup/net_cls,net_prio type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,net_prio,net_cls)
cgroup on /sys/fs/cgroup/memory type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,memory)
cgroup on /sys/fs/cgroup/hugetlb type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,hugetlb)
cgroup on /sys/fs/cgroup/devices type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,devices)
cgroup on /sys/fs/cgroup/cpuset type cgroup (rw,nosuid,nodev,noexec,relatime,seclabel,cpuset)
configfs on /sys/kernel/config type configfs (rw,relatime)
/dev/mapper/centos-root on / type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
selinuxfs on /sys/fs/selinux type selinuxfs (rw,relatime)
systemd-1 on /proc/sys/fs/binfmt_misc type autofs (rw,relatime,fd=35,pgrp=1,timeout=0,minproto=5,maxproto=5,direct,pipe_ino=14665)
debugfs on /sys/kernel/debug type debugfs (rw,relatime)
hugetlbfs on /dev/hugepages type hugetlbfs (rw,relatime,seclabel)
mqueue on /dev/mqueue type mqueue (rw,relatime,seclabel)
nfsd on /proc/fs/nfsd type nfsd (rw,relatime)
/dev/sdb1 on /images2 type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sda2 on /boot type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/sda1 on /boot/efi type vfat (rw,relatime,fmask=0077,dmask=0077,codepage=437,iocharset=ascii,shortname=winnt,errors=remount-ro)
/dev/mapper/centos-home on /home type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
/dev/mapper/centos-images on /images type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
sunrpc on /var/lib/nfs/rpc_pipefs type rpc_pipefs (rw,relatime)
tmpfs on /run/user/1002 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=789420k,mode=700,uid=1002,gid=1002)
tmpfs on /run/user/42 type tmpfs (rw,nosuid,nodev,relatime,seclabel,size=789420k,mode=700,uid=42,gid=42)
gvfsd-fuse on /run/user/1002/gvfs type fuse.gvfsd-fuse (rw,nosuid,nodev,relatime,user_id=1002,group_id=1002)
fusectl on /sys/fs/fuse/connections type fusectl (rw,relatime)

[centos@fogserver ~]$ df -h 
Filesystem                 Size  Used Avail Use% Mounted on
devtmpfs                   3.8G     0  3.8G   0% /dev
tmpfs                      3.8G   26M  3.8G   1% /dev/shm
tmpfs                      3.8G   10M  3.8G   1% /run
tmpfs                      3.8G     0  3.8G   0% /sys/fs/cgroup
/dev/mapper/centos-root     20G  8.3G   12G  42% /
/dev/sdb1                  466G   35G  432G   8% /images2
/dev/sda2                 1014M  386M  629M  39% /boot
/dev/sda1                  200M   12M  189M   6% /boot/efi
/dev/mapper/centos-home     10G  597M  9.5G   6% /home
/dev/mapper/centos-images  427G  356G   72G  84% /images
tmpfs                      771M   36K  771M   1% /run/user/1002
tmpfs                      771M   12K  771M   1% /run/user/42

[centos@fogserver ~]$ lsblk
NAME              MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda                 8:0    0 465.8G  0 disk 
├─sda1              8:1    0   200M  0 part /boot/efi
├─sda2              8:2    0     1G  0 part /boot
└─sda3              8:3    0 464.6G  0 part 
  ├─centos-root   253:0    0    20G  0 lvm  /
  ├─centos-swap   253:1    0   7.8G  0 lvm  [SWAP]
  ├─centos-home   253:2    0    10G  0 lvm  /home
  └─centos-images 253:3    0 426.8G  0 lvm  /images
sdb                 8:16   0 465.8G  0 disk 
└─sdb1              8:17   0 465.8G  0 part /images2

[Sebastian Roth]

@xburnerx00 That looks promising as the system is setup to use LVM already! Please run the following commands and post output here as well: pvs, vgs and lvs

[xburnerx00]

[root@fogserver ~]# pvs
  PV         VG     Fmt  Attr PSize   PFree
  /dev/sda3  centos lvm2 a--  464.56g 4.00m

[root@fogserver ~]# vgs
  VG     #PV #LV #SN Attr   VSize   VFree
  centos   1   4   0 wz--n- 464.56g 4.00m

[root@fogserver ~]# lvs
  LV     VG     Attr       LSize    Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert
  home   centos -wi-ao----   10.00g                                                    
  images centos -wi-ao---- <426.81g                                                    
  root   centos -wi-ao----   20.00g                                                    
  swap   centos -wi-ao----    7.75g

[Sebastian Roth]

@xburnerx00 Ok, here is what I would do in this situation. I am going to explain it and you can decide weather you want to go that route or not. Before you proceed I need to say that those kind of operations can have caveats that I might not be able to foresee not sitting in front of the machine myself but giving you the commands as this. Please be cautious when doing this and take a backup of it all first as in the worst case of the scenario data can go lost if there is something going wrong. I will triple check all the commands I post here but can’t be absolutely sure as I don’t have the same setup here to test with first. Although I don’t think it will go wrong I just wanted to make sure you are aware.

So please make sure you save all the commands you run and resulting outputs for us to be able to diagnose and help you if anything goes wrong. We won’t be able to help you if we don’t have detailed information on what has been done and what the outcomes were!

• Backup
  Take a backup of you FOG server. Either a full image backup if you want/need to be able to get back to that state quickly or a full dump of the FOG database along with the two images directories.

• Move date from /images2
  To be able to add this extra space to your FOG server a different way we will need to empty that disk first. Make sure non of the images is being used by someone else for imaging at the moment as we will be moving some of those around. Go to the FOG web UI -> images view -> pay attention to the “Storage Group” column here and edit one image definition after the other that is being set to storage group ImageStoreGroup2 at the moment -> go to “Storage Group” tab, click the checkbox “Check here to see what storage groups can be added” to see the storage group called default, add that to the image, reload the page to see it, make it the default by setting the checkbox in the second column of the table and remove ImageStoreGroup2 at the end. Check you have all the images in the list set to Storage Group default and then run the following commands as root to move the images on disk (do not say yes if asked to confirm a folder to be overwritten - say no and investigate first):

mkdir /images/tmp
mv /images2/dev /images/tmp/
mv /images2/postdownloadscripts /images/tmp/
mv -i /images2/* /images/
ls -al /images2

• Prepare disk
  If you see anything left in /images2 other than a file called .mntcheck when running the final ls command above you need to stop and take a closer look. If in doubt you can post commands and outputs here in the forums so we can help. Now I would think there is no trouble and we can proceed. The following commands (run as root) will remove the extra NFS share and fstab entry (vi commands - just put a # sign in front of the /images2/ definitions and save the files), unmount the disk and prepare it for usage as a logical volume (the last command will wipe anything from that disk and you will probably need to confirm a warning here):

systemctl stop nfs-server
vi /etc/fstab
vi /etc/exports
umount /images2
pvs
pvcreate /dev/sdb1
pvs

• Adding space to the LVM
  Now as a last step we want to add this volume/space to the existing logical volume group to add to your /images directory - run as root and always pay a close look at the pvs/vgs/lvs commands output to validate the operations done:

vgs
vgextend centos /dev/sdb1
vgs
df -h | grep images
umount /images
lvs
lvextend -l+100%FREE /dev/mapper/centos-images
lvs
xfs_growfs /dev/mapper/centos-images
mount /dev/mapper/centos-images /images 
df -h | grep images

• Reboot and check
  After you’ve confirmed the images directory is now extended, I would reboot the server instead of just starting nfs-server again just to make sure it comes up all fine again. If everything seems fine (output of df -h also after reboot and free space in the FOG web UI, doing a test deploy and capture) you may want to review the things that we moved to /images/tmp/ directory. I would hope that there is nothing much of value in there but please run ls -alR /images/tmp/ and post output here.

[xburnerx00]

Thank you for this information. I will have to come back to this at a later date though. I really do want to continue with this but our school district is in the middle of preparing machines and we’re relying on FOG heavily right now. I am not a Linux user so everything is new to me right now, so if we lose FOG, it will set us back a bit. Once projects are done I will come back to this. Thanks again.