FOG image drive failed. What now?

[cammykool]

@george1421 said in FOG image drive failed. What now?:

@cammykool Its still not clear how you have the FOG server setup. I can read what you posted two different ways.

Is your FOG OS installed on the 1TiB drive, or is it installed on a different disk? If the fog images and the FOG host system is on different disks then the update/replacement is pretty easy.

Lets start out with having you run these commands and posting the output.

1. lsblk
2. cat /etc/fstab
3. ls -la /images

Lets see how you have the system currently setup.

My apologies, the fog server is on an SSD and there’s a separate drive for storing the images. The drive starting the images have failed but the fog SSD is fine

[george1421]

@cammykool I’d still like to see the output of the commands I asked for.

On the surface replacing that drive is going to be a trivial task. It will take you longer to physically replace the drive than it will be to integrate that into FOG.

[cammykool]

@george1421 said in FOG image drive failed. What now?:

@cammykool I’d still like to see the output of the commands I asked for.

On the surface replacing that drive is going to be a trivial task. It will take you longer to physically replace the drive than it will be to integrate that into FOG.

Good copy. I’ll get that to you in the morning.

[cammykool]

lsblk:

user@fog:~$ lsblk
NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
loop0    7:0    0    55M  1 loop /snap/core18/1880
loop1    7:1    0  96.6M  1 loop /snap/core/9804
loop2    7:2    0  97.1M  1 loop /snap/core/9993
loop3    7:3    0  55.3M  1 loop /snap/core18/1885
loop4    7:4    0 160.2M  1 loop /snap/gnome-3-28-1804/116
loop5    7:5    0 161.4M  1 loop /snap/gnome-3-28-1804/128
loop6    7:6    0 255.6M  1 loop /snap/gnome-3-34-1804/33
loop7    7:7    0   2.4M  1 loop /snap/gnome-calculator/730
loop8    7:8    0   2.4M  1 loop /snap/gnome-calculator/748
loop9    7:9    0   276K  1 loop /snap/gnome-characters/539
loop10   7:10   0  62.1M  1 loop /snap/gtk-common-themes/1506
loop11   7:11   0   2.2M  1 loop /snap/gnome-system-monitor/145
loop12   7:12   0  54.8M  1 loop /snap/gtk-common-themes/1502
loop13   7:13   0   276K  1 loop /snap/gnome-characters/550
loop14   7:14   0   2.2M  1 loop /snap/gnome-system-monitor/148
loop15   7:15   0   956K  1 loop /snap/gnome-logs/93
loop16   7:16   0   956K  1 loop /snap/gnome-logs/100
loop17   7:17   0 255.6M  1 loop /snap/gnome-3-34-1804/36
sda      8:0    0 119.2G  0 disk
└─sda1   8:1    0 119.2G  0 part /
sdb      8:16   0 931.5G  0 disk
└─sdb1   8:17   0 931.5G  0 part /images
sdc      8:32   0 931.5G  0 disk
└─sdc1   8:33   0 931.5G  0 part

cat /etc/fstab:

user@fog:~$ cat /etc/fstab
# /etc/fstab: static file system information.
#
# Use 'blkid' to print the universally unique identifier for a
# device; this may be used with UUID= as a more robust way to name devices
# that works even if disks are added and removed. See fstab(5).
#
# <file system> <mount point>   <type>  <options>       <dump>  <pass>
# / was on /dev/sda1 during installation
UUID=cd18940f-0f0d-47ad-a856-9f42d3f522d4 /               ext4    errors=remount-ro 0       1
/swapfile                                 none            swap    sw              0       0
UUID=380fb981-2e16-4b48-ab3a-2625493b62c1 /images/        ext3    defaults          0       0

ls -la /images:

user@fog:~$ ls -la /images
total 20986204
drwxrwxrwx 21 fogproject root        4096 Feb  3 09:30 .
drwxr-xr-x 36 root       root        4096 Oct 13 09:27 ..
drwxrwxrwx  2 fogproject root        4096 Jan 22  2020 1-22RAMIED
drwxrwxrwx  2 fogproject root        4096 Aug 28  2019 1tb
drwxrwxrwx  2 fogproject root        4096 Sep 30  2019 3470-Norma
drwxrwxrwx  2 fogproject root        4096 Jun 23  2020 CloudReady
drwxrwxrwx  6 fogproject root        4096 Feb  9 14:31 dev
drwxrwxrwx  2 fogproject root        4096 Jun  2  2020 Elitedesk
drwxrwxrwx  2 fogproject root        4096 Jul 29  2020 EliteDeskStaff
drwxrwxrwx  2 fogproject root        4096 Aug 29  2019 Gateway
drwxrwxrwx  2 root       root        4096 Oct 15 17:08 Jeff-Master
drwxrwxrwx  2 fogproject root        4096 May 28  2020 Latitude3400
drwxrwxrwx  2 fogproject root        4096 May 28  2020 Latitude3470
drwxrwxrwx  2 root       root        4096 Feb  3 09:30 Latitude3470Staff
drwxrwxrwx  2 root       root        4096 Dec  7 10:41 LatitudeE5550
drwxrwxrwx  2 fogproject root       16384 Aug 28  2019 lost+found
-rwxrwxrwx  1 fogproject root           0 Aug 28  2019 .mntcheck
drwxrwxrwx  2 fogproject root        4096 May 27  2020 Optiplex790
drwxrwxrwx  2 fogproject root        4096 Sep 24 15:42 PHD
drwxrwxrwx  2 fogproject root        4096 Aug 28  2019 postdownloadscripts
-rwxrwxrwx  1 fogproject user 21468781622 Mar 23  2020 sdabak.img.gz
drwxrwxrwx  2 root       root        4096 Dec  1 16:25 Server2019
drwxrwxrwx  2 fogproject root        4096 Jan 13  2020 VanillaWin10

I don’t really care about the images. I have them set up now so they can be consolidated into just 3 or 4 images total not this huge list. so I don’t mind starting over as i have a machine ready to push new copies of images to the new drive.

[cammykool]

@george1421 said in FOG image drive failed. What now?:

@cammykool I’d still like to see the output of the commands I asked for.

On the surface replacing that drive is going to be a trivial task. It will take you longer to physically replace the drive than it will be to integrate that into FOG.

I posted them for you!

[george1421]

@cammykool OK good deal,

The lsblk tells us you have 3 hard drives installed at the moment

sda      8:0    0 119.2G  0 disk
└─sda1   8:1    0 119.2G  0 part /
sdb      8:16   0 931.5G  0 disk
└─sdb1   8:17   0 931.5G  0 part /images
sdc      8:32   0 931.5G  0 disk
└─sdc1   8:33   0 931.5G  0 part

/dev/sdb seems to be currently mounted on /images

I take it that /dev/sdc is the new disk that already has a partition on it?
Did you just create that partition or is it residue from another install?

[cammykool]

@george1421 said in FOG image drive failed. What now?:

@cammykool OK good deal,

The lsblk tells us you have 3 hard drives installed at the moment

sda      8:0    0 119.2G  0 disk
└─sda1   8:1    0 119.2G  0 part /
sdb      8:16   0 931.5G  0 disk
└─sdb1   8:17   0 931.5G  0 part /images
sdc      8:32   0 931.5G  0 disk
└─sdc1   8:33   0 931.5G  0 part

/dev/sdb seems to be currently mounted on /images

I take it that /dev/sdc is the new disk that already has a partition on it?
Did you just create that partition or is it residue from another install?

dev/sdc is a spare drive i had in there its even older then the drive i currently have set up for images and was in a computer running a ticker display. I don’t trust that drive at all but its in there just in case. The new drive should be here today its a 6tb surveillance drive

[george1421]

@cammykool Ok when you get it in replace /dev/sdc with the 6TiB drive then run the lsblk command again. I will give you a step by step instructions to get it setup.

[cammykool]

Sorry for the delay. Shipping was a nightmare on that drive. A 4TB bad drive is now on sdc

[cammykool]

@george1421
lsblk:

sda      8:0    0 119.2G  0 disk 
`-sda1   8:1    0 119.2G  0 part /
sdb      8:16   0 931.5G  0 disk 
`-sdb1   8:17   0 931.5G  0 part /images
sdc      8:32   0   3.7T  0 disk 
`-sdc1   8:33   0   3.7T  0 part 

[george1421]

@cammykool ok lets make sure the drive is blank with lsblk you should just see /dev/sdc with no partition. Make sure its the size you think it should be so you have the right drive. The next steps will be destructive if you pick the wrong drive.

Assuming /dev/sdc Do these commands as root to su to root (sudo su -)

fdisk /dev/sdc

# Print out existing partition table, should be blank
p
n
1
p
{pick all defaults until you get back to command prompt}

# display partition table
p

# if good then write it
w

#exit out
e

Now run lsblk again and you should see sdc1

Next thing is to format it.
mkfs.ext4 /dev/sdc1

Now lets connect it to a temp location

mkdir -p /mnt/nd
mount /dev/sdc1 /mnt/nd

Then run the df command to ensure its mounted
df -h

OK now comes the moving of data
cp -R /images/* /mnt/nd

So when that is done copying over you will have your original data on /dev/sdb1 (faulty drive) and on /dev/sdc1.

The last bit is drop the currently mounted drives and update the fstab.

umount /images
umount /mnt/nd

Now you need to update the fstab. You need to remove the line that references /dev/sdb1 (note could be by uuid so you need to pick the right drive)

Then add in this line

/dev/sdc1 /images ext4 defaults 0 0

Save and exit fstab.

Then run this command mount -a
Confirm that /dev/sdc1 is mounted with df -h

and you are done.

EDIT: I see that it already has the partition so you can skip to the format command in the sequence

[cammykool]

@george1421 yeah i already ran through and partitoned it for EXT through webmin.

[george1421]

@cammykool Ok I updated the instructions to use ext4. It looks like you just need to mount that sdc1 partition to a temp folder and copy your files over, then update fstab/mount the drive and you are done.

[cammykool]

@george1421 do I need to move images over? or will fog notice the new blank drive and just clear out what it has?

[george1421]

@cammykool Well if you do it will FOG will not even know you played with its files.

So what you are doing is copying the files from /dev/sdb1 to /dev/sdc1. Then finally mounting /dev/sdc1 in place of /dev/sdb1 (with the fstab changes). FOG only knows about /images directory and now how the OS gets there. It should work just like magic.

[cammykool]

@george1421 I’d like to clear out that long list and thin the heard. Most of those entries are for hardware we dont even have anymore.

[cammykool]

@george1421 or wopuld it be better to move everything over and thin it out using the FOG UI?

[george1421]

@cammykool said in FOG image drive failed. What now?:

I’d like to clear out that long list and thin the heard.

I guess two points here.

1. Cull the herd after the move. Save /dev/sdb1 in case something happens to the new drive.
2. FOG images are constructed of the meta data (in the fog database) and the raw files in /images. You have to delete both to clear everything out. The easiest way is via the web ui and GO INTO THE IMAGE definition and pick delete image from there. When you are in the image definition you have the ability to delete the raw data files. If you delete the image from the image list view it will only remove the metadata. This is by design.

[cammykool]

@george1421

I have started the drive transfer to cull the heard after. The state of the drive is showing itself though.

cp: error reading '/images/1-22RAMIED/d1p2.img': Input/output error

[cammykool]

@george1421 I have gotten a few more R/W errors. hopefully not enough for fog to not know they are there, and delete them.