1.3.0-RC-7: Image store corrupt, unable to locate MBR

[cbc-tgschultz]

We recently updated our Fog installation to 1.3.0-RC-7 because, after a debian dist-upgrade, there was some issue where Fog couldn’t seem to write new data to the database. Installing the latest git version seemed to be the path of least resistance on fixing it, so I did that, and it did indeed solve the database issue.

However, we are now finding that we can’t start the image that kicked off this whole thing. The error is:

Image store corrupt, unable to locate MBR, no file found (MBRFileName)
Args Passed: /images/Windows10Corp 1 mbrfilename
Variable set to: /images/Windows10Corp/d1.mbr
mbrfilename Variable set to: /images/Windows10Corp/d1.mbr

Thing is, there is no d1.mbr for Windows10Corp. I have verified with backups taken before the updates that in fact there never was. Several of our images do not have this file. I suspect this is because they might be GPT, and the error is caused by some change in the imaging system that no longer handles this correctly. I haven’t had much time to dig into it yet, so any help pointing me in the right direction would be appreciated.

[Quazz]

Can you show us what’s in the folder if there is no d1.mbr file?

[cbc-tgschultz]

d1.fixed_size_partitions
d1.original.fstypes
d1.original.partitions
d1.original.partitions.8
d1.original.swapuuids
rec.img.000
rec.img.000.8
sys.img.000

There seems to be code in the fog.download task for handling the situation of no MBR and using these files, but clearly that isn’t triggering. A Windows 10 image created a while after this one uses a different style:

d1.fixed_size_partitions
d1.mbr
d1.minimum.partitions
d1.original.fstypes
d1.original.swapuuids
d1p1.img
d1p2.img
d1.partitions

PS: It should be noted that we were able to work around the issue by copying the d1.mbr from the second image to the first. However, I cannot see why this should be necessary.

[Quazz]

@cbc-tgschultz I don’t believe the old style is very compatible with Windows 10, so if at all possible, create a new image of it.

[cbc-tgschultz]

@Quazz

Which style would be the “old” one?

I’m fairly confident this image was working as of the last git version we were using (64something? I’m afraid I lost track. It was pre-RC anyways). And, as I mentioned, there does appear to be code to handle this situation that just isn’t triggering. Since I have a workaround, I’m going to try and find a fix for it by patching up fog.download. If I manage to work it out I’ll post the results.

[Quazz]

@cbc-tgschultz

rec.img.000
rec.img.000.8
sys.img.000

is the old style.

The Web interface for images should list it as Partimage instead of Partclone I believe?

[cbc-tgschultz]

It appears to be a partclone image and is listed as such in the GUI.

Here’s what I think has happened. There are lines in fog.download and fog.upload like this:

if [ “$osid” == “5” ] || [ “$osid” == “6” ] || [ “$osid” == “7” ]; then

The code under those lines seem to handle this kind of image. 5, 6, and 7 are of course Windows 7, 8, and 8.1 respectively. Notably absent is 9, Windows 10. So Windows 10 is treated differently than 7-8.1.
As I recall, we did need to edit the d1.fixed_size_partitions in order for Windows 10 to deploy correctly in the last build we used, so it is probable that this change was intentional. Unfortunately, changing it between builds breaks any image taken while Windows 10 was still handled like 7-8.1, and changing it back would break any images taken with the current version.

[Wayne Workman]

@cbc-tgschultz Can you doublecheck the main server’s free space please?

df -h

[cbc-tgschultz]

@Wayne-Workman

Currently sitting at 38% utilization. I don’t know why you’ve asked, I’ve already identified the source of the issue. The only thing I need to know now is if this is the intended behavior or not.

[Wayne Workman]

@cbc-tgschultz You only have one partition? What’s the utilization of the / partition? I ask because there was a bug in rc-5 that would spam an error to the point that it would fill the root partition, and if the root partition is filled, Linux becomes very ill and wild things happen.

[cbc-tgschultz]

Again, I’ve already worked out what the issue is. And I’ve worked around it, so I don’t know why you asked a question that seems completely unrelated.

There is more than one partition, but “/” is the only relevant one as it also hosts “/images/”. None of them are anywhere near full.

Filesystem Size Used Avail Use% Mounted on
udev 983M 0 983M 0% /dev
tmpfs 201M 21M 180M 11% /run
/dev/mapper/fogcleary–vg-root 1006G 354G 601G 38% /
tmpfs 1001M 0 1001M 0% /dev/shm
tmpfs 5.0M 0 5.0M 0% /run/lock
tmpfs 1001M 0 1001M 0% /sys/fs/cgroup
/dev/sda1 236M 126M 98M 57% /boot
cgmfs 100K 0 100K 0% /run/cgmanager/fs
tmpfs 201M 0 201M 0% /run/user/1000
/dev/loop0 57M 49M 4.7M 92% /mnt

Before you say anything, the loopback fs is the mounted “init” image. I have it mounted so I can examine the download and upload scripts.

All I really need to know at this point is this:

Is the behavior I described previously, where Windows 10 is treated differently by fog.download and fog.upload than Windows 7, 8, and 8.1, the intended behvior, or not? If it is intended, then we will recreate the affected images. If not, then we will need to recreate the other images.

[Wayne Workman]

@cbc-tgschultz said in 1.3.0-RC-7: Image store corrupt, unable to locate MBR:

Is the behavior I described previously, where Windows 10 is treated differently by fog.download and fog.upload than Windows 7, 8, and 8.1, the intended behvior, or not? If it is intended, then we will recreate the affected images. If not, then we will need to recreate the other images.

Pinging @Tom-Elliott

I do think Win10 is treated differently, I know that the registry keys that the Early Hostname Changer feature touches is different with Win10, but this is post-imaging manipulation.

Apologies for the drive space question, When I asked, I had myself just dealt with a full root partition, and found a few other people here in the forums with the exact same issue. As soon as I read the line “fog couldn’t write data to the DB” in your OP, I immediately asked about drive space because that is sometimes a tale-tale sign of a full root partition.

[Tom Elliott]

@Wayne-Workman Windows 10 is not treated any differently.

However, the init’s that appear to be in use are NOT the init’s that 1.3.0-RC (any) were published with.

@cbc-tgschultz I say that because of the:

@cbc-tgschultz said in 1.3.0-RC-7: Image store corrupt, unable to locate MBR:

It appears to be a partclone image and is listed as such in the GUI.

Here’s what I think has happened. There are lines in fog.download and fog.upload like this:

if [ “$osid” == “5” ] || [ “$osid” == “6” ] || [ “$osid” == “7” ]; then

The code under those lines seem to handle this kind of image. 5, 6, and 7 are of course Windows 7, 8, and 8.1 respectively. Notably absent is 9, Windows 10. So Windows 10 is treated differently than 7-8.1.
As I recall, we did need to edit the d1.fixed_size_partitions in order for Windows 10 to deploy correctly in the last build we used, so it is probable that this change was intentional. Unfortunately, changing it between builds breaks any image taken while Windows 10 was still handled like 7-8.1, and changing it back would break any images taken with the current version.

posting.

For a while now, the init’s have used a much cleaner mechanism to check the osid. Add to that, and as you noticed the osid 9 (which is our Windows 10 definition) isn’t even in the mix.

Here’s the code that, more or less, handles that code now: https://github.com/FOGProject/fogproject/blob/dev-branch/src/buildroot/package/fog/scripts/bin/fog.download

And:
https://github.com/FOGProject/fogproject/blob/dev-branch/src/buildroot/package/fog/scripts/bin/fog.upload

[Wayne Workman]

@Tom-Elliott And that’s why I pinged you.

[cbc-tgschultz]

@Wayne-Workman

Ah, so that’s what you were basing that question on. I was really just very confused about why you were on about disk space, now that makes sense.

@Tom-Elliott

Interesting. I am fairly sure it was working before the latest git pull, so I’m not sure where the init image I’m looking at came from. I will update the init image, but I do wonder now what else might not have taken since the update. Perhaps it is time to rebuild that server.

Thank you all for your help.