deploy issue after copying image to an other FOG server

boombasstic · Mar 2, 2021, 5:05 PM

Hi,

Im using FOG for a bit more than a year, and it works great. my FOG server is using FOG 1.5.7 on CentOS 7. I had to create a new server, but decided to try it on Debian. Installation was smooth and i installed the latest version 1.5.9, and copied my Windows images from one server to the other.
However when i deploy my images, after reboot, the Windows OS seem to be “corrupted”. Windows starts but explorer crashes and windows is unusable.
i tried different things, but with same result.
I tried deploying images on different devices, they all act the same. Though deploy with my old server works fine with same image.
Images capture from Debian server and deployed from it are fine, but i would like to use my images from CentOS server and not re-create 50 images.
Is there a reason for same image to create such result on deploy with different version of FOG (1.5.7 / 1.5.9) running on different OSes (CentOS7 / Debian 10).

Thanks for your help.

Sebastian Roth · Mar 2, 2021, 9:16 PM

@boombasstic I think you run into an issue we fixed after the release of 1.5.9, see https://github.com/FOGProject/fos/commit/8d310f51d7d945c5dc1685eb6698d1aed8634dc7

I suggest you just grab the latest inits from our server and use those until we officially release the next version of FOG.

https://fogproject.org/inits/init.xz
https://fogproject.org/inits/init_32.xz

Put those into /var/www/html/fog/service/ipxe/ on your FOG server, renaming the original files.

boombasstic · Mar 3, 2021, 11:26 AM

@sebastian-roth Thanks for your answer.

I changed the inits, but i still have the same issue.
I wonder if it could come from a resize problem during deploy. And on boot there is no available space on the system drive, so windows crashes?

i noticed that with the images captured from and wich work on my 1.5.7 server, the d1.fixed_size_partitions is :1:2:5:1:2:5
On the image captured with 1.5.9 server it is 1:2

Sebastian Roth · Mar 3, 2021, 3:57 PM

@boombasstic Hmmm, ok. Seems like I was on the wrong track with the partclone issue. Now that I think about that issue again it might have been the deployed Windows did not even boot.

Good you mention the d1.fixed_size_partitions file. Can you please post the contents of d1.partitions and d1.minimum.partitions here as well so we know what your partition layout looks like.

boombasstic · Mar 3, 2021, 8:33 PM

if it helps, i found another post where they were having the same exact issue i’m having

https://forums.fogproject.org/topic/14735/odd-issue-with-win-10-uefi-images-after-updating-from-1-5-9-rc2-15-to-1-5-9-rc2-17?sort=oldest_to_newest

Windows boots, but no icon on desktop, and not possible to do anything even load task manager.
But the solution or re capturing (which works) is time consuming if i have to re deploy all my images with 1.5.7 and recapture them with 1.5.9.

here is the content of the images captured with 1.5.7 which “crashes” after deploy from 1.5.9

d1.partitions

label: gpt
label-id: 7D3BE2EA-722A-43A9-AEA0-1F36F17095D8
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=      532480, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1686215-19B9-4424-AE16-3405C9F3DEE1, name="EFI system partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      534528, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=935DDB3D-1186-4C7F-A7D0-903A2F7831CE, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      567296, size=   496336896, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=A1E84F32-54EE-481A-A40E-C8A6AE68AD64, name="Basic data partition"
/dev/nvme0n1p4 : start=   496904192, size=     1165312, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=5D48B483-9C7A-4094-BDA1-AD3DC4C1070A, name="attrs=\x22RequiredPartition GUID:63"
/dev/nvme0n1p5 : start=   498069504, size=     2048000, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=02B11D85-B439-4E37-9004-691A53DCD24F, name="Basic data partition", attrs="RequiredPartition GUID:63"

d1.minimum.partitions

label: gpt
label-id: 7D3BE2EA-722A-43A9-AEA0-1F36F17095D8
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=      532480, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=D1686215-19B9-4424-AE16-3405C9F3DEE1, name="EFI system partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=      534528, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=935DDB3D-1186-4C7F-A7D0-903A2F7831CE, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=      567296, size=    68387676, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=A1E84F32-54EE-481A-A40E-C8A6AE68AD64, name="Basic data partition"
/dev/nvme0n1p4 : start=   496904192, size=     1031462, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=5D48B483-9C7A-4094-BDA1-AD3DC4C1070A, name="attrs=\x22RequiredPartition GUID:63"
/dev/nvme0n1p5 : start=   498069504, size=     2048000, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=02B11D85-B439-4E37-9004-691A53DCD24F, name="Basic data partition", attrs="RequiredPartition GUID:63"

d1.fixed_size_partitions

:1:2:5:1:2:5

Sebastian Roth · Mar 4, 2021, 7:39 AM

@boombasstic Please do us a favour and deploy this image with 1.5.9 again but do it in debug mode this time. When you boot the host up you get to a command shell. Enter fog there to Start the deployment and step through it. If you notice a message between the normal output take a picture.

Then at the end you get back to the command shell. Enter sfdisk -d /dev/nvme0n1 and take a picture of the full output on screen. Post picture (s) here.

boombasstic · Mar 4, 2021, 9:52 AM

@sebastian-roth when i type the fog command in the shell i get the following error:

george1421 · Mar 5, 2021, 2:53 PM

@boombasstic The error would imply you entered debug mode from the advanced options. What Sebastian meant was for you to deploy the image as normally, but before you hit the schedule task button, tick the debug checkbox then submit the task. That will put the target computer in debug deploy mode to allow you to read errors or stop the imaging process with a ctrl-c.

boombasstic · Mar 5, 2021, 2:53 PM

@george1421 Thanks for the help
@Sebastian-Roth

i did as requested, please find below the pictures

Sebastian Roth · Mar 8, 2021, 1:04 AM

@boombasstic When updating the inits today I just noticed that I was on the wrong track. So far the inits on the webserver were still the ones not having the fix. I am sorry, didn’t remember I actually uploaded the fixed one separately: https://fogproject.org/inits/init-1.5.9-ignore_crc-fix.xz

As well you can re-download the current official inits that have the fix in them as well - as of now.

https://fogproject.org/inits/init.xz
https://fogproject.org/inits/init_32.xz

boombasstic · Mar 8, 2021, 1:25 PM

@sebastian-roth said in deploy issue after copying image to an other FOG server:

https://fogproject.org/inits/init.xz
https://fogproject.org/inits/init_32.xz

We replaced the inits with those, and we still having the issue.

Sebastian Roth · Mar 8, 2021, 4:26 PM

@boombasstic When you do a debug deploy what Init Version does it show? See the first picture you posted last time so you know what I mean.

boombasstic · Mar 9, 2021, 9:42 AM

@sebastian-roth said in deploy issue after copying image to an other FOG server:

@boombasstic When updating the inits today I just noticed that I was on the wrong track. So far the inits on the webserver were still the ones not having the fix. I am sorry, didn’t remember I actually uploaded the fixed one separately: https://fogproject.org/inits/init-1.5.9-ignore_crc-fix.xz

As well you can re-download the current official inits that have the fix in them as well - as of now.

https://fogproject.org/inits/init.xz
https://fogproject.org/inits/init_32.xz

With the init-1.5.9-ignore_crc-fix.xz init Version was: 2020927

With the current official inits , the Version was : 20200906

Wit both of the the problem still occurs.

boombasstic · Mar 9, 2021, 5:02 AM

So i ran some tests following my intuition that the “windows not loading” symptom came from a lack of disk space, and fog not resizing correctly the Data partition of Windows.
Among other things the result of the sfdisk -d /dev/nvme0n1 led me to think about this.

It show 68387676 blocks size for the C : system partition. which is the equivalent of the same partition size in the d1.minimum_partitions file.
its about 32GB which is the Used size on the partition.

I booted with a Live Gparted, and expanded the partition, then the computer boots normally.

So i think the issue lies with the resizing, not with CRC check.

Sebastian Roth · Mar 9, 2021, 7:11 AM

@boombasstic said:

I booted with a Live Gparted, and expanded the partition, then the computer boots normally.
So i think the issue lies with the resizing, not with CRC check.

Ok that is a valid point. So we would need to find out why it does not expand on deploy. Do you see any errors or wanrings. Maybe do another debug deploy and take a close look at the point where it says “Filling the disk”.

With the current official inits , the Version was : 20200906

That still looks like you are not using the very latest inits I uploaded two days ago! Did you re-download the files after I posted this? I can’t verify right now but you might run the following command to get us the SHA1 sums (checking both locations just in case there is something wrong with the link):

sha1sum /var/www/html/fog/service/ipxe/init*
sha1sum /var/www/fog/service/ipxe/init*

boombasstic · Mar 10, 2021, 3:40 PM

@sebastian-roth
there is no “filling disk” step in debug deploy task.
see picture below

i copied the latest inits once again and the Version show is still 20200906

here are the SHA1 sums

0d60fc165643a57c10120e0892f2a1e520b53809  /var/www/html/fog/service/ipxe/init_32.xz
bea70a25cd1f0ddfd15bd1124acafaed79408d1e  /var/www/html/fog/service/ipxe/init.xz
0d60fc165643a57c10120e0892f2a1e520b53809  /var/www/fog/service/ipxe/init_32.xz
bea70a25cd1f0ddfd15bd1124acafaed79408d1e  /var/www/fog/service/ipxe/init.xz

Sebastian Roth · Mar 10, 2021, 9:55 AM

@boombasstic said in deploy issue after copying image to an other FOG server:

there is no “filling disk” step in debug deploy task.

That would be before the blue partclone screens. I just had a look at the code to see what message it should print at this stage. Should be “Attempting to expand/fill partitions…”.

To see even more information you can set ismajordebug=1 in Host Kernel Arguments for this host you want to deploy to. It should print the partition table after it’s expanded to your disk.

Something is going wrong with the init download on your side I think. I just did the following and get different hash sums:

shell> wget https://fogproject.org/inits/init.xz
shell> wget https://fogproject.org/inits/init_32.xz
shell> sha1sum init*
e7a19cd587bd1d9f3a989308e66a421afcabe813  init_32.xz
350907ec31886fa2ca4f07ba203904d0c8efaf17  init.xz

boombasstic · Mar 11, 2021, 4:00 PM

getting the inits with wget the SHA1 sum are correct. maybe it was an issue with cahced files.

The version now shows 20210307

it still does not work

here is the filling disk step

the data partition after deploy still is
68387676 blocks while it should be 496336896

Sebastian Roth · Mar 11, 2021, 4:09 PM

@boombasstic To see even more information on the expand/fill operation set ismajordebug=1 in Host Kernel Arguments for this host you want to deploy to and schedule a debug deploy. Take a picture of the output on screen again.

boombasstic · Mar 12, 2021, 4:13 PM

@sebastian-roth here are the picture of the debug deploy task with kernel arguments.
It shows an error message "sfdisk failed in (applySfdiskPartitions)

deploy issue after copying image to an other FOG server

181

12.0k

17.3k

155.2k