Full C:\ After Imaging Inconsistent

[imagingmaster21]

@Sebastian-Roth

Ok, we just updated to version 1.5.6.5 and we are still having the same issue as in this post. This happens for any image, I am going to include information for two of our most used images.

Here is the first image we use it is a Windows 10 Enterprise 1809:

d1.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=3366A936-7B65-43E1-AFDF-9DE4F8FE13A7, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=0DBFD8B3-44AC-4600-926D-FDFB469EFF2A, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=BC00D195-83B5-4B15-BFF8-DDCEC9B3ADFF, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   498855936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=890554EA-E1AB-4D7F-8C9D-7F8C83FA2A62, name="Basic data partition"

d1.minimum.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=3366A936-7B65-43E1-AFDF-9DE4F8FE13A7, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=0DBFD8B3-44AC-4600-926D-FDFB469EFF2A, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=BC00D195-83B5-4B15-BFF8-DDCEC9B3ADFF, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=    79997904, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=890554EA-E1AB-4D7F-8C9D-7F8C83FA2A62, name="Basic data partition"

d1.fixed_size_partitions

:2:3:1

The other image that we are primarily using that it is also happening on is a Windows 10 Professional 1809 build.

d1.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 250069646

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=13FC5702-27F9-49D6-8A69-F39933A23D1D, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=12D97EEC-C05D-4AB2-85FC-E279C488B16C, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7DEE0FD7-4583-417C-9FCE-58C0396BD356, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   248807936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=CB038B33-4E54-441D-80A5-6F8614224912, name="Basic data partition"

d1.minimum.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 250069646

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=13FC5702-27F9-49D6-8A69-F39933A23D1D, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=12D97EEC-C05D-4AB2-85FC-E279C488B16C, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7DEE0FD7-4583-417C-9FCE-58C0396BD356, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   248807936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=CB038B33-4E54-441D-80A5-6F8614224912, name="Basic data partition"

d1.fixed_size_partitions file

:2:3:1

Let me know if you need anything else with this. I figured I would try updating to the latest version before anything else. Everything seems to be imaging fine, except for this hit or miss issue.

[Sebastian Roth]

@imagingmaster21 From what you posted it sounds like this is still an issue with the NVMe disks that we have fixed in 1.5.6.5 already. Very strange.

When you deploy to the machines do you see any warning message or something like that when it tries to expand the partition? Could you schedule a debug deploy task and step through it while taking pictures of the screen when it does the resize. Maybe there is something in the output that might help us here.

As well please run the following command when you are on the debug shell and post a picture of the output here: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

The partition layouts look pretty much alright I reckon.

[imagingmaster21]

@Sebastian-Roth

I ran one debug task so far on a Lenovo T440p with a Samsung SSD and it imaged fine did not have the disk space problem. But I got a bunch of pictures during the whole image debug task. I am going to be imaging more machines today, so I will keep running debug tasks and hopefully I will get one that results in the drive size issue. This issue seems to be really hit or miss, sometimes you get one and sometimes you don’t. Out of 45 machines we did the one day (T460’s and T440’s), only 4 out of the 45 had the issue.

I put the pictures I took on a Shared Google Drive Folder: https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharing

Also I ran an apache tail while this was going on and here is the results:

[Thu Jul 11 07:35:06.234826 2019] [mpm_prefork:notice] [pid 1624] AH00163: Apache/2.4.33 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Jul 11 07:35:06.234842 2019] [core:notice] [pid 1624] AH00094: Command line: '/usr/sbin/apache2'
[Thu Jul 11 09:00:18.200339 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52466] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=24
[Thu Jul 11 09:01:08.039396 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52511] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31
[Thu Jul 11 09:01:23.484370 2019] [proxy_fcgi:error] [pid 10452] [client 172.16.9.35:52520] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31
[Thu Jul 11 09:22:26.559418 2019] [proxy_fcgi:error] [pid 17656] [client 172.16.9.35:52826] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=46

Let me know what the next steps should be, in the mean time I will do more debug imaging and hopefully get one that has the drive size issue

I was just looking under /usr/share and there is no fog folder? However I did do a search in terminal and found the procsfdisk.awk file is actually located under /root/fogproject/src/buildroot/package/fog/scripts/usr/share/fog/lib/procsfdisk.awk

So when I do the debug should I run the command with this path that I found rather than the one you posted? @Sebastian-Roth

[Sebastian Roth]

@imagingmaster21 Please run the following command when you are on the debug shell and post a picture of the output here: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

[imagingmaster21]

@Sebastian-Roth

I put that command in on the computer when the debug screen came up, I took pictures as it went along. I put all the pictures for the first computer I did on Google Drive:
https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharing

Here was the initial output right after I put the command in on the machine in debug mode:

This is the first time I did a debug on an image, so not sure if I’m doing this the correct way.

[imagingmaster21]

@Sebastian-Roth

I also noticed that we have been getting this error on machines, no consistent model and not always happening when a machine gets the full C:, maybe they are related…

https://drive.google.com/file/d/1w4ToCKSSng0gvdNpFWlZD-9J7yZqeUaG/view?usp=sharing

Also as a side note when we updated te init files a month ago we changed the KERNEL RAMDISK SIZE from 127000 to 275000. Would it be worth changing it back to 127000?

[Sebastian Roth]

@imagingmaster21 Unfortunately you ran the command too early before you got to the shell. Please hit ENTER a couple of times and then run: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

Decreasing the KERNEL RAMSIZE down to 127000 will lead to a kernel error on boot as the new init simply is bigger und need that space.

I will have a look at all the pictures tomorrow.

[imagingmaster21]

@Sebastian-Roth said in Full C:\ After Imaging Inconsistent:

grep " part_number " /usr/share/fog/lib/procsfdisk.awk

Ok, I’m not sure exactly what I am looking for to put this command in at the correct time? Is it before the Partclone starts or after? This is something I am not familiar with doing, I want to make sure I do this the right way so I get you the right info.

[Tom Elliott]

@imagingmaster21 this is at the command prompt. When you boot into a debug session, before you type this command you’ll be asked to press enter two times. Then you will have a command prompt like

root@fogclient ~] #

Here is where you enter the grep command

[imagingmaster21]

@Tom-Elliott @Sebastian-Roth

Ok, I did this on a T460 and here is what I got:

Let me know if I did this command correct?

One thing I have been noticing is it set the hard drive size to the image size on FOG. So when I image with my Enterprise image on a machine that has the issue the disk size is always 38gb which the size of the image is 37.84. Like I said before this issue is very hit or miss, does not happen to any specific model or harddrive model.

[Sebastian Roth]

@imagingmaster21 Yeah you got it right this time and it shows that you are indeed using the patched version that shouldn’t have the NVMe bug anymore. Just wanted to make sure so we don’t run down the wrong rabbit hole.

I will look into the details you posted over the weekend and try to replicate the issue. Will let you know!

[Quazz]

Issue is likely similar to if not the same as

https://forums.fogproject.org/topic/13440/fog-1-5-6-auto-resize-is-unpredictable

[imagingmaster21]

@Quazz @Sebastian-Roth I actually ended up reading that post this morning. We do have two FOG servers on the same version. Our main one is stationed at our office which is the one I’ve been getting the info off of.

The other one is our mobile version (we like to call it that) simply installed on a laptop, it does have the same issue however it is on version 1.5.6.4, whereas our main one is on 1.5.6.5. I will run a debug task on our mobile one also while it is on 1.5.6.4 sometime this afternoon hopefully.

[imagingmaster21]

@Tom-Elliott @Sebastian-Roth @Quazz

I tried imaging a machine on our FOG Mobile server that has 1.5.6.4 that has the same hit or miss issue with incorrect partition/full C:. Here is the debug:

[Sebastian Roth]

@imagingmaster21 Yes @Quazz is right, really sounds like a similar issue to the one linked. Pretty sure the issue is being found and we’ll fix it in the next days. Will let you know here!

[Sebastian Roth]

@imagingmaster21 New inits are ready for testing. Download 64 Bit and 32 Bit and put those in /var/www/html/fog/service/ipxe/ dir on your FOG server. Probably good to rename the original ones instead of overwriting.

As well you need to check the RAM SIZE option in FOG configuration, FOg settings. Need to be 275000 with the newer inits.

[imagingmaster21]

@Sebastian-Roth @Tom-Elliott
Ok I did that and rebooted the FOG server and got this error message on a T440 and T490:

Would it be worth updating to 1.5.7?

[Sebastian Roth]

@imagingmaster21 My fault!! Sorry. Will fix that in the next two hours and let you know.

[Sebastian Roth]

@imagingmaster21 Alright, should be fixed now. Just re-download the inits from the same links as before and give it a try. Please let us know how things go.

[imagingmaster21]

@Sebastian-Roth
That error is fixed. We are imaging a large batch of computers now, I will let you know how the disk space is on them after we image and check them.