Full C:\ After Imaging Inconsistent

imagingmaster21 · Jul 9, 2019, 6:35 PM

So I have been having an odd issue lately of computers being imaged and their available C:\ space is 40gb after imaging and there is only 2% available. It can be fixed by doing and expand on C:\ under disk management. At first this only happened to two Lenovo T460 laptops with Intel SSD-SC2KF256H6L, and we imaged many other T460’s all with Samsung SSD’s in them with no issue. Now I just had it happen on one T460 and one T440. The T460 has a Samsung SSD MZ7LF192HCGS-000L1 and the T440 has a Toshiba MQ01ACF05Q (not a SSD).

I am able to fix the issue y right clicking on C:\ in Disk Management and expand it and the problem is resolved. But I am not sure on what would be the cause with this? Here are some screenshots…

https://drive.google.com/file/d/1Gb0ydZJNuicgRmMxhfN99VvUrXcESuJz/view?usp=sharing

https://drive.google.com/file/d/1T8HT4nS04HskXrZkUoyH_8NkiDkHP81s/view?usp=sharing

https://drive.google.com/file/d/1kR_xkBJ_G6mZxpXJ69UwyTOrSJLOegOd/view?usp=sharing

We are running 1.5.2, would updating FOG fix this? About a month ago we had a similar issue and it appeared to get fixed by updating the init files. We just started having this new issue a few days ago. It only has happened on a total of four computers out of 100 imaged. So its not wide spread at least.

Here is the link to the previous issue that we posted about…
https://forums.fogproject.org/topic/13336/partition-resizing/4

Let me know what the best idea is to resolve this.

Sebastian Roth · Jul 9, 2019, 7:23 PM

@imagingmaster21 Which image did you deploy to those machines. Best if you can provide the content of d1.partitions, d1.minimum.partitions and d1.fixed_size_partitions again. Plus it would be great if you could upload d1.mbr to a file share and post a link here.

imagingmaster21 · Jul 10, 2019, 6:11 PM

@Sebastian-Roth

Ok, we just updated to version 1.5.6.5 and we are still having the same issue as in this post. This happens for any image, I am going to include information for two of our most used images.

Here is the first image we use it is a Windows 10 Enterprise 1809:

d1.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=3366A936-7B65-43E1-AFDF-9DE4F8FE13A7, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=0DBFD8B3-44AC-4600-926D-FDFB469EFF2A, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=BC00D195-83B5-4B15-BFF8-DDCEC9B3ADFF, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   498855936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=890554EA-E1AB-4D7F-8C9D-7F8C83FA2A62, name="Basic data partition"

d1.minimum.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 500118158

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=3366A936-7B65-43E1-AFDF-9DE4F8FE13A7, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=0DBFD8B3-44AC-4600-926D-FDFB469EFF2A, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=BC00D195-83B5-4B15-BFF8-DDCEC9B3ADFF, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=    79997904, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=890554EA-E1AB-4D7F-8C9D-7F8C83FA2A62, name="Basic data partition"

d1.fixed_size_partitions

:2:3:1

The other image that we are primarily using that it is also happening on is a Windows 10 Professional 1809 build.

d1.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 250069646

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=13FC5702-27F9-49D6-8A69-F39933A23D1D, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=12D97EEC-C05D-4AB2-85FC-E279C488B16C, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7DEE0FD7-4583-417C-9FCE-58C0396BD356, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   248807936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=CB038B33-4E54-441D-80A5-6F8614224912, name="Basic data partition"

d1.minimum.partitions file

label: gpt
label-id: 066202D1-4FDA-44C2-A0A0-3C26A816B87E
device: /dev/nvme0n1
unit: sectors
first-lba: 34
last-lba: 250069646

/dev/nvme0n1p1 : start=        2048, size=     1021952, type=DE94BBA4-06D1-4D40-A16A-BFD50179D6AC, uuid=13FC5702-27F9-49D6-8A69-F39933A23D1D, name="Basic data partition", attrs="RequiredPartition GUID:63"
/dev/nvme0n1p2 : start=     1024000, size=      204800, type=C12A7328-F81F-11D2-BA4B-00A0C93EC93B, uuid=12D97EEC-C05D-4AB2-85FC-E279C488B16C, name="EFI system partition", attrs="GUID:63"
/dev/nvme0n1p3 : start=     1228800, size=       32768, type=E3C9E316-0B5C-4DB8-817D-F92DF00215AE, uuid=7DEE0FD7-4583-417C-9FCE-58C0396BD356, name="Microsoft reserved partition", attrs="GUID:63"
/dev/nvme0n1p4 : start=     1261568, size=   248807936, type=EBD0A0A2-B9E5-4433-87C0-68B6B72699C7, uuid=CB038B33-4E54-441D-80A5-6F8614224912, name="Basic data partition"

d1.fixed_size_partitions file

:2:3:1

Let me know if you need anything else with this. I figured I would try updating to the latest version before anything else. Everything seems to be imaging fine, except for this hit or miss issue.

Sebastian Roth · Jul 10, 2019, 9:52 PM

@imagingmaster21 From what you posted it sounds like this is still an issue with the NVMe disks that we have fixed in 1.5.6.5 already. Very strange.

When you deploy to the machines do you see any warning message or something like that when it tries to expand the partition? Could you schedule a debug deploy task and step through it while taking pictures of the screen when it does the resize. Maybe there is something in the output that might help us here.

As well please run the following command when you are on the debug shell and post a picture of the output here: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

The partition layouts look pretty much alright I reckon.

imagingmaster21 · Jul 11, 2019, 1:44 PM

@Sebastian-Roth

I ran one debug task so far on a Lenovo T440p with a Samsung SSD and it imaged fine did not have the disk space problem. But I got a bunch of pictures during the whole image debug task. I am going to be imaging more machines today, so I will keep running debug tasks and hopefully I will get one that results in the drive size issue. This issue seems to be really hit or miss, sometimes you get one and sometimes you don’t. Out of 45 machines we did the one day (T460’s and T440’s), only 4 out of the 45 had the issue.

I put the pictures I took on a Shared Google Drive Folder: https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharing

Also I ran an apache tail while this was going on and here is the results:

[Thu Jul 11 07:35:06.234826 2019] [mpm_prefork:notice] [pid 1624] AH00163: Apache/2.4.33 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations
[Thu Jul 11 07:35:06.234842 2019] [core:notice] [pid 1624] AH00094: Command line: '/usr/sbin/apache2'
[Thu Jul 11 09:00:18.200339 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52466] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=24
[Thu Jul 11 09:01:08.039396 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52511] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31
[Thu Jul 11 09:01:23.484370 2019] [proxy_fcgi:error] [pid 10452] [client 172.16.9.35:52520] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31
[Thu Jul 11 09:22:26.559418 2019] [proxy_fcgi:error] [pid 17656] [client 172.16.9.35:52826] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=46

Let me know what the next steps should be, in the mean time I will do more debug imaging and hopefully get one that has the drive size issue

I was just looking under /usr/share and there is no fog folder? However I did do a search in terminal and found the procsfdisk.awk file is actually located under /root/fogproject/src/buildroot/package/fog/scripts/usr/share/fog/lib/procsfdisk.awk

So when I do the debug should I run the command with this path that I found rather than the one you posted? @Sebastian-Roth

Sebastian Roth · Jul 11, 2019, 3:20 PM

@imagingmaster21 Please run the following command when you are on the debug shell and post a picture of the output here: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

imagingmaster21 · Jul 11, 2019, 9:34 AM

@Sebastian-Roth

I put that command in on the computer when the debug screen came up, I took pictures as it went along. I put all the pictures for the first computer I did on Google Drive:
https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharing

Here was the initial output right after I put the command in on the machine in debug mode:

This is the first time I did a debug on an image, so not sure if I’m doing this the correct way.

imagingmaster21 · Jul 11, 2019, 5:58 PM

@Sebastian-Roth

I also noticed that we have been getting this error on machines, no consistent model and not always happening when a machine gets the full C:, maybe they are related…

https://drive.google.com/file/d/1w4ToCKSSng0gvdNpFWlZD-9J7yZqeUaG/view?usp=sharing

Also as a side note when we updated te init files a month ago we changed the KERNEL RAMDISK SIZE from 127000 to 275000. Would it be worth changing it back to 127000?

Sebastian Roth · Jul 11, 2019, 9:17 PM

@imagingmaster21 Unfortunately you ran the command too early before you got to the shell. Please hit ENTER a couple of times and then run: grep " part_number " /usr/share/fog/lib/procsfdisk.awk

Decreasing the KERNEL RAMSIZE down to 127000 will lead to a kernel error on boot as the new init simply is bigger und need that space.

I will have a look at all the pictures tomorrow.

imagingmaster21 · Jul 11, 2019, 10:28 PM

@Sebastian-Roth said in Full C:\ After Imaging Inconsistent:

grep " part_number " /usr/share/fog/lib/procsfdisk.awk

Ok, I’m not sure exactly what I am looking for to put this command in at the correct time? Is it before the Partclone starts or after? This is something I am not familiar with doing, I want to make sure I do this the right way so I get you the right info.

Tom Elliott · Jul 12, 2019, 1:17 PM

@imagingmaster21 this is at the command prompt. When you boot into a debug session, before you type this command you’ll be asked to press enter two times. Then you will have a command prompt like

root@fogclient ~] #

Here is where you enter the grep command

imagingmaster21 · Jul 12, 2019, 7:36 AM

@Tom-Elliott @Sebastian-Roth

Ok, I did this on a T460 and here is what I got:

Let me know if I did this command correct?

One thing I have been noticing is it set the hard drive size to the image size on FOG. So when I image with my Enterprise image on a machine that has the issue the disk size is always 38gb which the size of the image is 37.84. Like I said before this issue is very hit or miss, does not happen to any specific model or harddrive model.

Sebastian Roth · Jul 12, 2019, 2:42 PM

@imagingmaster21 Yeah you got it right this time and it shows that you are indeed using the patched version that shouldn’t have the NVMe bug anymore. Just wanted to make sure so we don’t run down the wrong rabbit hole.

I will look into the details you posted over the weekend and try to replicate the issue. Will let you know!

Quazz · Jul 12, 2019, 2:46 PM

Issue is likely similar to if not the same as

https://forums.fogproject.org/topic/13440/fog-1-5-6-auto-resize-is-unpredictable

imagingmaster21 · Jul 12, 2019, 3:04 PM

@Quazz @Sebastian-Roth I actually ended up reading that post this morning. We do have two FOG servers on the same version. Our main one is stationed at our office which is the one I’ve been getting the info off of.

The other one is our mobile version (we like to call it that) simply installed on a laptop, it does have the same issue however it is on version 1.5.6.4, whereas our main one is on 1.5.6.5. I will run a debug task on our mobile one also while it is on 1.5.6.4 sometime this afternoon hopefully.

imagingmaster21 · Jul 12, 2019, 6:43 PM

@Tom-Elliott @Sebastian-Roth @Quazz

I tried imaging a machine on our FOG Mobile server that has 1.5.6.4 that has the same hit or miss issue with incorrect partition/full C:. Here is the debug:

Sebastian Roth · Jul 13, 2019, 7:12 AM

@imagingmaster21 Yes @Quazz is right, really sounds like a similar issue to the one linked. Pretty sure the issue is being found and we’ll fix it in the next days. Will let you know here!

Sebastian Roth · Jul 15, 2019, 1:48 PM

@imagingmaster21 New inits are ready for testing. Download 64 Bit and 32 Bit and put those in /var/www/html/fog/service/ipxe/ dir on your FOG server. Probably good to rename the original ones instead of overwriting.

As well you need to check the RAM SIZE option in FOG configuration, FOg settings. Need to be 275000 with the newer inits.

imagingmaster21 · Jul 15, 2019, 8:59 AM

@Sebastian-Roth @Tom-Elliott
Ok I did that and rebooted the FOG server and got this error message on a T440 and T490:

Would it be worth updating to 1.5.7?

Sebastian Roth · Jul 15, 2019, 4:21 PM

@imagingmaster21 My fault!! Sorry. Will fix that in the next two hours and let you know.

Full C:\ After Imaging Inconsistent

186

12.0k

17.3k

155.2k