Full C:\ After Imaging Inconsistent
-
So I have been having an odd issue lately of computers being imaged and their available C:\ space is 40gb after imaging and there is only 2% available. It can be fixed by doing and expand on C:\ under disk management. At first this only happened to two Lenovo T460 laptops with Intel SSD-SC2KF256H6L, and we imaged many other T460’s all with Samsung SSD’s in them with no issue. Now I just had it happen on one T460 and one T440. The T460 has a Samsung SSD MZ7LF192HCGS-000L1 and the T440 has a Toshiba MQ01ACF05Q (not a SSD).
I am able to fix the issue y right clicking on C:\ in Disk Management and expand it and the problem is resolved. But I am not sure on what would be the cause with this? Here are some screenshots…
https://drive.google.com/file/d/1Gb0ydZJNuicgRmMxhfN99VvUrXcESuJz/view?usp=sharing
https://drive.google.com/file/d/1T8HT4nS04HskXrZkUoyH_8NkiDkHP81s/view?usp=sharing
https://drive.google.com/file/d/1kR_xkBJ_G6mZxpXJ69UwyTOrSJLOegOd/view?usp=sharing
We are running 1.5.2, would updating FOG fix this? About a month ago we had a similar issue and it appeared to get fixed by updating the init files. We just started having this new issue a few days ago. It only has happened on a total of four computers out of 100 imaged. So its not wide spread at least.
Here is the link to the previous issue that we posted about…
https://forums.fogproject.org/topic/13336/partition-resizing/4Let me know what the best idea is to resolve this.
-
@Sebastian-Roth
That error is fixed. We are imaging a large batch of computers now, I will let you know how the disk space is on them after we image and check them.
-
@imagingmaster21 Alright, should be fixed now. Just re-download the inits from the same links as before and give it a try. Please let us know how things go.
-
@imagingmaster21 My fault!! Sorry. Will fix that in the next two hours and let you know.
-
@Sebastian-Roth @Tom-Elliott
Ok I did that and rebooted the FOG server and got this error message on a T440 and T490:
Would it be worth updating to 1.5.7?
-
@imagingmaster21 New inits are ready for testing. Download 64 Bit and 32 Bit and put those in
/var/www/html/fog/service/ipxe/dir on your FOG server. Probably good to rename the original ones instead of overwriting.As well you need to check the RAM SIZE option in FOG configuration, FOg settings. Need to be 275000 with the newer inits.
-
@imagingmaster21 Yes @Quazz is right, really sounds like a similar issue to the one linked. Pretty sure the issue is being found and we’ll fix it in the next days. Will let you know here!
-
@Tom-Elliott @Sebastian-Roth @Quazz
I tried imaging a machine on our FOG Mobile server that has 1.5.6.4 that has the same hit or miss issue with incorrect partition/full C:. Here is the debug:
-
@Quazz @Sebastian-Roth I actually ended up reading that post this morning. We do have two FOG servers on the same version. Our main one is stationed at our office which is the one I’ve been getting the info off of.
The other one is our mobile version (we like to call it that) simply installed on a laptop, it does have the same issue however it is on version 1.5.6.4, whereas our main one is on 1.5.6.5. I will run a debug task on our mobile one also while it is on 1.5.6.4 sometime this afternoon hopefully.
-
Issue is likely similar to if not the same as
https://forums.fogproject.org/topic/13440/fog-1-5-6-auto-resize-is-unpredictable
-
@imagingmaster21 Yeah you got it right this time and it shows that you are indeed using the patched version that shouldn’t have the NVMe bug anymore. Just wanted to make sure so we don’t run down the wrong rabbit hole.
I will look into the details you posted over the weekend and try to replicate the issue. Will let you know!
-
Ok, I did this on a T460 and here is what I got:
Let me know if I did this command correct?
One thing I have been noticing is it set the hard drive size to the image size on FOG. So when I image with my Enterprise image on a machine that has the issue the disk size is always 38gb which the size of the image is 37.84. Like I said before this issue is very hit or miss, does not happen to any specific model or harddrive model.
-
@imagingmaster21 this is at the command prompt. When you boot into a debug session, before you type this command you’ll be asked to press enter two times. Then you will have a command prompt like
root@fogclient ~] #
Here is where you enter the grep command
-
@Sebastian-Roth said in Full C:\ After Imaging Inconsistent:
grep " part_number " /usr/share/fog/lib/procsfdisk.awk
Ok, I’m not sure exactly what I am looking for to put this command in at the correct time? Is it before the Partclone starts or after? This is something I am not familiar with doing, I want to make sure I do this the right way so I get you the right info.
-
@imagingmaster21 Unfortunately you ran the command too early before you got to the shell. Please hit ENTER a couple of times and then run:
grep " part_number " /usr/share/fog/lib/procsfdisk.awkDecreasing the KERNEL RAMSIZE down to 127000 will lead to a kernel error on boot as the new init simply is bigger und need that space.
I will have a look at all the pictures tomorrow.
-
I also noticed that we have been getting this error on machines, no consistent model and not always happening when a machine gets the full C:, maybe they are related…
https://drive.google.com/file/d/1w4ToCKSSng0gvdNpFWlZD-9J7yZqeUaG/view?usp=sharing
Also as a side note when we updated te init files a month ago we changed the KERNEL RAMDISK SIZE from 127000 to 275000. Would it be worth changing it back to 127000?
-
I put that command in on the computer when the debug screen came up, I took pictures as it went along. I put all the pictures for the first computer I did on Google Drive:
https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharingHere was the initial output right after I put the command in on the machine in debug mode:
This is the first time I did a debug on an image, so not sure if I’m doing this the correct way.
-
@imagingmaster21 Please run the following command when you are on the debug shell and post a picture of the output here:
grep " part_number " /usr/share/fog/lib/procsfdisk.awk
-
I ran one debug task so far on a Lenovo T440p with a Samsung SSD and it imaged fine did not have the disk space problem. But I got a bunch of pictures during the whole image debug task. I am going to be imaging more machines today, so I will keep running debug tasks and hopefully I will get one that results in the drive size issue. This issue seems to be really hit or miss, sometimes you get one and sometimes you don’t. Out of 45 machines we did the one day (T460’s and T440’s), only 4 out of the 45 had the issue.
I put the pictures I took on a Shared Google Drive Folder: https://drive.google.com/drive/folders/1EyxMz7RF7LzVEDYPygshJKT_qRxN-jge?usp=sharing
Also I ran an apache tail while this was going on and here is the results:
[Thu Jul 11 07:35:06.234826 2019] [mpm_prefork:notice] [pid 1624] AH00163: Apache/2.4.33 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations [Thu Jul 11 07:35:06.234842 2019] [core:notice] [pid 1624] AH00094: Command line: '/usr/sbin/apache2' [Thu Jul 11 09:00:18.200339 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52466] AH01071: Got error 'PHP message: PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=24 [Thu Jul 11 09:01:08.039396 2019] [proxy_fcgi:error] [pid 10448] [client 172.16.9.35:52511] AH01071: Got error 'PHP message: PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31 [Thu Jul 11 09:01:23.484370 2019] [proxy_fcgi:error] [pid 10452] [client 172.16.9.35:52520] AH01071: Got error 'PHP message: PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=31 [Thu Jul 11 09:22:26.559418 2019] [proxy_fcgi:error] [pid 17656] [client 172.16.9.35:52826] AH01071: Got error 'PHP message: PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', referer: http://foggy.adelphoi.org/fog/management/index.php?node=host&sub=edit&id=46Let me know what the next steps should be, in the mean time I will do more debug imaging and hopefully get one that has the drive size issue :)
I was just looking under /usr/share and there is no fog folder? However I did do a search in terminal and found the procsfdisk.awk file is actually located under /root/fogproject/src/buildroot/package/fog/scripts/usr/share/fog/lib/procsfdisk.awk
So when I do the debug should I run the command with this path that I found rather than the one you posted? @Sebastian-Roth
-
@imagingmaster21 From what you posted it sounds like this is still an issue with the NVMe disks that we have fixed in 1.5.6.5 already. Very strange.
When you deploy to the machines do you see any warning message or something like that when it tries to expand the partition? Could you schedule a debug deploy task and step through it while taking pictures of the screen when it does the resize. Maybe there is something in the output that might help us here.
As well please run the following command when you are on the debug shell and post a picture of the output here:
grep " part_number " /usr/share/fog/lib/procsfdisk.awkThe partition layouts look pretty much alright I reckon.
