Failure to expand shrunken resizeable image from Linux machines



  • Server
    • FOG Version: 1.4.4
    • OS: CentOS 7 X64
    Client
    • Service Version: ?
    • OS: Linux Mint 18.1
    Description

    I’ve been trying to capture an image of this system (it has about 5~gb of used space on a 50gb drive) and then cast it back to two more systems, one with a 30gb drive, one with a 100gb drive.

    The image I captured is marked resizeable, and during the capture the drive is successfully shrunk to about the same 5gb of space thats on it, however when its ‘cast’ back onto any system, the drive is not expanded, it stays shrunk leaving massive amounts of free space.

    Even if I edit the “fixed size partitions” to tell it that the partition in question shouldn’t be a fixed size, it won’t realize it.

    So far every linux capture/cast I’ve tried has worked this same way, none of them ‘re-expand’ to fill the ‘free space’, I’ve tried with and without using LVM on the machines I’m capturing from, etc.

    I’ve got no idea what I’m doing wrong here… I’m going to venture that this is some sort of bug?


  • Developer

    Thanks to @Tom-Elliott again for coming up with a fix so quickly! I am marking this solved then. But I still think having a testcase suite would be really good as we could make changes to the awk code, run the testcases and just be sure to not have it broken for other known cases. I think we could collect data from the community fairly easy as the information is already there (in d1.partitions that people have lying around on their FOG servers by the dozens)…



  • @george1421 For my needs, I’m totally dropping LVM and agree, LVM really doesn’t seem to have a place in images at the moment.


  • Moderator

    Just thinking about this. If it was me and I planned on deploying Linux client OS to target machines, I would not choose to use LVM (with expanding partitions or not). The issues is that FOG / partclone can only capture / deploy the image as raw. This is a slowest way of deploying the image. I would use native partitions on the disk.

    Right now deploying that captured LM partition will take about 28 minutes to finish. The native partitions are much faster.


  • Moderator

    @tom-elliott said in Failure to expand shrunken resizeable image from Linux machines:

    I’ve tried for all I’m worth, but I’m only one man, sorry.

    Tom, to be honest, as I posted before about 90% of the deployments are MS Windows, with about 8% Mac and the rest other bits (my feels like numbers). With MS doing its Win10 crap hopefully more will start to migrate to Linux based desktops. When issues arise you jump on them with both feet. The key is knowing about the issues to fix.


  • Senior Developer

    @george1421 There’s already functions in place for the hopeful integration of LVM extending through fog (as well as imaging) though I’ve not made any major effort in actually getting this working. You’re more than welcome to have a go at it. The same resizing functions should work similarly. Would you be willing to re-update your init’s to the absolute latest using the same wget commands and seeing if things are a bit better for you?


  • Moderator

    @xipher said in Failure to expand shrunken resizeable image from Linux machines:

    @george1421 Close to my findings wherein the first partition (sda1) will grow proportionally, but the rest… no go.

    If you create the same layout but have the root partition where you currently have the boot partition, it will grow the root and ‘work’ as it seems it would be intended.

    If you were to make a home partition in the same place, it would grow but not the boot or root… etc

    IMO In the case of Mint with the lvm. I would consider the /boot and [swap] partitions to be fixed in size. This would equate the System partition of Windows. In my ignorance I would have expected the disk layout to match the source image and have 15GB of air left in the target disk. Since LVM partition is copied as raw and both the /boot and [swap] were fixed.

    My goal here was to see if I could come up with a way to extend the lvm disk.


  • Senior Developer

    @sebastian-roth I agree, but data is hard to come by, and I don’t think we can come up with every case potential.


  • Developer

    @Tom-Elliott Sorry, didn’t see that you pushed a new version. Confirmed that my error posted does not apply anymore! Looks good to me now.

    Still I think about creating a testcases suite for the procsfdisk.awk script. So we could always check a lot of different layouts against the most current script to see if everything still works.


  • Senior Developer

    @sebastian-roth said in Failure to expand shrunken resizeable image from Linux machines:

    ./procsfdisk.awk -v SECTOR_SIZE=512 -v CHUNK_SIZE=512 -v MIN_START=2048 -v action=filldisk -v target=/dev/sda -v sizePos=146800640 -v diskSize=146800640 -v fixedList=1:5 in1.txt

    Based on the most current this is what I see:

    ./procsfdisk.awk -v SECTOR_SIZE=512 -v CHUNK_SIZE=512 -v MIN_START=2048 -v action=filldisk -v target=/dev/sda -v sizePos=146800640 -v diskSize=146800640 -v fixedList=1:5 in1.txt
    # Partition table is consistent.
    label: dos
    label-id: 0x77265efa
    device: /dev/sda
    unit: sectors
    
    /dev/sda1 : start=        2048, size=     3940352, type=82, bootable
    /dev/sda2 : start=     3944446, size=    71485442, type=5
    /dev/sda5 : start=     3944448, size=     3940352, type=83
    /dev/sda6 : start=     7884800, size=    67425792, type=83
    

  • Developer

    Great you all have been heavily working on this. Looks like we are getting there although the initially posted disk layout (sda1=root, sda2=extended, sda5=swap) has changed. From the latest pictures posted it looks as if the expansion is still not working for this kind of layout (SWAP being on sda1).

    I was just able to replicate this issue. But I didn’t have the time to find what’s causing the issue in our script:

    $ cat in1.txt
    label: dos
    label-id: 0x77265efa
    device: /dev/sda
    unit: sectors
    
    /dev/sda1 : start=        2048, size=     3940352, type=82, bootable
    /dev/sda2 : start=     3942400, size=    69457920, type=5
    /dev/sda5 : start=     3944448, size=     3940352, type=83
    /dev/sda6 : start=     7886848, size=    65513472, type=83
    
    $ ./procsfdisk.awk -v SECTOR_SIZE=512 -v CHUNK_SIZE=512 -v MIN_START=2048 -v action=filldisk -v target=/dev/sda -v sizePos=146800640 -v diskSize=146800640 -v fixedList=1:5 in1.txt 
    ERROR: New start and size (150960130) on (/dev/sda1) is larger than the disk (146800640).
    # ERROR in new partition table, quitting.
    # ERROR: /dev/sda5 has an overlap.
    label: dos
    label-id: 0x77265efa
    device: /dev/sda
    unit: sectors
    
    /dev/sda1 : start=        2048, size=     3940352, type=82, bootable
    /dev/sda2 : start=     3944448, size=   142856192, type=5
    /dev/sda5 : start=   142856194, size=     8103936, type=83
    /dev/sda6 : start=   142856194, size=   134747648, type=83
    

    From my point of view the output of sfdisk -d /dev/sda is much more helpful than df -h.


  • Senior Developer

    @george1421 It only expands partitions that are “expandable” and not “fixed”. Meaning, if you have 1 ext4 partition, and it shrinks, (say boot here), it will try to expand the partitions somewhat evenly. However, this is not a perfect system, nor has it ever really been. I’ve tried for all I’m worth, but I’m only one man, sorry.



  • @george1421 Close to my findings wherein the first partition (sda1) will grow proportionally, but the rest… no go.

    If you create the same layout but have the root partition where you currently have the boot partition, it will grow the root and ‘work’ as it seems it would be intended.

    If you were to make a home partition in the same place, it would grow but not the boot or root… etc


  • Moderator

    I was just looking at my ubuntu example and I found something not right. The image is LVM so FOG can only copy that in raw mode. So no expansion can happen (this is A reason for using standard partitions). But when I went from a 50GB original image to a 65GB target computer is grew the /boot partition to by 15GB (the difference between 50 and 65). I would have thought that the /boot partition would be a fixed size partition (as with MS Windows). I created a new LM master image based on LVM and it happened the same as with ubuntu. My original intent was to have some extra space in the physical disk so I could test some LVM expansion tools.

    Linux Mint with LVM defaults.

    jondoe@jondoe-VirtualBox ~ $ lsblk
    NAME                MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
    sr0                  11:0    1 1024M  0 rom  
    sda                   8:0    0   65G  0 disk 
    ├─sda2                8:2    0    1K  0 part 
    ├─sda5                8:5    0 49.5G  0 part 
    │ ├─mint--vg-swap_1 253:1    0    1G  0 lvm  [SWAP]
    │ └─mint--vg-root   253:0    0 48.5G  0 lvm  /
    └─sda1                8:1    0 15.5G  0 part /boot
    jondoe@jondoe-VirtualBox ~ $ df -h
    Filesystem                 Size  Used Avail Use% Mounted on
    udev                       474M     0  474M   0% /dev
    tmpfs                      100M  3.6M   96M   4% /run
    /dev/mapper/mint--vg-root   48G  5.0G   41G  11% /
    tmpfs                      496M  168K  496M   1% /dev/shm
    tmpfs                      5.0M  4.0K  5.0M   1% /run/lock
    tmpfs                      496M     0  496M   0% /sys/fs/cgroup
    /dev/sda1                   15G   67M   15G   1% /boot
    cgmfs                      100K     0  100K   0% /run/cgmanager/fs
    tmpfs                      100M  4.0K  100M   1% /run/user/108
    tmpfs                      100M   20K  100M   1% /run/user/1000
    jondoe@jondoe-VirtualBox ~ $ 
    


  • @tom-elliott

    Ran the two commands in your prior post as root:

    wget -O /var/www/fog/service/ipxe/init.xz https://fogproject.org/inits/init.xz
    wget -O /var/www/fog/service/ipxe/init_32.xz https://fogproject.org/inits/init_32.xz

    Recaptured all 3 images, then casted them to the test system one after the other and recorded the results.

    Didn’t re-run the installer or anything else, system wasn’t rebooted, etc, pretty much just replaced the init per the above command provided, recaptured, recast.

    I could try providing a VM image of one of the systems I’m capturing from that exhibits the issue? Could provide the actual images too.


  • Senior Developer

    Also, you did update the init first? No need to rerun installer for these tests as it will write the “bad” inits if done.


  • Senior Developer

    @xipher what do you mean identical? Of course my test boxes are using the default layout of linux mint, but all are appropriately resizing where my tests yesterday before changes failed in almost the exact same way you described.



  • @tom-elliott Sorry it took so long, I put the new init in, but the results were identical sadly :C

    Also, totally agree on the SWAP partition issue, its a change I’ll make on the client to capture.


  • Senior Developer

    @xipher I believe the swap mount is unrelated to where the partition actually sits, rather Linux Mint seems to set the swap by uuid. While we do try to reset the UUID, this isn’t working and I’m not overly worried about fixing the UUID for SWAP partitions anyway. (Why you ask?) Because you can edit the /etc/fstab, change the swap partition from the UUID to the actual location. In my case /dev/sda5 is the drive, so replace the UUID bit with /dev/sda5 and all works properly on reboots.



  • Ok, want to break down the results in the same sort of output I’m seeing used here for the source and destination of each attempt :)

    Destination is always a 75gb machine which is always a bigger disk. The ‘smaller disk’ restore issue I think is already identified?

    This is the ‘custom’ partitioned layout I was using yesterday which caused a problem:

    Source:

    alt text

    Destination:

    alt text

    This is the ‘default’ layout Mint Mate 18.2 produces from its installer when LVM is selected:

    Source:

    alt text

    Destination:

    alt text

    This is the ‘working’ solution wherein the root partition is the first partition on the drive:

    Source:

    alt text

    Destination:

    alt text

    Something I noticed on all the machines though was that the SWAP partition ONLY mounted on the LVM destination machine…!? All other attempts without LVM lead to the destination machine not mounting its swap, really weird…

    Hope this helps understand what I’m running into :) and I’ll try the patched init today, though I have to run off for a bit at the moment, sorry!


Log in to reply
 

456
Online

38968
Users

10709
Topics

101631
Posts

Looks like your connection to FOG Project was lost, please wait while we try to reconnect.