FOG 1.3.5 RC 11

[Tom Elliott]

@URfog When did you update last?

[fogschut]

root@fog-vm:~/fogproject# git pull
Already up-to-date.

I updated about an hour (or two) ago and I still got the problem.

[Tom Elliott]

@fogschut Please rerun the installer. Git pull will not work in this instance. re-running the installer will download whatever the latest init’s are.

I just pushed a testing version using a very old version of the script that handles resizing modified slightly to adjust for some of the changes in the variables being passed in.

[fogschut]

I did run a git pull followed by running the install-script (installfog.sh) and the problem is not fixed.

I still get a raw partition as 2nd partition and the uploaded image on the server is way too small to contain all the data on the server.

[Tom Elliott]

@fogschut When did you try? After I posted my message and before you posted yours?

Just need clarity. Not even 20 minutes ago a new init was uploaded Information will help me help find the problem, or at the least narrow it down.

[fogschut]

immediately after your message. I’m working on another image on another PC at the moment so I can test immediately

[fogschut]

Just to take away any confusion:

root@fog-vm:~/fogproject# date
di mrt  7 16:19:11 CET 2017

root@fog-vm:~/fogproject# git checkout dev-branch
Already on 'dev-branch'
Your branch is up-to-date with 'origin/dev-branch'.
root@fog-vm:~/fogproject# git pull
Already up-to-date.
root@fog-vm:~/fogproject# cd bin
root@fog-vm:~/fogproject/bin# ./installfog.sh
Installing LSB_Release as needed
 * Attempting to get release information.......................Done


   +------------------------------------------+
   |     ..#######:.    ..,#,..     .::##::.  |
   |.:######          .:;####:......;#;..     |
   |...##...        ...##;,;##::::.##...      |
   |   ,#          ...##.....##:::##     ..:: |
   |   ##    .::###,,##.   . ##.::#.:######::.|
   |...##:::###::....#. ..  .#...#. #...#:::. |
   |..:####:..    ..##......##::##  ..  #     |
   |    #  .      ...##:,;##;:::#: ... ##..   |
   |   .#  .       .:;####;::::.##:::;#:..    |
   |    #                     ..:;###..       |
   |                                          |
   +------------------------------------------+
   |      Free Computer Imaging Solution      |
   +------------------------------------------+
   |  Credits: http://fogproject.org/Credits  |
   |       http://fogproject.org/Credits      |
   |       Released under GPL Version 3       |
   +------------------------------------------+


   Version: 1.3.5-RC-11 Installer/Updater


 * Found FOG Settings from previous install at: /opt/fog/.fogsettings

 * Performing upgrade using these settings


  Starting Debian based Installation



   ######################################################################
   #     FOG now has everything it needs for this setup, but please     #
   #   understand that this script will overwrite any setting you may   #
   #   have setup for services like DHCP, apache, pxe, tftp, and NFS.   #
   ######################################################################
   # It is not recommended that you install this on a production system #
   #        as this script modifies many of your system settings.       #
   ######################################################################
   #             This script should be run by the root user.            #
   #      It will prepend the running with sudo if root is not set      #
   ######################################################################
   #           ** Notice ** FOG is difficult to setup securely          #
   #        SELinux and IPTables are usually asked to be disabled       #
   #           There have been strides in adding capabilities           #
   #          The recommendations would now be more appropriate         #
   #    to set SELinux to permissive and to disable firewall for now.   #
   #  You can find some methods to enable SELinux and maintain firewall #
   #   settings and ports. If you feel comfortable doing so please do   #
   ######################################################################
   #            Please see our wiki for more information at:            #
   ######################################################################
   #             https://wiki.fogproject.org/wiki/index.php             #
   ######################################################################

 * Here are the settings FOG will use:
 * Base Linux: Debian
 * Detected Linux Distribution: Ubuntu
 * Server IP Address: 10.195.0.65
 * Server Subnet Mask:
 * Interface: eth0
 * Installation Type: Normal Server
 * Donate: 0
 * Internationalization:
 * Image Storage Location: /images
 * Using FOG DHCP: No
 * DHCP will NOT be setup but you must setup your
 | current DHCP server to use FOG for PXE services.

 * On a Linux DHCP server you must set: next-server and filename

 * On a Windows DHCP server you must set options 066 and 067

 * Option 066/next-server is the IP of the FOG Server: (e.g. 10.195.0.65)
 * Option 067/filename is the bootfile: (e.g. undionly.kpxe)


 * Are you sure you wish to continue (Y/N) Y

 * Installation Started

 * Installing required packages, if this fails
 | make sure you have an active internet connection.

 * Adding needed repository....................................OK
 * Preparing Package Manager...................................OK
 * Packages to be installed:

        apache2 bc build-essential cpp curl g++ gawk gcc gzip htmldoc lftp libapache2-mod-php7.1 libc6 libcurl3 m4 mysql-client mysql-server net-tools nfs-kernel-serv         er openssh-server php7.1 php7.1-bcmath php7.1-cli php7.1-curl php7.1-fpm php7.1-gd php7.1-json php7.1-ldap php7.1-mbstring php7.1-mcrypt php7.1-mysql php-gettext sysv         -rc-conf tar tftpd-hpa tftp-hpa unzip vsftpd wget xinetd zlib1g


 * Skipping package: apache2...................................(Already Installed)
 * Skipping package: bc........................................(Already Installed)
 * Skipping package: build-essential...........................(Already Installed)
 * Skipping package: cpp.......................................(Already Installed)
 * Skipping package: curl......................................(Already Installed)
 * Skipping package: g++.......................................(Already Installed)
 * Skipping package: gawk......................................(Already Installed)
 * Skipping package: gcc.......................................(Already Installed)
 * Skipping package: gzip......................................(Already Installed)
 * Skipping package: htmldoc...................................(Already Installed)
 * Skipping package: lftp......................................(Already Installed)
 * Skipping package: libapache2-mod-php7.1.....................(Already Installed)
 * Skipping package: libc6.....................................(Already Installed)
 * Skipping package: libcurl3..................................(Already Installed)
 * Skipping package: m4........................................(Already Installed)
 * Skipping package: mysql-client..............................(Already Installed)
 * Skipping package: mysql-server..............................(Already Installed)
 * Skipping package: net-tools.................................(Already Installed)
 * Skipping package: nfs-kernel-server.........................(Already Installed)
 * Skipping package: openssh-server............................(Already Installed)
 * Skipping package: php7.1....................................(Already Installed)
 * Skipping package: php7.1-bcmath.............................(Already Installed)
 * Skipping package: php7.1-cli................................(Already Installed)
 * Skipping package: php7.1-curl...............................(Already Installed)
 * Skipping package: php7.1-fpm................................(Already Installed)
 * Skipping package: php7.1-gd.................................(Already Installed)
 * Skipping package: php7.1-json...............................(Already Installed)
 * Skipping package: php7.1-ldap...............................(Already Installed)
 * Skipping package: php7.1-mbstring...........................(Already Installed)
 * Skipping package: php7.1-mcrypt.............................(Already Installed)
 * Skipping package: php7.1-mysql..............................(Already Installed)
 * Skipping package: php-gettext...............................(Already Installed)
 * Skipping package: sysv-rc-conf..............................(Already Installed)
 * Skipping package: tar.......................................(Already Installed)
 * Skipping package: tftpd-hpa.................................(Already Installed)
 * Skipping package: tftp-hpa..................................(Already Installed)
 * Skipping package: unzip.....................................(Already Installed)
 * Skipping package: vsftpd....................................(Already Installed)
 * Skipping package: wget......................................(Already Installed)
 * Skipping package: xinetd....................................(Already Installed)
 * Skipping package: zlib1g....................................(Already Installed)
 * Updating packages as needed.................................OK

 * Confirming package installation

 * Checking package: apache2...................................OK
 * Checking package: bc........................................OK
 * Checking package: build-essential...........................OK
 * Checking package: cpp.......................................OK
 * Checking package: curl......................................OK
 * Checking package: g++.......................................OK
 * Checking package: gawk......................................OK
 * Checking package: gcc.......................................OK
 * Checking package: gzip......................................OK
 * Checking package: htmldoc...................................OK
 * Checking package: lftp......................................OK
 * Checking package: libapache2-mod-php7.1.....................OK
 * Checking package: libc6.....................................OK
 * Checking package: libcurl3..................................OK
 * Checking package: m4........................................OK
 * Checking package: mysql-client..............................OK
 * Checking package: mysql-server..............................OK
 * Checking package: net-tools.................................OK
 * Checking package: nfs-kernel-server.........................OK
 * Checking package: openssh-server............................OK
 * Checking package: php7.1....................................OK
 * Checking package: php7.1-bcmath.............................OK
 * Checking package: php7.1-cli................................OK
 * Checking package: php7.1-curl...............................OK
 * Checking package: php7.1-fpm................................OK
 * Checking package: php7.1-gd.................................OK
 * Checking package: php7.1-json...............................OK
 * Checking package: php7.1-ldap...............................OK
 * Checking package: php7.1-mbstring...........................OK
 * Checking package: php7.1-mcrypt.............................OK
 * Checking package: php7.1-mysql..............................OK
 * Checking package: php-gettext...............................OK
 * Checking package: sysv-rc-conf..............................OK
 * Checking package: tar.......................................OK
 * Checking package: tftpd-hpa.................................OK
 * Checking package: tftp-hpa..................................OK
 * Checking package: unzip.....................................OK
 * Checking package: vsftpd....................................OK
 * Checking package: wget......................................OK
 * Checking package: xinetd....................................OK
 * Checking package: zlib1g....................................OK

 * Configuring services

 * Setting up fog user.........................................Already setup
 * Setting up fog password.....................................OK
 * Stopping FOGMulticastManager Service........................OK
 * Stopping FOGImageReplicator Service.........................OK
 * Stopping FOGSnapinReplicator Service........................OK
 * Stopping FOGScheduler Service...............................OK
 * Stopping FOGPingHosts Service...............................OK
 * Stopping FOGSnapinHash Service..............................OK
 * Stopping FOGImageSize Service...............................OK
 * Setting up and starting MySQL...............................OK
 * Backing up user reports.....................................Done
 * Stopping web service........................................OK
 * Removing vhost file.........................................OK
 * Setting up Apache and PHP files.............................OK
 * Testing and removing symbolic links if found................OK
 * Backing up old data.........................................OK
 * Copying new files to web folder.............................OK
 * Creating config file........................................OK
 * Getting checksum files for kernels and inits................OK
 * Downloading inits, kernels, and the fog client..............OK
 * Comparing checksums of kernels and inits....................Done
 * Enabling apache and fpm services on boot....................OK
 * Creating SSL Certificate....................................OK
 * Creating auth pub key and cert..............................OK
 * Resetting SSL Permissions...................................OK
 * Setting up SSL FOG Server...................................OK
 * Restarting Apache2 for fog vhost............................OK
 * Changing permissions on apache log files....................OK
 * Backing up database.........................................OK

 * You still need to install/update your database schema.
 * This can be done by opening a web browser and going to:

   http://10.195.0.65/fog/management

 * Press [Enter] key when database is updated/installed.

 * Setting up storage..........................................OK
 * Setting up and starting DHCP Server.........................Skipped
 * Setting up and starting TFTP and PXE Servers................OK
 * Setting up and starting VSFTP Server........................OK
 * Setting up FOG Snapins......................................OK
 * Setting up UDPCast..........................................OK
 * Configuring UDPCast.........................................OK
 * Building UDPCast............................................OK
 * Installing UDPCast..........................................OK
 * Installing FOG System Scripts...............................OK


 * Configuring FOG System Services


 * Setting FOGMulticastManager script executable...............OK
 * Enabling FOGMulticastManager Service........................OK
 * Setting FOGImageReplicator script executable................OK
 * Enabling FOGImageReplicator Service.........................OK
 * Setting FOGSnapinReplicator script executable...............OK
 * Enabling FOGSnapinReplicator Service........................OK
 * Setting FOGScheduler script executable......................OK
 * Enabling FOGScheduler Service...............................OK
 * Setting FOGPingHosts script executable......................OK
 * Enabling FOGPingHosts Service...............................OK
 * Setting FOGSnapinHash script executable.....................OK
 * Enabling FOGSnapinHash Service..............................OK
 * Setting FOGImageSize script executable......................OK
 * Enabling FOGImageSize Service...............................OK
 * Setting up FOG Services.....................................OK
 * Starting FOGMulticastManager Service........................OK
 * Starting FOGImageReplicator Service.........................OK
 * Starting FOGSnapinReplicator Service........................OK
 * Starting FOGScheduler Service...............................OK
 * Starting FOGPingHosts Service...............................OK
 * Starting FOGSnapinHash Service..............................OK
 * Starting FOGImageSize Service...............................OK
 * Setting up exports file.....................................OK
 * Setting up and starting RPCBind.............................OK
 * Setting up and starting NFS Server..........................OK
 * Ensuring node username and passwords match..................Done

 * Setup complete

   You can now login to the FOG Management Portal using
   the information listed below.  The login information
   is only if this is the first install.

   This can be done by opening a web browser and going to:

   http://10.195.0.65/fog/management

   Default User Information
   Username: fog
   Password: password


After this the problem persists. Check date command on top.

[Tom Elliott]

Figured out the problem.

This is because of 4k disks. I won’t go into the specifics, but I can try to show the math that equates out exactly what was happening:.

I was taking the size, making it a multiple of 1024:

9637387 * 1024 = 9868684288

On a 512 block disk (normal sized from long ago) The math than goes:
9868684288 / 512 = 19274774

Then we align it as needed (no need to show that work here).

On a 4096 block disk (4k) the math was being presented as:
9868684288 / 4096 = 2409346.75

Now if you make that number an int, you can immediately see it EXACTLY equals the number sectors trying to be copied.

Take that 2409346 and multiple it by 512 and you get about 1.2 GB of disk.

I’ve taken correction action on this and hope this will no longer be present.

As always, please re-run the installer and this capture issue (which appears to me to be only related to 4k disks) should be fixed.

[Tom Elliott]

Oh, and here’s the size that the (bad) was returning:

1233585536

In otherwords, 1 gb, 233 mb, 585 kb, 536b (or 1.2 GB)

[george1421]

@Tom-Elliott said in FOG 1.3.5 RC 11:

Figured out the problem.

This is because of 4k disks.

Ugh, 4kn drives… nuff said.

I would your legacy code will work for 512e drives since they are being presented externally as 512n drives but internally they are 4k.

[Tom Elliott]

I will, however, add that the “boot” partition issue you’re describing.

FOG does not resize partitions that have a label containing reserved, or recovery.

If the label isn’t there, it’s shouldn’t hurt anything to be resized. I don’t think the “non-boot” issue was due to this resizing anyway, now that I’m aware of the CHUNK_SIZE problem. If anything, it was this chunk_size issue that was causing the boot problems to begin with.

[fogschut]

I was indeed running on a 4k drive, those things are so common now I did not even think about mentioning it

The capture job worked correct now and the image was properly recognized as NTFS and a Block Size of 4096 bytes rather than raw and 512 bytes. I was able to restore the image without any problems.

It seems you solved the problem, thanks for your hard work!

[jc35]

Hello,
I test in debug mode, when I launch “fgo” and I have a message “* Running post init scripts…/bin/fog: line 20: fog.postinit: No such file or directory”

[URfog]

@Tom-Elliott My capture is running well now, thanks for your magic

[george1421]

@jc35 said in FOG 1.3.5 RC 11:

Hello,
I test in debug mode, when I launch “fgo” and I have a message “* Running post init scripts…/bin/fog: line 20: fog.postinit: No such file or directory”

Will you confirm that the following directory exits: /images/dev/postinitscripts

[jc35]

@george1421 Hello, thanks for your answer, Sorry, I change the path and I foget to copy postinitscripts.

[Tom Elliott]

All those that were having issues in the past (or those at least willing), would you mind downloading the init’s I’m working on? I’m trying to maintain a proper “alignment” system so a 4k disk would actually be working on 4k sectors as is “expected” to ensure performance and usability.

My hope, now, is I’ve figured out where the issues were present. This should, from what I can tell, only impact capture tasks, so if you can install and test a simple upload (If it fails you ahve not lost your now working image)? You could do this with another image definition too I suppose.

I just am hoping i’m converting the sector sizes as according to the disks now.

The init’s are currently located at:

http://mastacontrola.com/init.xz
http://mastacontrola.com/init_32.xz

You can temporarily replace the init’s with something like:

mv /var/www/fog/service/ipxe/init{,_back}.xz
mv /var/www/fog/service/ipxe/init_32{,_back}.xz
wget -O /var/www/fog/service/ipxe/init.xz http://mastacontrola.com/init.xz
wget -O /var/www/fog/service/ipxe/init_32.xz http://mastacontrola.com/init_32.xz

[URfog]

@Tom-Elliott
I’ve done a new capture just to test with your new inits, it does, but shows one error:

• Resizing partition /dev/sda1…awk: fatal: cannot file ‘-v’ for reading (No such file or directory)

Probably a mistype in the script

Regards

[Tom Elliott]

@URfog I’m fairly sure this error is fixed in those init. I’ll rebuild just to be sure.

[Tom Elliott]

Init’s rebuilt on my side.

They are located at the same urls as earlier on.