@edv-schuster The new problem is off-topic, please create a new thread with all the details. In the new thread, feel free to link to this one for reference. It’s important to create a new thread because this thread is solved, and therefore will be ignored by some moderators and testers (including myself). I only clicked on this thread because I was curious about how the issue was solved, I normally don’t do that. It’s also important for thread titles to accurately represent what’s in the thread.

We’re at 534 dev installations of FOG that have reported wiithin the last week. We started tracking this about 10 months ago. Coming up on a year of version tracking.

@george1421

Compression and decompression happens on the host. The default compression value was changed from 9 to 6 because 6 is faster (in probably 99% of cases).

If the OP can change his compression to 6 and then re-upload the image, he might find that his image deploys much faster afterwards.

@ch3i posted a script that will change the compression of an image without deploying/capturing again. It does it all on the server. It’s in the forums somewhere here. It needs to go in the wiki. Somebody hash tag it if they find it.

Server

• FOG Version: 1.4.2
• OS: Latest Arch

Description

Not a fog bug, but I’m putting this here so we can track it and others can find it.

During installation of any branch (working, dev-branch, master), FOG installer hangs at Setting up and starting MySQL

Looking deeper into it, MariaDB is crashing. Errors:

WARNING: The host 'Arch' could not be looked up with resolveip.
This probably means that your libc libraries are not 100 % compatible
with this binary MariaDB version. The MariaDB daemon, mysqld, should work
normally with the exception that host name resolving will not work.
This means that you should use IP addresses instead of hostnames
when specifying MariaDB privileges !
Installing MariaDB/MySQL system tables in '/var/lib/mysql' ...
2017-06-17 14:18:12 140206427066304 [Note] /usr/sbin/mysqld (mysqld 10.1.24-MariaDB) starting as process 894 ...
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: innodb_empty_free_list_algorithm has been changed to legacy because of small buffer pool size. In order to use backoff, increase buffer pool at least up to 20MB.

2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Using mutexes to ref count buffer pool pages
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: The InnoDB memory heap is disabled
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Mutexes and rw_locks use GCC atomic builtins
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: GCC builtin __atomic_thread_fence() is used for memory barrier
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Compressed tables use zlib 1.2.11
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Using Linux native AIO
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Using SSE crc32 instructions
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Initializing buffer pool, size = 128.0M
2017-06-17 14:18:12 140206427066304 [Note] InnoDB: Completed initialization of buffer pool
2017-06-17 14:18:13 140206427066304 [Note] InnoDB: Highest supported file format is Barracuda.
2017-06-17 14:18:13 140206427066304 [Note] InnoDB: 128 rollback segment(s) are active.
2017-06-17 14:18:13 140206427066304 [Note] InnoDB: Waiting for purge to start
2017-06-17 14:18:13 140206427066304 [Note] InnoDB:  Percona XtraDB (http://www.percona.com) 5.6.36-82.0 started; log sequence number 1616799
2017-06-17 14:18:13 140205812004608 [Note] InnoDB: Dumping buffer pool(s) not yet started
170617 14:18:15 [ERROR] mysqld got signal 11 ;
This could be because you hit a bug. It is also possible that this binary
or one of the libraries it was linked against is corrupt, improperly built,
or misconfigured. This error can also be caused by malfunctioning hardware.

To report this bug, see https://mariadb.com/kb/en/reporting-bugs

We will try our best to scrape up some info that will hopefully help
diagnose the problem, but since we have already crashed, 
something is definitely wrong and this may fail.

Server version: 10.1.24-MariaDB
key_buffer_size=16777216
read_buffer_size=262144
max_used_connections=0
max_threads=153
thread_count=0
It is possible that mysqld could use up to 
key_buffer_size + (read_buffer_size + sort_buffer_size)*max_threads = 137013 K  bytes of memory
Hope that's ok; if not, decrease some variables in the equation.

Thread pointer: 0x0
Attempting backtrace. You can use the following information to find out
where mysqld died. If you see no messages after this, something went
terribly wrong...
stack_bottom = 0x0 thread_stack 0x48400

I’ve created an issue with the MariaDB team here:
https://jira.mariadb.org/browse/MDEV-13116

You can also monitor the issue on the fog testing dashboard here:
http://perpetuum.io:20080/fog_distro_check/installer_dashboard.html

What would people think about a release? It’s been over a year since the last one. There are 248 adopters on version 1.5.9.111

I realize it’s not trivial to put a FOG release together. Perhaps all the steps can be captured to make it easier for others in the future to do it?

@Jarli is the server out of space? df -h

Moving a post from another thread about a bug to here. See below. @developers

Hello Fog Community!

I’m happy to inform that as the title says, Alma 8, Rocky 8, Fedora 35, and Debian 11 have been added to the daily installation tests. Fedora 32 has been removed.

In addition to this, I’ve re-enabled testing the working-1.6 branch as I see there is now some activity there. I’ve also implemented HTTPS for the installation results. And, I’ve added links at the bottom of the test results to the external reporting dashboard, and vice-versa as well. I’ve also done some extensive maintenance & care on the testing code base. It’s been upgraded from Python2 to Python3, among other improvements relating to speed and cost. If you’re interested, this was the pull request for the changes.

I’m thinking a lot on bringing back the streak counts, because they were really fun to watch. Though in addition to this, I want to capture execution time and other figures, and produce some pretty graphs for these things rather than just a number. Will be thinking on this.

Also thinking on adding some CI/CD stuff to the tests, where people can submit changes to the test suite and they “become real” shortly later in the test environment. Though for this, I do need to control it as things are not free in AWS. Thinking to make the test suit it’s own repo, and allowing others to submit pull requests. Upon pull request merge by me, Jenkins would implement the changes automatically. There’s both Terraform as well as the actual tests that could be affected.

Another thing I was thinking about is adding UI testing via selenium. The UI could be tested for every branch, with a fresh set of OS patches applied, every day, automatically. Though, I have very little experience with Selenium. Though the nice thing about such tests is we’d only need to have 1 set of tests as there’s only one UI, rather than independent code per operating system like I have to maintain for the installation tests. Selenium test results would be displayed on the same page as the installation tests.

I realize FOG is in maintenance mode. Though, I must admit I really enjoy doing these devops-ey type things for fog.

@Jarli Thanks to @george1421 we have an article on that. https://wiki.fogproject.org/wiki/index.php/Add_%26_Extend_a_2nd_Virtual_HDD

In a nutshell, just use your hypervisor to expand the disk, and then use the Linux OS to then expand the partition. Contigous space is required for it to work.

Otherwise just give the server a 2nd HDD via your hypervisor - and make it a respectable size like 500GB, and configure FOG to use it for images.

After a Normal Fog install i should be able to get to the iPXE Boot Menu right out of the Box using the Legacy or UEFI PXE boot, right?

Not necessarily. If FOG is doing DHCP and there isn’t another dhcp server operating with the same broadcast domain, then yeah you can pxe boot out of the box once fog is installed. This is the only case where it works ‘out of box’. All other scenarios require additional configuration somewhere.

What is the tftpboot location on FOG 1.4.4, is it just /tftpboot beside the other folders such as /home, /etc, /bin…?

The iPXE binaries are placed into /tftpboot automatically by the installer.

Why does the DHCP not need the “filename:undipxe.k/pxelinux.0” Option?

No idea. the .0 thing was a bug in the old dnsmasq versions - which has since been fixed by dnsmasq’s maintainer Simon Kelly. It’s interesting that you even ask this, because this suggests you have DHCP running on another server somewhere besides the FOG server.

As for all the other stuff - just start over. I have no idea where you went wrong, and don’t want to waste time trying to figure out what went wrong. There’s nothing important in your broke VM, so just delete the VM and make a new one. Also, use git this time instead of sourceforge. Instructions here: https://wiki.fogproject.org/wiki/index.php?title=Getting_FOG
Here’s a full video showing a Debian 8 setup that will also work with Debian 9: https://wiki.fogproject.org/wiki/index.php?title=Debian_8

Another thing I just now thought of - versioning the partition file we keep with the images. We can start out with v1.0.0. And if we change how we store that file, we simply increment the version. This would give us some more flexibility in regard to changes.

Thinking further, the scripts that collect the physical partitions are fine, we can just add JSON output for those.

And really, the LVM describing tools are what needs JSON output, so the effort there is targeted to just those.

I think I’m going to take a swing at adding JSON to an LVM describing tool (there are several).

@monasmith529 is the fog server virtualized too? is the client’s NIC type set to bridged? What’s doing DHCP ? Are you behind a home router device? Normally those do DHCP too, and they usually don’t support the needed options.

MOST people would use dnsmasq at home if they didn’t want to turn DHCP off for their home routing device.

Moved this to bugs, because I confirmed the problem in master branch.

@Sebastian-Roth I just tried doing an install on Debian9 with 1.4.4. If you specify the installer to setup dhcp, it fails when trying to start the dhcp server. And because the /tftpboot directory is setup after dhcp is, that doesn’t get done.

I’ve been digging into the problem, it’s IPv6 related as far as I can see right now. Here’s the error:
Starting ISC DHCPv6 server: dhcpd6check syslog for diagnostics. ... failed!

Snippet from journalctl -xe:

root@Debian9:/# journalctl -xe
Dec 30 14:14:34 Debian9 dhcpd[22356]: 
Dec 30 14:14:34 Debian9 dhcpd[22356]: 
Dec 30 14:14:34 Debian9 dhcpd[22356]: Not configured to listen on any interfaces!
Dec 30 14:14:34 Debian9 dhcpd[22356]: 
Dec 30 14:14:34 Debian9 dhcpd[22356]: If you think you have received this message due to a bug rather
Dec 30 14:14:34 Debian9 dhcpd[22356]: than a configuration issue please read the section on submitting
Dec 30 14:14:34 Debian9 dhcpd[22356]: bugs on either our web page at www.isc.org or in the README file
Dec 30 14:14:34 Debian9 dhcpd[22356]: before submitting a bug.  These pages explain the proper
Dec 30 14:14:34 Debian9 dhcpd[22356]: process and the information we find helpful for debugging..
Dec 30 14:14:34 Debian9 dhcpd[22356]: 
Dec 30 14:14:34 Debian9 dhcpd[22356]: exiting.
Dec 30 14:14:36 Debian9 isc-dhcp-server[22339]: Starting ISC DHCPv6 server: dhcpd6check syslog for diagnostics. ... failed!
Dec 30 14:14:36 Debian9 isc-dhcp-server[22339]:  failed!
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Control process exited, code=exited status=1
Dec 30 14:14:36 Debian9 systemd[1]: Failed to start LSB: DHCP server.
-- Subject: Unit isc-dhcp-server.service has failed
-- Defined-By: systemd
-- Support: https://www.debian.org/support
-- 
-- Unit isc-dhcp-server.service has failed.
-- 
-- The result is failed.
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Unit entered failed state.
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Failed with result 'exit-code'.

This is systemctl status isc-dhcp-server:

root@Debian9:/etc/dhcp# systemctl status isc-dhcp-server
● isc-dhcp-server.service - LSB: DHCP server
   Loaded: loaded (/etc/init.d/isc-dhcp-server; generated; vendor preset: enabled)
   Active: failed (Result: exit-code) since Sat 2017-12-30 14:14:36 CST; 11min ago
     Docs: man:systemd-sysv-generator(8)
  Process: 22339 ExecStart=/etc/init.d/isc-dhcp-server start (code=exited, status=1/FAILURE)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/isc-dhcp-server.service
           └─22351 /usr/sbin/dhcpd -4 -q -cf /etc/dhcp/dhcpd.conf

Dec 30 14:14:32 Debian9 dhcpd[22350]: Wrote 0 leases to leases file.
Dec 30 14:14:32 Debian9 dhcpd[22351]: Server starting service.
Dec 30 14:14:34 Debian9 isc-dhcp-server[22339]: Starting ISC DHCPv4 server: dhcpd.
Dec 30 14:14:34 Debian9 dhcpd[22356]: Wrote 0 NA, 0 TA, 0 PD leases to lease file.
Dec 30 14:14:36 Debian9 isc-dhcp-server[22339]: Starting ISC DHCPv6 server: dhcpd6check syslog for diagnostics. ... failed!
Dec 30 14:14:36 Debian9 isc-dhcp-server[22339]:  failed!
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Control process exited, code=exited status=1
Dec 30 14:14:36 Debian9 systemd[1]: Failed to start LSB: DHCP server.
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Unit entered failed state.
Dec 30 14:14:36 Debian9 systemd[1]: isc-dhcp-server.service: Failed with result 'exit-code'.

Snippet from the bottom of my fog error log:

Dec 20 19:59:42 Debian9 systemd[1]: Starting The PHP 7.0 FastCGI Process Manager...
Dec 20 19:59:42 Debian9 systemd[1]: Started The PHP 7.0 FastCGI Process Manager.
isc-dhcp-server.service is not a native service, redirecting to systemd-sysv-install.
Executing: /lib/systemd/systemd-sysv-install enable isc-dhcp-server
Job for isc-dhcp-server.service failed because the control process exited with error code.
See "systemctl status isc-dhcp-server.service" and "journalctl -xe" for details.
● isc-dhcp-server.service - LSB: DHCP server
   Loaded: loaded (/etc/init.d/isc-dhcp-server; generated; vendor preset: enabled)
   Active: failed (Result: exit-code) since Wed 2017-12-20 20:00:12 CST; 2s ago
     Docs: man:systemd-sysv-generator(8)
  Process: 21889 ExecStart=/etc/init.d/isc-dhcp-server start (code=exited, status=1/FAILURE)
    Tasks: 1 (limit: 4915)
   CGroup: /system.slice/isc-dhcp-server.service
           └─21901 /usr/sbin/dhcpd -4 -q -cf /etc/dhcp/dhcpd.conf

Dec 20 20:00:10 Debian9 dhcpd[21906]: than a configuration issue please read the section on submitting
Dec 20 20:00:10 Debian9 dhcpd[21906]: bugs on either our web page at www.isc.org or in the README file
Dec 20 20:00:10 Debian9 dhcpd[21906]: before submitting a bug.  These pages explain the proper
Dec 20 20:00:10 Debian9 dhcpd[21906]: process and the information we find helpful for debugging..
Dec 20 20:00:12 Debian9 isc-dhcp-server[21889]: Starting ISC DHCPv6 server: dhcpd6check syslog for diagnostics. ... failed!
Dec 20 20:00:12 Debian9 isc-dhcp-server[21889]:  failed!
Dec 20 20:00:12 Debian9 systemd[1]: isc-dhcp-server.service: Control process exited, code=exited status=1
Dec 20 20:00:12 Debian9 systemd[1]: Failed to start LSB: DHCP server.
Dec 20 20:00:12 Debian9 systemd[1]: isc-dhcp-server.service: Unit entered failed state.
Dec 20 20:00:12 Debian9 systemd[1]: isc-dhcp-server.service: Failed with result 'exit-code'.

Here’s the output of ip addr:

root@Debian9:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: ens3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:67:a2:b4 brd ff:ff:ff:ff:ff:ff
    inet 10.0.0.39/24 brd 10.0.0.255 scope global ens3
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe67:a2b4/64 scope link 
       valid_lft forever preferred_lft forever

Output of ifconfig

root@Debian9:/# ifconfig
ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.0.0.39  netmask 255.255.255.0  broadcast 10.0.0.255
        inet6 fe80::5054:ff:fe67:a2b4  prefixlen 64  scopeid 0x20<link>
        ether 52:54:00:67:a2:b4  txqueuelen 1000  (Ethernet)
        RX packets 163559  bytes 232881890 (222.0 MiB)
        RX errors 0  dropped 231  overruns 0  frame 0
        TX packets 44237  bytes 4542466 (4.3 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 46  bytes 1794 (1.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 46  bytes 1794 (1.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

This is my dhcpd.conf file:

root@Debian9:/etc/dhcp# cat dhcpd.conf 
# DHCP Server Configuration file\n#see /usr/share/doc/dhcp*/dhcpd.conf.sample
# This file was created by FOG
#Definition of PXE-specific options
# Code 1: Multicast IP Address of bootfile
# Code 2: UDP Port that client should monitor for MTFTP Responses
# Code 3: UDP Port that MTFTP servers are using to listen for MTFTP requests
# Code 4: Number of seconds a client must listen for activity before trying
#         to start a new MTFTP transfer
# Code 5: Number of seconds a client must listen before trying to restart
#         a MTFTP transfer
option space PXE;
option PXE.mtftp-ip code 1 = ip-address;
option PXE.mtftp-cport code 2 = unsigned integer 16;
option PXE.mtftp-sport code 3 = unsigned integer 16;
option PXE.mtftp-tmout code 4 = unsigned integer 8;
option PXE.mtftp-delay code 5 = unsigned integer 8;
option arch code 93 = unsigned integer 16;
use-host-decl-names on;
ddns-update-style interim;
ignore client-updates;
# Specify subnet of ether device you do NOT want service.
# For systems with two or more ethernet devices.
# subnet 136.165.0.0 netmask 255.255.0.0 {}
subnet 10.0.0.0 netmask 255.255.255.0 {
    option subnet-mask 255.255.255.0;
    range dynamic-bootp 10.0.0.10 10.0.0.254;
    default-lease-time 21600;
    max-lease-time 43200;
    option routers 10.0.0.1;
    option domain-name-servers 208.67.222.222;
    next-server 10.0.0.39;
    class "Legacy" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00000";
        filename "undionly.kkpxe";
    }
    class "UEFI-32-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00002";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-32-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00006";
        filename "i386-efi/ipxe.efi";
    }
    class "UEFI-64-1" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00007";
        filename "ipxe.efi";
    }
    class "UEFI-64-2" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00008";
        filename "ipxe.efi";
    }
    class "UEFI-64-3" {
        match if substring(option vendor-class-identifier, 0, 20) = "PXEClient:Arch:00009";
        filename "ipxe.efi";
    }
    class "SURFACE-PRO-4" {
        match if substring(option vendor-class-identifier, 0, 32) = "PXEClient:Arch:00007:UNDI:003016";
        filename "ipxe7156.efi";
    }
    class "Apple-Intel-Netboot" {
        match if substring(option vendor-class-identifier, 0, 14) = "AAPLBSDPC/i386";
        option dhcp-parameter-request-list 1,3,17,43,60;
        if (option dhcp-message-type = 8) {
            option vendor-class-identifier "AAPLBSDPC";
            if (substring(option vendor-encapsulated-options, 0, 3) = 01:01:01) {
                # BSDP List
                option vendor-encapsulated-options 01:01:01:04:02:80:00:07:04:81:00:05:2a:09:0D:81:00:05:2a:08:69:50:58:45:2d:46:4f:47;
                filename "ipxe.efi";
            }
        }
    }
}

Now at 610 Dev users. I’m thinking to myself - wow. That’s a lot of dev users.

I am super curious what will happen to these figures after the next release. I’m thinking a lot of people are using the dev branch simply because the installer works with the latest OS patches. But, I also think some unknown amount of users out there are sticking with the last stable release, and biding their time with updated kernels, inits, and ipxe binaries as necessary.

@george1421 said:

@Sebastian-Roth said:

Then run dmesg | grep -i eth and lspci (full output of Ethernet controller so we might see if it is really using tg3). Thanks!

I’m sure a few commands hard coded into a menu selection like this would be handy to have, especially if they were in the FOG advanced menu so a user could just run them upon request of a a dev.

+1 great idea. Maybe even attempt to throw the results into /images/dev too ?

At this point, it’s very likely this is a bug, so moved this to bug reports.

As the title says, RHEL 7 and CentOS 8 have been removed from the daily installation tests.

CentOS 8 had been failing on all branches for some days, something to do with no mirror URLs. @Sebastian-Roth recommended to remove it from the tests, so I went and got this done.

RHEL 7 - there was some problem with getting patches for new AWS Instances. I didn’t spend much time trying to figure it out, as I expect patches to just work when I run yum -y update in AWS. RHEL 7 was also pretty long in the tooth and is nearing it’s ultimate EOL. So, I removed it.

@mitzayapa said:

I would gladly update to 1.20 but the “quick image/host registration bypass” keeps me from doing it. I really need that option because the install process has to be fast (we do “on the spot” installations), we have 9 different windows images from 9 to 27 gigs , and registering every single client ain’t an option.

I have 22 images, all are 40 to 50 GB. I have 500 hosts in my building - they are all registered.

Some of the FOG users here have 6,000 hosts.

I think you are mistaking that this registration needs to happen all at once. it does not. Register as you image.

I also image “on the spot” - and a registered client in FOG Trunk will be done far before an unregistered one.

Automatic naming
Automatic domain joining
Automatic Windows Activation
Automatic printer deployment

Just imaging a 45GB image on Trunk takes 5 minutes, the booting and rebooting and activation maybe 2 minutes.

Can you beat that, doing it all manually? I really doubt it.

You need to try out FOG’s features. They will SPOIL you Rotten, I tell you!

I can image 30 PCs at once and just walk away and not worry about it. I’ll come back later and their all imaged, named, on the domain and ready to rock and roll.

My next goal for this summer is 500 computers in a day - using a combination of unicasting and multicasting.

@austinjt01 Did you upgrade the node also? The main server and all nodes must be on the exact same fog version to work.

I see Fedora 35 tests are now passing on dev-branch as well as working-1.6.