@sebastian-roth Thanks for your work, we are in the process of switching to the dev-branch on Ubuntu 22 now and will report any additional bugs we find.

@JJ-Fullmer and @george1421 thank you so much for the immediate responses. I will follow your advice and see if I can fix it.
Regarding the use case, this is for doing malware analysis on bare metal machines (resetting the machine after each analysis).

@Tom-Elliott I really appreciate that you are putting effort into providing more frequent releases, which makes it easier for everyone to deploy new security fixes in time. Keep up the good work!

Hi there,
Here is some information for anybody upgrading their installation to Ubuntu 18.04 and getting stuck because of the sysv-rc-conf package which does not exist on Ubuntu 18.

The thing is that if you do a fresh install of Ubuntu 18 and then install Fog, you don’t face this issue.

But if you are coming e.g. from Ubuntu 14, you get stuck and as a solution are told to remove the package from config.sh, see: https://forums.fogproject.org/topic/9998/upgrade-ubuntu-16-10-to-17-04-sysv-rc-conf-fail

This “riddle” has been asked but not answered here: https://github.com/FOGProject/fogproject/issues/258 before.

I noticed when comparing the two that for the install that gets stuck, the output is:

Installing package: sysv-rc-conf..............................Failed! (Will try later)

while for the fresh install that works it is:

Skipping package: sysv-rc-conf..............................(Does not exist)

I wondered what that meant and had a look at /lib/common/functions.sh

What happens here is that the installer calls:

eval $packagelist "$x"

which in this case is:

apt-cache pkgnames | grep sysv-rc-conf

Now on a fresh Ubuntu 18, this will give you no output and return code 1, while on an upgraded Ubuntu, it is

sysvr-rc-conf

and the return code is 0, because the package has been removed, but is still listed by dpkg and marked as “rc”.

To cut a long story short, before you install Fog, completely purge sysvr-rc-conf, and you will be fine:

sudo apt purge sysv-rc-conf

Have fun!

@JJ-Fullmer Just a short update on this issue: The NUC’s BIOS was outdated, and updating it failed for unknown reasons. We now replaced the NUC with a brand new one and hope to see the issue gone.

PR created in dev-branch

I randomly have the issue that after restoring, the FOG boot menu does not proceed at all.
Next to the entry “Boot from hard disk”, there is normally a counter from 3 to 1, and then it boots.
But in this case, there is no counter, nothing happens.
If I press Return, the boots process continues normally.
If reboot the machine once more, the “Boot from hard disk” also works again as expected.
What could be causing this?

I am running the latest FOG version.

@Tom-Elliott I really appreciate that you are putting effort into providing more frequent releases, which makes it easier for everyone to deploy new security fixes in time. Keep up the good work!

@Sebastian-Roth Thanks a lot for these troubleshooting tips!
I haven’t been using systemd-analyze, quite interesting.

It looks to me a bit like it’s something with the network. The SVG shows this in red:

Also, smbd was marked red by critical-chain. I have disabled this service now, it is not needed to start on boot really.

graphical.target @2min 2.330s
└─multi-user.target @2min 2.330s                                                                                                                                                                                                               
   └─smbd.service @2min 2.126s +203ms
      └─network-online.target @2min 2.060s                                                                                                                                                                                                           
          └─network.target @2.156s         

The output for all the PHP commands is identical on my machine.

I’ll wait for the next boot to see if disabling smbd maybe helped.

@Sebastian-Roth That’s nice - but as far as I understand, it still doesn’t protect against the new image needing 10GB, but there are only 9GB left.
Right?

I now also checked the other services, and the funny thing is that they all have the same long list of errors, but start successfully in the end (e.g. FOGImageReplicator). Also FOGSnapinHash itself.
But only FOGSnapinHash cycle is shown on boot.

I guess it could all be fixed by making /usr/bin/env php work, but it’s not clear to me how to do that.

@Sebastian-Roth I collected the journalctl log as you suggested and found this:

First there are many of this entry:

Nov 17 11:29:39 host systemd[1]: Started FOGSnapinHash.
Nov 17 11:29:39 host env[26388]: /usr/bin/env: ‘php’: No such file or directory
Nov 17 11:29:39 host systemd[1]: FOGSnapinHash.service: Main process exited, code=exited, status=127/n/a
Nov 17 11:29:39 host systemd[1]: FOGSnapinHash.service: Failed with result 'exit-code'.
Nov 17 11:29:40 host systemd[1]: FOGSnapinHash.service: Scheduled restart job, restart counter is at 762.
Nov 17 11:29:40 host systemd[1]: Stopped FOGSnapinHash.

The problem here is that /lib/systemd/system/FOGSnapinHash.service says:

ExecStart=/usr/bin/env php /opt/fog/service/FOGSnapinHash/FOGSnapinHash

but it doesn’t find php this way, probably because at this point a full shell isn’t available yet (?).

I remember reading about this issue in FOG project years ago, I don’t know if it was fixed?

There are more errors in the log afterwards, they all look like this:

Nov 17 11:29:42 host systemd[1]: Started FOGSnapinHash.
Nov 17 11:29:42 host env[27676]: PHP Fatal error:  Uncaught Exception: Missing one or more extensions. in /var/www/fog/commons/init.php:439
Nov 17 11:29:42 host env[27676]: Stack trace:
Nov 17 11:29:42 host env[27676]: #0 /var/www/fog/commons/init.php(306): Initiator::_extCheck()
Nov 17 11:29:42 host env[27676]: #1 /var/www/fog/commons/base.inc.php(46): Initiator::startInit()
Nov 17 11:29:42 host env[27676]: #2 /opt/fog/service/lib/service_lib.php(22): require('...')
Nov 17 11:29:42 host env[27676]: #3 /opt/fog/service/FOGSnapinHash/FOGSnapinHash(25): require('...')
Nov 17 11:29:42 host env[27676]: #4 {main}
Nov 17 11:29:42 host env[27676]:   thrown in /var/www/fog/commons/init.php on line 439
Nov 17 11:29:42 host systemd[1]: FOGSnapinHash.service: Main process exited, code=exited, status=255/EXCEPTION
Nov 17 11:29:42 host systemd[1]: FOGSnapinHash.service: Failed with result 'exit-code'.
Nov 17 11:29:43 host systemd[1]: FOGSnapinHash.service: Scheduled restart job, restart counter is at 764.
Nov 17 11:29:43 host systemd[1]: Stopped FOGSnapinHash.

I check the code in init.php, but the extension mentioned there are loaded in PHP (mysqli, gettext).
I assume it is related to the other error.

@sebastian-roth Unfortunately, this is all I could gather when the issue happened. It happens on every boot, but only on this server.
If you can let me know which logs I could collect, I’ll do that.

I have several Ubuntu 22.04 servers running the current dev-branch of FOG.

On one of these, the boot process loops a while and shows that it is starting and stopping FOGSnapinHash.
See the attached picture:

At some point, it stops doing that and finishes the boot.
FOG works normally then.

What could this be due to? How can I debug it?

@sebastian-roth Thanks for your work, we are in the process of switching to the dev-branch on Ubuntu 22 now and will report any additional bugs we find.

@zfeng +1 We are now moving our servers to Ubuntu 22.04 and rather want to install a somewhat stable release than a working-branch version that may change from day to day.