Failed to upload image (Process Hung)

[Keating178]

All,

We are having an issue where we start the upload of a new image to our FOG server and it starts the process but then it hangs until we manually kill it.

There is no explanation for it and the logs do not stand out in a way that tells me what is going on.

Has anyone had this issue, or have an idea on how we can fix it?

Log:

The time out line could have something to do with it but this all worked fine up until recently without issues.

I verified, checked, and re-established my mount points and the all work fine. I verified my images, and the directory and I can write to them with no issues. This leads me to think the mount point is fine .

[george1421]

What do you see on the screen of the target computer when its hangs. The logs really doesn’t give us a good picture of what is wrong. The FOS Linux error screen will.

[Keating178]

I will upload a few screenshots when my tech comes back and hopefully that will help out a bit.
As far as the task goes, it just keeps spinning.

[george1421]

@Keating178 Yes the task will keep spinning unless it gets an all done message or a stat update message from the target computer.

In fog imaging the target computer (and FOS Linux running on it) does all of the work. The fog server sends the image between the network adapter and local storage and manages the overall process, but the FOS Linux engine running on the target is where the action is. It will report actually what is going on or wrong.

[Sebastian Roth]

@Keating178 As George says, we definitely need pictures of what’s on the host’s screen to get an idea what’s wrong.

About the timeout messages you have: For me this sound like one of your drives/partitions is not found. It tries to mount the disk/partition and fails. This is probably something happening on server boot up and doesn’t have to do with the image hang I suppose. Unless this is the disk where you had stored your images…

Please run the following commands and post a screenshot here:

cat /etc/fstab
df -h
lsblk

[Keating178]

Sorry for the delay, this is what is happening on the client side.

[Keating178]

@Sebastian-Roth

cat /etc/fstab

lsblk

df -h

[Sebastian Roth]

@Keating178 Ok, the timeout error is just complaining about the SWAP partition UUID not matching. That shouldn’t cause the capture issue. If you want to fix that you can edit /etc/fstab and put in /dev/sda3 instead of UUID=a5ff26a8-1f8b-4659-b897-e4cead8414e5 in line 13. Just be aware that messing things up in /etc/fstab can lease you with an unbootable OS. So do this with caution and on your own risk.

Now about the “Invalid Storage Group” error. Please take a look at the apache and PHP-FPM error logs (see my signature) so see if there are any error messages in the logs while this happens.

As well login to the web UI, open the Storage Node and Storage Group view and edit those. Save the settings and then try again.

[Keating178]

This is all my “Storage” section shows, should I add another one and see if that helps?

Apache2 logs:
[Wed Nov 06 06:25:04.992901 2019] [mpm_prefork:notice] [pid 1295] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured – resuming normal operations
[Wed Nov 06 06:25:04.992969 2019] [core:notice] [pid 1295] AH00094: Command line: ‘/usr/sbin/apache2’
[Wed Nov 06 10:37:55.921388 2019] [proxy_fcgi:error] [pid 62954] [client 10.200.208.38:53305] AH01071: Got error ‘PHP message: PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n’, refe$

I am not sure what it is telling me here and if it is an error or just a warning.

PHP-FPM Logs:
x

[Sebastian Roth]

@Keating178 said in Failed to upload image (Process Hung):

This is all my “Storage” section shows, should I add another one and see if that helps?

No need to add a new one. Just click the “default” Storage Group here and click Update to save. As well click “All Storage Nodes” on the left -> “DefaultMember” -> Update.

Thanks for the logs doesn’t seem like there is anything of value in that. Though You might want to check an older file because the log file might have been rotated already:

cat /var/log/apache2/error.log.1
zcat /var/log/apache2/error.log.2.gz

[Keating178]

@Sebastian-Roth said in Failed to upload image (Process Hung):

cat /var/log/apache2/error.log.1

I have clicked update on the storage options and I have listed the logs below from where you mentioned.

error.log.1:

[Wed Nov 06 06:25:04.992901 2019] [mpm_prefork:notice] [pid 1295] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Wed Nov 06 06:25:04.992969 2019] [core:notice] [pid 1295] AH00094: Command line: '/usr/sbin/apache2'
[Wed Nov 06 10:37:55.921388 2019] [proxy_fcgi:error] [pid 62954] [client 10.200.208.38:53305] AH01071: Got error 'PHP message: PHP Warning:  Illegal offset type in /var/www/fog/lib/fog/fogmanagercontroller.class.php on line 296\n', refe$
[Thu Nov 07 06:25:04.913077 2019] [mpm_prefork:notice] [pid 1295] AH00171: Graceful restart requested, doing restart

zcat error.log.2.gz:

[Tue Nov 05 06:25:04.838487 2019] [mpm_prefork:notice] [pid 1295] AH00163: Apache/2.4.41 (Ubuntu) OpenSSL/1.1.1c configured -- resuming normal operations
[Tue Nov 05 06:25:04.838555 2019] [core:notice] [pid 1295] AH00094: Command line: '/usr/sbin/apache2'
[Tue Nov 05 15:32:27.701982 2019] [proxy_fcgi:error] [pid 17035] [client 10.200.208.71:64260] AH01071: Got error 'PHP message: PHP Warning:  array_diff(): Argument #2 is not an array in /var/www/fog/lib/fog/fogcontroller.class.php on line 1080\nPHP message: PHP Warning:  Invalid argument supplied for foreach() in /var/www/fog/lib/fog/fogcontroller.class.php on line 1100\nPHP message: PHP Warning:  array_diff(): Argument #2 is not an array in /var/www/fog/lib/fog/fogcontroller.class.php on line 1080\nPHP message: PHP Warning:  Invalid argument supplied for foreach() in /var/www/fog/lib/fog/fogcontroller.class.php on line 1100\nPHP message: PHP Warning:  array_diff(): Argument #2 is not an array in /var/www/fog/lib/fog/fogcontroller.class.php on line 1080\nPHP message: PHP Warning:  Invalid argument supplied for foreach() in /var/www/fog/lib/fog/fogcontroller.class.php on line 1100\n'
[Wed Nov 06 06:25:04.699292 2019] [mpm_prefork:notice] [pid 1295] AH00171: Graceful restart requested, doing restart

[Keating178]

@Sebastian-Roth

I haven’t heard back from anyone on the issues that I am having. I have tried verifying all storage paths both in the GUI as well as the server shell and all is working correctly.

Were you able to see anything from the logs I uploaded last?

[Tom Elliott]

@Keating178 The Invalid Storage Group error you were seeing should be fixed if you can upgrade to the latest within the dev-branch from GIT.

Hopefully this will fix the issue you were seeing too.