System will not image after update: mkfifo error related?

[CBT]

Server

Running version 13
OS: Ubuntu

Description

Issue started after upgrading to the latest Fog version. I will push a Windows installation to a client and fog will initialize like normally, however it will go through partclone and reboot without starting the imaging process. I have noticed an error with the following: “mkfifo: cannot create fifo ‘/tmp/pigz1’ : File Exists”

Additionally, the image management section within the fog web console shows that the image used is 0.00 iB after this failure, whereas it was a couple GiB in size prior to image process.

This behavior happens with all images that we have stored.

[CBT]

@Quazz

Update to 1.4.0 seems to have fixed the issue with imaging.

[george1421]

what did you upgrade from (fog version)?

How did you do the upgrade? Did you migrate from another server to a new one?

[apathetic_admin]

@george1421 I work with @CBT and can help you out here.
I can’t get you exact versions, but I can give you commits from the git log:
We were on 1.3 RC-11 commit 2718a13d2bd11d4d9ccd4be7f2f005a67000da3e and not experiencing the issue. We were on this one for quite a while, and were at a point where we needed functionality, so we stopped pulling new changes from Github.
2-3 weeks ago I updated to the 1.4 RC, commit ae137758eba57b3c8cbbe316142a628020231045 and began experiencing the issue CBT mentioned.
Last Friday, at CBTs request, I pulled again, commit 56c296e9ae2af0e0ba34b12a04b0b8db0a2dc423, which did not resolve the issue.

[george1421]

@apathetic_admin ok I just wanted to make sure we were updating from 1.3.x to 1.4.x so I don’t make the wrong assumption about things.

Well the error message says that a temp file exists in /tmp/pigz1. I would start out with this command ls -la /tmp/p* to see if the file exists and who its owned by. If its there and you are not currently imaging, go and delete it. FOG uses this file to compress or decompress the image being transferred to the target computer. It should not normally exist once imaging has been completed.

[apathetic_admin]

@george1421 That file doesn’t exist.

[george1421]

@apathetic_admin there are no pigz files in the tmp directory?

If you setup a debug deploy (schedule a deploy as you would normally but before you schedule the deploy task, check the debug check box). Then pxe boot the target computer, after a few enter key presses on the target computer you will be dropped to a linux command prompt. At the linux command prompt on the target computer key in fog. This will start the installtion process. You will have to press the enter at each break point.

What I’m guessing is that there is another message that is happening just before this message that we are missing. Also why you are in debug mode we might have the chance to look at the log files on the FOS engine. But lets see if we can trap the error just before the pigz one.

[CBT]

Possible error findings after running “fog” from command line:

Running post init scripts…/bin/fog: line 20: fog.post
init: No such file or directory
Done

mkfifo: cannot create fifo ‘/tmp/pigz1’: File exists
*Clearing ntfs flag…Done
*Resizing ntfs volume (/dev/sda2)…Done

[george1421]

@CBT Well, the postinit error is not important since you are not using them right now.

The error is still the pigz one. If you still have the FOS engine up and running through the imaging script press ctrl-c and then see if there is any other details in /var/logs that give an idea why its doing this.

Also for the devs, please post the configuration for this image you are trying to deploy. This is the settings from image management in the web gui.

Also from the linux server command line include the output of this command ls -la /images/<image_name> replace the <image_name> with the name of the image you are deploying. They will need to see how you have this image setup and if the files match what you have configured.

[CBT]

Reviewed var/logs, was unable to point out anything obvious

Image Management information from webgui:

Image Name: Windows 7 Enterprise December 2016
Operating System: Windows 7- (5)
Image Path: /images/Windows7EnterpriseDecember2016
Image Type: Single Disk - Resizable - (1)
Partition: Everything - (1)
Compression: 9
Protected: False
Image Enabled: True
Replicate?: True

Deploy Method: Partclone
Primary Storage Group: default

[Tom Elliott]

@CBT The error you’re seeing appears because the /tmp/pigz1 file already exists. This, itself, is not a problem at all. What’s interesting, is it doesn’t appear to try imaging. It doesn’t even appear to try mounting anythign?

[Tom Elliott]

@CBT what’s the output of:

cat /images/Windows7EnterpriseDecember2016/d1.{minimum.,}partitions

What’s output of fdisk -l /dev/sda

[CBT]

cat ouput for /images/Windows7EnterpriseDecember2016/d1.{minimum.,}partitions

fdisk -l out for /dev/sda

ls -al /tmp/ produces no results for pigz1

running fog from debug mode shows that the image properly mounts according to what’s outputted. From the host logs on the webgui it’s showing that the regular image and debug image events are successful

[Tom Elliott]

What is “Version 13?”

Did you git checkout working and came up with this?

[Tom Elliott]

For what it’s worth, I think I know why this isn’t working right.

For this image (and possibly others) you should see something in the “Image Manager”.

Based on what I’m seeing (so far) the compression type element is missing from your Image management page?

It should look similar to:

Why is this important when in the past it wasn’t? Because we have zstd, pigz, and uncompressed modes now. In the past, it only assumed partclone with gzip IF the type was not partimage. The check was basically:

case $format in
    1)
        pigz -dc </tmp/pigz1 | partimage restore ${target} stdin -f3 -b 2>/tmp/status.fog
        ;;
    *)
        pigz -dc </tmp/pigz1 | partclone.restore --ignore_crc -O ${target} -N -f 1
        ;;
esac

In the new method, it uses:

case $format in
    [5-6])
         zstdmt -dc </tmp/pigz1 | partclone.restore -n "Storage Location $storage, Image name $img" --ignore_crc -O ${target} -Nf 1
        ;;
    [3-4])
        cat </tmp/pigz1 | partclone.restore -n "Storage Location $storage, Image name $img" --ignore_crc -O ${target} -Nf 1
        ;;
    1)
        pigz -dc </tmp/pigz1 | partimage restore ${target} stdin -f3 -b 2>/tmp/status.
        ;;
    0|2)
        pigz -dc </tmp/pigz1 | partclone.restore -n "Storage Location $storage, Image name $img" --ignore_crc -O ${target} -N -f 1
        ;;
esac

Notice there is no *) line? My guess is the image is not sending the format code when the tasking begins. This should occur automatically if you run an update.

[CBT]

Compression currently set to 9, but there is not Image Manager option with our version it seems,

@apathetic_admin updated to 1.4 RC, commit ae137758eba57b3c8cbbe316142a628020231045 and then we noticed the issue, so he pulled commit 56c296e9ae2af0e0ba34b12a04b0b8db0a2dc423, which did not resolve the issue.

I will see if the update helps, thank you for your assistance.

[Quazz]

@CBT 1.4.0 release is out, it’s newer than the RC versions. I don’t believe there’s a new RC already available, only a working-branch atm.

It would be useful to know why you don’t have this option, though. Anything in the apache error logs?

[CBT]

apache2\error.log produces the following with varied URLs:

PHP Warning: Illegal offset type in /var/www/fog/lib/fog/fogpagemanager.class.php on line 152, referer: http://fog.rmgcom.local/fog/management/index.php?node=*

[apathetic_admin]

@Quazz I changed to the master branch, which is the 1.4.0 release, and we just installed it. @CBT is going to poke around it and he’ll get back with you.

[CBT]

@Quazz

Update to 1.4.0 seems to have fixed the issue with imaging.