Surface Go 2 (microsoft surface usb-c ethernet adapter) won't boot bzImage on fog 1.5.9.2 and Getting database update errors

[JJ Fullmer]

I just upgraded to fog 1.5.9 (technically 1.5.9.2 from the dev-branch)
I went to image a new surface go and it got stuck on bzImage. I tried updating the bzImage to the latest (that was probably the one included 4.19.143) and there was no change.

I reverted to 4.18.11 (it was the closest to 4.18.6 I saw in the official kernels, which is what I looked for after reading this post https://forums.fogproject.org/topic/12465/microsoft-surface-go-usb-c-to-ethernet-adapter-compatibility/5 probably didn’t need to go that far back )

So maybe we took out the microsoft driver in the latest update? These were imaging fine in 1.5.8

Wait, now I’m getting an error after the image finished…

[JJ Fullmer]

I double checked my sql settings in the .fogsettings file and all was correct.
Not sure what’s going on now with the database update error

[george1421]

Lets have you a few things.

1. Upgrade to FOS Linux 5.6.18 and test.
2. What is the hardware ID of this network adapter (as seen in the device manager in windows)?
3. From the fog iPXE menu did you run the hardware compatibility test? What was the results?

From what it looks like the kernel is running fine, but getting hung up updating the database. If it was a network issue it should have errored out before this point. If we can’t find the root of the issue you may need to schedule a debug deployment and then single step through the deployment until we can see a traceable error.

I assume all other systems deploy fine, its just this specific model that has an issue?

[JJ Fullmer]

@george1421 This was my first deployment since upgrading to fog 1.5.9 from 1.5.8 (technically 1.5.9.2 from 1.5.9 RC 2 I think)

I did switch to the 5.6.18 kernel yesterday and that got things through the imaging as well, but still had the database error. I’m working from home today so I can’t do any testing on the device till tomorrow, when I will gladly give a debug a try. I’m also wondering if there’s a matching init for the 5.6.18 kernel?

I can pull up logs and check the database and such today if it would help.

[Sebastian Roth]

@JJ-Fullmer I am wondering about the Error returned: Machine Details:- and I can’t imagine this to be a specific issue with this Surface device. This is not part of the FOG code as far as I know. Custom script?

[JJ Fullmer]

@Sebastian-Roth I don’t have any custom scripts at the fog level.
I also don’t think this is an issue with the surface device. I need to image another machine today and see if it’s happening everywhere or just on this one machine.
I’m gonna run a debug deploy on the surface this morning and a fresh image on another new machine.

[Sebastian Roth]

@JJ-Fullmer Please run the following command as root on your FOG server: find /var/www/html/fog -type f -exec grep "Machine Details" {} /dev/null \;

[JJ Fullmer]

@Sebastian-Roth said in Surface Go 2 (microsoft surface usb-c ethernet adapter) won't boot bzImage on fog 1.5.9.2 and Getting database update errors:

find /var/www/html/fog -type f -exec grep “Machine Details” {} /dev/null ;

 find /var/www/html/fog -type f -exec grep "Machine Details" {} /dev/null \;
/var/www/html/fog/management/languages/zh_CN.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/pt_BR.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/it_IT.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/es_ES.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/fr_FR.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/de_DE.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/en_US.UTF-8/LC_MESSAGES/messages.po:msgid "Machine Details"
/var/www/html/fog/management/languages/messages.pot:msgid "Machine Details"
/var/www/html/fog/lib/reg-task/taskqueue.class.php:            sprintf("%s:-\n", _('Machine Details')) => '',

[JJ Fullmer]

@Sebastian-Roth and @george1421
Started the debug task and it’s stuck at loading the init.

[Sebastian Roth]

@JJ-Fullmer Ok, I should have looked at the code myself before saying that this is not part of the FOG code. I was wrong! This part of the code is trying to send out an email on finishing a task. Did you play with those settings recently?

Started the debug task and it’s stuck at loading the init.

How long did yo wait? Does it boot beyond that point when running a non-debug task?

[JJ Fullmer]

@Sebastian-Roth I waited until just now and it got into the debug console. I’ve never seen it take 7 minutes to load the init. (full discolsure, I wasn’t watching it the whole time and it may have rebooted itself and worked the second time around)
It also started working as soon as I started deploying on another machine. Gonna wait to see if the other machine gets through without error before starting any debugging. Are there any debug commands that would help in troubleshooting this?

[JJ Fullmer]

@Sebastian-Roth The same errors are happening on the other machine =(
What’s odd is that it’s claiming to be a database issue, but I also am adding the hosts through the pxe menu and they’re getting added to the database just fine.

[JJ Fullmer]

@Sebastian-Roth Another thing I just noticed is that the debug mode loaded from the main server (i.e. the task in the gui showed ‘working with fogServer’) But the other deploy task was working with my storage node

edit: This is probably nothing, the error happened in both cases.

[JJ Fullmer]

@Sebastian-Roth @george1421
Ran the debug task.
It gets through everything normal all the way up to Task Complete
then it says
Updating Database.........................Failed
Then it repeats the Error Returned: Machine Details:- repeatedly. (It’s hard to count them with shift+ page up/page down but I believe it’s about 10 retries for updating the database if that helps to find where this is happening in the code)

[JJ Fullmer]

@Tom-Elliott @Sebastian-Roth @george1421

I gave something else a try. I looked at the commits included in the dev_branch since 1.5.9 and found that Tom had added the machine details array in the taskqueue.class.php file.

First thing I tried was adding a primary user to the host, in case that being empty was somehow a problem. No dice.

Looking at it, it’s just a prettier more informative email array. But the parameters in that object must be used somewhere else, because I reverted the file to its last commit (made a .old version of it on my server, made a new version of it with the previous version’s contents) just to see if that change maybe broke something.

My debug task was waiting for me to hit enter at another updating database attempt, so I hit enter, and viola!

So now we gotta figure out why in the world that change in the email sent would break the database update to say the task is completed.

edit:
I debugged it a bit further. Restored the newer version of the file and started commenting things out one at a time to confirm the problem.

The code here is where the problem lies (line 293-315 of taskqueue.class.php found at https://github.com/FOGProject/fogproject/blob/99bb17b168e929796a8d76526f73bcfdac11bb4b/packages/web/lib/reg-task/taskqueue.class.php)

$Inventory = self::$Host->get('inventory');
        $mac = self::$Host->get('mac')->__toString();
        $ImageName = $this->Task->getImage()->get('name');
        $ImageStartTime = self::niceDate($this->Task->get('checkInTime'));
        $ImageEndTime = self::niceDate();
        $duration = self::diff($ImageStartTime, $ImageEndTime);
        $Snapins = implode(',', (array)$SnapinNames);
        $email = array(
            sprintf("%s:-\n", _('Machine Details')) => '',
            sprintf("\n%s: ", _('Host Name')) => self::$Host->get('name'),
            sprintf("\n%s: ", _('Computer Model')) => $Inventory->get('sysproduct'),
            sprintf("\n%s: ", _('Serial Number')) => $Inventory->get('sysserial'),
            sprintf("\n%s: ", _('MAC Address')) => $mac,
            "\n" => '',
            sprintf("\n%s: ", _('Image Used')) => $ImageName,
            sprintf("\n%s: ", _('Snapin Used')) => $Snapins,
            "\n" => '',
            sprintf("\n%s: ", _('Imaged By')) => $engineer,
            sprintf("\n%s: ", _('Imaged For')) => $primaryUser,
            sprintf("\n%s: ", _('Imaging Started')) => $ImageStartTime,
            sprintf("\n%s: ", _('Imaging Completed')) => $ImageEndTime,
            sprintf("\n%s: ", _('Imaging Duration')) => $duration
        );

If I comment out the last 3 pieces of the array and try again, the database updates and all is well again

$Inventory = self::$Host->get('inventory');
        $mac = self::$Host->get('mac')->__toString();
        $ImageName = $this->Task->getImage()->get('name');
        $ImageStartTime = self::niceDate($this->Task->get('checkInTime'));
        $ImageEndTime = self::niceDate();
        $duration = self::diff($ImageStartTime, $ImageEndTime);
        $Snapins = implode(',', (array)$SnapinNames);
        $email = array(
            sprintf("%s:-\n", _('Machine Details')) => '',
            sprintf("\n%s: ", _('Host Name')) => self::$Host->get('name'),
            sprintf("\n%s: ", _('Computer Model')) => $Inventory->get('sysproduct'),
            sprintf("\n%s: ", _('Serial Number')) => $Inventory->get('sysserial'),
            sprintf("\n%s: ", _('MAC Address')) => $mac,
            "\n" => '',
            sprintf("\n%s: ", _('Image Used')) => $ImageName,
            sprintf("\n%s: ", _('Snapin Used')) => $Snapins,
            "\n" => '',
            sprintf("\n%s: ", _('Imaged By')) => $engineer,
            sprintf("\n%s: ", _('Imaged For')) => $primaryUser
            //sprintf("\n%s: ", _('Imaging Started')) => $ImageStartTime,
            //sprintf("\n%s: ", _('Imaging Completed')) => $ImageEndTime,
            //sprintf("\n%s: ", _('Imaging Duration')) => $duration
        );

[george1421]

Mod Note: Moving this over to the Bug Reports forum.

I know that was a late add-in feature just before 1.5.9 was released. Its strange how/why it is impacting imaging through.

@Developers thoughts?

ref: https://forums.fogproject.org/topic/14749/feature-modification-request-image-task-email-addition

[Sebastian Roth]

@JJ-Fullmer Thanks heaps. Somehow must have missed that one in the last days. I am onto this and will get back to you soon.

@george1421 It’s no in 1.5.9 but was added just after he release.

[Sebastian Roth]

@JJ-Fullmer Please try the following: Add back the three lines you commented and add some ->format(...) code to lines 296 and 297 to make them look like this:

...
        $ImageStartTime = self::niceDate($this->Task->get('checkInTime'))->format('Y-m-d H:i:s');
        $ImageEndTime = self::niceDate()->format('Y-m-d H:i:s');
...

I will push a fix soon!

[Tom Elliott]

@Sebastian-Roth I’ve added the change just to keep it up to snuff. Sorry I missed that part. @JJ-Fullmer and @Sebastian-Roth

[george1421]

@Sebastian-Roth said in Surface Go 2 (microsoft surface usb-c ethernet adapter) won't boot bzImage on fog 1.5.9.2 and Getting database update errors:

I will push a fix soon!

So for fog admins that have this issue would the instructions be to do a git pull on the local fog repo and then rerun the installer?