[RC19] Snapin Replication not working

mp12

Server

FOG Version: 1.3.0 RC-19 (6003)
OS: Ubuntu 14.04

Client

Service Version: 0.11.5
OS: Win7

Description

The FOGSnapinReplication does not replicate new created snapins to existing nodes.

Here my fogsnapinrep.log

[11-04-16 11:15:30 am]
        ___           ___           ___
       /\  \         /\  \         /\  \
      /::\  \       /::\  \       /::\  \
     /:/\:\  \     /:/\:\  \     /:/\:\  \
    /::\-\:\  \   /:/  \:\  \   /:/  \:\  \
   /:/\:\ \:\__\ /:/__/ \:\__\ /:/__/_\:\__\
   \/__\:\ \/__/ \:\  \ /:/  / \:\  /\ \/__/
        \:\__\    \:\  /:/  /   \:\ \:\__\
         \/__/     \:\/:/  /     \:\/:/  /
                    \::/  /       \::/  /
                     \/__/         \/__/

  ###########################################
  #     Free Computer Imaging Solution      #
  #     Credits:                            #
  #     http://fogproject.org/credits       #
  #     GNU GPL Version 3                   #
  ###########################################

[11-04-16 11:15:30 am] Interface Ready with IP Address: X.X.X.X
[11-04-16 11:15:30 am] Interface Ready with IP Address: XXX
[11-04-16 11:15:30 am]  * Starting SnapinReplicator Service
[11-04-16 11:15:30 am]  * Checking for new items every 600 seconds
[11-04-16 11:15:30 am]  * Starting service loop
[11-04-16 11:15:30 am]  * Starting Image Replication.
[11-04-16 11:15:30 am]  * We are group ID: 1. We are group name: default
[11-04-16 11:15:30 am]  * We are node ID: 1. We are node name: fog
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: BIM_Modul_1_IFC_Files
[11-04-16 11:15:30 am]  | I am the only member
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: bios_upgrade
[11-04-16 11:15:30 am]  | I am the only member
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: delete fog.log
[11-04-16 11:15:30 am]  | I am the only member
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: gpupdate
[11-04-16 11:15:30 am]  | I am the only member
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: Install CUDA 7.5
[11-04-16 11:15:30 am]  | I am the only member
[11-04-16 11:15:30 am]  * Not syncing Snapin between group(s)
[11-04-16 11:15:30 am]  | Snapin Name: InstallBuildUp19
[11-04-16 11:15:30 am]  | I am the only member

I am wondering about the entry in line six [11-04-16 11:15:30 am] * Starting Image Replication.

Tom Elliott

@mp12 There were two things going on here, which have no been republished as RC-20 to ensure all will be well.

I renamed a variable for ease, but where it set’s the snapin name was calling the “bad” variable (which was unset) leaving the snapin without a name. All other things worked right (the upload, args, etc…) but would report failed because it was setting the name to a null value which is not allowed.

The ID you see shows it actually did get created though.

The error message you posted had nothing to do with the create of the snapin, but rather the creating of the “File Exists” listing. My isAvailable method was working properly but not removing the requests from the url queue after it returned the available status. Both of these things are now fixed in RC-20 if you’d be so kind as to repull and reinstall.

Tom Elliott

The line six in non-issue. It’s just that the replication and snapin replication use the same method. I can fix it, but this is not the issue itself.

Can you provide any apache error logs? On ubuntu they’re located under: /var/log/apache2/error.log as well I believe Ubuntu 14 still used upstart (while still also used for 16 it’s easier to view things under systemd with systemctl -l status FOGSnapinReplicator) so this may be the ONLY way I can help find things.

Tom Elliott

Also, is this the whole log?

mp12

@Tom-Elliott
This is the whole log file.

apache2/error.log

[Fri Nov 04 06:39:02.854200 2016] [mpm_prefork:notice] [pid 24852] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2j configured -- resuming normal operations
[Fri Nov 04 06:39:02.854222 2016] [core:notice] [pid 24852] AH00094: Command line: '/usr/sbin/apache2'
[Fri Nov 04 11:14:07.491123 2016] [:error] [pid 6565] [client x.x.x.x:45734] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:07.491156 2016] [:error] [pid 6565] [client x.x.x.x:45734] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:07.579862 2016] [:error] [pid 2684] [client x.x.x.x:46615] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:07.579902 2016] [:error] [pid 2684] [client x.x.x.x:46615] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:08.256304 2016] [:error] [pid 5324] [client x.x.x.x:46661] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:08.256328 2016] [:error] [pid 5324] [client x.x.x.x:46661] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195, referer: http://x.x.x.x/fog/management/index.php?node=home
[Fri Nov 04 11:14:08.625397 2016] [:error] [pid 6958] [client x.x.x.x:50333] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:08.625415 2016] [:error] [pid 6958] [client x.x.x.x:50333] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:08.646214 2016] [:error] [pid 2418] [client x.x.x.x:50334] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:08.646233 2016] [:error] [pid 2418] [client x.x.x.x:50334] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:08.717218 2016] [:error] [pid 2947] [client x.x.x.x:50335] PHP Warning:  PDO::query(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:08.717240 2016] [:error] [pid 2947] [client x.x.x.x:50335] PHP Warning:  PDO::query(): Error reading result set's header in /var/www/fog/lib/db/pdodb.class.php on line 195
[Fri Nov 04 11:14:10.285651 2016] [:error] [pid 6981] [client x.x.x.x:30723] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 151
[Fri Nov 04 11:14:10.297790 2016] [:error] [pid 6564] [client x.x.x.x:30724] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 151
[Fri Nov 04 11:14:10.309616 2016] [:error] [pid 30359] [client x.x.x.x:30725] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 151
[Fri Nov 04 11:14:13.589262 2016] [:error] [pid 27742] [client x.x.x.x:52422] PHP Warning:  PDO::__construct(): MySQL server has gone away in /var/www/fog/lib/db/pdodb.class.php on line 151
[Fri Nov 04 11:14:16.699779 2016] [core:notice] [pid 24852] AH00051: child pid 2947 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Fri Nov 04 11:14:16.699840 2016] [core:notice] [pid 24852] AH00051: child pid 5324 exit signal Segmentation fault (11), possible coredump in /etc/apache2
[Fri Nov 04 11:14:16.699877 2016] [mpm_prefork:notice] [pid 24852] AH00169: caught SIGTERM, shutting down
[Fri Nov 04 11:14:51.672284 2016] [mpm_prefork:notice] [pid 9057] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2j configured -- resuming normal operations
[Fri Nov 04 11:14:51.672320 2016] [core:notice] [pid 9057] AH00094: Command line: '/usr/sbin/apache2'

Tom Elliott

@mp12 When I meant “whole log” i was referring to the replication log (just trying to clarify)?

mp12

@Tom-Elliott
I also checked the imagereplication.log. It seems that everything is okay.

I tried to start the FOGSnapinReplicator via php command.
Here is what I get

sudo /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator

PHP Fatal error:  Uncaught Error: Call to a member function get() on null in /var/www/fog/lib/service/snapinreplicator.class.php:203
Stack trace:
#0 /var/www/fog/lib/service/snapinreplicator.class.php(271): SnapinReplicator->_commonOutput()
#1 /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator(34): SnapinReplicator->serviceRun()
#2 {main}
  thrown in /var/www/fog/lib/service/snapinreplicator.class.php on line 203

I removed all snapins which followed after “InstallBuildUp19” (see first fogsnapinrep.log) and everything is replicating and working again.
Every new created snapin will crash the replication.

Tom Elliott

@mp12 Mind jumping on the working-RC-20 branch and see if this is now cleared?

Basically I’m an idiot and you should slap me…

The image replicator and snapin replicator services are essentially clones of one another. As imaging replication has been around for a lot longer, it’s relatively simple to know that I more or less copied that service and made modifications to make it “snapin” capable. In one of my many refactors, I forgot to adjust some of the copied lines (only two really I can find currently) that related to actually wanting the snapin, vs, image. I had a line (and you can confirm if you’d like first) at line 203 of /var/www/fog/lib/service/snapinreplicator.class.php that probably has something like $Image->get( when it really should’ve been $Snapin->get.

I’m sorry :(.

mp12

@Tom-Elliott

No one is going to slap you! Thanks for the great support. I update to RC-20.

Unfortunately I am get a new error when creating a snapin.

Snapin ID 110 is not valid

/var/log/apache2/error.log shows:

[Mon Nov 07 07:57:40.073360 2016] [:error] [pid 14613] [client x.x.x.x:28288] PHP Warning:  array_map(): Argument #2 should be an array in /var/www/fog/lib/fog/storagenode.class.php on line 216, referer: http://x.x.x.x/fog/management/index.php?node=snapin

Tom Elliott

@mp12 There were two things going on here, which have no been republished as RC-20 to ensure all will be well.

I renamed a variable for ease, but where it set’s the snapin name was calling the “bad” variable (which was unset) leaving the snapin without a name. All other things worked right (the upload, args, etc…) but would report failed because it was setting the name to a null value which is not allowed.

The ID you see shows it actually did get created though.

The error message you posted had nothing to do with the create of the snapin, but rather the creating of the “File Exists” listing. My isAvailable method was working properly but not removing the requests from the url queue after it returned the available status. Both of these things are now fixed in RC-20 if you’d be so kind as to repull and reinstall.

mp12

@Tom-Elliott

Thanks Tom!
Replication works fine again. I also deleted the snapin with ID 110 manully from the db.

mp12

@Tom-Elliott

Server

list itemFOG Version: 1.3.0 RC-21 (6014)
list itemOS: Ubuntu 14.04

Client

Service Version: 0.11.5
OS: Win7

Description

Replication stopped working
The Snapins which should get replicated won’t appear in the fogsnapinrep.log file.

Tom Elliott

@mp12 Should be re-fixed in the latest.

Had a cleanup bit for 21 and missed the count grading system. This should be corrected for now in the working-RC-22 branch (which if I get a test I will push up probably this evening.)

Tom Elliott

RC-22 officially released.

I’m seeing stuff replicating once again.

Thanks for informing.

mp12

@Tom-Elliott

Nodes are not replicating yet. Restarted the FOGSnapinReplication but nothing new.
fogsnapinrep.log shows only existing snapins but not new created ones.
apache2/error.log shows nothing new.

Tom Elliott

@mp12 Is the snapins set to replicate?

Thanks for reminding me on this. I realized recently (but had bigger bugs to tackle first) that the replicate checkbox is not set on fresh creations.

mp12

@Tom-Elliott

Found the error. After creating a new snapin and enabling it on creation, I rechecked the snapin and found that the checkbox Snapin Enabled was empty. Set a check mark, restarted replication and it uploaded the snapisn to the nodes.

Tom Elliott

I found the isEnabled issue. This was only present on new snapins. It’s fixed for RC-23 (while annoying it’s not as important as the functionality of the client which I’m more focused on anyway, among the other issues RC-22 was released to help correct).

[RC19] Snapin Replication not working

Server

Client

Description

300

11.7k

17.1k

154.7k