@Sebastian-Roth unfortunately remote sessions is not an option here, the outside traffic is controled/blocked, beyond my jurisdiction =/
Posts made by mronh
-
RE: Replication Issue
-
RE: Replication Issue
@Sebastian-Roth right…look at this
server side “git checkout replication
Already on ‘replication’
Your branch is up-to-date with ‘origin/replication’.”storage side “git checkout replication
Already on ‘replication’
Your branch is up-to-date with ‘origin/replication’.” -
RE: Replication Issue
@Sebastian-Roth on the server side
ls -al /var/www /var/www/html /var/www/fog /var/www: total 20 drwxr-xr-x 4 root root 4096 nov 12 16:06 . drwxr-xr-x 12 root root 4096 ago 28 13:16 .. drwxr-xr-x 10 www-data www-data 4096 nov 12 16:14 fog drwxr-xr-x 2 root root 4096 ago 28 13:22 html -rw-r--r-- 1 root root 41 out 10 11:10 index.php /var/www/fog: total 408 drwxr-xr-x 10 www-data www-data 4096 nov 12 16:14 . drwxr-xr-x 4 root root 4096 nov 12 16:06 .. drwxr-xr-x 2 www-data www-data 4096 nov 12 16:06 api drwxr-xr-x 2 www-data www-data 4096 nov 12 16:06 client drwxr-xr-x 2 www-data www-data 4096 nov 12 16:06 commons -rw-r--r-- 1 www-data www-data 370070 nov 12 16:06 favicon.ico lrwxrwxrwx 1 www-data www-data 13 nov 12 16:06 fog -> /var/www/fog/ drwxr-xr-x 2 www-data www-data 4096 nov 12 16:06 fogdoc -rw-r--r-- 1 www-data www-data 572 nov 12 16:06 index.php drwxr-xr-x 13 www-data www-data 4096 nov 12 16:06 lib drwxr-xr-x 10 www-data www-data 4096 nov 12 16:06 management drwxr-xr-x 3 www-data www-data 4096 nov 12 16:06 service drwxr-xr-x 2 www-data www-data 4096 nov 12 16:06 status /var/www/html: total 20 drwxr-xr-x 2 root root 4096 ago 28 13:22 . drwxr-xr-x 4 root root 4096 nov 12 16:06 .. lrwxrwxrwx 1 root root 13 ago 28 13:22 fog -> /var/www/fog/ -rw-r--r-- 1 root root 10701 ago 28 13:17 index.html
on the storage side
ls -al /var/www /var/www/html /var/www/fog /var/www: total 16 drwxr-xr-x 4 root root 4096 nov 12 15:59 . drwxr-xr-x 13 root root 4096 jul 18 11:56 .. drwxr-xr-x 10 www-data www-data 4096 nov 12 16:00 fog drwxr-xr-x 2 root root 4096 jul 18 12:03 html /var/www/fog: total 408 drwxr-xr-x 10 www-data www-data 4096 nov 12 16:00 . drwxr-xr-x 4 root root 4096 nov 12 15:59 .. drwxr-xr-x 2 www-data www-data 4096 nov 12 15:59 api drwxr-xr-x 2 www-data www-data 4096 nov 12 15:59 client drwxr-xr-x 2 www-data www-data 4096 nov 12 15:59 commons -rw-r--r-- 1 www-data www-data 370070 nov 12 15:59 favicon.ico lrwxrwxrwx 1 www-data www-data 13 nov 12 15:59 fog -> /var/www/fog/ drwxr-xr-x 2 www-data www-data 4096 nov 12 15:59 fogdoc -rw-r--r-- 1 www-data www-data 572 nov 12 15:59 index.php drwxr-xr-x 13 www-data www-data 4096 nov 12 15:59 lib drwxr-xr-x 10 www-data www-data 4096 nov 12 15:59 management drwxr-xr-x 3 www-data www-data 4096 nov 12 15:59 service drwxr-xr-x 2 www-data www-data 4096 nov 12 15:59 status /var/www/html: total 20 drwxr-xr-x 2 root root 4096 jul 18 12:03 . drwxr-xr-x 4 root root 4096 nov 12 15:59 .. lrwxrwxrwx 1 root root 13 jul 18 12:03 fog -> /var/www/fog/ -rw-r--r-- 1 root root 10701 jul 18 11:56 index.html
I’ll make the git pull to the replic rep e install again on the storage and return here
-
RE: Replication Issue
@Sebastian-Roth Hey man, make the upgrade from booth server and storage ( following ur previous instructions) and now we get some data to think bout. Justo to make it clear, I deleted all images in the storage to force a full replication from begining
I’ll up the logs from server and storage again, but ‘short history’ I see some “Erro fatal: max-retries exceeded (421 There are too many connections from your internet address.)” and “File size mismatch”
Server Side:
2_1542111667714_SERVER_php7.0-fpm.log
1_1542111667714_SERVER_fogreplicator.log
0_1542111667714_SERVER_error.logStorage Side:
1_1542111678025_STORAGE_php7.1-fpm.log
0_1542111678024_STORAGE_error.logBesides that, the improves are really good, logs more accurate and steps of the algorithm way more “solidified” way to go man!
-
RE: Replication Issue
@Sebastian-Roth right. I’ll do it next week then.
this week im on the leash again… oh boy how I hate the end of the year…=/ hahahaha
-
RE: Replication Issue
@Sebastian-Roth sure, right now Im using only the server due this issue.
if all became fixed my summer here in the next months will be sooooo much easier… hahaha
what I have to do?
-
RE: Replication Issue
@Sebastian-Roth Hi again, after a full round of the rep service, here it is:
I uploaded logs from booth server and storage
1_1541078475992_STORAGE_php7.1-fpm.log
0_1541078475992_STORAGE_error.log
1_1541078503271_SERVER_php7.0-fpm.log
0_1541078503271_SERVER_error.log
0_1541078552345_SERVER_fogreplicator.log
0_1541078653124_SERVER_fogreplicator.Sala209RebootRX.transfer.2 - Storage (YYY.YY.210.208).log
-
RE: Replication Issue
@Sebastian-Roth Hello! Right, I’ll make the push and update booth server and node, let the rep service make a full round and return the logs.
cheers
-
RE: Replication Issue
@Sebastian-Roth Hi again, one “dumb” question: why lftp was choosed instead of rsync?
cheers
-
RE: Replication Issue
@Sebastian-Roth ok, theres some defunct processes in the fog server after some replication ( delete parts and etc… )
17914 ? Z 0:00 [sh] <defunct>
20702 ? Z 0:00 [sh] <defunct>
21632 ? Z 0:00 [sh] <defunct>
24658 ? Z 0:00 [sh] <defunct>
27305 ? Z 0:00 [sh] <defunct>
28423 ? Z 0:00 [sh] <defunct>
31260 pts/0 S+ 0:00 grep defunct -
RE: Replication Issue
@Sebastian-Roth yeap, Debian on server, ubuntu on the storage
result=> 7174 pts/0 S+ 0:00 grep --color=auto defunct ( edit: I’ll let the replication process make a full round and then take the result here again)
-
RE: Replication Issue
@Sebastian-Roth yeah, I see this too, i put my expected “top speed” of the lan in the speed limit of the replication config ( inside fog GUI), thinking its a issue with lftp ( with none value, its take a default value instead of limitless)
but, as none changed for better (at least)… I’ll see with the infrastructure guy here, maybe someone (with no knowledge) tried to “fix” some switches / bgp configs
I’ll keep in touch, tks!
-
RE: Replication Issue
@mronh apache error log pasted here cause has “entity error” when tried to upload (1200 lines of it)
[Tue Oct 09 00:06:05.071128 2018] [mpm_prefork:notice] [pid 4797] AH00163: Apache/2.4.34 (Ubuntu) OpenSSL/1.1.0h configured -- resuming normal operations [Tue Oct 09 00:06:05.071170 2018] [core:notice] [pid 4797] AH00094: Command line: '/usr/sbin/apache2' [Tue Oct 09 10:03:55.253531 2018] [proxy_fcgi:error] [pid 16424] [client YYY.YY.211.13:60152] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:03:56.483605 2018] [proxy_fcgi:error] [pid 16420] [client YYY.YY.211.13:60160] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:03:57.679877 2018] [proxy_fcgi:error] [pid 16422] [client YYY.YY.211.13:60168] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:03:58.862615 2018] [proxy_fcgi:error] [pid 16421] [client YYY.YY.211.13:60192] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:04:00.075334 2018] [proxy_fcgi:error] [pid 16423] [client YYY.YY.211.13:60208] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:04:01.293366 2018] [proxy_fcgi:error] [pid 16424] [client YYY.YY.211.13:60224] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:04:02.526186 2018] [proxy_fcgi:error] [pid 16420] [client YYY.YY.211.13:60240] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10:04:03.757965 2018] [proxy_fcgi:error] [pid 16422] [client YYY.YY.211.13:60248] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 10 ........ ......... [Tue Oct 09 13:50:11.661757 2018] [proxy_fcgi:error] [pid 2761] [client YYY.YY.211.13:32864] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Tue Oct 09 13:50:12.895096 2018] [proxy_fcgi:error] [pid 1109] [client YYY.YY.211.13:32872] AH01071: Got error 'PHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 109\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\nPHP message: PHP Notice: A non well formed numeric value encountered in /var/www/fog/status/bandwidth.php on line 110\n' [Wed Oct 10 00:06:37.231462 2018] [mpm_prefork:notice] [pid 4797] AH00171: Graceful restart requested, doing restart AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message
-
RE: Replication Issue
@Sebastian-Roth Hi man, sorry by the new delay ( end of the year put some wood in the fire here =/ )
I’ll attach the log u need, i let the storage node inactive till we find the bug/whatever
thanks in advance dudes
0_1539706517590_php7.1-fpm.log -
RE: Replication Issue
@mronh what im doing now to try to bypass this: set the sleeptime to 7200, and set a bandwitch limit of the replication of the server and storage node to 10000 Kbps ( maybe, just maybe, lftpd set a default value if its not set in the parms, and pull it down limit)
while this, i’ll seek for some bug in my infra structure…
cheers
-
RE: Replication Issue
Hi there, the problem still, sadly… as asked, bellow the logs ( I guess about the time spend X speed of the lan but guessing)
cheers
1_1539088564934_fogreplicator.Sala209RebootRX.transfer.2 - Storage (YYY.YY.210.208).log
-
RE: Replication Issue
@Sebastian-Roth my apologies by the delay in the answer, I had to rollback to 1.5.2 to get back on the business here ( my boss giving me headaches…)
about ur questions:
IMAGEREPSLEEPTIME was 600, put 3600. i’ll try again ( but my net here is a giga conection… a LAN in fact, server and storage in the same switch giga port)
this weekend i’ll update fog and test it again
cheers
-
RE: Replication Issue
@Sebastian-Roth
No good, get the working branch, and still keeping replicating image parts thats already replicate… and worse… deploys made with images in the storage node make corrupted windows inits (goes to the restoration screen)now I disabled the storage node… thinking of rollback to 1.5.3 (or 1.5.2)
-
RE: I'm away, but back?
best luck in the new “settle up”
dont forget to flush DNS before leave :V heheheh
Cheers!
-
RE: Replication Issue
@Sebastian-Roth make the git pull in the server and the node side…
now happens this with the sync
“Type: 8, File: /var/www/fog/lib/service/fogservice.class.php, Line: 484, Message: Undefined variable: itemType”
I’ll make some deploy to test the images
edit: now the ftp server hangs due the max-conections reaches
mirror: d1p5.img: Erro fatal: max-retries exceeded (421 There are too many connections from your internet address.)