Fog 1.6.0.14 Master Storage Node not handing off to Slave Storage Nodes
-
-
Main and all nodes are on version 1.6.0.181
administrator@tv-fog-main:~$ sudo mysql fog -e "SELECT hmPrimary, hmPending, COUNT(*) FROM hostMAC GROUP BY hmPrimary, hmPending; SHOW COLUMNS FROM hostMAC;" [sudo: authenticate] Password: +-----------+-----------+----------+ | hmPrimary | hmPending | COUNT(*) | +-----------+-----------+----------+ | 0 | 0 | 3 | | 1 | 0 | 1295 | +-----------+-----------+----------+ +-----------------+-------------+------+-----+---------+----------------+ | Field | Type | Null | Key | Default | Extra | +-----------------+-------------+------+-----+---------+----------------+ | hmID | int(11) | NO | PRI | NULL | auto_increment | | hmHostID | int(11) | NO | MUL | NULL | | | hmMAC | varchar(59) | NO | UNI | NULL | | | hmDesc | longtext | NO | | '' | | | hmPrimary | tinyint(1) | NO | | 0 | | | hmPending | tinyint(1) | NO | | 0 | | | hmIgnoreClient | tinyint(1) | NO | | 0 | | | hmIgnoreImaging | tinyint(1) | NO | | 0 | | +-----------------+-------------+------+-----+---------+----------------+[Wed Nov 18 00:50:12.685862 2026] [mpm_prefork:notice] [pid 15288:tid 15288] AH00163: Apache/2.4.66 (Ubuntu) OpenSSL/3.5.5 configured -- resuming normal operations [Wed Nov 18 00:50:12.685895 2026] [core:notice] [pid 15288:tid 15288] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND' [Wed Nov 18 07:25:19.794937 2026] [proxy_fcgi:error] [pid 207505:tid 207505] [client 10.220.99.11:63745] AH01071: Got error 'PHP message: FOG storage node availability: the group has no enabled master node. Group Adamsville Storage Group (id 2); node id 4 chosen anyway. The probe is a reachability test, not a test of the image share, so tasking continues -- if the task fails, check that this node is reachable.', referer: https://tv-fog-main.tvschools.local/fog/management/index.php?node=host&sub=list- I don’t recall seeing any errors.
-
-
@FlareImp Thank you, that helps. Your database is correct: 1295 MACs are marked primary. So the flag is lost after FOG reads it. My lab runs the same code and shows the primary MAC, so I need to see what your server sends back.
Please post these:
- Sign in to the FOG UI. Open a host that shows no primary MAC, and note the
id=number in its address bar. Then open this address in the same browser, with that number at the end, and post what the page shows:
https://tv-fog-main.tvschools.local/fog/management/index.php?node=host&sub=getMacaddressesList&id=<id> - Open this address, with one of that host’s MACs, and post what the page shows. It is the boot script the host gets at PXE boot:
https://tv-fog-main.tvschools.local/fog/service/ipxe/boot.php?mac=<mac> - The PHP and database versions, and whether the update upgraded them:
php -v; mysql --version; grep -E " (upgrade|install) .*(php|mysql|mariadb)" /var/log/dpkg.log | tail -20
One separate item from your log: “the group has no enabled master node” for Adamsville Storage Group (id 2). Every storage group needs one enabled master node. Captures upload to the master, and replication copies images from it. Open that storage group’s nodes and set one of them as the master.
- Sign in to the FOG UI. Open a host that shows no primary MAC, and note the
-
{"draw":0,"recordsTotal":1298,"recordsFiltered":1,"truncated":false,"data":[{"id":384,"DT_RowId":"row_384","hostID":383,"hostLink":"(383) - 5260-Test<\/a>","mac":"e4:b9:7a:f0:a2:df","description":"","pending":0,"primary":1,"clientIgnore":0,"imageIgnore":0,"mac_vendor":""}],"_searchtypes":{"id":"num","DT_RowId":"num","hostID":"num","hostLink":"num","mac":"string","description":"string","pending":"num","primary":"num","clientIgnore":"num","imageIgnore":"num","mac_vendor":"string"},"_unadjustednote":"recorded before or near the time this server moved to UTC; it can be in UTC time or in UTC","_lang":"macaddressassociation","recordsReturned":1,"firstUrl":null,"prevUrl":null,"nextUrl":null,"lastUrl":null}-
I got “This page isn’t working HTTP ERROR 500”
PHP 8.5.4 (cli) (built: Oct 19 2026 14:35:18) (NTS) Copyright (c) The PHP Group Built by Ubuntu Zend Engine v4.5.4, Copyright (c) Zend Technologies with Zend OPcache v8.5.4, Copyright (c), by Zend Technologies mysql from 11.8.6-MariaDB, client 15.2 for debian-linux-gnu (x86_64) using EditLine wrapper 2026-11-05 15:24:47 upgrade php-curl:all 2:8.3+93ubuntu2 2:8.5+99ubuntu1 2026-11-05 15:24:48 install php8.5-gd:amd64 <none> 8.5.4-0ubuntu1.5 2026-11-05 15:24:48 upgrade php-gd:all 2:8.3+93ubuntu2 2:8.5+99ubuntu1 2026-11-05 15:24:48 install php8.5-ldap:amd64 <none> 8.5.4-0ubuntu1.5 2026-11-05 15:24:48 upgrade php-ldap:all 2:8.3+93ubuntu2 2:8.5+99ubuntu1 2026-11-05 15:24:50 install php8.5-mbstring:amd64 <none> 8.5.4-0ubuntu1.5 2026-11-05 15:24:50 upgrade php-mbstring:all 2:8.3+93ubuntu2 2:8.5+99ubuntu1 2026-11-05 15:24:51 install php8.5-mysql:amd64 <none> 8.5.4-0ubuntu1.5 2026-11-05 15:24:51 upgrade php-mysql:all 2:8.3+93ubuntu2 2:8.5+99ubuntu1 2026-11-05 15:24:58 upgrade mariadb-common:all 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:24:59 upgrade mariadb-server:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:25:01 upgrade mariadb-server-core:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:25:02 upgrade mariadb-client-core:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:25:02 upgrade mariadb-client:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:25:03 upgrade libmariadb3:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:27:42 upgrade mariadb-plugin-provider-bzip2:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:27:42 upgrade mariadb-plugin-provider-lz4:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:27:42 upgrade mariadb-plugin-provider-lzma:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:27:42 upgrade mariadb-plugin-provider-lzo:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 2026-11-05 15:27:43 upgrade mariadb-plugin-provider-snappy:amd64 1:10.11.14-0ubuntu0.24.04.1 1:11.8.6-5ubuntu0.1 -
-
@FlareImp Thank you. A correction first: I said my lab shows the primary MAC. I had checked the database and the data feed, not the page. On the page, my lab shows the same problem you see.
The MAC tab: your primary MACs are saved correctly. The page fails to show them. When the grid measures its columns, it briefly adds a copy of the table, and that copy unchecks the real radio button. The fix is https://github.com/FOGProject/fogproject/pull/2105. It changes only the page. Your saved data needs no change.
The PXE boot: this is a separate problem. Your
boot.phppage answered with HTTP 500, which means PHP stopped with an error. Please open thatboot.php?mac=address again. Then, right away, post the last 20 lines of/var/log/apache2/error.log. The error should be there. -
de. Group Adamsville Storage Group (id 2); node id 4 chosen anyway. The probe is a reachability test, not a test of the image share, so tasking continues -- if the task fails, check that this node is reachable.', referer: https://tv-fog-main.tvschools.local/fog/management/index.php?node=storagenode&sub=list [Wed Nov 18 08:25:58.622405 2026] [mpm_prefork:notice] [pid 1289:tid 1289] AH00170: caught SIGWINCH, shutting down gracefully AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message [Wed Nov 18 08:26:18.208295 2026] [mpm_prefork:notice] [pid 6900:tid 6900] AH00163: Apache/2.4.66 (Ubuntu) OpenSSL/3.5.5 configured -- resuming normal operations [Wed Nov 18 08:26:18.208351 2026] [core:notice] [pid 6900:tid 6900] AH00094: Command line: '/usr/sbin/apache2 -D FOREGROUND' [Wed Nov 18 08:28:33.963128 2026] [proxy_fcgi:error] [pid 12762:tid 12762] [client 10.220.99.11:56139] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /va r/www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html /fog/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location \\Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boo t\\IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:36.157208 2026] [proxy_fcgi:error] [pid 6916:tid 6916] [client 10.220.99.11:49457] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:36.919035 2026] [proxy_fcgi:error] [pid 7092:tid 7092] [client 10.220.99.11:61872] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:37.413317 2026] [proxy_fcgi:error] [pid 13106:tid 13106] [client 10.220.99.11:58000] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /va r/www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html /fog/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location \\Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boo t\\IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:37.597657 2026] [proxy_fcgi:error] [pid 7092:tid 7092] [client 10.220.99.11:56075] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:37.814380 2026] [proxy_fcgi:error] [pid 6916:tid 6916] [client 10.220.99.11:50029] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:38.053148 2026] [proxy_fcgi:error] [pid 13122:tid 13122] [client 10.220.99.11:62517] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /va r/www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html /fog/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location \\Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boo t\\IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:38.282779 2026] [proxy_fcgi:error] [pid 6914:tid 6914] [client 10.220.99.11:63151] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:38.506414 2026] [proxy_fcgi:error] [pid 13106:tid 13106] [client 10.220.99.11:56121] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /va r/www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html /fog/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location \\Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boo t\\IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:38.883973 2026] [proxy_fcgi:error] [pid 13105:tid 13105] [client 10.220.99.11:53142] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /va r/www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html /fog/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location \\Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boo t\\IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' [Wed Nov 18 08:31:39.088506 2026] [proxy_fcgi:error] [pid 6916:tid 6916] [client 10.220.99.11:61844] AH01071: Got error 'PHP message: PHP Fatal error: Uncaught Exception: No nodes available in /var/ www/html/fog/src/Items/StorageGroup.php:455\nStack trace:\n#0 /var/www/html/fog/lib/plugins/location/src/Items/Location.php(183): FOG\\Items\\StorageGroup->getOptimalStorageNode()\n#1 /var/www/html/f og/lib/plugins/location/src/Hooks/LocationChangeItems.php(204): FOG\\Plugins\\Location\\Items\\Location->getStorageNode()\n#2 /var/www/html/fog/src/Base/HookManager.php(219): FOG\\Plugins\\Location\\ Hooks\\LocationChangeItems->bootItemSettings()\n#3 /var/www/html/fog/src/Boot/IpxeBootMenu.php(921): FOG\\Base\\HookManager->processEvent()\n#4 /var/www/html/fog/service/ipxe/boot.php(86): FOG\\Boot\ \IpxeBootMenu->__construct()\n#5 {main}\n thrown in /var/www/html/fog/src/Items/StorageGroup.php on line 455' -
@FlareImp Thank you, that log shows the cause.
What happens: your hosts belong to a location with “Boot files from” turned on. That location has no Storage Node set, so FOG picks a node from the location’s Storage Group. That group has no storage nodes in it, or it no longer exists. FOG finds no node, and the boot page stops with “No nodes available”. So the host cannot load its boot menu and shows as not registered. I made the same setup on my lab and got the same error.
My earlier suggestion to clear the location’s Storage Node is what sent FOG down this path. I’m sorry for that.
Fix it now: open Location Management, then the location those hosts use. Set its Storage Group to the group that holds the Adamsville nodes, then save. Or set its Storage Node back to the node it had before. Then PXE boot one host.
What I’m changing: a location set up like this should not stop every PXE boot. I’m fixing the location plugin, so it skips that location’s node and writes one line to the error log instead.
-
@Tom-Elliott YAY! that seems to have fixed the issue at Adamsville location! I will test my other locations and let you know if everything is 10/10.
-
@FlareImp Great, thank you for testing it.
Both fixes are now in dev-branch. Run
updatefog.sh --branch dev-branchagain on the main server to get them:- The MAC tab shows the primary MAC as selected.
- A location whose Storage Group has no nodes no longer stops the PXE boot. FOG uses its own node and writes one line to the error log that names the location.
Let me know how the other sites do, and whether the deploys spread across the nodes.
-
@Tom-Elliott Per the original post, I was able to image 12 of the computer at the Adamsville location. 10 of them kicked off using the Master node but the other two sat in the que rather than moving to one of the other two free nodes.
-
@FlareImp Thanks. That should not happen on 1.6.0.181: when a host PXE boots and the master already has 10 tasks, the host should get a free node in the same group. I ran the same setup on my lab, 12 hosts with Max Clients 10, booted one at a time and all at once. Every time, the extra hosts moved to the free node.
When FOG looks for a free node, it skips a node without a message if any of these is true:
- The node is not enabled, or its Max Clients is 0.
- The node is not in the location’s Storage Group.
- The main server cannot reach the node. FOG tests the node’s SSH port (22 unless you changed it) first, then port 443 (or 80 if you use http).
Please check these for the Adamsville storage group:
- In Storage Node Management, for each of the three nodes: Storage Group, Is Enabled, Is Master Node, Max Clients.
- In Location Management, for that location: Storage Group and Storage Node.
- On the main server, run this for each of the two other nodes (replace the IP):
nc -zv -w 1 <node-ip> 22; nc -zv -w 1 <node-ip> 443; nc -zv -w 1 <node-ip> 80If all three fail for a node, FOG treats that node as offline, and that is why the hosts stayed in the queue.