Posts made by EuroEnglish

@george1421 I will take care of that, but it might be tomorrow. We just handed out 150 laptops a couple of days ago, but due to the FOG issues we were forced to hand the machines out without all the software and modifications required, now we are busy trying to make GPO and Intune changes to make the modifications we couldn’t push as an image. As soon as we are caught up I will disassemble one of the laptops to remove the drive and test again.

Like I said though, I did take a perfectly good Windows 10 machine, FOG menus were responding well, pushed Windows 11 to it, then put it back onto the FOG server - The menus were unusable. The only change was the OS, BIOS was untouched. Bitlocker wasn’t on before and isn’t on in the image on the server. We created a new image for the new FOG server and are seeing the same results.

I am perplexed as you are, how FOG menus are being manipulated by the OS I have no idea, but I will run the requested tests and let you know. Removing the drive will be an interesting test to see what might be going on.

@JJ-Fullmer We are running the FOG alpha-branch version 1.5.10.29 (it was the most recent at the time of installation), I upgraded to this as we were having the same problem on 1.5.9 and 1.5.10 regular. It is the Fog menu that grinds to a halt when deploying an image, it takes 3 to 4 minutes to switch to deploy and then enter a username/password, after that, the image pushes normally. We don’t register hosts, we simply PXE boot and deploy whichever image we need.

The L13s use USB-C Ethernet adapters, the same ones are used no matter what machine, current OS, or deployed OS choice. MAC address pass-thru is not enabled. We have the BIOS preconfigured by Lenovo, including all security passwords and boot options (The only thing they missed was disabling secure boot, so we manually turn that off)

FOG kernel is DefaultMeember FOG Version 1.5.10.29, not sure what Initrd version we are using, but it wasn’t changed from stock installation on any of the FOG servers. iPXE is 1.21.1+

At first, I thought it was my Fog server, but I spun up my old one, updated everything, and ran into the same problem. So I built a new Ubuntu VM, added Webmin for ISC-DHCP, and added a clean version of Fog, but still the same.

If I PXE into the server with any other OS - Win10, Linux, no OS (blank drive) everything responds perfectly, however, if the host connecting via PXE has Win11 on it, the whole system bogs down to where it is almost impossible to use. To test further, I pulled out an older laptop with Windows 10 (L13 Yoga from two years ago), it loaded fine and FOG was behaving normally, pushed Windows 11 and then reconnected it to FOG, and the system ground to a halt. I unboxed a brand-new laptop (also Lenovo L13 Yoga) that came from the factory with Windows 11 and the system slowed down to a crawl. I added Windows 10 to the brand-new machine and FOG behaved normally again.

Does FOG simply not play well with Windows 11? Or am I missing something stupid? Three FOG servers that work with every machine that isn’t Windows 11, but fail with any machine that is.

So, I have fixed the login issues, apparently installing the Dev Branch cured that problem. However, it now fails after selecting which image to push. I am receiving a few messages:

Could not select: Exec format error (https://ipxe.org/2e008081)
Could not boot: Exec format error (https://ipxe.org/2e008081)
Chainloading failed, hit ‘s’ for the for the iPXR shell; reboot in 10 seconds

Those errors appear no matter what image I select, or laptop I use. All the images and the laptops were working before this issue started.

I realized that I didn’t include any version information:

Ubuntu now version - 20.04.6 LTS
Fog now (Master) - 1.5.10.29

Also, trying to image a computer brings up the regular FOG screen, PXE is working fine, but then it locks up - You can kind of keep hitting down to get to deploy, but even that takes about 5 minutes, entering a username password has proven impossible as it lags so much.

I updated both Fog and Ubuntu today, then restarted the server (Hyper-V VM) and the regular login page came up, but none of my accounts let me log in. I checked the login/var/www/fog and have a list of errors saying simply that “username:fog failed to login” (and other usernames that were attempted.

All the accounts were working prior to the update. I checked MySQL and reset the password for FOG, no change. I created a new user in MySQL and that is unable to login either. I have plenty of free space, RAM is at about 50% max usage. Everything has been working perfectly for a few years, but now we are moving to Windows 11 I figured it was time to upgrade. I am at a loss, any suggestions?

@george1421
I would imagine that the most limiting factor would be the 1Gb limitation on the master server NIC itself. I looked into bonding a couple of NICs on my master server, however most reviews seems to suggest that this actually causes problems and can slow things down - As they would share a common IP address in Ubuntu/Fog, but two different IP addresses in the Hyper-V host computer it is running on, therefore confusion about packets sending and receiving out of order.

Ultimately the rest of the network is running 1Gb fiber as well, so even increasing the server to 2 bonded 1Gb NIC’s would still bottleneck.

I guess at this stage I am running about as fast a possible with Fog, and it is much faster than my old WDS system. I can remove all the storage nodes, as we have a single campus and deploy without registering clients, so everything will go to the master server anyway. All I need to do is make sure that I make a good backup of my Hyper-V virtual machine, then make checkpoints before doing any updates to both Ubuntu and/or Fog server, that way I am covered in the event of a crash.

This imaging solution is far superior to WDS and Ghost, creating and capturing images is easier and faster, deploying is much faster. It will save weeks of work over this summer alone, and moving forwards it will certainly get better and better. I only wish I had know about this a few years ago, I would have far less grey hairs

I am going to mark this as resolved, given that I was trying to do something it simply wasn’t built to do that way. However I really want to thank both you and Wayne for your help, it has helped find and resolve some other issues I didn’t even know I had. Plus given me a better understanding of Ubuntu and Fog, which is just as important for a newbie like myself to both systems.

Regards

Peter

@Wayne-Workman

Hi Wayne,
I just created a simple multicast session, a great suggestion from George, for 6 hosts as a test, it ran really well and seems much faster than 6 individual unicast sessions. I am wondering how Fog processes multicast sessions, when the multicast was running I monitored the Fog Management home page and noticed that my main server was the only thing showing transmit data, however both my 2 storage nodes seemed to be receiving the same amount of data. Does multicast push from the main server to the hosts? Or, does the main server push to the storage nodes and they distribute the session? My bandwidth was fluctuating between 500 Mbps and 3000 Mbps, which tells me that all three servers must be pooling the multicast session, but maybe the bandwidth monitor on the dashboard isn’t accurate?

I am trying to work out, once I start deploying 20 or more hosts at a time, what the best method would be. If multicasting utilizes the normal server and all storage nodes I can create a larger number of storage nodes to increase the combined available bandwidth. If multicasting only uses the normal server though, I may as well not have more than one storage node (a good way to make sure I have backups of images in case of normal server failure).

I am not finding any information online about how the multicasting distributes workload and this could define whether smaller unicast deploys or larger multicast would work better for me.

Thanks again for your help, really helped on the issues the Wiki didn’t cover.

Regards

Peter

Thanks again for all your help with this.

Peter

@george1421

Hi George,
I just created a simple multicast session for 6 hosts as a test, it ran really well and seems faster than 6 individual unicast sessions. I am wondering how Fog processes multicast sessions, when the multicast was running I monitored the Fog Management home page and noticed that my main server was the only thing showing transmit data, however both my 2 storage nodes seemed to be receiving the same amount of data. Does multicast push from the main server to the hosts? Or, does the main server push to the storage nodes and they distribute the session? My bandwidth was fluctuating between 500 Mbps and 3000 Mbps, which tells me that all three servers must be pooling the multicast session, but maybe the bandwidth monitor on the dashboard isn’t accurate?

Thanks again for all your help with this.

Peter

@Wayne-Workman

Hi Wayne,
After a few small issues sorting out user names and passwords, I reran the Fog Installer and then set the ownership/permissions. That all looks good now. Thanks for that help.

I ran into a problem with my second storage node and deleted it from Storage Management, then added it back in, which put it at the top of the list in “All storage nodes”. Now when I deploy to the 6 computers all of them deploy from this storage node, previously all pulled from my main server (which was at the top of the storage management list before).

I think that my problem is that I am misunderstanding the ‘Max Clients’ option, I presumed that it would send an image to the host from one node, until it hits the max client number, then start sending from the next node automatically, and so on. I am not registering my hosts and simply selecting deploy, therefore it seems to be ignoring settings as the hosts aren’t running as clients. Could that be right?

Thanks again

Peter

@Wayne-Workman

I have deleted the ‘Default’ storage group, it will never be used and so I guess there is no point leaving it there. I will add another group, ‘Teacher’, to separate the different images, but that can wait until I have everything running correctly. That should clear up those log file entries I would imagine.

@george1421

Well spotted on the owner difference, I didn’t even notice that one - Strange as it pretty obvious now you mentioned it. All the image replication was done by Fog itself, however it could be that I messed something up when I was setting up the different users on the different servers - Again, could be part of the problem.

@george1421

Based on your thoughts about the Image replication log files I used Webmin to check the Images folder on each server. It seems that you might be correct, it shows that there is a small file size difference between the main server and the two storage servers. It also seems that rights didn’t transfer the same, both storage servers have different rights to the files.

I marked each file manager with the server for reference:

@george1421

Thanks for the idea regarding multicasting, I will look into setting that up in the Wiki - Seems almost everything you would ever need about Fog is there somewhere. We were running WDS on a Windows server for the last few years, but something went wrong and it will not capture new images. Still deploys, but not a good idea to have old images that take longer to update than it did to deploy the image

I was researching how to fix the WDS server and came across a forum for Fog, this solution is a life-saver, no more sysprep and capturing images is much easier. I also find that deployment is much faster with Fog. We image the machines over the summer break, working in a high school, and bandwidth isn’t much of a problem, individual 1Gb Ethernet to each server and fiber point to point between the server room switches and my imaging location switches. However, I agree that multicasting might still speed things up.

@Wayne-Workman

Hi Wayne,
Here is the location management screenshot.

@Wayne-Workman

Hi Wayne,
Sorry about that, I have now taken a screenshot of that section and attached it. I also checked the other 2 images I have and all are the same as the one attached.

@Wayne-Workman

Hi Wayne,
I have attached the screenshots you requested and also the most recent Image Replication log file, in case that shows any issues. Right now it isn’t a problem, as I am only testing a few machines at a time, however once summer arrives I need to deploy around 300 laptops. I am hoping that setting up a main server and 3 storage nodes will allow me to push around 20 at a time without too much slow down (5 per node).

0_1491952936259_Fog Image Replicator Log.txt

@george1421

Hi George,
It seems that the hosts are still deploying from the main server only, I attached a screenshot showing the bandwidth and it shows that only FOG-Ubuntu (my normal server) is sending data, my 2 slave (storage servers) nodes are doing nothing.

I am also watching ‘tcptrack’ on all three servers, there is a lot of activity between the 2 storage nodes and the main server, but all very small and very short connections (speed less than 5 KB/s), nothing like the traffic on the main server - I see all the small activities, but also the 6 deployments running around 20 MB/s.

I am wondering if this is because I am only selecting deploy at the Fog PXE window on the hosts and not registering the clients when they first connect?

@george1421

Hi George,
I just watched the video tutorial on the location plugin, then installed it on the main server. I have created both the main server and two storage nodes in the location management tab. As all three are Fog servers, one normal and two storage, I selected the option to use kernels and inits from all locations. Time to test, I will post up once I have restarted my 6 test hosts and seen where they pull their deployment images from now.

Thanks for the information, I didn’t know about the plugins as I only started Fog (to replace my WDS server) less than a week ago.