@nonamejason Why do you have two IP addresses assigned to the network interface eth0? Is there a good reason for this?

@Demache What’s the difference between 1st/2nd storage node and the 3rd one? I suppose the WAN tunnel between head quarter and 3rd location doesn’t allow the communication needed to query the information from the 3rd storage node. For both size and hash the replication service first uses HTTP or HTTPS (depending on how you installed) and if it doesn’t receive a proper answer it also tries to retrieve the size and hash (only for files smaller than 10 MB) information via FTP. So if all those protocols are blocked the replication cannot work.

Now as I re-think what I wrote this doesn’t make sense because you said that it does replicate the files (using FTP!) properly. So we probably need to take a closer look.

As I said, it checks size first which seems to fail in most cases considering the log you posted. BUT there is one file where it seems to be ok with size and then goes ahead to match the checksums and fails on that. Seems to be a bit random.

Well, using the information given you might try to see if the WAN tunnel might oppose some restrictions that might cause this.

As well you want to check apache access and error log on the 3rd storage node to see if those requests actually ever read that node.

@Jimbohello said in Very Slow:

but for now, i’ve created a samba share and i get full speed of what a gigabit nic can give so arround 112-120 mb/s
with acronis

We did have reports of another user who had fast speeds with acronis as well. Turns out Acronis is using a fairly old kernel which doesn’t seem to have the issue. If you are keen to give an older kernel a try:
https://fogproject.org/kernels/bzImage-4.9.51
fogproject.org/inits/init-4.9.x.xz

Latest inits should have all the fixes included. Marking as solved.

@Alexandre @Tom-Elliott I am wondering if using git reset --hard ; git pull would be nicer as it does not remove old install logs that might be of interest later on.

Beside that we ought to find a nice solution to this as more and more people use HTTPS and will run into this on further updating when using git. Will be thinking about this.

@tec618 said:

... /usr/local/sbin/udp-sender --max-wait 600 ... d1p1.img;/usr/local/sbin/udp-sender ... --max-wait 10 ... d1p2.img; ...

Good I asked about the log and had a closer look at this now. I remember that we have changed the timeouts but this was a long time ago so I didn’t remember. Currently the first timeout is set to 10 minutes so hosts that boot quicker than others to the multicast don’t rush off and let others behind. But for subsequent partitions the timeout is only 10 seconds. This was made because when you have one out of 20 hosts that fails on the first partition (for whatever reason) then the whole set of hosts would wait for 10 minutes on each partition.

Now possibly the 10 seconds is too little for some of your machines. Edit the file /var/www/html/fog/lib/service/multicasttask.class.php line 662 and change number 10 to 30 sec.

Cancel all multicast tasks that might still be running and restart the service: systemctl restart FOGMulticastManager

@Tywyn It would be really helpful if you’d remember some more detailed infos on this. What did the unicode characters look like?

@george1421 Without more details I am not even sure this is still an issue in latest dev-branch. You know that we have changed the random password generator since 1.5.7. Might be fixed already but I can’t be sure without more details.

@TheBitFighter said in Custom boot config per host/group:

Is it possible to set different refind.conf files for different hosts?
Would there be a different solution to only effect the HP hosts with this solution?

No for both as far as I know. Though I have to admit that I am not part of the project since the beginning and there might be something I am not aware of yet.

And if both answers are no, would it be possible to implement such a feature in FOG?

Absolutely. FOG is open source and you can implement anything you want. Just start by looking at bootmenu.class.php (line 160). Here you see an array of the entries which will be available for selection in the web UI. So you can just add your custom entry here:

...
        $refind_hp = sprintf(
            'imgfetch ${boot-url}/service/ipxe/refind_hp.conf%s'
            . 'chain -ar ${boot-url}/service/ipxe/refind_x64.efi',
            "\n"
        );
...
        self::$_exitTypes = array(
            'sanboot' => $sanboot,
            'grub' => $grub['basic'],
            'grub_first_hdd' => $grub['basic'],
            'grub_first_cdrom' => $grub['1cd'],
            'grub_first_found_windows' => $grub['1fw'],
            'refind_efi' => $refind,
            'refind_efi_hp' => $refind_hp,
            'exit' => 'exit',
        );
...

As well you need to add your custom exit type to another file called service.class.php for it to be select-able in the host/group settings:

...
        $types = array(
            'sanboot',
            'grub',
            'grub_first_hdd',
            'grub_first_cdrom',
            'grub_first_found_windows',
            'refind_efi',
            'refind_efi_hp',
            'exit',
        );
...

@rogalskij Please run file /var/www/html/fog/service/ipxe/bzImage533 on your FOG server command line and post output here.

EDIT: Reading the earlier posts I think this is a typo: bzImage533 vs. bzImage553

@processor What do you see in the task list on the web UI?

As well, have you run the DB cleanup lately? https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL#Database_Maintenance_Commands

@Kiweegie Thanks for sharing your solution!

Without having put in much though I’d probably just force a redirect from http://hostname/ to http://hostname/fog/

@george1421 said in PXE Boot not working properly from Storage Node:

So the root cause of this:

The fog install script does not honor https for storage node installs?
The fog installer did not provide the necessary command line switches when installing the storage node to enable https?

As far as I can tell right now the only issue was that the storage node was not installed with HTTPS enabled in the first place and the buildipxe script in 1.5.7 has a bug that would not compile the cert into the binaries correctly (fixed for 1.5.8 already).

@rogalskij Seems like I was actually wrong about you having a different issue than discussed in the other topic. Nevertheless I think it was good to have two topics to not mix up the information on who had tested what.

Thanks for your effort. I will update the official binaries tomorrow.

@Magiq This is “hidden” in the image settings. Each image is either defined as being resizable or non-resizable or even RAW. In your case I’d just leave your current image as is and create a fresh image definition. Make it type RAW, associate to the hosts (source and destination), capture and deploy. Be aware the RAW image will be much bigger so you need enough space on your FOG server.

@hancocza said:

maybe you have a better way of doing this?

In your case you better just comment or remove those five lines from the PHP code. The fog-client code is made to check if “url” ($location in the PHP code) is empty and defaults to the values specified in settings.json on the client.

Not sure if there should be a change to use the hostname instead of IP if specified in the .fogsettings file

Using DNS name is adding another layer of complexity and I wouldn’t want to add that as default for every FOG user. You can run FOG without setting up proper DNS and I see this as a good default to not add another hurdle for most of our users. If you want to switch to DNS/hostname you wre welcome to do so.

If there are issues in the code that we need to fix preventing you from using the hostname instead of IP, let us know. Best to open an issue on github and point out where things are going wrong.

@Chris-Whiteley @george1421 I really like George’s truth table as it’s the best way to figure out those kind of things. To have it always on top I updated George’s information and put it into a table in the initial post. Let’s try to keep it updated there.

This is solved in latest dev-branch.

@Hanz Reaching out to you via chat.

@Chris-Whiteley @george1421 Just wanna keep this topic up on the screen…

@Arrowtron Overall from what you wrote I would imagine this is happening due to the clients checking in “too often” in your setup. See there are many variables that influence this, some you can alter very easily but others that are just hard facts (like your network speed and load - the longer one request takes the more need to be handled in parallel).

Restarting the server, CPU usage isn’t as bad, eventually goes to 100% the longer it is on

Sure client connections are all dropped and come back over time.

Took a snapshot, upgraded to the latest dev-branch a couple weeks back (i think ~1.5.8.1), CPU usage was still hovering quite high, reverted back

Hmm? FOG 1.5.8.1 is only available since yesterday. You probably were using a different version then.

Ran MySQLTuner, results below

Sorry, I am not an expert on this. Others might give you hints on this. @george1421 @EduardoTSeoane Sure thing, FOG does not do a great job in optimizing the DB backend as of now. This is something we need to pay more attention to in the future.

Ran Database Maintenance Commands on https://wiki.fogproject.org/wiki/index.php/Troubleshoot_MySQL

Good thing to do although it didn’t help.

I haven’t changed any polling/refresh rates for hosts, I’ve seen on these forums that has helped for others

This is the first thing I’d suggest you do to actually find out if it’s the clients causing the load. At least double the checkin time (FOG web UI settings) and wait for half an hour so every client has pulled and adjusted to the new setting.

https://forums.fogproject.org/topic/12392/fog-server-high-cpu/23?_=1582602629372 - I’ve tinkered with the settings below but no drop in CPU

That might be a hint on this being the database backend causing it and not the webserver. Though we will see.