More info about this 🙂

Now we have configurated the client cheout time = 275 seconds. In September we had to increase it to 900, but after the changes we have setup it as I tell, 275 seconds.

This is a capture of goaccess tool:

0_1540379551079_goaccess_RHEL.png

The capture is after 6 minutes of activity, we can see that:
Total request: 8436 -> 23,5 req/sec
1378 visitors requested the /fog/service/getversion.php?NewServise&json page. This says that 1378 client are connected simultaneous.
1 visitor requested /fog/service/progress.php 114 times. This client is doing a download task.

Is necessary take in account that when a cleint is doing a dowload, capture or multicast task, this client asks or reports to the server his advance (yes is very pretty and cool see the progress bar) but this info has a price. The client reports his advance more or lees every three or four seconds, when you have one tasks is “pecata minuta”, but when you have 100 or more client doing download or cpature tasks is a problem, because the php server can not process all request simultaneous and is not only the php server, is the mysql server too.

For example in this capture of htop command (like atop or top):

0_1540380690148_Captura de pantalla de 2018-10-24 13-08-54.png

We can see that the vCores are busy, but not at 100%, the load is high, mysqld is using 221% of CPU. In this moment the server is proccessing only the FOG client requests of the computers, there is no any tasks (When the technicians send download, multicast or capture tasks, the server is burning … literally. I saw the load at 60 or more, the server could not attend the all request and refused them ), In this capture shows the activity of the two NUMA nodes clearly.

Node 0: 1, 2 ,3 and 4
Node 1: 5 ,6, 7 and 8

Where is working the mysqld proccess?

# ./numa-maps-summary.pl < /proc/1787/numa_maps N0 : 636053 ( 2.43 GB) N1 : 14336 ( 0.05 GB) active : 352378 ( 1.34 GB) anon : 649153 ( 2.48 GB) dirty : 649153 ( 2.48 GB) kernelpagesize_kB: 1016 ( 0.00 GB) mapmax : 480 ( 0.00 GB) mapped : 1276 ( 0.00 GB)

In the Node 0.

I downloaded a little script, i forgot from where, that shows the usage of RAM of each proccess:

# ./ps_mem.py Private + Shared = RAM used Program 4.0 KiB + 12.5 KiB = 16.5 KiB agetty 4.0 KiB + 15.0 KiB = 19.0 KiB mysqld_safe 4.0 KiB + 47.5 KiB = 51.5 KiB rpc.statd 4.0 KiB + 49.5 KiB = 53.5 KiB rpc.idmapd 4.0 KiB + 57.0 KiB = 61.0 KiB lvmetad 36.0 KiB + 31.0 KiB = 67.0 KiB atd 4.0 KiB + 73.5 KiB = 77.5 KiB VGAuthService 88.0 KiB + 32.0 KiB = 120.0 KiB rhsmcertd 92.0 KiB + 41.5 KiB = 133.5 KiB systemd-udevd 112.0 KiB + 22.5 KiB = 134.5 KiB sleep 88.0 KiB + 55.0 KiB = 143.0 KiB vsftpd 88.0 KiB + 65.0 KiB = 153.0 KiB gssproxy 148.0 KiB + 23.0 KiB = 171.0 KiB udp-sender 156.0 KiB + 30.0 KiB = 186.0 KiB crond 164.0 KiB + 30.0 KiB = 194.0 KiB in.tftpd 180.0 KiB + 20.0 KiB = 200.0 KiB numad 192.0 KiB + 16.0 KiB = 208.0 KiB rhnsd 128.0 KiB + 83.5 KiB = 211.5 KiB master 176.0 KiB + 35.5 KiB = 211.5 KiB xinetd 188.0 KiB + 54.5 KiB = 242.5 KiB auditd 232.0 KiB + 29.0 KiB = 261.0 KiB irqbalance 192.0 KiB + 88.5 KiB = 280.5 KiB qmgr 208.0 KiB + 87.0 KiB = 295.0 KiB sh 240.0 KiB + 109.5 KiB = 349.5 KiB rpcbind 588.0 KiB + 36.5 KiB = 624.5 KiB systemd-logind 668.0 KiB + 75.5 KiB = 743.5 KiB dbus-daemon 596.0 KiB + 311.5 KiB = 907.5 KiB polkitd 800.0 KiB + 175.5 KiB = 975.5 KiB vmtoolsd 916.0 KiB + 64.5 KiB = 980.5 KiB FOGpxe.sh 936.0 KiB + 135.0 KiB = 1.0 MiB dnsmasq 1.1 MiB + 386.0 KiB = 1.5 MiB NetworkManager 1.4 MiB + 413.5 KiB = 1.8 MiB pickup 2.0 MiB + 101.0 KiB = 2.1 MiB rpc.mountd 2.2 MiB + 67.5 KiB = 2.3 MiB systemd 2.6 MiB + 308.0 KiB = 2.9 MiB tuned 2.9 MiB + 355.5 KiB = 3.2 MiB mysql 2.7 MiB + 846.0 KiB = 3.5 MiB bash (6) 3.0 MiB + 822.5 KiB = 3.8 MiB FOGSnapinReplic (2) 3.1 MiB + 682.5 KiB = 3.8 MiB FOGImageReplica (2) 4.2 MiB + 80.0 KiB = 4.2 MiB nsrexecd 4.1 MiB + 718.5 KiB = 4.8 MiB FOGSnapinHash (2) 3.6 MiB + 1.3 MiB = 4.9 MiB sudo (3) 5.4 MiB + 1.0 MiB = 6.4 MiB FOGTaskSchedule (2) 7.5 MiB + 118.5 KiB = 7.6 MiB glusterfsd 1.9 MiB + 6.1 MiB = 8.0 MiB sshd (7) 2.5 MiB + 7.1 MiB = 9.5 MiB rsyslogd 10.3 MiB + 713.5 KiB = 11.0 MiB FOGImageSize (2) 7.0 MiB + 9.9 MiB = 16.9 MiB systemd-journald 21.5 MiB + 791.0 KiB = 22.2 MiB FOGPingHosts (2) 25.6 MiB + 1.0 MiB = 26.6 MiB FOGMulticastMan (2) 315.4 MiB + 14.6 MiB = 330.0 MiB php-fpm (51) 2.5 GiB + 289.0 KiB = 2.5 GiB mysqld 5.1 GiB + 14.9 MiB = 5.1 GiB httpd (12) --------------------------------- 8.0 GiB =================================

Any reason why you’re using raw image over something like resizable? Is there something special about the partitions or disk layout?

It also wouldn’t hurt to update FOG to the latest 🙂

@Quazz Nice explanation of the issues regarding the order of the partitions and the starting positions. I think that’s exactly what was going on. Since the actual usage of the original 1TB drive was only ~150GB, I went with a 240GB SSD which has always worked for me before. It gave me that “Seems like you are trying to restore to an empty disk…” error every time I tried to deploy that image to anything smaller than 2TB. Now that the two little partitions at the end of the original drive are gone, I recaptured it and it deployed just fine to a 240GB SSD.

@DJ Amazing we found this after such a long time of intense digging in packet dumps and testing. Thanks for posting the information on the switch so people can find this here in the forums.

I have though about this but don’t really see why we should “fix” NFS mount to use high source ports. I might be wrong here but from what I have seen in the packet dumps it seems like this is general behavior and changing this would need a fixup deep down in the NFS client libs code - something that we would need to maintain over a long time since I don’t see this change making it into the official code line. What do you think?

@dustindizzle11 Thanks for the heads up. Possibly I need to revisit what I said. Possibly there is a huge difference and it does make FOG run much faster in your case (and possibly for others as well). What I could think of is the php-fpm issue we still have in FOG 1.5.4 (see George’s post here). What is yours set to on Debian?

@CJR_ie Here you go. FOG detects your windows system partition to be fixed as well and therefore does not expand it on deployment. Edit the file d1.fixed_size_partitions by hand using a text editor and make it read

:1

Then schedule a new deploy task for your host and it should expand to the full SSD size properly.

I’ve found that there are a couple of things to do with the configuration.
I’m running 1.5.4 version and I’m experiencing the same issue.

To put this in context, I’ve installed the fog server and then the IP of this machine needed to be changed, originally it had 192.168.27.40 and now it’s 10.0.0.1

With this in mind, I found the same error than you.

As @Tom-Elliott stated I haven’t created the /home/fog/images neither /home/fog/images/dev, using his advice I created them and also created the .mntcheck files.

Then I modified the /etc/exports to reflect the changes, updating the paths to the new ones.

After that, as @Kiweegie wrote I executed exportfs -a, please notice that in the second line he proposes to modify the line of the /etc/exportfs adding an async modifier that nowadays is in place out of the box.

After all that changes, my client started but gave me a NFS timeout, looking at the error message I’ve noticed that the IP of the NFS export and the path weren’t correct yet.

Looking at file /var/www/html/fog/lib/fog/config.class.php I found that there are two values that should be updated
STORAGE_DATADIR
STORAGE_DATADIR_CAPTURE

I added the right path in both.

Tried to reboot the client but I received the same timeout.

So I ended up checking the configuration of my storage nodes at the UI.

I found that, in the storage node definition form there were a couple of fields where still had the wrong values:

IP Address Image Path FTP Path

I updated them and now it seems to be working properly, at least the image capture step.

I hope this can help anyone.

@Sebastian-Roth Very nice.
On a side note do those inits also address the intel raid-1 monitor application too? That one was related to a bootroot issue. Disregard, I see you answer the question in that thread too.

@Robert-H As we have the mdmon included in the official inits I deleted the special ones now. Find the official inits here:

https://fogproject.org/inits/init.xz https://fogproject.org/inits/init_32.xz

@george1421

Perfect!! Thank you George and Sebastian for your help and hopefully this helps others in the future.

-Joe

It works now! Thank you for all your help!

@Sebastian-Roth Thanks, I will have my Network admin redeploy the FOG client. Then entire environment was lost to a bad upgrade. This solved my problem

The wiki page you linked earlier should allow you to log into the web console. You might have to reboot he FOG server after you reset the password in the database because of the way FOG cache’s recent activity.

https://wiki.fogproject.org/wiki/index.php?title=Reset_WebUI_FOG_password

From the wiki this command should do it for you.

UPDATE users SET uPass = MD5('password') WHERE uName = 'fog'; exit;

I do have to say FOG 1.2.0 running on ubuntu 12.04 is really old. Unless you really need access to the images stored on that server, I (personally) would just spin up a new FOG server using a current build. FOG 1.2.0 doesn’t support win10, uefi firmware, nvme disks, or current hardware. If you need the images from the FOG 1.2.0 server you can copy them over to a new fog server and use them.

@John-L-Clark This is out of FOGs scope as we find WIFI not to be a reliable connection for deploying or capturing an image to/from a PC. So years back it was decided to ignore WIFI adapters altogether. We don’t have drivers for WIFI enabled in the iPXE boot nor have we included drivers in the Linux kernel that we use in our FOS imaging environment booted on the client machines.

Would be a huge task to add this and make it work reliably in most cases (thinking about different WIFI standards, driver issues and encryption stuff).

Why do you want WIFI to be the default/primary MAC?

@sapeurorca Try to reset encryption for that host, try to re-install the fog client on that host.

It is a fresh install, on that part.
I did mess with settings on the Imaging section (changed the DefaultMember to /images/FOGimages) I did not change the FOG settings

@Fernando-Gietz said in Multicast data address not change from one task to another one:

Se añade esta linea para que asigne direcciones IP diferentes a cada tarea multicast

I’ve added the patch, but a little more checking involved. This has been added to both the working and working-1.6 branches. It tests the set value for the $address variable. If this variable is set, it will calculate the address. Here’s the snippet of lines:

if ($address) { $address = long2ip( ip2long($address) + ( ( $this->getPortBase() / 2 + 1 ) % self::getSetting('FOG_MULTICAST_MAX_SESSIONS') ) ); }

Hopefully this will address the problem people have been seeing and allow the use of multiple sessions.

Just in the fwiw bucket, we recently found out that if you have 2 or more simultaneous multicasts streams going at one time, the trasfer rates drop off quite a bit. I do remember seeing someone create a patch to change the multicast base IP address with each new stream. That seemed to resolve the issue.

[edit] Updated info: https://forums.fogproject.org/topic/12300/multicast-data-address-not-change-from-one-task-to-another-one/16

@tesparza Which version of FOG do you run? I know we have a fair amount of users doing Win 10 with FOG and don’t see all of them reporting this issue. Not saying that FOG isn’t doing this but I guess it’s more likely something with your image.

Is bitlocker turned off?

manage-bde -status manage-bde -off C: