Posts made by benjamind

@george1421 said in High load average when capturing image:

@benjamind Some additional questions/tasks.

1. How is the virtual machine host server connected to its storage, over iscsi or nfs. I’m talking about the proxmox server connected to the VM’s storage.

Over NFS.

2. Will you capture another process capture just like you did for the static load but capture it while the target is still acquiring the image. The partclone screen should be on the target computer.

Here is the capture :

The top processes do not seem revealing to me because they change very regularly.

@benjamind said in High load average when capturing image:

@george1421 said in High load average when capturing image:

@benjamind What does your static load look like (without capture or deploy)?

There is a capture in progress. I will post the static load average when it is finished.

Here is the static load average without capture in progress :

@george1421 said in High load average when capturing image:

@benjamind said in High load average when capturing image:

Tell me a bit more about your hypervisor server. How many network interfaces does it have. Are they teamed 1 GbE or a 10GbE network interface.

It has two 1Gb/s network interfaces. One is used for the administration of the proxmox server itself and the other is used for VM traffic.

So you have NAS traffic from the hypervisor using the same single network interface as the FOG server uses for imaging and communications with FOG clients? I’m just trying to understand your system design. I can say on a physical machine I can saturate a 1 GbE link with just 3 simultaneous capture/deploys to a modern target computer. I think I would like to see at least 2 physical teamed nics for the hypervisor to use (understand this is just an opinion) especially since you are using NAS storage for your hypervisor.

Traffic to the NAS goes through the 1Gb/s hypervisor administration link. Deployment and capture traffic goes through the other 1Gb/s link dedicated to the VM Fog.

There still are some tweaks we can make to the fog environment to help a little but in my mind that 1 GbE link a concern. Your NAS sounds like its sufficiently sized for the load.

Lowering your vCPU count may not help much if the hypervisor is dedicated to the FOG server. Maybe taking it to 4 vCPU then would be advised since it doesn’t have to compete with other VMs for resources. That will leave 4 vCPU available for the hypervisor to use for system maintenance.

I would start by increasing the fog client check in time to see what impact that has on the static load on the server. My guess is that based on what I know now you won’t see much of a change. But lets see where we end up.

This manipulation ?

@george1421 said in High load average when capturing image:

Lets start debugging this by changing the fog client check in time from 300 seconds to 900 seconds. This will change the check in time from 5 minutes to 15 minutes. This is done under the fog settings -> fog configuration menu. Wait about 30 minutes and see if your static load on the fog server decreases.

@george1421 said in High load average when capturing image:

Lets start debugging this by changing the fog client check in time from 300 seconds to 900 seconds. This will change the check in time from 5 minutes to 15 minutes. This is done under the fog settings -> fog configuration menu. Wait about 30 minutes and see if your static load on the fog server decreases.

Do you still want me to try this solution?

Now you said this was a VM. with 8 vCPU and 16GB of ram. How does that compare with the physical machine the VM is running on? Over committing vCPU is bad for the VM’s performance too. I would think that 2 vCPU should be adequate for 100 client computers. I understand with your high cpu usage you might think adding more vCPUs would help, but it looks like from your first screen shot only 2 cores are really taking the load.

The VM uses all of the hypervisor’s RAM and CPUs. It is a hypervisor dedicated to the Fog server. We have a Proxmox cluster of Fog servers which allows us to have high availability.

I reduced the vCPUs to 2 (1 socket, 2 cores) but it requires a restart to be effective. I will keep you informed.

Tell me a bit more about your hypervisor server. How many network interfaces does it have. Are they teamed 1 GbE or a 10GbE network interface.

It has two 1Gb/s network interfaces. One is used for the administration of the proxmox server itself and the other is used for VM traffic.

On your NAS is that a single 1GbE interface or do you have them teamed also?

It has 2x10Gb/s teamed network interfaces.

How many other VMs are running on this hypervisor host?

None

How many hypervisor hosts are using this single NAS?

~30

On the NAS, what is the disk subsystem? raid-10, raid-6?

RAID DP

During a fog image capture, what does the nas CPU and disk usage look like?

A Fog capture does not significantly affect the resources of the NAS.

@benjamind I am sure this will help you: https://forums.fogproject.org/topic/13948/database-stress-when-cloning-on-big-environments

Hi @Sebastian-Roth,

Can you please confirm that what I have set up seems correct to you?

touch /images/dev/postinitscripts/fog.dbstress
chmod 755 /images/dev/postinitscripts/fog.dbstress
cat /images/dev/postinitscripts/fog.dbstress
#!/bin/bash

sed -i 's/usleep 3000000$/usleep 30000000/g' /bin/fog.statusreporter

cat /images/dev/postinitscripts/fog.postinit
#!/bin/bash
## This file serves as a starting point to call your custom pre-imaging/post init loading scripts.
## <SCRIPTNAME> should be changed to the script you're planning to use.
## Syntax of post init scripts are
#. ${postinitpath}<SCRIPTNAME>
. ${postinitpath}/fog.dbstress

If you confirm the modification, I will post the evolution of the load average a few hours after the end of the current capture.

@george1421 said in High load average when capturing image:

@benjamind What does your static load look like (without capture or deploy)?

There is a capture in progress. I will post the static load average when it is finished.

My intuition is telling me that you have this load when not actively capturing or deploying just its more obvious when doing so.

We have been using this Fog server for a long time and this problem appeared only very recently, although we have not made any modifications recently.

Your disk subsystem on this FOG server, what is it a single (1TiB) HDD or a raid disk?

Describe the fog host server, is it a vm or physical machine. What is your processor (vCPU count) and memory installed. If physical what mfg and model of computer.

The server is a virtual machine running on a Proxmox hypervisor. It has 8vCPU (2 sockets, 4 cores) and 16Gb of RAM.

The virtual machine has two disks:

• one of 20Gb for the /
• one of 900Gb for the /images

These disks are network disks on a NetApp filer with RAID.

Also can you show me a screen shot using top sorted by processor? I a bit more familiar with that output.

Hi @george1421 ,

Thank you for your help.

I specified that the problem appeared during capture but I learned that it also appears during deployment.

This server is only used for that. There is indeed an ossec service but it is only the agent.

Regarding the /images and /images/dev directories, they are on the same disk (network) and on the same partition.

There are a hundred computers that have the Fog client installed.

root@fog2:/images# lsblk
NAME   MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda      8:0    0   20G  0 disk 
└─sda1   8:1    0   20G  0 part /
sdb      8:16   0  900G  0 disk 
└─sdb1   8:17   0  900G  0 part /images
sr0     11:0    1 1024M  0 rom

Hi everybody,

We run FOG 1.5.7 on a Debian 10 server.

When we take an image capture from our Fog server, we see a very .

What information can I provide you with to investigate?

Thank you in advance for your feedback.

@Sebastian-Roth No, i have changed the permissions to fogproject:root and delete the directories in /images/dev/ that looks like a MAC address.

It works now.

@Sebastian-Roth

Yes, i do. That’s why i ask if i have to set the permissions to fogproject:www-data as they do in this topic.

@Sebastian-Roth said in Capture problem:

@benjamind Run ls -alR /images und post the output here.

root@fog4:/images# ls -alR /images
/images:
total 20
drwxrwxrwx  5 fogproject root 4096 oct.   3 13:52 .
drwxr-xr-x 21 root       root 4096 août  29 16:48 ..
drwxrwxrwx  6 fogproject root 4096 oct.   4 09:05 dev
drwxr-xr-x  2 root       root 4096 août  29 16:57 ISYS_W10_AvecSoftsPartition1
-rwxrwxrwx  1 fogproject root    0 août  29 16:48 .mntcheck
drwxrwxrwx  2 fogproject root 4096 août  29 16:48 postdownloadscripts

/images/dev:
total 24
drwxrwxrwx 6 fogproject root 4096 oct.   4 09:05 .
drwxrwxrwx 5 fogproject root 4096 oct.   3 13:52 ..
drwxrwxrwx 2 root       root 4096 oct.   4 11:30 1ca0b87640e0
drwxrwxrwx 2 root       root 4096 sept.  9 10:34 1ca0b87640ec
drwxrwxrwx 2 root       root 4096 oct.   4 10:09 dc4a3e73805e
-rwxrwxrwx 1 fogproject root    0 août  29 16:48 .mntcheck
drwxrwxrwx 2 fogproject root 4096 août  29 16:48 postinitscripts

/images/dev/1ca0b87640e0:
total 201346432
drwxrwxrwx 2 root       root         4096 oct.   4 11:30 .
drwxrwxrwx 6 fogproject root         4096 oct.   4 09:05 ..
-rwxrwxrwx 1 root       root            0 oct.   4 09:05 d1.has_grub
-rwxrwxrwx 1 root       root      1048576 oct.   4 09:05 d1.mbr
-rwxrwxrwx 1 root       root           47 oct.   4 11:30 d1.original.swapuuids
-rwxrwxrwx 1 root       root 202232012011 oct.   4 11:26 d1p1.img
-rwxrwxrwx 1 root       root   3945636748 oct.   4 11:30 d1p2.img
-rwxrwxrwx 1 root       root          250 oct.   4 09:05 d1.partitions

/images/dev/1ca0b87640ec:
total 1040
drwxrwxrwx 2 root       root    4096 sept.  9 10:34 .
drwxrwxrwx 6 fogproject root    4096 oct.   4 09:05 ..
-rw-r--r-- 1 root       root       0 sept.  9 10:34 d1.has_grub
-rw-r--r-- 1 root       root 1048576 sept.  9 10:34 d1.mbr
-rw-r--r-- 1 root       root      20 sept.  9 10:34 d1p1.img.000
-rw-r--r-- 1 root       root     250 sept.  9 10:34 d1.partitions

/images/dev/dc4a3e73805e:
total 204602528
drwxrwxrwx 2 root       root         4096 oct.   4 10:09 .
drwxrwxrwx 6 fogproject root         4096 oct.   4 09:05 ..
-rwxrwxrwx 1 root       root            0 oct.   4 08:04 d1.has_grub
-rwxrwxrwx 1 root       root      1048576 oct.   4 08:04 d1.mbr
-rwxrwxrwx 1 root       root 209511898862 oct.   4 10:09 d1p1.img
-rwxrwxrwx 1 root       root          250 oct.   4 08:04 d1.partitions

/images/dev/postinitscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 août  29 16:48 .
drwxrwxrwx 6 fogproject root 4096 oct.   4 09:05 ..
-rwxrwxrwx 1 fogproject root  249 août  29 16:48 fog.postinit

/images/ISYS_W10_AvecSoftsPartition1:
total 201427840
drwxr-xr-x 2 root       root         4096 août  29 16:57 .
drwxrwxrwx 5 fogproject root         4096 oct.   3 13:52 ..
-rwxr-xr-x 1 root       root      1048576 août  29 16:57 d1.mbr
-rwxr-xr-x 1 root       root 206261019571 août  29 17:27 d1p1.img
-rwxr-xr-x 1 root       root          191 août  29 16:57 d1.partitions

/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fogproject root 4096 août  29 16:48 .
drwxrwxrwx 5 fogproject root 4096 oct.   3 13:52 ..
-rwxrwxrwx 1 fogproject root  235 août  29 16:48 fog.postdownload

@george1421 said in Capture problem:

@benjamind https://forums.fogproject.org/topic/13303/image-capturing-error-invalid-storage-group/2

Should i change the permissions to fogproject:www-data ?

Hi @george1421,

I didn’t change the fogprokect password.

Thereis a directory in /images/dev/ that looks like a MAC address.

What should be the permission for the directory /images/ ? I hacve some files with root:root and other with fogproject:root.

@Sebastian-Roth Thank you for your reply.

We can still access the Web UI.

We have 399 Hosts with fog-client installed.

The version actually installed is 1.5.7.

Hi everyone,

We got a problem when trying to catpure a new image :

Does anyone please got an idea ?

Thank you.

Hi @Sebastian-Roth and @EduardoTSeoane,

Thank you for your answers.

@Sebastian-Roth said in Multicast with multi-NIC FOG server:

@benjamind May I still ask you to post the full output of the command ip a s. I don’t like to do all the guesswork.

Sorry, but i have removed the virtual NIC to put the server directly in the subnet of the machines.

Thank you @Sebastian-Roth,

I have deployed several FOG for each of my VLANs.

If i understand the link provided by @EduardoTSeoane , i have to change the Storage Node’s ip also ?

@george1421

I also have changed the storage node IP.

I used to deploy via multicast over VLANs but our new Cisco Merka routers can’t do this unless there are stacked. Mines are in two different places.

@Sebastian-Roth

I have restarted the service and reboot the server with no effects. FOG still uses ens18 for multicast.

@EduardoTSeoane

The link you have provided seems to be broken.

Hello,

I’m trying to configure a FOG server to deploy images by multicast in different VLAN. The server has many virtual network interfaces for each VLAN.

I have change the interface from ens18 to ens19 in FOG Configuration => FOG Settings => Multicast Settings but when i start a deployement task it still uses the “main” interface as i hace see it in the /opt/fog/log/multicast.log :

/usr/local/sbin/udp-sender --interface ens18 --min-receivers 13 --max-wait 600 --portbase 53250 --full-duplex --ttl 32 --nokbd --nopointopoint --file /images/ISYS_W10_AvecSoftsPartition1/d1p1.img;

Did i forget something ?

Thaank you in advance for your help.

@Sebastian-Roth Great !!! Thank you again.

@Sebastian-Roth The deployment is finished and the PC has correctly joined the domain.

Here is the logs file : fog.log.

Does everything seems good to you ?

Thank you very much for your help.