FOG Web GUI speed and default storage activity

[Wayne Workman]

@jgallo said in FOG Web GUI speed and default storage activity:

could I raise the number of vCPU on Hyper-V from 2 to 4? Will this help improve performance?

That would only help if your host system isn’t overburdened. If you have too many VMs on it already with too many cores assigned, and not enough cores available, it’ll just make things worse. But if you have plenty of resources, then it would help.

Also, set your client checkin time to something like 300 seconds (five minutes) and see if that makes a difference. Keep in mind the change here isn’t immediate - the clients have to checkin once more to actually get the new setting.

[JGallo]

@wayne-workman

I don’t have too many VM’s. I do have 4 but each of those has either 1 or 2 vCPU’s allocated to them. I have now set 4 vCPU’s for the fog server VM and I still have that issue. I also set the client checkin time to 300 with same issue.

Here is what I noticed. I see that the FOG server disk I/O is about 15% give or take on a constant basis. I also noticed that all the disk activity is from apache2 with user www-data and mysql is using up to 5GB of memory at times just on idle. Could this be some programming bug or my database needs to be cleared?

[Wayne Workman]

@jgallo How many hosts do you have in your environment that have the FOG Client installed?

[JGallo]

@wayne-workman

I don’t have any. We use Group Policy to manage printers, settings, etc. Back in the day we used the FOG Client when autojoin domain features were utilized. We don’t anymore due to large amounts of chromebooks replacing aging PC’s.

[Wayne Workman]

@jgallo What are the link speeds between the main fog server and the other nodes? How many images do you have? What’s the FOG IMAGE REP SLEEP TIME in fog settings set to?

[JGallo]

@wayne-workman

At the secondary schools, the connection speed to our district office is 1Gb and the primary schools are at 100mb. The FOG IMAGE REP SLEEP TIME is set to 10800.

[Wayne Workman]

@jgallo How consistent is the problem? The “a valid connection cannot be established” problem. Any rhyme or pattern? Is this when imaging is happening?

[JGallo]

@wayne-workman

on working branch 57 it was very consistent even with all the changes made to fog and the vCPU’s. I have been updating all the storage nodes and fog to working branch 64 today and the problem is still persistent. The only pattern I have observed is that upon rebooting the fog sever, the valid connection messages do not appear for about an hour or so. Then the messages begin to appear for random nodes that I have the graph enabled. At random times, the messages tend to go away but then come back upon selecting another storage node on the dashboard.

[Wayne Workman]

@jgallo Do you know what version of fog this problem started with?

[JGallo]

@wayne-workman

I know that I went from 1.4.4 to 1.5.0 RC1 if I recalled. I know when I upgraded I made a huge leap. New interface and all. During that time there were replication issues and eventually updated to 1.5.0 RC7 which still had replication issues. I then upgraded to 1.5.0 RC9 which replication had major issue that was resolved in a working branch. So I have been on working branch 57 until about an hour ago I went to 64 with all nodes and fog server. I have been observing this issue since I have been on 1.5.0 from the new interface. I never had this issue in 1.4.4

[Wayne Workman]

@jgallo said in FOG Web GUI speed and default storage activity:

The only pattern I have observed is that upon rebooting the fog sever, the valid connection messages do not appear for about an hour or so.

I’m thinking about this - I’d like you to try to restart only Apache and see if it has the same effect or not. On CentOS/Fedora/RHEL it’s systemctl restart httpd and on Ubuntu 16/Debian8/Debian9 it’s systemctl restart apache2 and on Ubuntu 14-,debian7- it’s service apache2 restart

[JGallo]

@wayne-workman

Tried that and still get the database connection message.

[Wayne Workman]

@jgallo During when the problem is happening, what does this command return? free -h;uptime

[JGallo]

@wayne-workman
I have rebooted due to upgrade to working branch. FYI.

             total        used        free      shared  buff/cache   available
Mem:            15G        510M        181M         47M         14G         14G
Swap:          4.0G        268K        4.0G
 15:59:54 up  1:56,  2 users,  load average: 0.24, 0.19, 0.21```

[Wayne Workman]

@jgallo You have 14GB cached in RAM, which I’ll say is substantial. You can clear that with the below command:
sudo sh -c "sync; echo 3 > /proc/sys/vm/drop_caches"
Does the issue resolve after this? Also, check free -h afterwards.

[JGallo]

@wayne-workman

went down to 139MB. LOL dam that was a lot of cache. Wish I had it in my pocket.

[JGallo]

@wayne-workman

This was just a minute of running free -h

     total        used        free      shared  buff/cache   available
Mem:            15G        427M         14G         47M        289M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        434M         14G         47M        289M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        484M         14G         47M        292M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        485M         14G         47M        292M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        486M         14G         47M        301M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        486M         14G         47M        301M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        487M         14G         47M        301M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        487M         14G         47M        301M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        488M         14G         47M        301M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        488M         14G         47M        302M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        490M         14G         47M        302M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        489M         14G         47M        302M         14G
Swap:          4.0G          0B        4.0G
administrator@VUSD-FOG:~$ free -h
              total        used        free      shared  buff/cache   available
Mem:            15G        491M         14G         47M        302M         14G```

[Wayne Workman]

@jgallo Most importantly, are the nodes reporting properly?

[JGallo]

@wayne-workman

I rebooted server just in case. Went home for the day. I will keep an eye on the nodes and see how they are in the morning. Also see how fast the cache grows. It has gone up to 305MB right now but the web UI is substantially snapier.

[Wayne Workman]

@jgallo If this solves the problem, we can setup a cron job that does this every hour.