@dclark said in Increased network traffic since 1.5.0 upgrade (Snapins):

[03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber
[03-06-18 9:34:10 pm]  | 46388736 0 /opt/fog/snapins/CiscoJabberSetup.msi ftp://fog:ejOVmFXgU4rbglxcETxqt7AVI%2BBh6h%2BDnWv0ksrwBpA%3D@10.27.25.165/opt/fog/snapins/CiscoJabberSetup.msi
[03-06-18 9:34:10 pm]  | Files do not match.
[03-06-18 9:34:10 pm]  * Deleting remote file: /opt/fog/snapins/CiscoJabberSetup.msi
[03-06-18 9:34:10 pm]  | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRSMSFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.27.25.165
[03-06-18 9:34:10 pm]  * Started sync for Snapin Cisco Jabber

The above, to me, seems to indicate proper action. as you can see, the file sizes don’t match up, meaning either the remote node this was testing could not be contacted, or the file did not exist. (I say these because the filesize was 0 on the remote side.)

[03-06-18 9:33:51 pm]  * Started sync for Snapin Cisco Jabber
[03-06-18 9:33:57 pm]  | Cisco Jabber: No need to sync CiscoJabberSetup.msi file to IRLNEFOG01
[03-06-18 9:33:57 pm]  | CMD:
			lftp -e 'set xfer:log 1; set xfer:log-file "/opt/fog/log/fogsnapinrep.Cisco Jabber.transfer.IRLNEFOG01.log";set ftp:list-options -a;set net:max-retries 10;set net:timeout 30;  mirror -c --parallel=20 -R -i "CiscoJabberSetup.msi" --ignore-time -vvv --exclude ".srvprivate" "/opt/fog/snapins" "/opt/fog/snapins"; exit' -u fog,[Protected] 10.22.25.165

This line interests me as I don’t think i’ve ever seen this issue ever before. So while I’ve never seen it, I hope I may have found the reason to why it was happening and have a hopeful fix for it. Essentially my code was setting a test variable if it wasn’t already set. The idea was if other nodes needed the file, it would be able to send it. I guess this was an oversite? I’m not sure what impacts this will have, but I think it will work for the problem you were seeing.

Please try installing the working-1.5.1 branch? I know it’s not in RC or anything, I’m only pushing bug fixes there. I don’t have multiple nodes to test if this is working so I’m just hoping your input/help would be okay as, like I said, you’re also the first i’ve seen of this particular issue.

@george1421 Just php-fpm.

I don’t know how much help memcache would give, though I suppose it wouldn’t hurt to try it.

So, worked remotely and was able to help determine part of the problem.

Installed apachetop and took a look, found a lot of things still looking at servicemodule-active.php. We moved away from this call as it had some weird issues on its own. Moved the file and the GUI become much less sluggish. Worked to get phpmyadmin working as well. This wasn’t a problem with php-fpm vs. libapache2-mod-php (though I did make sure phpmyadmin would access using the php-fpm system). Part of the problem that was happening was the rewrite rules that handles api system.

Moved the rewrite rules into the directory stanza of the fog.conf. Made this update in the working-1.5.1 branch as well. I haven’t had time to test whether this move actually still allows the system to work, but it should. Will work on that this weekend (testing to see it works properly).

Also, we are still working to help purge the db of old stuff that’s no longer needed.

@hvaransky turn compression down to 19. 20 and 22 on zstd requires a lot of memory. While the selector lets you choose that high, it is not recommended. 19, I’ve found, is the highest you can go without causing an issue with the client machine.

So I took a little time today to try to see what was happening. I’m happy to report, at least based on how I could test, that I believe I found a solution.

Please, if either of you could be so kind, install the working-1.5.1 branch of FOG? This should contain the fix for ldap users and authentication without having to delete the users from the database every iteration.

@hvaransky There’s a lot of variables to consider in deploy, or capture, speed.

First is your network.
Second is the disks writing to/reading from.
Third is the compression.

As @Junkhacker stated, finding the “goldilocks” zone of compression is also useful. For example, gzip at -9 will take a long time to capture and you don’t really gain much compression increase. That and speed to deploy isn’t much better (partly due to the compression already reaching it’s peak).

Less data to deploy = faster network transfer, but if your disk is really old or slow the speed could be limited there. (This on deploy and capture).

I’ve found -11 on Zstd to be a good zone, though I don’t have much disk space, so I use zstd on 19 (which i find is still faster than gzip on 9) during capture.

As I said, there’s a lot of variables to consider.

Also, if your fog server is replicating images and files at the same time as you performing a capture or deploy, chances are likely the slowdown is due to the server being used at the same time as the leads have to jump around the server’s hard drive.

You could try rebooting the fog server though. After all, the server is still a computer, and while it’s not necessarily a normal requirement, rebooting might solve many of the problems you’re seeing.

Also, look at your network, if you have a 1Gbps network, but a switch is a 100/Mbps, the maximum capture/deploy across network to that machine would be limited to 100 Mbps (about 2.5GB/min) Where 1Gbps would give about 7.5GB/min. (This is for uncompressed data though.) The speed, is also (as stated earlier) limited to the hard drives of both the client and server machines. Most often I’ve found the slowdown is not the networks, rather they’re the hdd’s reading/writing from/to.

@itcc I don’t know what’s going on with groups sorry for that but the git branch working-1.5.1 has the fix for power management deletion from groups

@itcc correct

If I had to guess it’s to do with selinux.
https://wiki.fogproject.org/wiki/index.php?title=CentOS_7#CentOS_7_pre-config

I could be wrong but epel is a fedora/centos thing, not a redhat (native) thing.

You could try this:
https://access.redhat.com/discussions/3140721

Particularly (or as root):

sudo rpm -ivh https://dl.fedoraproject.org/pub/epel/epel-release-latest-7.noarch.rpm
sudo yum update

@gordon-taylor please try editing the /var/www/fog/service/ipxe/refind.conf file and set the scan_delay to 0. I’ve heard of this seeming to be a fix for this type of problem.

@gordon-taylor The binary might have changes (refind.efi) though we don’t know what changed (we don’t write the refind program stuff.)

I’ve made scan_delay = 0 the default when 1.5.1 releases.

I think this is not such a good idea.

For one, snapins aren’t an installing/uninstalling utility. Snapins are just (in the simplest of terms) a way of running some executable, be it a script, program, or what have you.

If anything, the message of snapins should really be (in my opinion) “Running snapin…”

And the status of that snapin should be displayed on the task management page.

These status’ are able to be configured, and I’ll request @lee-rowlett to take a look at this as I know he’s doing similar types of things for snapins and status messages.

Using the “Running snapin” message you could name your snapins so it more pertains to the message.

For example, the name being “Google Chrome Uninstaller” you would see the message:

“Running Google Chrome Uninstaller”

Mind trying the working-1.5.1 branch?

I’ve made a lot of headway in performance issues from 1.5.0 that I think is now addressed in 1.5.1, but need more people to test live to ensure the fixes are indeed working as expected.

You would cd to where your current git environment is. Then you do:

git checkout working-1.5.1
git pull
cd bin
./installfog.sh -y

(changing installfog.sh with your normally run arguments if you have any differences.)

@sebastian-roth I’ve made a patch that includes these fixes. I’m currently building the new init’s which will take quite a long time, but I have uploaded the patch to our fos repository. Sorry it will take a while to get some inits built.

I don’t know what the problem was. All I did was patch the ntfs-3g files with the suggested fixes from Sebastian and rebuilt the inits with the changes.

The code in php is:

     /**
      * Returns hash of passed file.
      *
      * @param string $file The file to get hash of.
      *
      * @return string
      */
     public static function getHash($file)
     {
         usleep(50000);
         $file = escapeshellarg($file);
         $filesize = self::getFilesize($file);
         if ($filesize <= 10485760) {
             return trim(
                 shell_exec("sha512sum $file | awk '{print $1}'")
             );
         }
         return trim(
             shell_exec(
                 sprintf(
                     "(%s -c %d %s; %s -c %d %s) | sha512sum | awk '{print $1}'",
                     'head',
                     10486760,
                     $file,
                     'tail',
                     10486760,
                     $file
                 )
             )
         );
     }

The function is not necessary to be in a class file. The url that gets called is: http://{nodeip}/fog/status/gethash.php

The passed data (the $file parameter) comes from the sent link: ($_POST['file']).

If you can create a file on the storage node in /path/to/web/files

mkdir -p fog/status

Create a php file named gethash.php

Add the lines:

<?php
// Delay for 50 milliseconds.
usleep(50000);
// File is passed base64 encoded, decode it here.
$debase_file = base64_decode($_POST['file']);
// We can't trust user inputs, so escape the file we will check.
$file = escapeshellarg($debase_file);
// The file doesn't exist, return blank.
if (!file_exists($file)) {
    return '';
}
// Get the filesize.
$filesize = trim(
    shell_exec("du -b $file | awk '{print $1}'")
);
// If the file is less than one meg, return the hash of the full file.
if ($filesize <= 10485760) {
    return trim(
        shell_exec("sha512sum $file | awk '{print $1}'")
    );
}
// Otherwise return the first and last meg hashed -- hashing full files takes a long time
return trim(
    shell_exec(
        sprintf(
            "(%s -c %d %s; %s -c %d %s) | sha512sum | awk '{print $1}'",
            'head',
            10486760,
            $file,
            'tail',
            10486760,
            $file
        )
    )
);

This of course assumes a few things.

FIrst:
PHP is installed on the NAS (probably is)
du is installed and available.
awk is installed and available.
sha512sum is installed and available.
head is installed and available.
tail is installed and available.

Hopefully this helps.

Sorry I posted this originally in the wrong thread, too much reading I guess.

I’ve updated the ipxe files to use be built with FTP and NFS. These files will be in 1.5.1, and are currently in the working-1.5.1 branch of github repository.

I’ve confirmed and fixed this. Just retagging the 1.5.1 branch.