Best posts made by jkozee

@Tom-Elliott Sorry for not seeing this sooner. PAGE_SIZE is defined as 4096, so the mask is being set to 4095, which is the same value that iscsi_iser.c uses (~MASK_4K).

From the notes in LIS, I suspect that setting blk_queue_virt_boundary is supposed to insure that there are no gaps in th sg list, but they are still present, so the bounce buffer needs to be put back in place or the gaps need to be eliminated elsewhere.

The patch author responded this morning and is looking into the slowdown report. I’ll post any updates as I hear them.

Looks good now. Thanks for the speedy fix.

Anecdotally, it appears that both image captures and image deploys take longer in 6299 than my previous 5315 installation. I am using Windows 10 clients under Hyper-V.

Is this expected/explainable behavior?

If not, I can bring up both installations and provide some metrics between the two versions. I haven’t measured the overall capture/deploy times, but it definatelly takes longer for the partclone step to begin.

Thanks.

I don’t think anything was changed between versions, but I will verify. I will perform a detailed analysis and report my findings.

I ran some tests that will hopefully prove useful.

Both the client and server are VM’s on the same server. I used a single checkpoint on the client to run all of the tests. The server was tested from a checkpoint running 5315 and then upgraded to 6303 with a new checkpoint created, so that I can easily do additional tests if needed. The upgraded VM gives similar results as a new install on a VM that I originally observed the slow behavior. So, there should be no appreciable differences between the test scenarios, except for the updated FOG version.

Deployment went from 6:03 to 8:54, with the most time increase seen during “Formatting initialized partition” before Partclone and “Resizing ntfs volume” after.

Capture went from 18:03 to 25:52, with the most time increase seen during “Resizing filesysten” before Partclone and “Resizing ntfs volume” after.

I will include additional data and times in separate posts for each test for closer inspection.

Please let me know if you have any ideas, or anything else you would like to see tested.

Thanks!

Sorry, long post as I have a limit on how often I can post

5315-Capture
#0:00

• Verifying network interface configuration…Done

• Checking Operating System…Windows 10

• Checking CPU Cores…1

• Send method…NFS

• Checking In…Done

• Mounting File System…Done

• Preparing to send image file to server…Done

• Checking Mounted File System…Done

• Using Image: delme

• Preparing backup location…Done

• Looking for Hard Disks…Done

• Re-reading Partition Tables…Done

• Using Hard Disk: /dev/sda

• Clearing part (/dev/sdal)…Done

• Mounting partition (/dev/sdal)…Done

• Removing page file…Done

• Removing hibernate file…No hibernate found

• Clearing ntfs flag…Done

• Saving original partition table…Done

• Saving Partition Tables (MBR)…Done

• Possible resize partition size: 11263111 k

• Running resize test /dev/sdal…Done

• Resize test was successful

• Resizing filesystem…Done

• Clearing ntfs flag…Done

• Resizing partition dev/sda1…Done

• Checking Hard Disks…Done

• Clearing ntfs flag…Done

• Now FOG will attempt to upload the image using Partclone.

• Processing Partition: /dev/sdal (1)

• Using partclone.ntfs
  #0:23
  <<PARTCLONE>>
  #18:00

• Image uploaded

• Restoring MBR…Done

• Resizing ntfs volume (/dev/sdal)…Done

• Clearing ntfs flag…Done

• Stopping FOG Status Reporter…Done
  #18:03

6303-Capture
#0:00

• Verifying network interface configuration…Done
• Checking Operating System…Windows 10
• Checking CPU Cores…1
• Send method…NFS
• Attempting to check in…Done
• Mounting File System…Done
• Checking Mounted File System…Done
• Checking img variable is set…Done
• Preparing to send image file to server
• Preparing backup location…Done
• Setting permission on /images/00155d016673…Done
• Removing any pre-existing files…Done
• Using Image: delme
• Looking for Hard Disk…Done
• Reading Partition Tables…Done
• Using Hard Disk: /dev/sda
• Now FOG will attempt to upload the image using Partclone
• Checking for fixed partitions…Done
• Getting Windows/Linux Partition Count…Done
• NTFS Partition count of: 1
• EXTFS Partition count of: 0
• Setting up any additional fixed parts
• Saving original partition table…Done
• Saving original disk/parts UUIDs…Done
• Shrinking Partitions on disk
• Clearing part (/dev/sda1)…Done
• Mounting partition (/dev/sdal)…Done
• Removing page file…Done
• Possible resize partition size: 11263111 k
• Running resize test /dev/sdal…Done
• Resize test was successful
  #0:18
• Resizing filesysten…Done
  #4:53
• Resizing partition /dev/sdal…Done
• Clearing ntfs flag…Done
• Saving shrunken partition table
• Saving Partition Tables (MBR)…Done
  #4:53
  <<PARTCLONE>>
  #22:44
• Image Uploaded
• Restoring Original Partition Layout…Done
  #22:44
• Resizing ntfs volune (/dev/sda1)…Done
  #25:49
• Clearing ntfs flag…Done
• Stopping FOG Status Reporter…Done
• Task Complete
• Updating Database…Done
• Rebooting system as task is conplete
  reboot: Restarting system
  #25:52

5315-Deploy
#0:00

• Verifying network interface configuration…Done

• Checking Operating System…Windows 10

• Checking CPU Cores…1

• Send method…NFS

• Attempting to send inventory…Done

• Checking In…Done

• Mounting File System…Done

• Checking Mounted File System…Done

• Starting Image Push

• Using Image: delme

• Looking for Hard Disks…Done

• Checking write caching status on HDD…Enabled

• Erasing current MBR/GPT Tables…Done

• Restoring Partition Tables (MBR)…Done

• Extended partitions…Done

• Expanding partition table to fill disk…Done

• Processing Partition: /dev/sdal (1)
  #0:28
  <<PARTCLONE>>
  #6:00

• Clearing ntfs flag…Done

• Stopping FOG Status Reporter…Done

• Resizing ntfs uolume (/dev/sda1)…Done

• Clearing ntfs flag…Done

• Backing up and replacing BCD…Done

• Changing hostname…Done

• Updating Computer Database Status

• Database Updated!

• Task is completed, computer will now restart.

reboot: Restarting system
#6:03

6303-Deploy
#0:00

• Verifying network interface configuration…Done
• Checking Operating System…Windows 10
• Checking CPU Cores…1
• Send method…NFS
• Attempting to check in…Done
• Mounting File System…Done
• Checking Mounted File System…Done
• Checking img variable is set…Done
• Attenpting to send inventory…Done
• Using Image: delme
• Looking for Hard Disk…Done
• Using Disk: /dev/sda
• Write caching not supported
• Preparing Partition layout
• Wiping /dev/sda partition information
• Erasing current MBBA3PT Tables…Done
• Creating disk with new label…Done
• Initializing /dev/sda with NTFS partition…Done
  #0:20
• Formatting initialized partition…Done
  #3:53
• Erasing current MBR/GPT Tables…Done
• Restoring Partition Tables (MBR)…Done
• Inserting Extended partitions…Done
• Attempting to expand/fill partitions…Done
  #3:57
  <<PARTCLONE>>
  #5:53
• Clearing ntfs flag…Done
  #5:53
• Resizing ntfs volume (/dev/sda1)…Done
  #8:51
• Clearing ntfs flag…Done
• Resetting UUIDs for /dev/sda
• Resettings swap systems
• Stopping FOG Status Reporter…Done
• Mounting directory…Done
• Changing hostname…Done
• Task Complete
• Updating Database…Done
• Rebooting system as task is complete
  reboot: Restarting system
  #8:54

I should also note that 5315 == kernel 4.3.0 and 6303 == kernel 4.4.1, as that’s probably relevant. I have tested tried an older kernel on 6303, but can if needed.

New tests indicate the slowdown exists in kernel 4.4.0 (x86_64) and 4.4.1 (x86_64), but 4.3.0 (x86_64) appears to be fine.

@Sebastian-Roth said:

Do you mean 6303 with kernel 4.3.0 is as fast as 5315?? Can you please verify if you see such drastic differences (where exactly? still resize ntfs…?) just by using older/newer kernel!

Actually it may be faster. I ran a deploy test using the same VM’s with 6307 using kernel 4.3.0, and it completed in 4:18. To be accurate, I would need to repeat all of the tests to compare 5315/4.3.0 and 6307/4.3.0 under the same server load. But it’s probably safe to say it’s as fast using the older kernel.

Here are the metrics comparing 6307 using kernel 4.3.0 and 4.4.1. Looking at the numbers, it’s safe to say that 6307/4.3.0 is faster than 5315/4.3.0 and far faster than 6307/4.4.1 when using a VM client on Hyper-V.

Let me know if there are any more measurements required. I’ll keep the VMs around for a day or two.

6307-4.3.0-Capture
#0:00
#0:18

• Resizing filesysten…Done
  #0:18
  #0:19
  <<PARTCLONE>>
  #14:19
  #14:20
• Resizing ntfs volune (/dev/sda1)…Done
  #14:20
  #14:25

6307-4.4.1-Capture
#0:00
#0:17

• Resizing filesysten…Done
  #4:53
  #4:54
  <<PARTCLONE>>
  #22:42
  #22:43
• Resizing ntfs volune (/dev/sda1)…Done
  #25:49
  #25:53

6307-4.3.0-Deploy
#0:00
#0:20

• Formatting initialized partition…Done
  #0:20
  #0:24
  <<PARTCLONE>>
  #4:11
  #4:11
• Resizing ntfs volume (/dev/sda1)…Done
  #4:12
  #4:14

6307-4.4.1-Deploy
#0:00
#0:20

• Formatting initialized partition…Done
  #3:53
  #3:57
  <<PARTCLONE>>
  #5:42
  #5:42
• Resizing ntfs volume (/dev/sda1)…Done
  #8:49
  #8:52

@Tom-Elliott I kicked off a script to build the kernels. Assuming they build and boot, I’ll report my findings.

@Sebastian-Roth
Looks like my script wasn’t copying the .config file, so I was building with the defaults.

I updated it and it built 4.3.2 and it boots fine now. I used the latest config and my script does “yes ‘’ | make oldconfig”. I’ll let it build the ones I mentioned earlier and test them. I’m about out of time for now, so I’ll post the results later.

@Tom-Elliott
I did not have time to write a sed script to include the additional patches from the wiki, but I can do that later or apply them by hand, once I have a chance to test the scripted builds.

Sound reasonable?

Build script finished quicker than I expected. Looks like it was introduced between 4.3.5 and 4.4.1. I’ll look at git bisect when I can make the time.

@Sebastian-Roth and @Tom-Elliott

The change to the kernel is actually in the scsi driver.

The commit that introduced the delay is 81988a0e6b031bc80da15257201810ddcf989e64, which applies changes to drivers/scsi/storvsc_drv.c.

I can confirm that reverting the diff on 4.4.2 brings the performance on the hyper-v client on par with 4.3.2. I can’t speak to the commit itself, as I just blindly reverted it and didn’t spend any time on digesting the patch itself.

My timings on the patched 4.4.2 was 2:14 for the deploy and 18:20 for the capture. That means the deploy is 50% faster and the capture is 27% slower than my tests for 4.3.2. @Tom-Elliott I did not include the additional patches you mentioned either, so I would need to retest both kernels under the same server conditions (and with the additional patches applied to 4.4.2) for more accurate results.

Hope this proves useful.

Still seems more like the issue should be with the block device, rather than the scsi driver. Seems like it would be related to caching or block size/block alignment of the ssd.

To me, these lines from the commit look most interesting:
/* Ensure there are no gaps in presented sgls */
blk_queue_virt_boundary(sdevice->request_queue, PAGE_SIZE - 1);

So, adding that line to 4.3.2 results in performance degradation and removing it from 4.4.2 results in performance increase. Guess all that’s left is to figure out (understand) what it actually does…

Maybe this makes more sense: blk_queue_virt_boundary(sdevice->request_queue, sdevice->page_size - 1);

I reached out to the author of the patch. I’ll post if any new information becomes available.

No resolution on this issue yet. (One of?) the author of the patch has confirmed the behavior and is investigating a kernel solution that doesn’t re-introduce the bounce buffers. No indication on how long this might take.