We are back on track!
Here a snapshot with dev-branch kernel and the new init_partclone_0.3.13.xz files. The image we are deploying was previosly captured with dev kernel an dev init.xz.
At the moment we are also running a multicast with around 12Gb/min - 13GB/min. Maybe everything has even become a little faster.
Again great support and excellent work! Thank you guys! If there are further tests let us know.
We are still using FOG Server dev-branch 1.5.7.112.
All binaries (1.5.3 up to 1.5.7) used the partclone version 0.2.89. Maybe thats the problem? The binaries from dev-branch where running on partclone 0.3.12.
We are running several tests at the moment. Only using the Binaries which can be downloaded from https://fogproject.org/binaries1.5.x.zip.
FOG running dev-branch 1.5.7.112
Here are some results.
Binaries 1.5.7: deploy speed around 12GB/min. bzImage-1.5.7: Linux kernel x86 boot executable bzImage, version 4.19.48 (jenkins-agent@Tollana) #1 SMP Sun Jul 14 13:08:14 CDT , RO-rootFS, swap_dev 0x7, Normal VGA
Binaries 1.5.6: deploy speed around 12GB/min. bzImage-1.5.6: Linux kernel x86 boot executable bzImage, version 4.19.36 (jenkins-agent@Tollana) #1 SMP Sun Apr 28 18:10:07 CDT , RO-rootFS, swap_dev 0x7, Normal VGA
Binaries 1.5.5: deploy speed around 12GB/min. bzImage-1.5.5: Linux kernel x86 boot executable bzImage, version 4.19.1 (sebastian@Tollana) #1 SMP Fri Feb 22 01:04:27 CST 2019, RO-rootFS, swap_dev 0x8, Normal VGA
Binaries 1.5.4: deploy speed around 12GB/min. bzImage-1.5.4: Linux kernel x86 boot executable bzImage, version 4.16.6 (builder@4c3c12e8cfd6) #4 SMP Wed May 9 22:08:36 UTC 201, RO-rootFS, swap_dev 0x7, Normal VGA
Binaries 1.5.3: deploy speed around 12GB/min. bzImage-1.5.3: Linux kernel x86 boot executable bzImage, version 4.15.2 (builder@c38bc0acaeb4) #5 SMP Tue Feb 13 18:30:08 UTC 20, RO-rootFS, swap_dev 0x7, Normal VGA
@Sebastian-Roth
We did two re-captures now. One with 1.5.3 binaries and one with current binaries from dev-branch 1.5.7.112.
We have no improvments in deploying with 1.5.7. The speed of the binaries 1.5.7.112 is around 5 GB/min.
Luckily the binaries 1.5.3 boost up to 13 GB/min. Thats the speed we had before.
For now we will stick to 1.5.3 binaries running behind FOG 1.5.7.112.
If there are improvments please let us know so we can test them.
@Sebastian-Roth @Quazz @george1421
We only have computers with the following specs:
Made ten iperf3 tests in a row and got an average of 9.7 retries. Is that really so bad?
We also checked the binaries from 1.5.3 to 1.5.6. with the old image and recieved the following error:
read image_hdr device_size error
Now we are capturing a new image with the 1.5.3 binaries. The above error may have something to do with a mismatching partclone version. Hopefully the deploy speed will be at 12GB/min. If so we will try to capture an image with the current binaries.
Linux debian 5.3.0-1-amd64 #1 SMP Debian 5.3.7-1 (2019-10-19) x86_64 GNU/Linux
@Quazz said in Slowdown Unicast and Multicast after upgrading FOG Server:
I vaguely recalled someone having this problem before with similar outcomes.
Linking here for further reference: https://forums.fogproject.org/topic/13733/hp-elitebook-830-gen-6-issues-capturing-images-and-deploying-images/10
Tried the bzImage529 and checked if RAMDISK size is correct.
Also tried the following Kernels which all end up with a KERNEL PANIC:
4.13.4, 4.11.1, 4.10.1, 4.9.11 and 4.8.11
Other kernels starting with 4.15.2 and above seem to work but not with the sufficient speed.
@Quazz said in Slowdown Unicast and Multicast after upgrading FOG Server:
Are you on the dev-branch
Yes I am on the dev-branch 1.5.7.109 (bzImage Version: 4.19.101)
@Sebastian-Roth @Quazz @george1421
I have some good and bad news.
First the good ones:
I created a deploy using the FOG.Wiki https://wiki.fogproject.org/wiki/index.php/Debug_Mode#Win_7
Therefor I booted a Clonezilla (2.6.4-10-amd64) Flashdrive and mounted the NFS-Share from FOG Server.
I started the deploy with the following command:
cat /images/IMAGEPATH/d1p2.img | zstd -d | sudo partclone.restore -O /dev/sda2 -N -f -i
Tried a deploy with the fog client and still one-third of the expected speed.
I think there is something wrong in the deploy process.
Main difference I can see is that the FOS uses Partclone 0.3.12 and Clonezilla 0.3.13.
Did my best. After trying I noticed that the “frozen state” was active. Removing the SSD power cable while PC was powered on removed the frozen state. Then I was able to do a secure erase. A dd from ramdisk to /dev/sda was at the same speed as before. So no luck at all.
Server
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from x.x.x.x, port 50672
[ 5] local x.x.x.x port 5201 connected to x.x.x.x port 50674
[ ID] Interval Transfer Bandwidth
[ 5] 0.00-1.00 sec 108 MBytes 903 Mbits/sec
[ 5] 1.00-2.00 sec 112 MBytes 942 Mbits/sec
[ 5] 2.00-3.00 sec 112 MBytes 942 Mbits/sec
[ 5] 3.00-4.00 sec 112 MBytes 942 Mbits/sec
[ 5] 4.00-5.00 sec 112 MBytes 942 Mbits/sec
[ 5] 5.00-6.00 sec 112 MBytes 942 Mbits/sec
[ 5] 6.00-7.00 sec 112 MBytes 942 Mbits/sec
[ 5] 7.00-8.00 sec 112 MBytes 942 Mbits/sec
[ 5] 8.00-9.00 sec 112 MBytes 942 Mbits/sec
[ 5] 9.00-10.00 sec 112 MBytes 942 Mbits/sec
[ 5] 10.00-10.04 sec 4.35 MBytes 937 Mbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bandwidth Retr
[ 5] 0.00-10.04 sec 1.10 GBytes 939 Mbits/sec 11 sender
[ 5] 0.00-10.04 sec 1.10 GBytes 938 Mbits/sec receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Client
Connecting to host x.x.x.x, port 5201
[ 5] local x.x.x.x port 50674 connected to x.x.x.x port 5201
[ ID] Interval Transfer Bitrate Retr Cwnd
[ 5] 0.00-1.00 sec 113 MBytes 947 Mbits/sec 3 258 KBytes
[ 5] 1.00-2.00 sec 112 MBytes 943 Mbits/sec 0 364 KBytes
[ 5] 2.00-3.00 sec 112 MBytes 939 Mbits/sec 2 232 KBytes
[ 5] 3.00-4.00 sec 112 MBytes 943 Mbits/sec 1 318 KBytes
[ 5] 4.00-5.00 sec 112 MBytes 943 Mbits/sec 2 211 KBytes
[ 5] 5.00-6.00 sec 112 MBytes 943 Mbits/sec 0 364 KBytes
[ 5] 6.00-7.00 sec 112 MBytes 943 Mbits/sec 1 267 KBytes
[ 5] 7.00-8.00 sec 112 MBytes 943 Mbits/sec 1 364 KBytes
[ 5] 8.00-9.00 sec 112 MBytes 943 Mbits/sec 0 366 KBytes
[ 5] 9.00-10.00 sec 112 MBytes 943 Mbits/sec 1 282 KBytes
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval Transfer Bitrate Retr
[ 5] 0.00-10.00 sec 1.10 GBytes 943 Mbits/sec 11 sender
[ 5] 0.00-10.00 sec 1.10 GBytes 942 Mbits/sec receiver
iperf Done.
/dev/sda:
Timing cached reads: 29692 MB in 1.99 seconds = 14911.18 MB/sec
Timing buffered disk reads: 1614 MB in 3.00 seconds = 537.87 MB/sec
Here the output:
/dev/sda:
ATA device, with non-removable media
Model Number: Samsung SSD 860 EVO 500GB
Serial Number: S3Z2NB1KA50028H
Firmware Revision: RVT01B6Q
Transport: Serial, ATA8-AST, SATA 1.0a, SATA II Extensions, SATA Rev 2.5, SATA Rev 2.6, SATA Rev 3.0
Standards:
Used: unknown (minor revision code 0x005e)
Supported: 11 8 7 6 5
Likely used: 11
Configuration:
Logical max current
cylinders 16383 16383
heads 16 16
sectors/track 63 63
--
CHS current addressable sectors: 16514064
LBA user addressable sectors: 268435455
LBA48 user addressable sectors: 976773168
Logical Sector size: 512 bytes
Physical Sector size: 512 bytes
Logical Sector-0 offset: 0 bytes
device size with M = 1024*1024: 476940 MBytes
device size with M = 1000*1000: 500107 MBytes (500 GB)
cache/buffer size = unknown
Form Factor: 2.5 inch
Nominal Media Rotation Rate: Solid State Device
Capabilities:
LBA, IORDY(can be disabled)
Queue depth: 32
Standby timer values: spec'd by Standard, no device specific minimum
R/W multiple sector transfer: Max = 1 Current = 1
DMA: mdma0 mdma1 mdma2 udma0 udma1 udma2 udma3 udma4 udma5 *udma6
Cycle time: min=120ns recommended=120ns
PIO: pio0 pio1 pio2 pio3 pio4
Cycle time: no flow control=120ns IORDY flow control=120ns
Commands/features:
Enabled Supported:
* SMART feature set
Security Mode feature set
* Power Management feature set
* Write cache
* Look-ahead
* Host Protected Area feature set
* WRITE_BUFFER command
* READ_BUFFER command
* NOP cmd
* DOWNLOAD_MICROCODE
SET_MAX security extension
* 48-bit Address feature set
* Device Configuration Overlay feature set
* Mandatory FLUSH_CACHE
* FLUSH_CACHE_EXT
* SMART error logging
* SMART self-test
* General Purpose Logging feature set
* WRITE_{DMA|MULTIPLE}_FUA_EXT
* 64-bit World wide name
Write-Read-Verify feature set
* WRITE_UNCORRECTABLE_EXT command
* {READ,WRITE}_DMA_EXT_GPL commands
* Segmented DOWNLOAD_MICROCODE
* Gen1 signaling speed (1.5Gb/s)
* Gen2 signaling speed (3.0Gb/s)
* Gen3 signaling speed (6.0Gb/s)
* Native Command Queueing (NCQ)
* Phy event counters
* READ_LOG_DMA_EXT equivalent to READ_LOG_EXT
* DMA Setup Auto-Activate optimization
Device-initiated interface power management
* Asynchronous notification (eg. media change)
* Software settings preservation
Device Sleep (DEVSLP)
* SMART Command Transport (SCT) feature set
* SCT Write Same (AC2)
* SCT Error Recovery Control (AC3)
* SCT Features Control (AC4)
* SCT Data Tables (AC5)
* reserved 69[4]
* DOWNLOAD MICROCODE DMA command
* SET MAX SETPASSWORD/UNLOCK DMA commands
* WRITE BUFFER DMA command
* READ BUFFER DMA command
* Data Set Management TRIM supported (limit 8 blocks)
* Deterministic read ZEROs after TRIM
Security:
Master password revision code = 65534
supported
not enabled
not locked
frozen
not expired: security count
supported: enhanced erase
4min for SECURITY ERASE UNIT. 8min for ENHANCED SECURITY ERASE UNIT.
Logical Unit WWN Device Identifier: 5002538e408a5e55
NAA : 5
IEEE OUI : 002538
Unique ID : e408a5e55
Device Sleep:
DEVSLP Exit Timeout (DETO): 50 ms (drive)
Minimum DEVSLP Assertion Time (MDAT): 30 ms (drive)
Checksum: correct
Here the results:
I also copied some rows out of mysql fog.tasks. Here you can see the difference between speed and duration.
First a normal multicast with FOG 1.5.3.* (dev-branch). Sorry I don’t know the exact version anymore.
| 17919 | Multi-Cast Task | 2020-01-24 08:11:41 | 2020-01-24 08:12:33 | 143 | 389 | 4 | 0 | fog | 0 | 0000-00-00 00:00:00 | 8 | 0000000100 | 12.42GB | 00:21:00 | 00:00:00 | 249.006 GiB | 100 | 249.124 GiB | 1 | 1 | | 0 | 1 | | |
This is the last single deploy I made with FOG 1.5.3.* (dev-branch):
| 18027 | Deploy Task | 2020-02-10 15:16:44 | 2020-02-10 15:17:34 | 214 | 391 | 4 | 0 | fog | 0 | 0000-00-00 00:00:00 | 1 | 0000000100 | 12.72GB | 00:20:38 | 00:00:00 | 250.320 GiB | 100 | 250.650 GiB | 1 | 9 | | 0 | 1 | | |
Here a multicast after upgrading to FOG 1.5.7.109 (dev-branch):
| 18032 | Multi-Cast Task - 04-634-Reihe-1 | 2020-02-10 16:40:43 | 2020-02-10 16:42:14 | 214 | 391 | 4 | 0 | fog | 0 | 0000-00-00 00:00:00 | 8 | 0000000100 | 1.13GB | 03:52:47 | 00:00:00 | 250.422 GiB | 100 | 250.650 GiB | 1 | 1 | | 0 | 1 | | |
And this single deploy was created with FOG 1.5.7.109 (dev-branch) the day after:
| 18042 | Deploy Task - 04-628-36 | 2020-02-11 12:53:24 | 2020-02-11 12:54:13 | 347 | 392 | 4 | 0 | fog | 0 | 0000-00-00 00:00:00 | 1 | 0000000100 | 4.61GB | 00:56:53 | 00:00:00 | 250.371 GiB | 100 | 250.620 GiB | 1 | 9 | | 0 | 1 | | |
It is a normal 2.5 in. SATA Samsung 860 EVO with 500GB (Model: MZ-76E500B/AM). Any other suggestions?
Will quit work for now and restart at 8 a.m. CET
@Sebastian-Roth
No performance tweaks. Deployment around 1 hour.
Hardware Client: Dell Optiplex 9010, i7, 16GB RAM, SATA Samsung EVO 860 500GB
So we are ready to try the other things 
@george1421
We tried different two older kernels but the results where even worse.
FOG-Server: VM with 6 cores and 8GB RAM
Ubuntu 16.04, FOG dev-branch 1.5.7.109 (before: 16.04, FOG dev-branch 1.5.3.?)
We are having a huge slowdown in deploying our images. Here is a comparison of Unicast and Multicast:
Unicast FOG 1.5.3: 21 minutes 20 seconds
Unicast FOG 1.5.7: 1 hour 7 minutes 40 seconds
Multicast FOG 1.5.3: 22 minutes 36 seconds
Multicast FOG 1.5.7: 3 hours 53 minutes 31 seconds
Image Size on Client around 250GB.
Compression Format: Partclone Zstd.
We just startet a new capture to see if the problem still exists.
Otherwise we have no clue what happened.
Anyone else having this problem?
@Sebastian-Roth
Thanks for the reply! Did an update to 1.5.5.1 two weeks ago. Everything works fine so far!