Another slow Deployment qustion

[Wayne Workman]

@george1421 I didn’t suggest one at a time, I suggested two at a time.

I can test this in the lab and give hard numbers, I know this because I’ve had max clients set to 10 before and it was terrible, and changed it to 2 and now it’s performing great.

Also, pretty sure the figures Partclone displays is read/write speed, not network transfer speed.

[Wayne Workman]

@george1421 Take this video for example, look at the video @ 5:34

We see the space used on the image is 31.7, write speed is 2.29GB/min.

We see elapsed time is 13 minutes and 31 seconds.

31.7 divided by 2.29 = 13.84, or 13 minutes and 50 seconds.

Therefore the rate that Partclone displays is write speed (or read speed), not network transfer rate. The image is sent across the network in compressed form, so there is much more room available on the network. Once network saturation is reached, it’s reached. However, if you continue to pile on imaging tasks, eventually the HDD seek time is maxed, and you actually start loosing network saturation. This is my reasoning for setting Max Clients to 2 or 3, not 10.

The idea is to saturate the available bandwidth for imaging tasks, but not exceed the HDD’s seek ability to keep up. If you exceed the HDD’s ability to keep up due to seek times, then you’re going backwards.

The optimal setting for max clients should saturate the network fully, but not exceed the HDD’s ability to seek and keep up with the full network speeds.

https://www.youtube.com/watch?v=OgKwhuNB1cs

[Arsenal101]

Great replies guy! I thought that the rate was always the network transfer speed and not the write speed…

@Wayne-Workman I am in now way discrediting your method. I just don’t think its the best set up for the needs that we have right now. I want to stay away from multicast for now as we dont have out layer 3 switch/router configured to pass broadcast packets as we have had several issues with staff Looping the network and bringing down everything instead of just one building (this was also before we turned spanning tree on), and I know my boss wont be a fan of increasing the broadcast domain even if was just for the summer.

on top of that we don’t have the time to build more storage nodes for fog so it’s kind of a trade off yes they image slower but at the same time the you’re doing 10 at a time per storage node (20 at a time in my case 1 master + 1 storage) and 800MB-PM is plenty acceptable for what we need it for.

@george1421 Maybe at some point I could convince my boss to throw a 10gb NIC and some decent hardware for our fog server in the budget. I could try some of the stuff you are suggesting. and that would really boost the speed right up.

If we could ever get it 10gb to the end PC the only thing slowing the process down would be the human factor!

[Wayne Workman]

@Arsenal101 I was not suggesting multicast? But ok.

[Arsenal101]

@Wayne-Workman Sorry I concatenated @Sebastian-Roth’s and your reply in my head.

[george1421]

@Arsenal101 said in Another slow Deployment qustion:

If we could ever get it 10gb to the end PC the only thing slowing the process down would be the human factor!

While 10G to the desktop would be really nice, its not necessary and a bit of a waste because I would suspect on the target end, the disk or CPU is your limiting factor and not the network. For the server managing multiple data streams I can see the network and then the disk subsystem being the bottleneck.

[Arsenal101]

@george1421 Pretty much, It would just be cool to say we have 10gb to the desktop… A SSD drive would be sweet though!

[george1421]

@Wayne-Workman said in Another slow Deployment qustion:

@george1421 Take this video for example, look at the video @ 5:34

We see the space used on the image is 31.7, write speed is 2.29GB/min.

We see elapsed time is 13 minutes and 31 seconds.

31.7 divided by 2.29 = 13.84, or 13 minutes and 50 seconds.

Therefore the rate that Partclone displays is write speed (or read speed), not network transfer rate.

Wayne I fully agree with you and understand that the part clone display is not actual network usage, but its the best metric we have without getting into to much tech. So the point it it’s not accurate but its the best we have (like the Windows Performance Index, at least its some metric that we can use as a baseline).

My testing with the intel nuc as the FOG server and deploying to a e6400 with a HHD and SSD. You can see the speed difference in just switching the target from a HDD to SSD with everything else being the same.

https://forums.fogproject.org/topic/6373/fog-mobile-deployment-server-for-200-usd-finished/3

“I replace the seagate rotating hard drive in the e6400 with a Crucial MX100 256GB SSD I had laying around. I again redeployed the same image as in the previous tests, this time the transfer rate was 7.8GB/m (130MB/s {!!faster than wire speed!!} ) according to partclone. As compared to 5.1GB/m with a rotating disk in the target. I booted the e6400 back into debug mode and ran hdparm -Tt /dev/sda hdparm reported 242MB/s for buffered disk reads as compared to 80MB/s with rotating media.”

[Wayne Workman]

@george1421 We can get a exact metric if we turn on FTP_Image_Size on the server, it’ll display the image size. Then we can use total time elapsed during imaging to calculate throughput.

[Tom Elliott]

@Wayne-Workman Throughput is already displayed on the dashboard (granted not per host), but if the network is NOT saturated as this is trying to lead to, you should see “plenty” of available bandwidth of the network there.

The part that bothers me is this still feels more like a networking issue than a seek/io issue. While I do totally understand IO as being a part of this, your network is most likely the first culprit. Primarily considering the mount point is used across the network to begin with.

[Sebastian Roth]

@Arsenal101 said:

Maybe at some point I could convince my boss to throw a 10gb NIC and some decent hardware for our fog server in the budget.

As George already said this is not very wise. I might add that multicast would solve all that I am pretty sure. I don’t understand why everyone is so afraid to get multicast running?? What’s that drawback that I don’t seem to see? Please tell me…

[Quazz]

@Sebastian-Roth Only downside is reduced flexibility, that is to say if you have 20 computers and they need 7 different images, multicast won’t really be that useful.

Other than that, multicast is pretty great. Is there any word on the state of the torrent mechanic or is that on hold/abandoned? I personally think multicast would be better than the torrenting from a network saturation/resources perspective, but maybe I’m wrong?

[Sebastian Roth]

@Quazz said:

Only downside is reduced flexibility, that is to say if you have 20 computers and they need 7 different images, multicast won’t really be that useful.

While you are right about the impossibility to deploy different images via multicast I don’t see this as a fair argument. A network/switch can handle unicast and multicast at the same time so it’s not an either or thing. Use whichever is appropriate for the task you want to run. Multicasting when you actually want different images is stupid and unicasting when you want the same image is not very wise either.

Other than that, multicast is pretty great. Is there any word on the state of the torrent mechanic or is that on hold/abandoned? I personally think multicast would be better than the torrenting from a network saturation/resources perspective, but maybe I’m wrong?

There has been a discussion on and off on the forum. Just search for ‘torrent’ and I am sure you’ll find it.

[Wayne Workman]

While I still stand by what I’ve already said - just try multicast and see if it works - for same images at once. Stick computers into a group, and initiate multicast from the group.