FOG Torrent

[Wayne Workman]

@Tom-Elliott Ok I get it. Where is the image stored in the mean time? I guess thats why you all were talking about knowing exactly how large the image is and the resizing and stuff - because the image would need temporarily stored on the HDD because it’s too big to fit in RAM.

Bet this would work a lot better for two-drive systems…

[Tom Elliott]

@Wayne-Workman two disks would be good to know getting the image but as we all know cross drive copying is VERY slow. Slower than same drive copying.

[Tom Elliott]

Specifically for spinner disks of course.

[Sebastian Roth]

@Tom-Elliott said:

Downloading the image could become faster but it still has to be put on the HDD once the image was downloaded, which would only be at HDD speed after the download. Speed difference may be faster to download the image but you still had to wait to put it on the HDD.

I can confirm that in my tests the speed kind of picked up as more and more of the image arrived at the clients. But the network/switch also had to handle more and more.

BUT if I remember correctly I used a modified torrent client which was able to write the image straight to disk. Yes, I found that link! Very much to the end you see ./btdownloadheadless.py /tmp/bittorrent/currentimage.torrent --saveas /dev/sda1

@Junkhacker Re-reading the old thread on torrent imaging… I was somehow aware of this in the back of my mind but I actually didn’t remember that you were using it even at the cost of having to download first before dumping the image to disk. Must have been really useful for you. Are you still not using multicast? If you are keen to get this back into FOG (including writing straight to disk - otherwise does not make sense to me) I offer to put in what I know and tested a while ago.

[Junkhacker]

@Sebastian-Roth said:

In my case it really hammered the network equipment as I remember. Lots and lots of TCP connections all over the place and - as you said - not much speed…

that is true, but if you were imaging a lab of computers it didn’t take long before most of that traffic was on a relatively local switch instead of coming all from the server room.

@Wayne-Workman said:

Where is the image stored in the mean time? I guess thats why you all were talking about knowing exactly how large the image is and the resizing and stuff - because the image would need temporarily stored on the HDD because it’s too big to fit in RAM.

what it was doing was creating a partition at the end of the drive to put the image on, then writing the downloaded files to the other partitions. when imaging was done, it would remove that partition and extend the partition before it to the end of the drive. works for the 100-350mb partition 1 remainder partition 2 setups that windows uses most of the time

@Sebastian-Roth said:

@Junkhacker Re-reading the old thread on torrent imaging… I was somehow aware of this in the back of my mind but I actually didn’t remember that you were using it even at the cost of having to download first before dumping the image to disk. Must have been really useful for you. Are you still not using multicast? If you are keen to get this back into FOG (including writing straight to disk - otherwise does not make sense to me) I offer to put in what I know and tested a while ago.

i’m still not using multicast. to be honest, unicast is perfectly fast enough for the way we use fog. the torrent-cast method was mostly an experiment and i hoped a solution to the problems most people have with multicast. it just never worked out that way.

if your method writes straight to disk, does the end client still seed somehow? one of the biggest advantages of the torrent-cast method was that if you had a slow link to the server, your local hosts could help make up for it (old buiding they won’t upgrade the uplink to, remote office, that kind of thing). i also had an idea about how the fog client on computers that weren’t being imaged could be pre-seeded with images for an upcoming imaging task, but that also never came to fruition.

[Wayne Workman]

@Junkhacker If you could get it working again that would be great. And it’s perfectly fine if the data is not received sequentially in my mind - this is just the nature of torrenting - it’s designed to be this way and has many advantages. Torrent imaging should work like torrenting and not like unicasting (sorry sebastian).

[Sebastian Roth]

@Junkhacker said:

if your method writes straight to disk, does the end client still seed somehow?

Why not? The modified torrent client thinks it is using a normal file - writing and reading random blocks within that file/block device. If I remember correctly more and more seeders and speed build up over time…

[Wayne Workman]

@Sebastian-Roth Oh. The way you described it before seemed like the clients only accepted pieces in sequential order… If they just write random pieces to where they should go on the drive, that’d be the best solution.

[Junkhacker]

@Sebastian-Roth said:

Why not? The modified torrent client thinks it is using a normal file - writing and reading random blocks within that file/block device. If I remember correctly more and more seeders and speed build up over time…

ah, so this is a completely uncompressed block device torrent file, i see. my method uses the same images as standard Fog imaging

[Sebastian Roth]

@Junkhacker Good point! Thanks for mentioning. Now more and more I remember why I totally dismissed torrent for imaging for me. I guess there is no way you could combine random block torrent transfer with partclone (filesystem aware) imaging that is writing to disk on the fly. Only raw would be possible. With current HDs being up to 1 TB and more even in clients this would be a huge wast of network traffic…

Reading through some old mails with a college I remembered the HD in the clients being a massive bottleneck as well. Disk IO literally dropped down to the floor as clients upload (read) and download (write) random blocks at the same time.

[Junkhacker]

@Sebastian-Roth well, we could still shrink the partition to the size of the actual data usage like we do now. that would keep you from having 1 TB images unless there’s actually a TB of data on it. but you still couldn’t compress it, and it would be a torrent only image, you couldn’t do normal unicast/multicast deployments with it.

[Wayne Workman]

@Junkhacker What if a regular image was decompressed & then mounted on the FOG server, and then it’s raw data sent out via a torrent task ?

I still think that sticking the standard partition data at the end of the drive is perfectly acceptable. Torrent task type is made to overcome network issues, not HDD issues.