Upgrade from 1.5.7 to 1.5.8 issues

[george1421]

@JJ-Fullmer I appreciate your feed back and clarification. I also want to add a clarification to your post to not confuse others that may read this in the future. Hold on I feel this is going to be a wall of text…

You have a 2 link bonded connection. That doesn’t imply that it gives you 2Gb/s of bandwidth (i.e. twice as fast as a single link). LAG/bonded/teamed groups don’t work that way (at least with today’s technology). A 2 link bonded group would give you 2 1GbE links into that device. It works the same way as adding and additional lane to a highway. Your road can carry more traffic, but the speed limit is still 70mph. Also assuming we are talking about lacp/802.3ad/mlt links there is a hashing algorithm that is used to decide which traffic flows across which link. Once the link route has been determined (i.e. link 1 or link 2) that link route does not change during the lifetime of the communication between the two devices (assuming that port based hashing is not used). So the guidance is between any two devices you will never have any faster communication than the speed of a single link. With only 2 actors (FOG server and target computer) you will only have the best speed as a 1GbE link. So having a LAG/MLT/bonded link on the target computer will not help you one bit for imaging. Having a LAG/MLT/bonded link on the server end will not help you when there is only a FOG server and target computer involved. Having a LAG group on the FOG server when more than 2 actors are involved will help you to spread the load across the links based on the link hashing protocol.

See I told you it was going to be a wall of text.

[JJ Fullmer]

@george1421 While I agree and understand how it all works. I have found that we did get an increase in speed when we setup the aggregated adapter on the storage node. Even with just one client going. But perhaps that’s really just agreeing with your statements. As like on a highway if it was 1 lane, you often slowdown cause of the other slow drivers and perhaps I just opened up a metaphorical passing lane for fog images to go the full speed limit at. You do also have to consider all the switches it goes through and yada yada. And of course it’s all more complicated. Point being, I didn’t get a 2x boost when we aggregated the server link, but we did get a boost, so I wouldn’t deter anyone with the equipment capable of it from giving it a try.

[Sebastian Roth]

@Chris-Whiteley said in Upgrade from 1.5.7 to 1.5.8 issues:

After a test with the new init I am still having the issues of speed decrease. It is almost double what it used to take. My images being pushed out was around 2:30 minutes and now it is 4:17.

Ok, then we need to start looking at other things I suppose. Did you try going back to use the inits from 1.5.7 (only inits, not the kernel as a first try) as suggested by @george1421 yet? https://fogproject.org/binaries1.5.7.zip

What if there is something in your network that changed?

[Chris Whiteley]

@Sebastian-Roth Nothing on the network has changed and I did try @george1421 suggestion as well. It was a 2 minute install/upgrade of 1.5.8 and the snapshot revert took about 2 minutes max. Sorry, I have been on vacation the last couple of days but I wanted to make sure and get back to you on those things.

[Sebastian Roth]

@Chris-Whiteley So what’s the outcome of testing George’s suggestion? Sorry for nagging. Take your time…

[Chris Whiteley]

@Sebastian-Roth it was the same as the new init

[Sebastian Roth]

@Chris-Whiteley So that would leave us with an issue in the Linux kernel or FOG server. Kernel is easier to test. Leave the inits from 1.5.7 in place and grab the kernel binaries (bzImage*) from the archive to use. See if it’s back to speed like this.

[george1421]

@Sebastian-Roth said in Upgrade from 1.5.7 to 1.5.8 issues:

So that would leave us with an issue in the Linux kernel or FOG server.

-OR- something outside of FOG causing the delay.

[Chris Whiteley]

@Sebastian-Roth I just tested with the old bzimage for 1.5.7 and the speed was much faster and what I am used to.

[george1421]

@Chris-Whiteley If you are testing the bzImage from 1.5.7 and 1.5.8 on different days, could you test the bzImage from 1.5.8 now? I’m just trying to rule out other variables, because what you’ve told us, should not be. I’m not saying it can’t happen, only its a bit strange in the same series (4.19.x)

[Chris Whiteley]

@george1421 I am testing it now. I found the 1.5.8 binaries.

[george1421]

@Chris-Whiteley That was the 4.19.101 version.

But maybe now I’m confused. I’m trying to build a truth table and I thought we zeroed in on something

1.5.8:bzImage 4.19.101 1.5.8:init.xz == Slow (partclone 0.3.13)
1.5.8:bzImage 4.19.101 1.5.7:init.xz == Slow (partclone 0.3.13)
1.5.7:bzImage 4.19.65 1.5.7:init.xz == Fast (partclone 0.2.89)
1.5.8:bzImage 4.19.101 1.5.7:init.xz == ?? (partclone 0.2.89)

[Chris Whiteley]

@george1421 This is correct so far. This last test proved to be faster this time…hmmm…This has also been the most stable as far as not going from ridiculous speeds down to what it actually should be. Now it starts where it normally rests at.

[george1421]

@Chris-Whiteley said in Upgrade from 1.5.7 to 1.5.8 issues:

This last test proved to be faster this time

Just as fast, or faster than last time?

Also this is why I wanted you to test the same day with the same set of circumstances. Just in case there was something in network land that was different than last week.

[Chris Whiteley]

@george1421 Just as fast as when I had 1.5.7 and it was working. Both tests from normal SSD and NVMe worked as I think they should. How do I now get an output of what versions of things I have so that you guys can figure out what you need to?

[george1421]

@Chris-Whiteley well for bzImage that’s easy file bzImage will tell you the current version of bzImage.

For init.xz its not as easy but not hard either. If we use md5sum utility we can get the fingerprint of init.xz file.

This is for 1.5.7

md5sum init.xz
913326f3317b577be3cb65a7bf332afb  init.xz

If you have the chance, since something is different, can you test with the init.xz from 1.5.8? Its best to do all in one day if possible.

[Chris Whiteley]

@george1421

bzimage - 4.19.101
init.xz - 9133326

[Chris Whiteley]

@george1421 just tested with the 1.5.8 init.xz and got the slowness again

[george1421]

@Chris-Whiteley said in Upgrade from 1.5.7 to 1.5.8 issues:

just tested with the 1.5.8 init.xz and got the slowness again

Ok so what the developers (or myself) need to do is pluck out partclone from the 1.5.8 inits and install it in the 1.5.7 inits. This will point exactly to partclone.

The issue with 1.5.8 (not a problem just many things change) is that the system used to create the virtual hard drive (init.xz) was upgraded, this also upgraded a number of support modules inside the init.xz. So on the surface we don’t know if the latest version of gzip is doing this or partclone. At least in my head.

[Chris Whiteley]

@george1421 Thanks for the update! Let me know what you guys find and if I can be of anymore help.