Imaging Issue
-
Well, here’s what I’m seeing…
You’re FOG server, and your FOG storage nodes are on the same segment. All the computers in all the buildings use this ONE fog server.
The problem computer works with that fog server in building B, but not in building A.
Additionally, it’s just these 60 machines…
Hmm…
Let’s do a test…
Grab a computer that works fine, one from another part of building A.
Take it to where those 60 machines are (the problem ones) and plug it in using one of their network ports.[U]See if it works.[/U] If it doesn’t, you’ve pinpointed the switch being the problem, or the trunk config for wherever it’s up-link goes to.
-
ALSO,
Take one of the problem computers,
And plug it into a network port that a working computer was using. See if it works there. If it does, again you’ve pinpointed the switch being the issue.
-
Also a thing to try is using the realtek.pxe file. There was an issue with rtl8169 cards with the eeprom that caused all kinds of weird issues.
-
[quote=“SeqSupport@Edkey, post: 46075, member: 27616”]Also a thing to try is using the realtek.pxe file. There was an issue with rtl8169 cards with the eeprom that caused all kinds of weird issues.[/quote]
But the computers work fine with the same FOG server, but different physical location… I’m not sure the realtek.pxe file would help, but I’m sure it wouldn’t hurt anything to try.
-
[quote=“Wayne Workman, post: 46077, member: 28155”]But the computers work fine with the same FOG server, but different physical location… I’m not sure the realtek.pxe file would help, but I’m sure it wouldn’t hurt anything to try.[/quote]
I just threw it out there because we have had issues with certain older managed/unmanaged switches not liking tftp and our new rtl8169 nics.
-
[QUOTE]Grab a computer that works fine, one from another part of building A.
Take it to where those 60 machines are (the problem ones) and plug it in using one of their network ports.[/QUOTE]I did try this earlier - that computer works fine on the same port as the problem computer. It’s so weird - like, everything points to it being a switch port issue, but then I take another machine to that same port and it’s okay.
[QUOTE]Also a thing to try is using the realtek.pxe file. There was an issue with rtl8169 cards with the eeprom that caused all kinds of weird issues.[/QUOTE]
I’ll give this a try hopefully soon. Both of those labs are full at the moment with classes.
-
Do the reverse when you can.
Take one of the problem computers, move it to a known-good port in another part of the building. One that’s [B]not[/B] connected to the same switch that the other 60 are on.
-
Test done - realtek.pxe and realtek.kpxe both do the same thing - link down, network starting, and then link up.
[QUOTE]Take one of the problem computers, move it to a known-good port in another part of the building. Preferably one that’s not connected to the same switch that the other 60 are on.[/QUOTE]
I did that earlier too, guess I forgot to mention that. Problem computer in Building A, moved to another room on another switch, same behavior.
-
Then the problem must be the top-level device in building A.
Because the Lenovo M72e Tiny works in Building “B”, with FOG server/nodes
But not in Building “A” with the [B]same[/B] FOG server/nodes.YET, other clients in building “A” work with the [B]same[/B] FOG server/nodes; even if they are connected to the Lenovo M72e Tiny’s network ports.
I’m very sure this is a network issue.
-
I’d like to believe that and it really does appear to be a network issue. But what I don’t understand is the issue started happening immediately after the update.
2 weeks ago I was on r2961 and imaged the lab without a problem. This week I updated to r3287 and this problem appears. Nothing else changed - no switch configurations, no dhcp server changes. The only variable that changed here is the revision of fog.
I think tomorrow I’ll fire up another virtual server and install r2961. I’d like to have that running and temporarily point dhcp option 66 to the new server just to (hopefully)prove or deny that the network is still functioning correctly. I’ll let you know what I find.
-
If it does work, we should slowly iterate up in the revisions until we find where it breaks.
This will help the developers with creating a fix.
-
What’s most intriguing to me is that it doesn’t work at one level but works fine elsewhere?
-
[QUOTE]What’s most intriguing to me is that it doesn’t work at one level but works fine elsewhere?[/QUOTE]
That’s very interesting to me too. And yet there are other machines connected to the same switch that are working, exact same port configurations.So the latest update: I did a fresh install of Ubuntu 12.04 and fog r2961. Much to my surprise, I’m seeing the same behavior - the network is starting before the link is up, and it’s not getting an IP. “Downloading kernels and inits…OK” Is it downloading the latest and greatest kernel/init, no matter which revision I’m on? I did go into settings and revert back to a kernel from February, but same results. What does the init.xz file do, and is there a way I can get a version of that from back in February?
-
It is getting the latest kernels and init’s every time. No it’s not possible to download one from february anymore, but this is not where things are failing…directly. Nor is it the kernel if the interface comes up outside of that scope. The only thing I can think is the other Nodes/Servers you have are on older Init’s?
-
This post is deleted! -
Okay, good to know. I’m going to try to get those files off the VM backup just to see if they make a difference.
-
Where is this new r2961 build located, physically? Building “A” or the troubled building “B”
-
[QUOTE]Where is this new r2961 build located, physically? Building “A” or the troubled building “B”[/QUOTE]
The new server is physically located in building A. All of our virtual infrastructure is there.So I do have good news - it’s working, with some recovered files. Undionly.kpxe appears to have nothing to do it. The problem appears no matter which version of that I have.
The culprits seem to be init.xz and bzImage. I restored both of them from a backup from 2 weeks ago. When those restored files are in place, everything works happily on both r2961 and r3278. As Tom pointed out earlier, it looks like the revision version of FOG does not apply to these 2 files. It’s worth noting that replacing one or the other didn’t fix the problem - I had to replace both of them for it to work.
I see one major difference in debug mode, I attached a picture for clarity. The difference is I see the sending discover. It sends one discover, then the link comes up, another discover, and bingo - it gets an IP. That sending discover process does not show up in the bzImage/init.xz file combination that the installer script is currently downloading.
[url=“/_imported_xf_attachments/1/1920_Debug.JPG?:”]Debug.JPG[/url]
-
Ok now we’re getting somewhere, good job! It’s not your network after all.
From here, it’s up to Tom.
I’d recommend making a copy of those two files and put them in a safe spot for the future…
-
While on that prompt, what’s the output of uname -rm?
