Horrible performance on migrated server
-
Hi there,
Like many I ran Fog for the first time on a basic desktop just to make sure it would work. Well it worked like a champ, a few issues here and there but for the most part did everything I needed it to. So, as with many, I upgraded to a much bigger piece of hardware and began with migrating everything over.
I’m not sure where I screwed up but I copied the /opt/fog folder over intact from the old servers hard drive slaved in, I used phpmyadmin to dump and import the old database (sorry I still like a GUI when I can get it). I used the same IP addresses on both, the same login and just changed the outside user’s password, not the fog user password but apparently that’s what the TFTP server uses, so I had to reset that in the back end after clients could register but I could not start a task because of the TFTP failure. That fixed the TFTP failure and the first 2 machines I queued began their boot process.
The image is an XP image with Office, Adobe CS3, and antivirus and updates installed, that’s it, really straight forward and one I’ve used hundreds of times, about 20 gigs resized. The first client is only handling about 50 Mib/min (estimating 6 hours to complete) and the other one starts, transfers about 50 MB and then stops, says it’s still running but isn’t transferring any data. With the old box with lesser hardware I could use this same image and deploy 5 at a time in about 12 minutes to the same lab not using multicast. The network isn’t the issue because all of the switches are the same, the only hardware difference is the new Fog server. Apparently there’s something else I’m missing on the migration here. Any ideas?[B]Basic info:[/B]
Fog Server 0.32
Ubuntu 11.10 (same on both old and new)
FogAdmin (outside Ubuntu user)
IP address the same
Fog Server name is identical -
Try to mount the NSF volume onto a test machine and pull an image down via file copy. What speeds do you get?
-
I’ve never tried mounting one of these. Is there a walk through on the wiki somewhere? Thanks in advance.
-
Try the debug an image deployment wiki article. Also, you can google “mount nfs volume linux” as it is not specific to FOG.
-
Ok, I tried running in debug mode but it hangs at different points during the deployment but everything up until that deployment point works fine. It’s been different percentages 30 percent, then 82 percent, then the last one was 9 percent. I’ve tried to mount the image using
mount -o soft,intr,rsize=8192,wsize=8192 192.168.10.168/images/ nfs-test
but receive the following error response:
mount: special device 192.168.10.168/images/ does not existbut, using the showmount command returns the following:
root@FOG-SRVR-01:/home/fogadmin# showmount -e
Export list for FOG-SRVR-01:
/images/dev *
/images *
So I know that the images folder is a valid mount point. I’ve also tried mounting the path all the way up to and including the image file, but receive that same error message.Any ideas?
-
Ok, I just realized the mistake. I forgot the colon after the IP address. Its working now and I’m connected to the share with my Macbook and I’m trying to copy the image down now.
I tried another image that wasn’t having an issue and brought down an 11gig image in between 3 and 4 minutes, the one that is a problem is our main image and is quite large, over 50 gigs. I’ll report back here after finding out what happens with it.
-
Ok, so the problematic image’s size on the hard drive was about 42 gigs, instead of 50. When I transferred down to my Macbook, it took about 11 minutes while uploading it through FOG took about an hour. The network is basically made up of HP Pro Curves with fiber run between the buildings (12), completely gigabit all the way throughout.
The FOG server is a brand new custom box with a 6 core AMD Chip and 4 Gigs of DDR III with a Terabyte SATA II hard drive in it.
So, does this mean that the FOG server is working correctly and I have a network problem or does this indicate a problem with the FOG Server?
-
Are you imaging from the same access layer switch all the time or are you moving between two different switches? I only ask because a had a very similar problem with one of our Cisco POE switches. I would take the laptop and move around and check the switches move outward from the FOG server until you get back to your access switch where you are having the slow imaging. Mine was a cheaply done uplink into a a 10/100 port instead of a gig uplink. Try that out and let us know how it goes.
-
[quote=“djm79, post: 8686, member: 1568”]Are you imaging from the same access layer switch all the time or are you moving between two different switches? I only ask because a had a very similar problem with one of our Cisco POE switches. I would take the laptop and move around and check the switches move outward from the FOG server until you get back to your access switch where you are having the slow imaging. Mine was a cheaply done uplink into a a 10/100 port instead of a gig uplink. Try that out and let us know how it goes.[/quote]
Actually the FOG server and I are in the same switch, basically our IT office is run off the main Pro Curve and both the FOG server and my imaging station are in the same office. At some point in the future I would like to be able to use the VPro chips and RDP over KVM and remote into a machine and image straight from the users location if there is a problem but for right now, I’m removing the machine, bringing it to the office and imaging from here.
An interesting thing to note here is we use primarily Lenovo workstations and laptops, I have 3 laptop images, an X230, and X200 and an X201. Both the X200 and the X230 both deploy pretty well and there’s generally no issues, it just seems to be the X201s that are the problem. This could be the image or the hardware or the fog server is my thought but not sure what to think anymore as I’ve gone round and around on this.
-
[B]UPDATE[/B]:
Started over with a completely new installation of FOG server, same IP address, did not migrate the images, used the master computers for each image to create new images. Uploaded a new image from the X201 master which went much faster and then attempted to download to 2 different X201 machines (all have the same hardware) and both of them bomb out at different points during the deployment. This is the only image that doesn’t complete and they stop at different points during the deployment. I tested the NFS setup again with the same results, took about 11 minutes to copy this image straight down to the desktop. These images were working like a charm from our initial temporary FOG server, so the only thing that’s new is the new FOG server.Any ideas? Reloading our machines from scratch is a nightmare with all the additional software and I have to get a bunch of these machines knocked out soon.
-
Have you tried Ubuntu 10.10 or less because I have seen post of people saying they had issues but not sure if it was imaging. So one image uploads fine but the other two crap out at different points but never fully upload? Have you thought of trying a different kernel for the PXE upload and download part?
-
I actually haven’t tried reversing the install, but maybe that’s the way to go and I haven’t changed any of the Kernels, so far this is the first time I have had any trouble with our network or the FOG server. I’ll have to try and find the older versions of the Ubuntu downloads, as I’m pretty sure I don’t have a CD of it.
-
[B]UPDATE[/B]: Ok, so after a lot of struggling, and some physical and OS changes, here’s the final solution (at least I hope so). I rolled back the OS via complete reinstall of the OS to Ubuntu 11.04 as 10.04 didn’t include the NIC drivers for this new of a motherboard and I wasn’t going to try and compile drivers for the Network Card (I already have enough on my plate without that task).
I also physically moved the FOG server into a rack in the server room and am direct now into the Pro Curve which eliminates any in wall wiring or cabling issues with it in my office. So far I’ve uploaded 2 of the problem images in less than 12 minutes and downloaded and imaged 2 PC’s with the troublesome X201 image in less than 10 minutes each with no trouble on either one.
Unfortunately I was so exasperated with the issues I ran across I didn’t do this in stages to find out what the actual problem is so I’m not sure this post will be useful for anyone looking at the same issues.
One thing I wanted to ask, what does mounting the NFS volume actually indicate, since I was able to successfully mount it and copy down the image with a very respectable speed? Basically does that indicate network problems or FOG problems? I’m sorry I’m not following the diagnostic pattern on this test.
Regardless of all of this thank you both for your help with this, I’m pretty sure I’m back and imaging happily.
-
mounting the volume NFS and bringing down the file helps to determine if the problem is in the FOG kernel on the client or something with the server, or something in the network. Bringing up a liveCD version of ubuntu and doing this removes the FOG client kernel from the mix and let’s you know whether the slow down is on the server or network. If the file transfer over NFS to a liveCD was slow, then the problem is in the network or the server, not the client. If the speed was not slow, then the problem is most likely in the FOG client kernel.
If the problem was found not to be on the client, then you can troubleshoot the network by removing any intermediate equipment by imaging directly from the switch to which the server is connected. If the problem still existed, you could try an “out of the box” switch with just the server and client connect and no uplink to the normal network. The would help rule out a switch config problem. If it’s still slow after all this, it’s probably something on the FOG server, such as disk performance or NIC driver issues.
-
Hi , iam able to boot with the network by pressing F12 iam able to see the logo of FOG but not able to boot with the images some where i don’t know where to copy the iso images i think its 100 % problem with the images i need some one help to sort it out asap.
-
Manjunath, please start your own thread and include what OS the server is running, what version of FOG, what model and brand of clients you are trying to image, etc. The more information you provide, the easier it will be to help.
-
Hi Dave, could you please confirm that you have been able to upload from and deploy to your Lenovo X230? Since this particular model is not on the ‘approved’ FOG list, I’m NOT looking forward to getting 25 of them tommorrow… Any particular setttings used?
-
Hi Klaus, probably already too late here but for notifications sake, I’m able to do the X230’s no problem. I had to restructure the master to a single partition and make sure the boot flag is set to the primary OS after Windows 7 sets up it’s utility partition. I also changed the AHCI settings to compatibility mode. Other than those things it went smoothly.
-
This post is deleted!