OK, so I’ve put everything back on gigabit and my numbers are consistent with your “Before” values (7GB/min or so), I’m on 1.1.2 by they way.
I strongly suspect 100BaseT is being negotiated somewhere between your client and fog server. ethtool on the fog server and in the debug task on your client should help you rule them out, then it is just the switch(es) that would need to be checked.
[quote=“braindead, post: 31694, member: 24282”]
Before the update, I was getting 5-7 GB/min for my downloads, and anywhere from 2-4 GB/min uploads. Now, I get ~1.39 GB/min downloads and ~750 MB/min to 1.3 GB/min uploads.
[/quote]
Is 5-7 GB/min aggregate for deploying multiple images simultaneously? And are these the numbers reported by fog/partclone or have you also noticed that an actual image is taking longer?
I’m on a single [S]gigabit[/S] fast ethernet (100Mb/s) link and partclone consistently tells me that it is running at about 1-2GB/min for a download at the moment - I haven’t noticed a significant change from other versions. The numbers that partclone reports will be for a the compressed image transfer so will be quite variable.
If you wanted to dig a bit deeper you could take a look at the link negotiation on the switch and make sure it isn’t falling back to 100MB/s somewhere. The debug task in fog might also help you check the link speed and do some copies to check the numbers, for gigabit ethernet the max is somewhere around 117MB/s without jumbo frames, in the real world for NFS I’d expect somewhere around 90MB/s.
EDIT ethtool is available in the fog debug task, “ethtool eth0” should tell you what link speed was negotiated without messing with your switch. Mine is reporting 100MB/s for some depressing reason 
Has your iSCSI performance dropped? or is it just imaging? The second set of numbers you gave look typical of a single interface to me so my first guess would be your interface bonding has come undone during the update.
This is more of a workaround than a solution
I had the same issue with the installer, the mysql configuration in /var/www/html/fog/lib/fog/Config.class.php ended up being incorrect this led to the upgrade script in the fog GUI not detecting my current settings and trying to do a CREATE as in the first post. Putting the correct information allowed the update script to complete successfully.
Hi Jamie, I just noticed in your syslog
[CODE]
refused mount request from 192.168.3.33 for /fog/images (/): not exported
[/CODE]
The fog clients are trying to mount the wrong location, that is almost certainly the problem. Check the image path in your storage configuration, you would either need to update that or change your export list, once they match your NFS problem should go away (and probably the FTP one as well).
[CODE]
Jun 26 13:22:56 troi rpc.mountd[1691]: authenticated mount request from 192.168.3.33:930 for /images (/images)
[/CODE]
Does this correspond with the fog debug task and mount command? did you get any output from that command? Could you turn on debugging for NFS, edit the /etc/default/nfs-kernel-server file to have the line
[CODE]
RPCMOUNTDOPTS=“–manage-gids --debug all”
[/CODE]
Then “service nfs-kernel-server restart” and retry the mount - syslog should get more information this time.
OK, so it looks like syslog would be the where NFS mounts are logged on your system, is it Ubuntu? I have an ubuntu machine image and I’m deploying it now. Once it is installed I’ll take a look at the NFS setup so we can compare. I’ll post back when it is up and running.
When I said I thought the network was the problem I meant the networking on the fog server you are running: the installer script runs yum updates and my guess is that a new version of one of these packages broke something.
The message [CODE]mount: mounting 192.168.3.8:/fog/images/ on /images failed: Permission denied.[/CODE] is from NFS so it shouldn’t care about FTP or FTP passwords. I still suspect it is a network problem somewhere - your NFS configuration should be able to function with all of the fog services turned off. If you boot one of your clients with a debug task, once it gives you a shell, can you try
[CODE]
$ mkdir /images
$ mount -o nolock,vers=3 IP.OF.YOUR.FOG:/images /images
[/CODE]
This should be the NFS mount that your client is doing during an image. When you run that command you should be able to see the server logging the mount, e.g. in /var/log/messages on RHEL
[CODE]
Jun 26 10:00:56 fog rpc.mountd[23650]: authenticated mount request from IP.OF.YOUR.CLIENT:720 for /images (/images)
[/CODE]
The log file location (and message) might be slightly different (depending on your distribution), but running “ls -lhrt /var/log” should tell you which log file was most recently changed.
For the disk image sizes, it looks like those are partimage (fog < 1.0) images. This is a known problem, see[URL=‘http://fogproject.org/forum/threads/disk-information-and-image-size-on-server-not-showing-any-data.10953/’] this thread[/URL] for more details on it.
Hi David, I’ve seen similar things when there is a problem with the image on the fog server. I’m assuming that you have done the following
[LIST=1]
[]Defined an image in the fog interface “Image Mangement->Create new Image”
[]Associated that image with the host you are trying to upload from (under host mangement)
[*]Created an upload task for that host
[/LIST]
Is that correct? Also could you be specific about exactly when you see the error message?
Hi Fhajad, you have nothing to worry about, the memory is all going to cached which is a good thing!
Basically accessing memory is much faster than accessing disk so if there is memory not being used, the kernel will keep recently accessed files in there, this speeds up the system considerably. When you need that memory for a running process the caches are dropped automatically and your process is given the memory. The relevant numbers in your output are on the “-/+ buffers/cached” line, you have 437m used and 7305 “free”.
This might explain better: [url]http://www.thomas-krenn.com/en/wiki/Linux_Page_Cache_Basics[/url]
Incidentally the idea is similar to filesystems like btrfs and ZFS which use copy-on-write to turn “empty” space into something a bit more useful.
What are your cached and buffer numbers though? e.g. in mine above the “free” should be 15111 + 47 + 406. I just rebooted that machine so it’s not a great example, here is a better one
[CODE]
free -m
total used free shared buffers cached
Mem: 7768 6726 1042 0 530 5136
-/+ buffers/cache: 1058 6710
Swap: 8191 56 8135
[/CODE]
My free for this machine would be 1042 + 530 + 5136 = 6708.
I’m on RHEL 6.5 and I’m not seeing the same thing,
[CODE]
$ free -m
total used free shared buffers cached
Mem: 15947 835 15111 0 47 406
-/+ buffers/cache: 381 15565
Swap: 16383 0 16383
[/CODE]
What are you using to judge the free memory? Memory management in linux is quite complex. From memory, the kernel will regard any free space as available for caching, so with longer uptimes you will see little free memory, the actual memory available for applications is the sum of free, buffers and cached.
You shouldn’t need to reinstall, it sounds like just an NFS problem. NFS(3) is pretty simple so we should be able to debug the problem here.
Which distribution are you running? “service nfs restart” should work for RHEL based distributions but on ubuntu you might do something like “service nfs-kernel-server restart”. Alternatively you can just quickly reboot the fog server and let that restart the nfs service if you’re not sure. When it comes back up could you include the output from rpcinfo, e.g.
[CODE]
$ rpcbind -p
program vers proto port service
100000 4 tcp 111 portmapper
100000 3 tcp 111 portmapper
100000 2 tcp 111 portmapper
100000 4 udp 111 portmapper
100000 3 udp 111 portmapper
100000 2 udp 111 portmapper
100024 1 udp 48324 status
100024 1 tcp 60634 status
100011 1 udp 875 rquotad
100011 2 udp 875 rquotad
100011 1 tcp 875 rquotad
100011 2 tcp 875 rquotad
100005 1 udp 51151 mountd
100005 1 tcp 56945 mountd
100005 2 udp 38168 mountd
100005 2 tcp 56201 mountd
100005 3 udp 49959 mountd
100005 3 tcp 41228 mountd
100003 2 tcp 2049 nfs
100003 3 tcp 2049 nfs
100003 4 tcp 2049 nfs
100227 2 tcp 2049 nfs_acl
100227 3 tcp 2049 nfs_acl
100003 2 udp 2049 nfs
100003 3 udp 2049 nfs
100003 4 udp 2049 nfs
100227 2 udp 2049 nfs_acl
100227 3 udp 2049 nfs_acl
100021 1 udp 57018 nlockmgr
100021 3 udp 57018 nlockmgr
100021 4 udp 57018 nlockmgr
100021 1 tcp 48912 nlockmgr
100021 3 tcp 48912 nlockmgr
100021 4 tcp 48912 nlockmgr
[/CODE]
Once you are sure that the services are running and listening on the correct ports, that really only leaves things that can terminate inbound connections like tcpwrappers.
It looks like the problem is between the clients and your fog server then, first could you check that you aren’t using selinux, e.g.
[CODE]
$ sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: permissive
Policy version: 24
Policy from config file: targeted
[/CODE]
Next, make sure that you aren’t using tcp_wrappers or configure it to allow rpcbind, portmap, statd etc. if you are.
Do you have any other linux clients on your network that you could try the NFS mount from? If not you can schedule a debug task on one of the clients and run the mount commands there, it might give you a better idea of where the problem is. e.g.
[CODE]
$ mount -vvv -o vers=3,rw IP.OF.YOUR.FOG:/images /mnt
[/CODE]
It is worth checking your firewall and restarting the nfs service(s). Restarting the NFS service(s) depends on your system, but on mine looks like.
[CODE]
$ service nfs restart
[/CODE]
Also, could you send the output of exportfs -va, e.g.
[CODE]
$ exportfs -ua
$ exportfs -va
[/CODE]
Also, another good test is to see if you can mount the NFS share on the server itself under a temporary location, e.g.
[CODE]
$ mount IP.OF.YOUR.FOG:/images /mnt
$ ls /mnt
dev Mint17x64PIMS1 U1404GAH87NWIFIIAM W7x32Stat55GBDeploy-XXX
fog-math RHL6x64PIMS50GB u1404PIMS
lost+found RHL7x64BetaPIMS1 W7x32Stat55GBDeploy
*** If it worked, remember to unmount it ****
$ umount /mnt
[/CODE]
There is something similar in the wiki: [url]http://www.fogproject.org/wiki/index.php?title=Error_Messages[/url]. Could you try what they suggest and see if it makes a difference? They are suggesting that there is a problem in the startup scripts for debian but that temporarily removing the fog startup scripts from /etc/init.d seemed to work.
Hi Jason, the fog script fog.download does an “mkswap” on the partition during the restore. I think that will change the UUID. The alternative would be to just dd the entire swap space which is probably what Clonezilla is doing (though I’m guessing :)).
Reading what you are doing (congratulations by the way - a very laudable project!) there are a couple of ways you could solve this. They all come in at the time you are expanding the ext4 filesystem. At the same time you would update either the fstab or the UUID in the swap partition. After the next reboot (or after a swapon/swapoff) your swap partition would show up.
To edit the fstab I might do something like
[CODE]
#!/bin/sh
OLDUID=grep swap /etc/fstab |cut -d' ' -f1
NEWUID=blkid | grep swap | cut -d' ' -f2
sed “s/$OLDUID/$NEWUID/” /etc/fstab
[/CODE]
before I expand the ext4fs.
If you wanted to modify the swap UUID instead, then something like
[CODE]
#!/bin/sh
TARGETUID=grep swap /etc/fstab |cut -d' ' -f1 | cut -d'=' -f2
DEVICE=blkid | grep swap | cut -d':' -f1
swaplabel -U “$TARGETUID” $DEVICE
swapon -a
[/CODE]
A third alternative is to use labels rather than UUIDs or device names.
To access a directory you have to have read and [B]execute[/B] - in the context of a directory it basically means “access metadata” rather than execute. That explains the 7 instead of 6, but you would still expect 676 to work, assuming that your tftp service runs as root.
Hi Jason, bear with me, I’m not really sure I understand the question. The system doesn’t add swap automatically, installers sometimes do, but it is a reversible or modifiable choice at your discretion. The swapon and swapoff commands will let you turn any of your swap files/partitions on and off.
If the UUID of the swap partition doesn’t match the fstab it won’t be used. To get around that, you would either need to modify the fstab or modify the swap partition (swaplabel will let you change the UUID).
What I was suggesting at the top of the thread was that if your client systems don’t vary too much, then the device names will not change (i.e. /dev/sda will always be /dev/sda) and you can put the device names in fstab instead of UUIDs - this avoids the need for editing the fstab or the swap uuid in the future.
Does that help?