Posts made by ianabc

I’m not sure at the moment. I’ll take a look at your syslog later, it certainly still looks like a memory leak in some process to me (my guess would be gnome somewhere - if you don’t need a desktop on your fog server just uninstall it).

If you don’t want to uninstall gnome just yet make sure you apt-get update && apt-get upgrade the system. It’s possible whatever program is leaking has been fixed already.

If you want to try the system without a desktop for a while you could try
[CODE]
$ sudo service lightdm stop
[/CODE]
You should end up at a black console with white text. Run your upload/download tests and see if the problem is fixed.

t.mayer: Could you try the omping commands above just to confirm that multicast routing is working for you.

Hi Sparky, Operating systems are now associated with images not hosts. If you look under images you should be able to see the OS.

[quote=“p38fln, post: 33443, member: 2516”]I dropped the fog db and reinstalled and the DefaultMember configuration looks exactly the same as before, and same errors - It mounts the NFS volume, then says it isn’t mounted.[/quote]

How are you confirming that it is actually mounted?

You can schedule a debug tasks for one of the clients and boot into the imaging environment. Under “Basic Tasks” for the client you are trying to image select “Advanced”, then “Debug task”, when the client finishes booting it will leave you at a shell. Try the mount command manually from there, e.g.
[CODE]
$ mount -o nolock,proto=tcp,rsize=32768,wsize=32768,intr,noatime 10.0.0.36:/images/dev /mnt
[/CODE]
That will let you watch for errors on the NFS server without rebooting the client.

No problem. I hope it works for you

What do you see in the PXEboot process, can you see it getting a DHCP entry?

There are a bunch of changes in NFS (and everything else) waiting in CentOS 7, not sure which ones have been integrated with fog yet. As X32 said, you should give us your /etc/exports and maybe check NFS mounts on the fog server itself, e.g.
[CODE]
$ mkdir /tmp/images
$ mount -o nolock,vers=3 192.168.222.100:/images /tmp/images
$ ls /tmp/images

C652  client1  client3  dev  lost+found  opt  postdownloadscripts  teststretch  teststretch2

$ umount /tmp/images
$ rmdir /tmp/images
[/CODE]
Once you know that the NFS server is running properly we can move on to clients.

Check the task list in fog to make sure that the task has actually be scheduled and is waiting.

Also, I just noticed that I can test multicast with the plain vanilla ping command as long as icmp_echo_ignore_broadcasts is set to 0, e.g.
[CODE]
fog-server$ echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts
fog-client$ echo 0 > /proc/sys/net/ipv4/icmp_echo_ignore_broadcasts

fog-server$ ping -c 5 224.0.0.1
PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data.
64 bytes from 192.168.222.100: icmp_seq=1 ttl=64 time=0.044 ms
64 bytes from 192.168.222.102: icmp_seq=1 ttl=64 time=0.393 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=2 ttl=64 time=0.043 ms
64 bytes from 192.168.222.102: icmp_seq=2 ttl=64 time=0.414 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=3 ttl=64 time=0.041 ms
64 bytes from 192.168.222.102: icmp_seq=3 ttl=64 time=0.402 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=4 ttl=64 time=0.042 ms
64 bytes from 192.168.222.102: icmp_seq=4 ttl=64 time=0.424 ms (DUP!)
64 bytes from 192.168.222.100: icmp_seq=5 ttl=64 time=0.036 ms

--- 224.0.0.1 ping statistics ---
5 packets transmitted, 5 received, +4 duplicates, 0% packet loss, time 3999ms

rtt min/avg/max/mdev = 0.036/0.204/0.424/0.182 ms

[/CODE]
On a network that doesn’t permit multicast, the same thing gives me
[CODE]

fog-server$ ping -c 5 224.0.0.1
PING 224.0.0.1 (224.0.0.1) 56(84) bytes of data.
64 bytes from 192.168.222.100: icmp_seq=1 ttl=64 time=0.046 ms
64 bytes from 192.168.222.100: icmp_seq=2 ttl=64 time=0.041 ms
64 bytes from 192.168.222.100: icmp_seq=3 ttl=64 time=0.044 ms
64 bytes from 192.168.222.100: icmp_seq=4 ttl=64 time=0.040 ms
64 bytes from 192.168.222.100: icmp_seq=5 ttl=64 time=0.042 ms

--- 224.0.0.1 ping statistics ---
5 packets transmitted, 5 received, 0% packet loss, time 3999ms
rtt min/avg/max/mdev = 0.040/0.042/0.046/0.007 ms

[/CODE]
i.e. Only the machine sending the pings responds, the client never sees them.

[B]N.B. I should also point out that I have basically no idea what I’m doing when it comes to multicast so YMMV :-)[/B]

I found it useful to start debugging with omping and udpcast directly before trying to debug fog - it means you get instant answers, you don’t have to wait for a client to reboot.

If you can install omping somewhere (like a spare linux client), then you can try
[CODE]
fog-server$ omping FOG.CLIENT.IP FOG.SERVER.IP
…

fog-client$ omping FOG.SERVER.IP FOG.CLIENT.IP
[/CODE]
On my system I see output on the fog server similar to
[CODE]
192.168.222.102 : multicast, seq=37, size=69 bytes, dist=0, time=0.437ms
192.168.222.102 : unicast, seq=38, size=69 bytes, dist=0, time=0.482ms
192.168.222.102 : multicast, seq=38, size=69 bytes, dist=0, time=0.495ms
192.168.222.102 : unicast, seq=39, size=69 bytes, dist=0, time=0.485ms
192.168.222.102 : multicast, seq=39, size=69 bytes, dist=0, time=0.499ms
192.168.222.102 : unicast, seq=40, size=69 bytes, dist=0, time=0.274ms
192.168.222.102 : multicast, seq=40, size=69 bytes, dist=0, time=0.291ms
192.168.222.102 : unicast, seq=41, size=69 bytes, dist=0, time=0.446ms

192.168.222.102 : multicast, seq=41, size=69 bytes, dist=0, time=0.455ms
[/CODE]
When multicast is working. When it isn’t working I only see the unicast responses.

If it is just being slow then I would agree that it is probably a driver issue, but in one of your other threads you mentioned that you fixed it…
[QUOTE]I got the problem fixed that made the fog boot menu take forever to load by setting permissions to the tftpboot folder to 777 and the owner is fog and the restarting the xinetd service.

Now when I try to upload/deploy an image, bzimage will not go past 0%.[/QUOTE]
Does that workaround no longer work? I mean does it EVER complete or is it just slow?

0.0.0.0 in that context basically means listen on all available addresses on your machine. You should leave it at that value.

Can you confirm exactly what you see when you boot one of the problematic clients, e.g. Do you see it get a DHCP entry and then grab the iPXE image?

ooff…this is getting ugly! Could you try running the same commands as the fog user, i.e. log in to the fog server and try this
[CODE]
$ sudo su - fog
$ cd /tmp
$ echo “test” > testFile

$ ftp 10.120.88.146
Connected to 10.120.88.146 (10.120.88.146).
220 (vsFTPd 2.2.2)
Name (10.120.88.146:fog): fog
331 Please specify the password.
Password: ********
230 Login successful.
Remote system type is UNIX.
Using binary mode to transfer files.
ftp> put testFile /var/www/fog/service/ipxe/testFile
local: testFile remote: /var/www/html/fog/service/ipxe/testFile
227 Entering Passive Mode (10,128,128,146,125,20).
150 Ok to send data.
226 Transfer complete.
ftp> quit
221 Goodbye.

$ ls -l /var/www/fog/service/ipxe/testFile
-rw-r–r–. 1 fog fog 0 Jul 15 20:47 /var/www/fog/service/ipxe/testFile
[/CODE]

Additionally, just confirm that the [FONT=Ubuntu][COLOR=#555555]FOG_TFTP_HOST[/COLOR][/FONT] in your fog configuration is set to 10.120.88.146. Also Since that section of code relies on Javascript and ajax, can you confirm that the browser you’re using to do this doesn’t have javascript disabled.

That’s basically all that portion of the fog code is doing (ajax/kernel-fetcher.php), so I’m not sure what could be going wrong. The permissions on some of the files under services/ipxe look a bit odd, could you send the output of “ls -ld /var/www/html/fog/service/ipxe”, i.e. the ipxe directory, NOT it’s contents.

Hmmm… I think that might be transferring over ssh (port 22). If you know FileZilla well you should be able to tell it to use ftp (port 23) somewhere, but if not, you should be able to run the commands above on your fog server after installing an ftp client, e.g.
[CODE]
$ apt-get install ftp
$ ftp IP.OF.YOUR.FOG
[/CODE]

You should be prompted for your username and password, then you should try “putting” a file as above “put <FILENAME>”.

Any (small) file in the directory where you ran the ftp command should work, I created one by echoing some text into a file called myfile before running the ftp command (echo ‘test Test test’ > myfile).

Could you try that ftp transfer that I did in my last post. Make sure you use the username and password from the fog settings page.

OK, if you test the ftp manually can you put files etc., go to a client machine with vsftp installed and try to “put” a file
[CODE]
$ echo “test Test test” >mytest
$ ftp IP.OF.YOUR.FOG
Connected to IP.OF.YOUR.FOG
220 (vsFTPd 2.2.2)
Name (IP.OF.YOUR.FOG:iana): fog
331 Please specify the password.
Password:<PASSWORD OF YOUR FOG TFTP USER HERE>
230 Login successful.
Remote system type is UNIX.

Using binary mode to transfer files.

ftp> put mytest
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.

7 bytes sent in 0.00 secs (159.0 kB/s)
ftp> quit
[/CODE]
You should be able to see that file in your fog user’s home directory
[CODE]
$ ls -l /home/fog/
total 4

-rw-r–r–. 1 fog fog 7 Jul 15 16:03 myfile
[/CODE]
Does that work on your system?

OK, the network sounds clear and those error messages you were getting from tftp.hpa are very suspicious.

I don’t have an ubuntu system to test with, but on CentOS systems tftpd is started by inetd, if ubuntu is the same then when a client makes an incoming connection to the fog server for tftp, inetd spawns an instance of in.tftpd to handle the connection. It looks like, in your case, when that happens your system can’t allocate memory for tftpd so it never gets spawned, that is why your tftp sessions fail. The difficult part is figuring out why there is no memory for tftpd.

As I mentioned above, I don’t think it is a physical problem, 4GB seems like plenty of RAM - I get by with much less. It is possible that some runnaway process is eating up all of your ram, but if your system was genuinely memory starved I would expect more problems than just a failing tftpd server.

Could you attach a copy of your /var/log/messages or /var/log/syslog file? We might be able to spot the root cause in there.

Also could you periodically have a look for any tftp processes
[CODE]
$ ps -ef | grep tftp
root 5007 1138 0 12:38 ? 00:00:00 in.tftpd -s /tftpboot
root 5010 4980 0 12:38 pts/0 00:00:00 grep tftp
[/CODE]
It would be interesting to see if it is running and if more than one process is spawned.

Finally after your machine has been running for a while you can get a list of the most memory hungry processes by running ps, e.g.
[CODE]
$ ps -eo pid,rss,vsz,comm= | sort -n -k 3 | tail -n 20
5461 688 100944 tail
5459 996 108124 ps
1192 1540 108168 mysqld_safe
4980 1924 108308 bash
1406 1360 117300 crond
5460 868 155432 sort
955 1560 249084 rsyslogd
1413 11140 250640 FOGTaskSchedule
1427 10944 250640 FOGImageReplica
1420 11212 250896 FOGMulticastMan
1398 9544 271660 httpd
1462 10548 370800 httpd
1460 10576 370808 httpd
1458 10480 371024 httpd
1461 10764 371024 httpd
1457 10724 371040 httpd
1463 18528 378748 httpd
1459 18280 378752 httpd
1456 19196 379668 httpd
1294 27708 640800 mysqld
[/CODE]
If you could do this around the time that tftp dies we might have a chance of catching the reason why.

I think you should be OK for memory, 4GB seems like plenty. As I said, my setup is only using 1GB and I’m not seeing the problems, before you dig too deep looking for answers, you should check that the problem isn’t firewalling.

If 0.32 was working and there have been no network changes since then there probably isn’t a network firewall, but you should double/triple check that the fog server doesn’t have one. On Ubuntu 11.10 you should check
[CODE]
$ sudo iptables -nL
Chain INPUT (policy ACCEPT)
target prot opt source destination

Chain FORWARD (policy ACCEPT)
target    prot opt source              destination       

Chain OUTPUT (policy ACCEPT)

target    prot opt source              destination   

$ sestatus
[/CODE]
You should also check SELINUX, make sure it is either disabled or in permissive mode
[CODE]
$ sestatus
SELinux status: enabled
SELinuxfs mount: /selinux
Current mode: permissive
Mode from config file: permissive
Policy version: 24

Policy from config file:        targeted

[/CODE]
And you might want to just double check that you are NOT using tcp_wrappers: just check that /etc/hosts.allow and /etc/hosts.deny are empty or only have comments in them. Once you have confirmed those things we can start tracking down the problem without going in circles