Kernel update 6 hours later

[NDNtech]

I started a kernel update at about 6 pm before I left work and let it run, hoping it would finally finish. Now, 5 hours later, I remote in and its still right where I left it. It says: “Download Complete! Moving file to TFTP Server…”

Earlier I noticed it was taking a while, but I thought maybe I wasn’t being patient enough. I’m also having problems loading files up for pxe boot, it takes a while to get to the fog menu and/or to deploy/upload an image.

This makes me want to drop everything, again, and go with WDS. lol

What could I do to check and fix TFTP speeds or block sizes and fix this?

[ianabc]

The kernel updates are usually very quick, I’ve never seen it take more than a minute or so. Is this a prebuilt kernel or one that you are creating yourself?

If I do [Fog configuration]->[Kernel Update], then click the download icon under “[FONT=Ubuntu][COLOR=#555555]Kernel - 3.14.2 TomElliott[/COLOR][/FONT]”, I get a dialog for the new kernel name, I give it the name “bzImage-TomE”, then hit next, about 15 seconds later I get a message that my new kernel has been installed. If I look in the tftp root on my fog server I can see it
[CODE]
$ ls -lhrt /var/www/html/fog/service/ipxe/
-rwxr-xr-x. 1 fog apache 44K Jul 12 10:24 bg.png
-rwxr-xr-x. 1 fog apache 503 Jul 12 10:24 advanced.php
-rw-r–r–. 1 fog apache 11M Jul 12 10:24 init.xz
-rwxr-xr-x. 1 fog apache 5.9M Jul 12 10:24 bzImage32
-rwxr-xr-x. 1 fog apache 756 Jul 12 10:24 boot.php
-rwxr-xr-x. 1 fog apache 1.8M Jul 12 10:24 memtest.bin
-rwxr-xr-x. 1 fog apache 25K Jul 12 10:24 memdisk
-rw-r–r–. 1 fog apache 9.2M Jul 12 10:24 init_32.xz
-rwxr-xr-x. 1 fog apache 230K Jul 12 10:24 grub.exe
-rw-r–r–. 1 fog apache 5.9M Jul 12 10:24 bzImage
drwxr-xr-x. 2 fog fog 4.0K Jul 14 21:33 backup

-rw-r–r–. 1 fog fog 5.9M Jul 14 21:33 bzImage-tomE
[/CODE]
My new kernel is at the bottom (5.9M). How does this compare to your setup?

If your new kernel doesn’t show up there I would suspect permissions on the web directory, but you might find some useful information in the webserver logs (/var/log/httpd/error_log).

N.B. If you are running under ubuntu rather than CentOS the paths in the post above might be incorrect (your web root is probably /var/www/fog/service/ipxe and your error log is probably /var/log/apache2/error.log)

[NDNtech]

I get as far as naming the file, which I leave as bzimage, the file downloads, and then it stalls when moving to tftp server. I’m using the latest published kernel from the FOG Configuration section.

[CODE]> ls -lhrt /var/www/fog/service/ipxe/
total 33M
-rw-r–r-- 1 fog www-data 25K Jul 14 11:42 memdisk
-rw-r–r-- 1 fog www-data 756 Jul 14 11:42 boot.php
-rw-r–r-- 1 fog www-data 9.1M Jul 14 11:42 init_32.xz
-rw-r–r-- 1 fog www-data 503 Jul 14 11:42 advanced.php
-rw-r–r-- 1 fog www-data 10M Jul 14 11:42 init.xz
-rw-r–r-- 1 fog www-data 5.9M Jul 14 11:42 bzImage32
-rw-r–r-- 1 fog www-data 230K Jul 14 11:42 grub.exe
-rw-r–r-- 1 fog www-data 1.8M Jul 14 11:42 memtest.bin
-rw-r–r-- 1 fog fog 155K Jul 14 16:59 bg.png
drwxr-xr-x 2 fog fog 4.0K Jul 14 17:25 backup
-rw-r–r-- 1 fog fog 5.9M Jul 14 17:25 bzImage[/CODE]

And nothing shows in apache2 log

[ianabc]

Does 17:25 sound sensible to you for when you updated the image? Could you try giving it a different name just to make sure that your webserver is creating the file, e.g. bzImage-TomE

[NDNtech]

17:25 sounds like about the time one of the upgrades claims to have went through, shortly before I left work. I’m trying to update using a different name as you suggested, and it sticks on the ‘moving to tftp’ step and no file is created in that dir

[NDNtech]

And I notice our group names are different in our ls, do those matter?

[ianabc]

The short answer is I think you have a problem with your tftp server (it might not be running or accepting connections), the long answer is…

Some of the relevant files seem to be…
[LIST]
[]fog/management/js/fog.about.kernel-update.js
[]fog/management/js/fog.about.kernel.js
[*]fog/management/ajax/kernel-fetcher.php
[/LIST]
Looking at them, the kernel file is downloaded to a temporary location then moved into fog/services/ipxe when it is ready. It is moved by tftp so if you have a problem with your tftp server you are going to hit problems.
As a first step I would give the machine a quick reboot, then CONFIRM that you have disabled any firewalls (iptables/ufw whatever) and SELINUX disabled. After that we need to confirm that your tftp setup is OK, the best way to do this would be to follow the code in kernel-fetcher.php you could try doing the tftp commands manually and see where the problem lies. If that isn’t an option I would try some simple tftp get commands just to confirm that it is running, e.g. on your fog server
[CODE]
$ yum install tftp
$ cd /tmp
$ rm -f undionly.kpxe
$ tftp IP.OF.YOUR.FOG.SERVER
tftp> get undionly.kpxe
$ ls -lhrt undionly.kpxe
-rw-r–r–. 1 root root 100K Jul 14 22:46 undionly.kpxe
$ ps -ef | grep tftp
root 2854 1139 0 22:46 ? 00:00:00 in.tftpd -s /tftpboot
[/CODE]

[Tom Elliott]

Check your FOG Configuration->FOG Settings->TFTP Settings->FOG_TFTP_FTP_USERNAME/PASSWORD settings. Make sure they’re set correctly. Check the /var/www/fog/service/ipxe folder and make sure it’s permissions are fog:www-data.

[NDNtech]

This post is deleted!

[NDNtech]

Today I did a clean install, again, and this issue is back. There’s gotta be something else wrong here.

[CODE]# apt-get install tftp
Reading package lists… Done
Building dependency tree
Reading state information… Done
tftp is already the newest version.
0 upgraded, 0 newly installed, 0 to remove and 2 not upgraded.

cd /tmp

rm -f undionly.kpxe

tftp 10.120.88.146

tftp> get undionly.kpxe
Received 104394 bytes in 0.0 seconds
tftp> quit

ls -lhrt undionly.kpxe

-rw-r–r-- 1 root root 101K Jul 15 17:11 undionly.kpxe

ps -ef | grep tftp

root 19173 1 0 16:55 ? 00:00:00 /usr/sbin/in.tftpd --listen --user root --address 0.0.0.0:69 -s /tftpboot
root 20969 20894 0 17:12 pts/0 00:00:00 grep --color=auto tftp[/CODE]

[NDNtech]

I checked to make sure the username and password were correct as well.

[Tom Elliott]

Which issue is back? The kernel download issue or tftp open timeout?

If it’s the kernel download you’re referring to, what is the output of:
[code]ls -l /var/www/fog/service/ipxe[/code]

[ianabc]

OK, if you test the ftp manually can you put files etc., go to a client machine with vsftp installed and try to “put” a file
[CODE]
$ echo “test Test test” >mytest
$ ftp IP.OF.YOUR.FOG
Connected to IP.OF.YOUR.FOG
220 (vsFTPd 2.2.2)
Name (IP.OF.YOUR.FOG:iana): fog
331 Please specify the password.
Password:<PASSWORD OF YOUR FOG TFTP USER HERE>
230 Login successful.
Remote system type is UNIX.

Using binary mode to transfer files.

ftp> put mytest
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.

7 bytes sent in 0.00 secs (159.0 kB/s)
ftp> quit
[/CODE]
You should be able to see that file in your fog user’s home directory
[CODE]
$ ls -l /home/fog/
total 4

-rw-r–r–. 1 fog fog 7 Jul 15 16:03 myfile
[/CODE]
Does that work on your system?

[NDNtech]

Well, both I guess. The Kernel looks like it’s downloading okay from the looks of the /tmp directory, but tftp won’t transfer it to /var/www/fog/service/ipxe.

[CODE]> ls -l /var/www/fog/service/ipxe
total 33640
-rwxrwxrwx 1 fog www-data 503 Jul 15 16:42 advanced.php
-rwxrwxrwx 1 fog www-data 44908 Jul 15 16:42 bg.png
-rwxrwxrwx 1 fog www-data 756 Jul 15 16:42 boot.php
-rwxrwxrwx 1 fog www-data 6148400 Jul 15 16:42 bzImage
-rwxrwxrwx 1 fog www-data 6157296 Jul 15 16:42 bzImage32
-rwxrwxrwx 1 fog www-data 234697 Jul 15 16:42 grub.exe
-rwxrwxrwx 1 fog www-data 10429892 Jul 15 16:42 init.xz
-rwxrwxrwx 1 fog www-data 9541876 Jul 15 16:42 init_32.xz
-rwxrwxrwx 1 fog www-data 25340 Jul 15 16:42 memdisk
-rwxrwxrwx 1 fog www-data 1839104 Jul 15 16:42 memtest.bin[/CODE]

[ianabc]

Could you try that ftp transfer that I did in my last post. Make sure you use the username and password from the fog settings page.

[NDNtech]

I wasn’t sure exactly how to do all of that stuff you listed, so I used FileZilla because I know it allowed for manual sftp transfers, I hope that’s the same thing. Here’s the output from FileZilla on a Windows machine:

[CODE]Status: Connecting to SC-Image…
Response: fzSftp started
Command: open “fog@SC-Image” 22
Command: Pass: *****
Status: Connected to [EMAIL]SC-Image@domain.org[/EMAIL]
Status: Starting upload of C:\Users\user\Pictures\Graphics\ISDTech Serv.png
Command: cd “/home/fog”
Response: New directory is: “/home/fog”
Command: put “C:\Users\user\Pictures\Graphics\ISD Tech Serv.png” “isd.png”
Status: local:C:\Users\ccolley\Pictures\Graphics\ISD Tech Serv.png => remote:/home/fog/jisd.png
Status: File transfer successful, transferred 16,384 bytes in 1 second[/CODE]

Output from FOG server:

[CODE]fog@SC-Image:~$ ls -l /home/fog
total 20
-rw-rw-r-- 1 fog fog 18304 Jul 15 18:24 isd.png
[/CODE]

[ianabc]

Hmmm… I think that might be transferring over ssh (port 22). If you know FileZilla well you should be able to tell it to use ftp (port 23) somewhere, but if not, you should be able to run the commands above on your fog server after installing an ftp client, e.g.
[CODE]
$ apt-get install ftp
$ ftp IP.OF.YOUR.FOG
[/CODE]

You should be prompted for your username and password, then you should try “putting” a file as above “put <FILENAME>”.

Any (small) file in the directory where you ran the ftp command should work, I created one by echoing some text into a file called myfile before running the ftp command (echo ‘test Test test’ > myfile).

[NDNtech]

From Windows CMD:
[CODE]H:>ftp 10.120.88.146
Connected to 10.120.88.146.
220 (vsFTPd 2.3.5)
ftp> user
Username fog
331 Please specify the password.
Password:
230 Login successful.
ftp> put C:/dfinstall.log /home/fog/dfinstall.log
200 PORT command successful. Consider using PASV.
150 Ok to send data.
226 Transfer complete.[/CODE]

From SSH to FOG Server
[CODE]fog@SC-Image:~$ ls
dfinstall.log isd.png[/CODE]

[Tom Elliott]

try:
[code]ftp 10.120.88.146
user fog
enter pass
put c:/dfinstall.log /var/www/fog/service/ipxe/dfinstall.log
[/code]

[NDNtech]

Tom:

It transferred correctly