Kernel Update fails ... Oh no, not again!

[sudburr]

SELinux is already configured and functions properly for all servers. Imaging is not the problem. The problem is updating the kernel on 2 of 18 physical servers.

[jstephen]

@sebastian-roth Following your steps, I get this:

ftp> mkdir /var/www/fog/service/ipxe/backup
550 Create directory operation failed.

It seems that I am able to rename the files though.

I also notice that all of the other files in the ipxe folder are owned by fog:www-data; however, the files the Fog is trying to create for the new kernel are owned by fog:fog. Could this be part of the problem?

One more observation: the fog GUI is reporting that the operation failed to ftp_mkdir and ftp_rename, however the file exists in the ipxe directory and has 8069680 bytes which is probably about right, right? Is the file being copied elsewhere also and failing there? Or, is the GUI error… in error?

[Sebastian Roth]

@sudburr said in Kernel Update fails ... Oh no, not again!:

Completed the ftp steps detailed and update via GUI continues to fail.

Can you please post a listing or picture of the commands and output?

@jstephen said:

ftp> mkdir /var/www/fog/service/ipxe/backup
550 Create directory operation failed.

This might be due to the fact that the directory already exists. See with ls -al /var/www/fog/service/ipxe on the command line.

[sudburr]

@sebastian-roth Aside from changing your

chmod 644 var/www/fog/service/ipxe/testfile

to

chmod 644 /var/www/fog/service/ipxe/testfile

The commands and results were word for word identical to what you posted.

[Sebastian Roth]

@sudburr Are you behind a proxy server by any chance? Do you still see the very same ftp_* messages in the apache error log?

[sudburr]

@sebastian-roth No proxy server. The error message has changed one one server at /var/log/httpd/error_log.

PHP Warning:  ftp_login(): Login incorrect. in /var/www/html/fog/lib/fog/fogftp.class.php on line 463, referer: http://xyzfog/fog/management/index.php?node=about&sub=kernel&file=aHR0cHM6Ly9mb2dwcm9qZWN0Lm9yZy9rZXJuZWxzL0tlcm5lbC5Ub21FbGxpb3R0LjQuMTMuNC42NA==&arch=64

The value for “Fog Configuration / Fog Settings / TFTP Server / FOG_TFTP_FTP_PASSWORD” changed. It now does not match the password as found in “/opt/fog/.fogsettings” or “Storage Management / Management Password”. I changed it to match, then ran the kernel update again and it worked!

The other server has exhibited no change in errors or passwords.

[Wayne Workman]

Cross-linking these threads as I think they are related.

https://forums.fogproject.org/topic/11375/kernel-update-fails-oh-no-not-again
https://forums.fogproject.org/topic/11409/newer-to-fog-need-a-little-assistance

[Sebastian Roth]

@sudburr said in Kernel Update fails ... Oh no, not again!:

I changed it to match, then ran the kernel update again and it worked!

Kind of weird. Well, good to know it works now. Marking as solved.

[sudburr]

@sebastian-roth It’s solved for one server, but not the other.

[sudburr]

Something to look at from the FTP logs.

Wed Feb  7 12:37:04 2018 [pid 15591] CONNECT: Client "<FOGIP>"
Wed Feb  7 12:37:04 2018 [pid 15591] FTP response: Client "<FOGIP>", "220 (vsFTPd 3.0.2)"
Wed Feb  7 12:37:05 2018 [pid 15591] FTP command: Client "<FOGIP>", "USER fog"
Wed Feb  7 12:37:05 2018 [pid 15591] [fog] FTP response: Client "<FOGIP>", "331 Please specify the password."
Wed Feb  7 12:37:05 2018 [pid 15591] [fog] FTP command: Client "<FOGIP>", "PASS <password>"
Wed Feb  7 12:37:05 2018 [pid 15590] [fog] OK LOGIN: Client "<FOGIP>"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "230 Login successful."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "PASV"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "227 Entering Passive Mode (<FOGIP>,209,195)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "TYPE A"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "200 Switching to ASCII mode."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "LIST -a //var/www/html/fogservice/ipxe"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "150 Here comes the directory listing."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "226 Transfer done (but failed to open directory)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "MKD //var/www/html/fogservice/ipxe/backup/"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "550 Create directory operation failed."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FAIL MKDIR: Client "<FOGIP>", "//var/www/html/fo[Wed Feb 07 12:37:05.182303 2018] [:error] [pid 1410] [client <CLIENT_IP>:45345] PHP Warning:  ftp_mkdir(): Create directory operation failed. in /var/www/html/fog/lib/fog/fogftp.class.php on line 492, referer: http://xyzfog/fog/management/index.php?node=about&sub=kernel&file=aHR0cHM6Ly9mb2dwcm9qZWN0Lm9yZy9rZXJuZWxzL0tlcm5lbC5Ub21FbGxpb3R0LjQuMTMuNC42NA==&arch=64
gservice/ipxe/backup/"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "PASV"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "227 Entering Passive Mode (<FOGIP>,96,38)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "LIST -a /var/www/html/fogservice/ipxe"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "150 Here comes the directory listing."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "226 Transfer done (but failed to open directory)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "PASV"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "227 Entering Passive Mode (<FOGIP>,232,42)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "LIST -a /var/www/html/fogservice/ipxe"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "150 Here comes the directory listing."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "226 Transfer done (but failed to open directory)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "RNFR /tmp/bzImage"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "550 RNFR command failed."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FAIL RENAME: Client "<FOGIP>", "/tmp/bzImage"
[Wed Feb 07 12:37:05.184327 2018] [:error] [pid 1410] [client <CLIENT_IP>:45345] PHP Warning:  ftp_rename(): RNFR command failed. in /var/www/html/fog/lib/fog/fogftp.class.php on line 769, referer: http://xyzfog/fog/management/index.php?node=about&sub=kernel&file=aHR0cHM6Ly9mb2dwcm9qZWN0Lm9yZy9rZXJuZWxzL0tlcm5lbC5Ub21FbGxpb3R0LjQuMTMuNC42NA==&arch=64
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "TYPE I"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "200 Switching to Binary mode."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "PASV"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "227 Entering Passive Mode (<FOGIP>,47,255)."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "STOR /var/www/html/fogservice/ipxe/bzImage"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "553 Could not create file."
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FAIL UPLOAD: Client "<FOGIP>", "/var/www/html/fogservice/ipxe/bzImage", 0.00Kbyte/sec
[Wed Feb 07 12:37:05.185077 2018] [:error] [pid 1410] [client <CLIENT_IP>:45345] PHP Warning:  ftp_put(): Could not create file. in /var/www/html/fog/lib/fog/fogftp.class.php on line 707, referer: http://xyzfog/fog/management/index.php?node=about&sub=kernel&file=aHR0cHM6Ly9mb2dwcm9qZWN0Lm9yZy9rZXJuZWxzL0tlcm5lbC5Ub21FbGxpb3R0LjQuMTMuNC42NA==&arch=64
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP command: Client "<FOGIP>", "QUIT"
Wed Feb  7 12:37:05 2018 [pid 15595] [fog] FTP response: Client "<FOGIP>", "221 Goodbye."

[Sebastian Roth]

@sudburr Great log file you posted. See the path is /var/www/html/fogservice/ipxe and I reckon there is a slash missing. Check TFTP_PXE_KERNEL_DIR in the FOG web UI -> FOG Configuration -> FOG Settings -> TFTP Server - should be /var/www/html/fog/service/ipxe/

[sudburr]

@sebastian-roth That did the trick! That server can now update its kernel.

It’s interesting to see how two different causes have similar results.