Unable to upload images

[george1421]

@Doctrg said in Unable to upload images:

@Tom-Elliott We only have the 1 master node.

As its defined on the master FOG server?

[Doctrg]

@george1421 We have the 1 master node and 3 storage nodes on different subnets. At the moment we are trying to fix uploading to the master node and then we will update the other storage nodes from 1.3.0 to 1.3.2.

[Tom Elliott]

@Doctrg On the current system that you’ve tested everything, can you verify that the image is uploaded to the /images/dev/<macofuploadinghost> location?

[Doctrg]

@Tom-Elliott I can confirm that it does upload to that file location.

[Tom Elliott]

@Doctrg Can you try running;

chown -R fog:root /images

[Doctrg]

@Tom-Elliott Unfortunately I’m getting the same failure after running chown -R fog:root /images.

[Tom Elliott]

@Doctrg Can you provide your apache error log please?

Maybe even the access log?

[Doctrg]

@Tom-Elliott 0_1484688152981_error_log I’ve attached the error log, however the access log has not been modified since June 2016.

[Tom Elliott]

@Doctrg Did somebody make your images immutable by chance?

[Doctrg]

@Tom-Elliott No, but there are 2 images that I have set to locked to make sure no-one else can make changes or delete them.

[Tom Elliott]

@Doctrg Are you uploading on one of these images? What do you mean set to locked I guess?

[Doctrg]

@Tom-Elliott I’m not trying to change those images and they are on one of the other storage nodes. There’s the option in the Image Management to set the image as Protected.

[Tom Elliott]

Here’s what I’m seeing is the issue:

[Tue Jan 17 14:58:50 2017] [error] [client 10.16.10.73] PHP Fatal error:  Uncaught exception 'Exception' with message 'Type: 2, File: /var/www/html/fog/lib/fog/fogftp.class.php, Line: 708, Message: ftp_put(): Could not create file., Host: 10.16.0.220, Username: fog' in /var/www/html/fog/lib/fog/fogftp.class.php:367\nStack trace:\n#0 /var/www/html/fog/lib/fog/fogftp.class.php(772): FOGFTP->ftperror(Array)\n#1 /var/www/html/fog/lib/reg-task/taskqueue.class.php(366): FOGFTP->rename('//images/dev/74...', '//images/LABNUR...')\n#2 /var/www/html/fog/lib/reg-task/taskqueue.class.php(413): TaskQueue->_moveUpload()\n#3 /var/www/html/fog/service/Post_Stage2.php(24): TaskQueue->checkout()\n#4 {main}\n  thrown in /var/www/html/fog/lib/fog/fogftp.class.php on line 367

Can you manually remove the folder that’s trying to be placed in the image location?

I don’t know the full name of the image but it looks like something:

rm -rf /images/LABNUR<other_stuff_I_cannot_see>

[Doctrg]

@Tom-Elliott Ok. I was able to manually delete the folder. I then had a spark in the brain to try something. I created a new image and unchecked replicate image. It was able to upload the image correctly. I then put the check mark back in to replicate the image and it failed. So it seems that since we are not replicating our images we should uncheck that because it seems to be the issue.

[Tom Elliott]

@Doctrg It could be, but replication doesn’t play (shouldn’t) a value in dealing with uploads.
I’ll see if I can confirm but I assure you this shouldn’t be the case. Only example I could think that might make it a problem, is if lftp is actually running, essentially making the file unable to be removed.

[Doctrg]

@Tom-Elliott It would great if there were a global setting for disabling replication. I have updated one of our other storage nodes that was failing before 1.3.2 and with replication unchecked it uploaded without an issue.

[Tom Elliott]

@Doctrg I wonder if this isn’t a bug, but it cannot remove the original image file because it’s trying to push the images down to the other nodes?

As a test would you be able to do say:

service FOGImageReplicator stop (but keep the image with replication on?)

The image replicator service wouldn’t be trying to do anything and any stuck transfers would be killed on the stopping of the replicator service. Can you upload then?

[george1421]

@Doctrg said in Unable to upload images:

@Tom-Elliott It would great if there were a global setting for disabling replication…

Stop the FOGReplicator service and then all replication will stop.

But I see Tom’s point the only time the image will be locked is when it is currently in the replication state.

[Doctrg]

@Tom-Elliott I stopped the Replicator service and tried uploading with replicate checked. It failed uploading.

[Doctrg]

@Doctrg I’m told by one of our other techs that even with the service off and the replicate image unchecked that it still fails.