Posts made by zachu

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu THe reason the path is specified (not allowing slashes) is because it can give access to other parts of the filesystem. This doesn’t mean it WILL happen, but from a security perspective what’s to ensure you don’t write a specific file or overwrite the root folder? I know there’s some other obfuscated methods of protection here, but let’s just imagine, you decide to put the image path be be as /images/../../../../bin/init or /images/../../../../usr/bin/init or some other binary name.
While technically it shouldn’t be possible, with the wrong setup it is a possibility. Now your fog server will likely work for a while. Then start having some funny issues.

I perfectly understand why it’s like this. I was more like toying with different ways to go around the problem I had. I’m ok with the way it works now.

The Wiki is perfectly accurate. You would, essentially, change the images location to go from /images to /images/newdisk. No where in the wiki article, from what I’ve skimmed through, would state that you are placing the image on the disk in the way you originally described. Changing the way move_upload is not quite that simple.

Oh right, I think I then read too far there between the lines. With the guide on the wiki article’s last lines, it’s meant to move all your images under the “newdisk”, not just one or part of them. I get it now

Changing the way move_upload is not quite that simple.

Yeah, I glanced just quickly at the codebase there and realized that it isn’t just a minor change, but I think it is doable still. All the needed actions are moving (renaming) and deleting files and folders, just as it is done now. Now it’s really simple and pretty when you can just move the whole directory instead of doing multiple actions to the files.

Would you accept a pull request of changing the move_upload process, if I find time to put it together? I’m not making any promises, but just thinking about it. I’ve never really contributed to an open source project but I might have the skills to put that one together. But if it fights the way you want it to work, then I wont bother

I really thank you for your time!

Btw, I settled for a process that I just move the image files to the dedicated HDD always after upload. It’ll do now, since it’s only temporary

Okay so imo this topic can be marked as solved. I think @Tom-Elliott’s approach is currently the closest one to go for my needs. Really big thanks to others too that helped me with this. For others with similar needs, @george1421’s approach may be good way to go too.

So a wrap-up:
My goal was to mount a device /dev/sdb1 to /images/Gizmok4 so that when I upload a image Gizmok4, it goes to a dedicated HDD. I assumed this was a valid approach since it is hinted so in Fog wiki

Fog wiki in Moving_your_images_directory/Adding_Storage_to_the_Images_directory states:

If you want to just add capacity to your images directory, you could instead mount your new device as a subfolder of the /images directory. In this case your wouldn’t need to move the original directory to /images1, and your fstab line would look like:

UUID=XXXXXXXXXXXXXXXXXXXXXX    /images/newdisk      ext3 defaults 0 0

As it turned out, you can’t do this, since after an upload task fog deletes the image path and renames the uploaded image directory in place of the old image directory the image path and renames the uploaded image directory in place of the old image directory. The task fails, because the mountpoint isn’t deletable. One approach at fogprojects end to fix or allow this, is instead of deleting and renaming, just move the content of /images/dev/<macaddr> to /images/dev/<imagename> (<imagename> being Gizmok4 in my case).

I figured this limitation could be circled around by putting one directory before the image directory as a mountpoint. Mounting /dev/sdb1 to something like /images/sdb and have <imagename> folder under that (again Gizmok4 in my case). Then I thought to just change the Gizmok4 image path to /images/sdb/Gizmok4 and everything would be fixed. Now the image directory is removable and the process of deleting old files and renaming new files on the same place would work. In other words, I tried something like this:

$ tree /images
/images
├── dev
├── postdownloadscripts
│   └── fog.postdownload
└── sdb
    └── Gizmok4
        ├── d1.mbr
        ├── d1p1.img
        ├── d1p2.img
        └── d1.partitions

The problem in this approach comes from Fog WebUI, more specifically when validating the image path input in fog.image.js. Only alphanumeric input and a plus (+) character is allowed there.

Out of curiosity, I did disable that part of the javascript and uploaded an image to a path sdb/Gizmok4 (full path being /images/sdb/Gizmok4). Now for some reason I ended up having a file /images/sdb/Gizmok4 instead of a directory. All the real image files were kept in the dev/<macaddr>

Conclusion… and a workaround request

All in all, I think the wiki article (or the last part of it) is misleading, since it isn’t possible to add storage to /images/ by just mounting another device as a subfolder, as it suggests. Well, you can do it, but you can’t use it My opinion is that this could be very handy addition though and should be implemented. I think the best way to do that would be reworking the move_upload() function in taskqueue.class.php to work as following:

1. Empty the image directory. (Currently the directory gets deleted)
2. Move the contents of /images/dev/<macaddr> to the image directory. (Currently the whole directory gets moved/renamed)
3. Remove the directory /images/dev/<macaddr>. (Currently it gets renamed, so no removing needed)

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

So now I need further understanding.

If I’m reading you properly, you’re trying to make /images/Gizmok4 essentially take the place of /images? (Or any of the Gizmok things).

No. I have an image called Gizmok4 which locates in /images/Gizmok4 as it does on default settings.

I also have an images named Gizmok5 and Gizmok6, which locates in /images/Gizmok5 and /images/Gizmok6 respectively. They are on same settings as the Gizmok4-image with the exception of Gizmok6 being multiple disk image. Although that’s irrelevant here

Why not mount /dev/sdb1 (or your relevant items) to /images?

I need the images to be on different physical disks (one disk for one image) for reasons explained before. Mainly because of portability (need to physically carry the image to another location) and because of disk space (all the images don’t fit on one disk).

To understand further, Gizmok4 is intended to be the <imagename> placeholder?

Yes, Gizmok4 is the name of one of my images.

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu make sure the fog (linux) user has full read/write access to those attached disks.

The other thing you could do is login to the fog server using ftp. And then try to create a directory in the /images/dev (which should work) and then create a folder in /images and then /images/Gizmok4 See if the fog user has rights to do this.

I tried again the FTP on the fog user. I surely can create, write and delete on /images/dev/ and on /images/ and on /images/Gizmok4/. What I can’t do through ftp, is delete /images/Gizmok4 since it is a mountpoint. So I still guess that’s what the fog is trying to do on the “Updating database…” part. It deletes the /images/<imagename> and renames /images/dev/<macaddr> to /images/<imagename>. It fails because it cant delete the mountpoint.

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu If it was me I’d go about it a little differently.

What I would do is create an LVM volume (for the FOG server only at this point). And then add all of the disks to the logical volume. Let linux span the disks and manage all of that bits. That way you can follow the guide for adding additional storage to the fog server. That way your FOG server will work as you need it. With LVM if you get a new hard drive, just install it and add it to the lvm volume and then expand the logical volume.

This is actually about how we normally do things, and then transfer the images on network, but because of temporary special circumstances, I have to be on a slow net connection

Now for the portable FOG. Get a usb portable hard drive (or hard drive with a removable caddie) and copy the images to that portable hard drive. I know it will be a bit slower to pack up and go. But this way on both ends the fog server will be setup in a normal manner.

Yeah indeed this sounds easiest way to go, but since this is a temporary situation, I can’t get my hands on a usb drive easily Not until it’s not needed again. But good advices anyway. I think I’ll head that way if I can’t get this work some other way.

Maybe this whole thing could work also if I’d put one directory after the mountpoint. Like this: mount /dev/sdb1 /images/sdb, and then I’d change the image path to /images/sdb/Gizmok4. If the problem is that fog cannot remove the mountpoint as I’ve guessed it is, then that would probably solve the problem. I’ll test this.

In other words, the only thing that diffrences here on just default out-of-box fog usage, is this:
I would like that /images/<imagename> would be contained on a single HDD device that I can take away from the server and carry to another place.

• /images/Gizmok4 => /dev/sdb
• /images/Gizmok5 => /dev/sdc
• /images/Gizmok6 => /dev/sdd

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu if I’m reading what you’re attempting to do is just make a bunch of disks that contain the image as uploaded. Tell me if I’m wrong please, but you’re expecting that when you upload the image to the now mounted drive will be a clone of the system that uploaded the image?

Umm… no I don’t need the mounted drive to be a clone on the system. I need it to contain the image files so I can deploy it elsewhere

I have 3 completely different Windows7 hosts registered to my fog server. They all have different images. I’m trying to clone one of them and have the image files on the fog server /images/Gizmok4 folder. Another one would be on /images/Gizmok5 and the third on /images/Gizmok6. This is all regular fog usage?

Now it would help me tremendously if that /images/Gizmok4/ (as well as the other image locations) would be a mountpoint for a single HDD that I coud take with me to another site when I’m deploying the image to a bunch of computers. There I also have a fog storage node that I can use to deploy the image. The transfering part is just a problem now, since I’m temporarily sitting on slow connection.

Clarified anything?

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I mean, you said “I had mounted /dev/sdb1 to /images/Gizmok4”, but where is /dev/sdb1? Is /dev/sdb1 on the host doing the download/upload? Is /dev/sdb1 on the server?

If /dev/sdb1 is on the server, and /images/Gizmok4 is the mount point, how is /dev/sdb1 mounted to /images/Gizmok4? I see in the fstab that it’s using UUID and appears to be mounted VIA ext4 meaning is presented to the system as a device, but is it protected from being written to?

Also, remember, that the /images Folder is read only when used in NFS mode. My guess, is wherever /dev/sdb1 is located (if I had to guess it’s a SAN?) it’s not got the same permissions which would not allow the fog user to write to it.

The /dev/sdb1 is the UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba. It’s just a regular 1TB HDD in the fog server. The whole fog server is just one physical ATX computer lying under my desk No virtualization or anything.

@george1421 said in Upload Task: Updating database... failed. FTP problems:

@zachu I have to ask, what is the logic of creating your disk structure this way? IMO you have a fundamental storage issue here not something specific to fog.

So again, why do you have these disks mounted in this configuration?

I have multiple images which all wouldn’t fit into a one 1TB disk (and currently I don’t have bigger disks laying around). I also need to transfer these images around the city, and I’m currently sitting on a really slow Internet connection. The fastest way to transfer those images, would be to take the disk with me and take a bus to the place I’m going to need the image.

Because I need different image on different location, I thought that placing a single image on single disk would be perfect way to set up this configuration. And like I said in previous post, the wiki page here, the last line, does implicate that this would be possible.

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

Where is /dev/sdb1?

Now I don’t know what you mean If you mean now that the upload task worked, it’s still unmounted and not in use.

@Tom-Elliott said in Upload Task: Updating database... failed. FTP problems:

@zachu I don’t know what you mean.

Please explain.

Well wrapping up this whole thread:

• I had mounted /dev/sdb1 to /images/Gizmok4
• Uploading Gizmok4 image caused “Upload Task: Updating database… failed” problem
• /var/log/apache2/error.log was filled with PHP Warnings of ftp_delete and ftp_rmdir failing
• I unmounted /images/Gizmok4, made it a normal directory instead of a mountpoint to /dev/sdb1 and tried again. Everything worked.

@zachu said in Upload Task: Updating database... failed. FTP problems:

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

This seemed to work. So something goes wrong when i have an mounted device dedicated to one image. I wonder if fog tries to ftp_remove and ftp_rmdir that mountpoint and afterwards rename /images/dev/<macaddr> to /images/<imagename>. That could cause the behaviour we see. Instead imo fog should move the contents of dev to the image folder.

Wiki page Adding Storage to the Images directory does state that mounting /images/<imagename> should be possible, so do we concider this as a bug?

Removed lost+founds Didn’t help Also running foginstall again to make sure that ftp credentials are correct didn’t help either

Next I’ll try to unmount /images/Gizmok4/ so that it’s part of the root device and not its own dedicated HDD. I’ll report back when I’ve got some sleep first

@Quazz said in Upload Task: Updating database... failed. FTP problems:

@zachu I notice you have some lost+found folders in some of your images folders.

It would be best to remove those. (and you might want to ask yourself why they’re there in the first place, did you have a lot of file system issues on your linux server?)

The lost+found directories are created when a partition is formatted as ext4. I think I can delete them and remove the fsck checks so they shouldn’t be recreated.

I already have an upload task going on, after doing fog install over again to fix the possibly-wrong-credentials. It shouldn’t matter that I deleted the lost+founds now during the upload task, since the problem occurs at the end with ftp, right?

Lets see if that fixes things

@Quazz

running out of RAM. How much is in your system?

8 GB

$ cat /proc/meminfo
MemTotal:        8175644 kB

Can you also post the output of df -h

$ df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            3.9G  4.0K  3.9G   1% /dev
tmpfs           799M  792K  798M   1% /run
/dev/sda1       909G  4.4G  859G   1% /
none            4.0K     0  4.0K   0% /sys/fs/cgroup
none            5.0M     0  5.0M   0% /run/lock
none            3.9G     0  3.9G   0% /run/shm
none            100M     0  100M   0% /run/user
/dev/sdd1       917G   72M  871G   1% /images/Gizmok6
/dev/sdc1       917G  405G  466G  47% /images/Gizmok5
/dev/sdb1       917G   72M  871G   1% /images/Gizmok4

As for the segfaulting, I think it has something to do with the fact that fog spams the ftp_delete and ftp_rmdir REALLY MUCH. I mean the first error entry on the error.log has timestamp 21:17:21.586806 and the last 22:35:10.321097. In that timespan of less than 20 minutes 1 hour and 20 minutes, I have 996M of error.log just spamming the ftp_delete and ftp_rmdir warnings. There is 5351072 lines in my error.log on that timespan

edit: To clarify more, I think the segfaulting isn’t the root cause here. The ftp_rmdir and ftp_delete failing is. Or that’s what I think is happening

edit2: Time flies… Corrected the timespan of the error messages

@Wayne-Workman said in Upload Task: Updating database... failed. FTP problems:

How did you manually check?
Sometimes an invalid login error is thrown, sometimes not. Sometimes the problem lies deeper.
We have a troubleshooting article that addresses about >95% of what could go wrong:
https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP

Well I manually checked the fields that the wiki article here https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP#Credentials_.2F_Passwords points out. I also logged in on the local shell with the password shown in there, and it worked. Also I tried ftp’ing with that username and password inside, and it worked too. I tried to point out on my initial post that I think I’ve done everything the troubleshooting article tells me to.

I’ll do installfog.sh -y now and retry the upload as you suggested.

@Wayne-Workman No I don’t have any email notifications configured, and as far as I know, the credentials are correct since I haven’t changed anything after the install. And by manually checking, they seemed correct. Wouldn’t the error message come from ftp_connect ftp_login rather than ftp_delete or ftp_rmdir with invalid credentials anyway?

Hi!

This might be duplicate or related of Image Capture/Upload gets to 100%, then Updating Database task fails and FTP Delete an image - failing but I couldn’t figure it out.

I’m having problems with fresh install of Fog. Running Version 1.3.0-RC-2, SVN Revision: 5937 on Ubuntu 14.04.4.

I’ve registered 3 hosts and added 3 Windows-7 images. Nothing more. I’m trying to upload and image to the fog server, but get the “Updating database… Failed” error.

My apache2 error.log is flooded with PHP warnings ftp_delete() and ftp_rmdir() failing, with occasionally the apache2 process segfaulting.

$ head -n 4 /var/log/apache2/error.log
[Tue Jul 26 21:17:21.586806 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
[Tue Jul 26 21:17:21.586899 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
[Tue Jul 26 21:17:21.588240 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_delete(): Delete operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 59
[Tue Jul 26 21:17:21.588307 2016] [:error] [pid 29337] [client 192.168.88.200:42738] PHP Warning:  ftp_rmdir(): Remove directory operation failed. in /var/www/fog/lib/fog/fogftp.class.php on line 173
...
...
...
[Tue Jul 26 21:21:34.467151 2016] [core:notice] [pid 24528] AH00051: child pid 28846 exit signal Segmentation fault (11), possible coredump in /etc/apache2

I think the permissions should allow deleting, since they are all 777 permissioned, just in case.

$ ls -laR /images/
/images/:
total 28
drwxrwxrwx  7 fog  root 4096 Jul 26 19:42 .
drwxr-xr-x 25 root root 4096 Jul 26 15:39 ..
drwxrwxrwx  3 fog  root 4096 Jul 26 15:59 dev
drwxrwxrwx  2 fog  root 4096 Jul 26 18:16 Gizmok4
drwxrwxrwx  3 fog  root 4096 Jul 25 16:52 Gizmok5
drwxrwxrwx  3 fog  root 4096 Jul 25 16:56 Gizmok6
-rwxrwxrwx  1 fog  root    0 Jul 26 15:39 .mntcheck
drwxrwxrwx  2 fog  root 4096 Jul 26 15:39 postdownloadscripts

/images/dev:
total 12
drwxrwxrwx 3 fog root 4096 Jul 26 15:59 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
drwxrwxrwx 2 fog root 4096 Jul 26 21:17 bc5ff4473427
-rwxrwxrwx 1 fog root    0 Jul 26 15:39 .mntcheck

/images/dev/bc5ff4473427:
total 450560628
drwxrwxrwx 2 fog root         4096 Jul 26 21:17 .
drwxrwxrwx 3 fog root         4096 Jul 26 15:59 ..
-rwxrwxrwx 1 fog root      1048576 Jul 26 18:54 d1.mbr
-rwxrwxrwx 1 fog root      8912823 Jul 26 18:54 d1p1.img
-rwxrwxrwx 1 fog root 461364057634 Jul 26 21:17 d1p2.img
-rwxrwxrwx 1 fog root          190 Jul 26 18:54 d1.partitions

/images/Gizmok4:
total 8
drwxrwxrwx 2 fog root 4096 Jul 26 18:16 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..

/images/Gizmok5:
total 424530580
drwxrwxrwx 3 fog root         4096 Jul 25 16:52 .
drwxrwxrwx 7 fog root         4096 Jul 26 19:42 ..
-rwxrwxrwx 1 fog root      1048576 Jul 22 20:02 d1.mbr
-rwxrwxrwx 1 fog root      8570406 Jul 22 20:52 d1p1.img
-rwxrwxrwx 1 fog root 434709616470 Jul 22 20:52 d1p2.img
-rwxrwxrwx 1 fog root          190 Jul 22 20:02 d1.partitions
drwxrwxrwx 2 fog root        16384 Jul 22 19:57 lost+found

/images/Gizmok5/lost+found:
total 20
drwxrwxrwx 2 fog root 16384 Jul 22 19:57 .
drwxrwxrwx 3 fog root  4096 Jul 25 16:52 ..

/images/Gizmok6:
total 24
drwxrwxrwx 3 fog root  4096 Jul 25 16:56 .
drwxrwxrwx 7 fog root  4096 Jul 26 19:42 ..
drwxrwxrwx 2 fog root 16384 Jul 25 16:56 lost+found

/images/Gizmok6/lost+found:
total 20
drwxrwxrwx 2 fog root 16384 Jul 25 16:56 .
drwxrwxrwx 3 fog root  4096 Jul 25 16:56 ..

/images/postdownloadscripts:
total 12
drwxrwxrwx 2 fog root 4096 Jul 26 15:39 .
drwxrwxrwx 7 fog root 4096 Jul 26 19:42 ..
-rwxrwxrwx 1 fog root  233 Jul 26 15:39 fog.postdownload

I do have every image on their own HDD, if that might be the reason?

$ tail -n3 /etc/fstab
UUID=59ea75b7-e91e-4965-814c-4f54e2c15fe4       /images/Gizmok6 ext4    defaults        0       2
UUID=9fb7e4b0-6237-4c8b-b746-22212d89ad10       /images/Gizmok5 ext4    defaults        0       2
UUID=beb59605-cbfb-4ee1-a943-561c8c5c5bba       /images/Gizmok4 ext4    defaults        0       2

As I checked, the FTP credentials seemed to be correct in all the places shown in here, as they were set on the install process. I haven’t touched them. Also I tested the FTP as shown here and all the steps went through with no problem.

I would appreciate a poke to the right direction Thanks!