FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS

[Fog_Newb]

@Tom-Elliott said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

@Fog_Newb

Manually moving a directory (which couldn’t possibly fit on vda) from /images/dev (vdc) to /images (vdb) caused the virtual disk size on vdb to increase. IMHO, it seems clear the directory was initially stored on vdc. Am I wrong in assuming this confirms that the mounts are correctly configured and functioning?

Yes. vdb is /images, so moving a file from /images/dev (vdc) to /images (vdb) should cause the disk to increase on vdb? The sice on vdc wouldn’t decrease but the amount of usable space off vdc should be increased by the amount of the file that was moved to now vdb?

That’s what I’m understanding here. I don’t know if this means there is an issue or if this is working as intended, but it seems like it is?

That is what I was thinking - I was trying to establish that the mount points are set up correctly and that it’s not an issue with fstab.

It only takes a few minutes or so to spin one of these VM’s. I spun up another VM, same parameters and mount points. The only things added other than the ssh server during the Ubuntu install were - the Ubuntu updates, dnsmasq, pip, tzlocal, tz… and of course the latest FOG dev version 1.5.10.1614. All cloud services, disabled

I exported it to OVF right before registering a host and capturing an image… which exhibited the same problems.

If you or anyone else wants to check it out

https://drive.google.com/file/d/1GPJH0ZuJFZaTqmGvIe1EM2HBCVlde4FI/view?usp=sharing

username: test
password: 123456

All 3 VM drive images are included with the OVF. A 2.8 GB zip since the drives have barely anything on them.

Fog web interface - user and password are defaults

It takes a ridiculously long time to import the OVF into VMWare Workstation. But it will import

If you want it quick and dirty create a new UEFI based Linux VM and chose to use the existing vmdk drives contained in the zip.

A copy of the conf file used by dnsmasq is in the home directory

[JJ Fullmer]

@Fog_Newb
I think this might be a limitation on the NFS side and the purposely limited permissions of the host when booted into FOS (Fog Operating System) for a task.
I know I once tried to do a separate disk within the /images NFS mount to have drivers I inject on a separate disk. I ran into similar issues, in the end I just made a bigger disks for /images.

Also, just an FYI, it’s not random numbers, it’s the mac address of the host being captured.

So anyway it sounds like.

• FOS is mounting both nfs shares correctly.
• FOS is writing to the /images/dev share
• FOS can’t move the files from one share to another when the shares are on different disks.

There could be an issue with the nfs exports

If you have another linux machine you could try mounting /images and /images/dev via nfs and see if you can copy in those use cases

[JJ Fullmer]

@JJ-Fullmer Maybe if we mounted the shares in fos differently so that they are in different mount paths and not the same on the client/host then this might work, but that’s a bigger change than it sounds like I think.

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

@Fog_Newb

• FOS is writing to the /images/dev share
• FOS can’t move the files from one share to another when the shares are on different disks.

exactly

FOS is mounting both nfs shares correctly.

In my case, there’s no NFS shares involved, unless I’m misunderstanding.

/images and /images/dev are ‘mount points’ of the 2nd and 3rd local hard-drives on the VM. Maybe I’ve been using the term ‘mount points’ incorrectly

Also, just an FYI, it’s not random numbers, it’s the mac address of the host being captured.

When I mentioned random numbers I was talking about the numbers listed during the ubuntu install for the disk drive device numbers.

They weren’t the UUID’s of the drives or partitions and looked too long to be MAC addresses

[JJ Fullmer]

@Fog_Newb Oh those are random, I thought you were talking about the /images/dev/{mac} that is used when capturing.

So /images and /images/dev are 2 nfs shares on your FOG server. You’ll see this defined in /etc/exports

If you aren’t able to manually move the file on the FOG server from /images/dev to /images then there’s some other weird permissions issue on the server.

When you are capturing an image though, the host boots to FOS, and part of the capture process is mounting the 2 NFS shares from the FOG server, /images and /images/dev. The capture process has partclone writing the captured image files to /images/dev/{mac}/*
Once complete, it moves them over to /images (technically it deletes the old version if any in /images and then moves the new version into its place).
I don’t recall if in the stable/dev branch if we are using sftp yet, so it may be doing ftp (1.6 for sure uses sftp). Point being it does technically remote into the server to do the move operation because the /images nfs share is a read only share.
When you have nfs shares mounted in the same folder that are actually on different disks, I’ve seen weirdness like you’re experiencing is what I’m saying.

So all of that is to say

• Try moving/copying a file from /images/dev to /images on the server directly, confirm that works.
• Try booting into a debug capture session, capture the image (use the command fog and hit enter to step through. Capture any errors.

Technically if you’ve captured the image, you can manually move it on the server to the /images/{imageName} folder and you should be able to deploy as a workaround while we figure this out.

You could also try installing 1.6-beta.

[Fog_Newb]

This post is deleted!

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

So /images and /images/dev are 2 nfs shares on your FOG server.

They are both local drives on the FOG server formatted GUID ext4. Everything is on that machine. There aren’t any network drives or shares.

Try moving/copying a file from /images/dev to /images on the server directly, confirm that works.

Yes, moving and copying from /images/dev to /images and visa versa works fine.

Try booting into a debug capture session, capture the image (use the command fog and hit enter to step through. Capture any errors.

There are no errors during the debug capture process.

Oddly enough, FOG/FOS will attempt to create the directory on /images, but it creates a file instead.

So, If I try to capture a new image named “BigBox”, after the capture there will be a file, not a directory, in /images named “BigBox”, - but that is it. Everything else stays in /images/dev/{mac}/*

Updated - I’ve tried FOG 1.5.10.1615 (coming from FOG 1.5.10.1612) and 1.6.0-beta.2130. Same results and no errors during debug-capture.

*1.6.0-beta.2130 does not create a file named after the image in /images

P.S. I now see what you mean by “the random numbers” being MAC addresses (the temp directories for the images). I thought you were talking about the other ‘random numbers’

[JJ Fullmer]

@Fog_Newb Fog is creating NFS shares from your 2 disks at those mount points. If they were not shared, imaging would not work. On the server the paths are mounted to different disks. And then each of those paths are shared, you can see this if you run cat /etc/exports where the shares are defined.

That said, the actual moving of files takes place with a remote command of ftp or ssh/sftp not via the nfs share. So it should be working the same.

So in 1.6 does the image stay in the dev folder?
What is the size of the file when it creates a file instead of a folder?
What is your php version? php - v

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

@Fog_Newb Fog is creating NFS shares from your 2 disks at those mount points. If they were not shared, imaging would not work. On the server the paths are mounted to different disks. And then each of those paths are shared, you can see this if you run cat /etc/exports where the shares are defined.

Oh I see

Why is /images ro? Shouldn’t it be rw? How do I change that? I’m going to edit the /etc/exports file and change /images to rw and test

So I edited it - from ro to rw saved and rebooted but same results.

I guess ro is the way /images is supposed to be as far as /etc/exports

https://github.com/FOGProject/fogproject/security/advisories/GHSA-3xjr-xf9v-hwjh

So in 1.6 does the image stay in the dev folder?

Yes

What is the size of the file when it creates a file instead of a folder?

0 bytes

What is your php version? php - v

8.3.6

[JJ Fullmer]

@Fog_Newb

I guess ro is the way /images is supposed to be as far as /etc/exports

https://github.com/FOGProject/fogproject/security/advisories/GHSA-3xjr-xf9v-hwjh

I’m glad you found the security advisory and already changed that back to RO, I panicked a little when I read that.

So in 1.6 does the image stay in the dev folder?

Yes

Then you could at least manually move the image from dev to images to workaround the issue while we work this out.
This feels like a permissions issue, but you already did that step it sounds like.

What is the size of the file when it creates a file instead of a folder?

0 bytes

And what does ls -l /images show, I’m curious if it’s a file or a symlink. Either way it’s not what should be happening

What is your php version? php - v

8.3.6

Might be worth upgrading to the latest php version (8.3.11), there was a bug in 8.3.10 (Which I realize you’re not using) that caused an issue with the part of capture where we delete the file that was fixed in 8.3.11, but it was likely fine in 8.3.6.

I’m going to test some of this on my dev fog server and see what happens.

[JJ Fullmer]

@Fog_Newb I am able to recreate the problem in the latest stable version (1.5.10.1615). Working it out. it looks like the image does stay in dev on my end.
I’ll debug this when I can this week.

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

@Fog_Newb I am able to recreate the problem in the latest stable version (1.5.10.1615). Working it out. it looks like the image does stay in dev on my end.
I’ll debug this when I can this week.

Awesome. Thanks.

Then you could at least manually move the image from dev to images to workaround the issue while we work this out.

I have another FOG server set up where /images and /images/dev are on the same drive so I can continue using that until things get sorted.

[JJ Fullmer]

@Fog_Newb I think I found the problem.

Normally the ftp_rename command successfully handles moving the captured image from /images/dev to /images but for some reason that command ( a built in php/ftp thing) doesn’t like it when the directories are mounted on separate disks. Not sure why, doesn’t make a ton of sense, but that’s the issue.

In Fog, when this fails, we attempt an ftp_put command aka an upload.
This should work, but there’s a path mismatch at the moment.
During capture the “local” file where it is captured is at /images/{mac} instead of /images/dev/{mac} and the put command is getting the latter because that’s what we already have for the rename command that happens completely remotely.
So I’m testing some stuff to fix this in the dev-branch.

Basically I think you may have found a bug in a failsafe method for uploading a captured image.

[Tom Elliott]

@Fog_Newb @JJ-Fullmer I wonder if this would work better separated out:

mkdir /images1
mkdir /imagesdev1

mv /images/dev/* /imagesdev1 #everything here to imagesdev1 for example
rm -rf /images/dev #only after moving everything first (including . files/directories)
mv /images/* /images1

rm -rf /images #only after moving everything first of course)

ln -s /images1 /images
ln -s /imagesdev1 /images/dev

Does this work?

Of course your fstab would need adjusting if it does, but jus trying to think what may be causing issue is that a root system (/images for example) doesn’t know how to handle the inode movement occurring from another filesystem (/images/dev for example) If you were to move /imagesdev1/file to /images1/file it should work just fine because you’re crossing filesystems which doesn’t pose a problem due to the root structures actually being at 2 different physical points. I believe this is a limitation of linux, not necessary a limitation of FOG persay.

[Fog_Newb]

@Tom-Elliott said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

I will definitely try this. Just to make sure I understand how you want these new directories mounted…

sudo mkdir /images1
sudo mkdir /imagesdev1

Then adjust fstab so ‘/images1’ is a mount point for one drive (where /images was mounted), and ‘/imagesdev1’ is a mount point for the other drive (where /images/dev was mounted. Mount them, once mounted then I do…

After mounting these, new mount points, everything that was in /images and /images/dev is now in /images1 and /imagesdev1

sudo rm -rf  /images
sudo rm -rf  /images/dev
sudo ln -s /images1 /images
sudo ln -s /imagesdev1 /images/dev

Then capture an image and see what happens

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

@Fog_Newb I think I found the problem.

Normally the ftp_rename command successfully handles moving the captured image from /images/dev to /images but for some reason that command ( a built in php/ftp thing) doesn’t like it when the directories are mounted on separate disks. Not sure why, doesn’t make a ton of sense, but that’s the issue.

That makes sense to me especially if that command is only able to rename.

Basically I think you may have found a bug in a failsafe method for uploading a captured image.

Cool cool?

[Fog_Newb]

@Tom-Elliott

Ok I think I have it set up how you asked

New mount points

Symlinks

Gonna try to capture will report back

Same thing.

[JJ Fullmer]

@Tom-Elliott @Fog_Newb

I tried this symlink method on my test server and didn’t have success sadly. Seems nfs doesn’t follow symlinks.
My theory on the put function was also wrong.

How big of an issue is this disk size inflation for you? Typically you’re only capturing 1 image at a time, so you need your /images disk to have
(enough space for all your images) + (the size of your biggest image) + (maybe 10% of that for flexibility)
So if your current images take up say 500 GB, and the biggest of them was 100GB (which would be a very big image)
Then provisioning 660 GB should be sufficient if you didn’t want to just double the provisioned space.
If you want to have space to capture them all at the same time, then 1.1 TB would do the trick.
If your images are just really really big, then consider using snapins or other software deployment methods to keep your images to just the base os and a limited number of default apps so that they stay on the smaller side. For examples, my Windows 11 images take up just under 20 GB on the server and around 30 GB on my hosts and then I add the rest per use case within my environment.

You can also play with the different compression levels of the image. I’ve found a zstd compression level of 11 to be a good balance of speed and size.
You could also throw caution to the wind and not care about the speed of deploy and capture and just put the compression to max if you’re more concerned about taking less server disk space.

The only thing I can think of to make this work natively would require database schema changes so you can define separate paths for /images and /images/dev and reworking the install script to make sure the /etc/exports is created correctly. Sadly I don’t think that amount of effort would benefit any other use case and this is an uncommon use case that’s not in our realm of support right now, especially since it’s mostly due to limitations in linux.

The is one other possibility, and that would be a postinit script, but those run before imaging starts not after like postdownload scripts. But there’s other security issues there with how you’d get the ro /images share to be mounted as rw when your task only has permissions to open /images/dev which does have rw. So you’d have to open an ssh or sftp session and send the command to move the folder from /images/dev to /images that way. But securely creating that connection at that point is a different beast. It’s odd that linux/php are having this limitation but that’s where we’re at.

[JJ Fullmer]

@Fog_Newb said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

That makes sense to me especially if that command is only able to rename.

Just for clarity. Rename in this context also handles moving the file. By “renaming” the file from its old path of /images/dev/$mac to /images/$imagename it’s really moving the file. But since those paths are on different filesystems, it doesn’t work.
The php ftp_put attempts to use an upload of the file from the server to the other path, but that is when it’s actually making the empty file because it gets confused by the paths being on different disks.

[Fog_Newb]

@JJ-Fullmer said in FOG has issues if the temp image location is on another drive. FOG 1.5.10.1612 Ubuntu Server24.04.1 LTS:

How big of an issue is this disk size inflation for you?

It’s not a big deal. I’ve since stopped capturing non-OS drives and now use file sync / mirror program to back those up. I actually like that better than huge image captures once a week. Currently, FOG is just capturing 2 relatively small OS drive images.

All this is running on a QNAP box someone gave me and QNAP’s Virtualization Station software is lacking features to cleanup/shrink or compact virtual machine disks.

This all started out as a Feature Request

https://forums.fogproject.org/topic/17651/local-time-and-alternate-temp-directory-for-capture-option

It’s odd that linux/php are having this limitation but that’s where we’re at.

Is this something the devs would change if it was brought to their attention?

Thanks