OK, it’s my fault for skimming code — getHardDisk does not print anything to stdout but saves the output to a variable. Checking that variable shows it does properly find the NVME drive.
I think this request can be closed then as I can use this library to access things, like the imaged drive and make my postimaging scripts much more robust.
Thanks @george1421 & @Tom-Elliott
@Tom-Elliott – no output (see image)
@george1421 – machine images properly; postdownload steps fail (due to hard-coded /dev/sda2)
@george1421 said in get imaged disk information for post download scripts:
@mrayzies said in get imaged disk information for post download scripts:
Undocumented feature…
The best ones always are.
Actually one of the developers tuned up the script I wrote when the NVMe disks came on line. The NVMe naming convention is a bit different. The funcs.sh has always been in FOS, its just hasn’t been end user reachable.
Can you confirm when that was? I’m running 1.4.4, but when I source that library on a debug installation, getHardDisk returns nothing.
@george1421 – sorry, I’m a couple versions behind I guess; when did the shared library funcs.sh get introduced?
I don’t see anything on the forum for this library either:
If you need to image a particular device, you can use the “Host Primary Disk” argument to define something (e.g. “/dev/sdb”). However, if you specify nothing, FOS chooses a disk to image by some mechanism. In either case, the information of what disk FOG images (and what partitions it creates) are seemingly not available to post-imaging scripts; this leads to hard-coded values, e.g. osdiskpart=“/dev/sda2”;. The problem with this is that if the hard-coded value does not correspond to the imaged disk, post-imaging tasks fail (in the best case, everything errors out and in the worst case, data on a non-imaged disk gets mutated/overwritten). In my particular case, our image was setup with a typical SSD and worked well until we bought a laptop with an NVME drive, which enumerates as /dev/nvme0n1 .
If the disk information was shared/stored, post-imaging scripts could be rewritten intelligently to always run on the proper device, e.g. “mount $fog_disk_part_2 /mnt”.
This should work both for multiple disk systems, e.g. /dev/sda & /dev/sdb as well as with non-typical devices, e.g. /dev/nvme0n1.
For this image, what do you have defined for “Image Type” and “Partition”? Perhaps you have something misconfigured here which is why the UUID of the SWAP space is not set properly?
I’d guess that you should have “Multiple Partition Image - Single Disk (Not Resizable) - (2)” and “Everything - (1)” respectively.
The following might be lies, so take it with a grain of salt til someone more qualified comments:
When your scripts are running, they are running from the perspective of the client; however, the file system you see is not the actual file system on the disk of the client; it’s a temporary file system that comes from the PXE file/kernel you boot. So when you operate on “/etc/fstab”, you are making a change to the fstab of something that is running memory, and thus no changes persist. If you really want to operate on files on the host, you need to mount the disk device. This is not an entirely fool-proof way, but for my scripts, I mount /dev/sda to /mnt and then can operate on files within that and those changes end up in the host. So for your example, you’d need to do something like this in your script:
#!/bin/bash
mount -t ext4 /dev/sda1 /mnt
ORIGINAL=$(grep -v '#' /mnt/etc/fstab | grep swap | cut -d ' ' -f1 | cut -d '=' -f2)
....
/dev/sda has worked well enough for me, though I am sure there are situations in which it would fail.
Secondly, to the heart of your issue, I believe your analysis is correct in that the UUID issue isn’t something you should be fixing in postdownload scripts. UUID’s, if I understand correctly, are unique, based on the hardware ID of the device. To my understanding, FOG should be handing that during the imaging process (similarly to how it handles growing the file system).
@Wayne-Workman said in Splitting up FOG pieces:
@mrayzies said in Splitting up FOG pieces:
I verified I was able to complete all steps listed in the “testing” section (https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP) and still see the error.
Any other ideas?The main server will first try to make a web query to the storage node. If the node were an actual storage node setup by the fog installer, it would respond. Since its not a full storage node, the web query fails, and the web server is supposed to fall back to FTP.
when testing, test from cli of the web server, use the same credentials you put in the storage management area for the node. Did you set the ftp users home directory to where the images are?
My new server is failing the FTP commands. But so is my old server.
What I believe is happening is my customization is coming back to bite me – I am pretty confident that I hardcoded some values into the old install that hijacked what would have been requests to the storage nodes and re-wrote them to use the local FOG server; so the client would make an FTP request to the FOG server, it would work fine, and then it would try and mount the remote server anyway – and I had added all the necessary permissions to make that work.
So I can either straighten out my install to be more standard or go through that headache again. In either case, I’d consider this “resolved” and not spend anymore time on it. Thanks for the help.
Debug output, like @george1421 mentioned, should help you diagnose the problem. Debug deploys can help you get it, but you can also cheat it by waiting for input:
#!/bin/sh
set -x
. ${postdownpath}update_swap_uuid.sh
read -n1 -r -p "[$(date)] press any key to continue"
Off hand, I’m guessing it’s probably that “${postdownpath}” doesn’t end in a trailing slash, so your client is trying to execute something like /images/postdownloadscriptsupdate_swap_uuid.sh which likely doesn’t exist.
@Wayne-Workman said in Splitting up FOG pieces:
@mrayzies said in Splitting up FOG pieces:
However, I am unable to schedule any deploy tasks as the webUI returns: “Type: 8, File: /var/www/fog/lib/fog/fogbase.class.php, Line: 793, Message: Undefined index: value, Host: 10.52.2.31, Username: fog”.
That’s an FTP issue. Look in the wiki for Troubleshoot FTP.
I verified I was able to complete all steps listed in the “testing” section (https://wiki.fogproject.org/wiki/index.php?title=Troubleshoot_FTP) and still see the error.
Any other ideas?
Regarding the silent installer – you had me install it once, then change settings and re-run it; is it supported to change the settings beforehand and then run it?
Lastly, regarding the non-standard setup, that’s good information about the progress of FOG and I’m glad to see it’s doing well. I’ll consider moving to the supported setup, but I won’t waste time trying to debug my non-standard setup.
“Hacks” was probably a poor term. To get it to play nice the way we want with Ansible, I dug into your installer script and tore it apart such that FOG was installed/configured without the use of the FOG installation script.
It seems like I’ll have to do that sort of mutilation again if I want an unattended installation.
In any case, I tried out FOG 1.3.0 per your recommendation, following your instructions, and I was able to see my existing images without the errors I was seeing in FOG 1.2.0. However, I am unable to schedule any deploy tasks as the webUI returns: “Type: 8, File: /var/www/fog/lib/fog/fogbase.class.php, Line: 793, Message: Undefined index: value, Host: 10.52.2.31, Username: fog”. The code offers no clue out of context (to me): "
/**
* Check if isLoaded
*
* @param string|int $key the key to see if loaded
*
* @return bool|string
*/
protected function isLoaded($key)
{
$key = $this->key($key);
$result = isset($this->isLoaded[$key]) ? $this->isLoaded[$key] : 0;
(793) $this->isLoaded[$key]++;
return $result ? $result : false;
}
On a separate note, is there are a particular reason you recommend storage replication via FOG nodes? Our images are hosted on a ZFS server with several bells and whistles to make it extremely efficient, so even if I had more FOG hosts, I’m inclined to just point all of them to it. The server internally handles data replication with its peer node, so it isn’t a single point of failure.
A while ago, I set up our FOG server. It was back when 1.2.0 wasn’t quite finished, so I used an arbitrary commit at the time. I hacked around in it quite a bit to strip out the components of FOG to better integrate into our environment; ultimately, FOG became only the home for the PXE files and web interface (the database was put on a database cluster, the images directory became an NFS mount from another host) and everything worked great.
I now need to create another FOG server in an isolated development subnet. To that end, I figured I’d use the actual 1.2.0 FOG release and mutate my process so that I could use our deployment tools (i.e. Ansible) to create a server for any subnet/isolated setup that I needed.
However, the code has changed substantially from when I hacked at it to the 1.2.0 release and it would essentially be another large investment of my time to hack it all apart.
So my question is, is it possible in 1.2.0 to split out the components of FOG such that I can tell it to just use a different host for some function?
I’ve tried to play nice by adding the other host as a storage node, but even with a database import, the images are listed with no available storage node. I must be making something unhappy as well with that as frequently the webui will lock up and demand I update my database schema (which promptly fails).
If this functionality isn’t available in 1.2.0, is it something that is being worked on in 1.3.0?
I’ll try to get a video/screenshots of the situation shortly.
As far as trying an updated FOG and/or a custom kernel… I’ll add it to my queue, but it may be a while before I can get to it; I’ll try to keep you posted.
The machines are already set to use ipxe.pxe from our DHCP server: filename "ipxe.pxe";. I can promise that DHCP server is the one controlling what file they boot.
As far as the HDD issue, maybe I am misunderstanding, but I don’t think it’s the linux kernel. If I have a VM with a virtio HDD, I can boot it just fine. It’s when FOG is trying to push/pull an image that FOG itself states there is no HDD. Thus I think it’s the kernel FOG is using.
If you think it would be useful, I can try and find some time to spin up a test FOG server with the latest and greatest version to see how it behaves.
While I found this ticket (https://forums.fogproject.org/topic/453/virtio-drivers), it does not have much response and it is very old.
So, I’d like to ask about the feasibility of supporting virtio drivers (NIC and HDD). Currently, when I have a VM that has a virtio NIC, it fails to ever load the FOG menu, sitting on bringing up “net0” before ultimately rebooting the machine. When I have a VM that has a virtio HDD (for either pulling or pushing images), FOG starts the task and then freaks out with a “no HDD found”.
I am running an older build of FOG (#5688)
Thanks for the reply Sebastian and sorry for the super long delay.
I am observing this behavior when using “ipxe.pxe” – I have not used any other one binary (not since I switched it to a production server a couple months ago).
Also, to clarify, the transfer rate is good once the transfer begins, it is only when it is “configuring net0” that it seems to be a slow link. However, I assume it is some hardware dependent issue as I just tried a random computer model and saw it was configuring at the expected 1Gb link speed.
So I guess the tl;dr is that this isn’t really a problem.
I must have misunderstood your response – I thought you were saying the original MAC would still be the primary MAC of the host ?
If the original MAC becomes an additional MAC, then yes, just removing it and creating a new host is simple and a fine solution.
I found several other issues talking about configuration of net0, but did not see this particular error mentioned.
After having upgraded our FOG server (from the ancient .32) to a more recent release (build # 5688) and using it for awhile, I’ve noted that machines when booting to FOG link at a very low 10Mb. It seems to be all computers. This happens across many different computer models. It also appears that this causes the net0 configuration step to fail on occasion, making the computer boot-loop 3 or 4 times before it gets in. This is not a limitation of the computer because once the OS has loaded, the computer will link at 1 Gb.
Why is it linking at such a slow speed? How can I get it to link at a faster 100Mb or 1Gb speed?
If I have read your response correctly, editing the primary MAC will now result in that newly entered MAC ending up as an additional MAC for the same host?
While that prevents the duplicate host issue which is my primary concern; it is still not ideal as that means I will have to manually delete the host and re-add to fix the issue. This is not the end of the world and if the cost to changing this behavior is as high as you say, then I’ll live with it.
Sorry I’ve been unresponsive – I’ve been quite busy with tasks at work and probably wont’ update anytime soon. If you think it’s fixed, I’d say it’s fine to close this ticket ----- if/when I upgrade, I can re-open/comment if I still find it to be an issue.