Posts made by george1421

I would take a look at the apache log file. It sounds like you came across a bug. tail the contents of /var/log/httpd/error_log and post the last few lines here. That may give us a better idea to the source of the issue.

When I installed the access control plugin and then activated it the page turned white with no response for a minute (probably longer since I did not want to wait).

In the http error log there was this error.

PHP Fatal error:  Call to a member function isValid() on null in /var/www/html/fog/lib/plugins/accesscontrol/hooks/RemoveMenuItems.hook.php on line 13, referer: http://10.12.1.88/fog/management/index.php?node=plugin&sub=install&run=bd6bf52053b1ddb064bd3c493bfc47ef&plug_name=accesscontrol&install=bd6bf52053b1ddb064bd3c493bfc47ef&plug_name=accesscontrol

Refreshing the management page created this error.

PHP Fatal error:  Call to a member function isValid() on null in /var/www/html/fog/lib/plugins/accesscontrol/hooks/RemoveMenuItems.hook.php on line 13

I seem to remember this same issue when I upgraded from a svn trunk to the next svn trunk sometime in the past.

The previous posts dealt with adding a new disk using the traditional partitions. Most linux distributions today use LVM to manage the disks. With LVM one could create a new vmdk, create a partiton on it, add it to the volume group, and then extend the logical volume to the new size, and then run the resize2fs command and be done with it.

But that really doesn’t fix the issue in that the /images and /opt/fog/snapins are still mounted on the root file system. All the above will do is delay the OS getting whacked for some period of more time. The right answer is to create a new disk/partition LVM or physical and then use the mount point method outlined below to mount the new disk over the /images directory. This way we are sure that the OS will never fill up with captured host images.

I would recommend that you create this new disk on its own vmdk without any additional partitions. Because there are some linux magic tricks we can do to grow this new disk if we ever run out of space in out /images folder again. Understand this time we will not crash the FOG server since all of our images are NOT on the root partition that the OS uses.

These next steps should only be taken by a seasoned linux professional, because there is a risk that if done wrong you can loose all of your host images. I’m not going to give all of the steps because if you are a linux professional you will know what to do. So step one and two of this next part is make a full system backup and make a second full system backup. I should also note that these steps only work for FOG servers running on a virtual machine. The physical guys can’t expand a hard drive larger that it was made as.

3. The next step is to go into your hypervisor and expand the vmdk file we created earlier. I’m not going to give the steps to do this since each hypervisor is different.
4. Now you should have a larger vmdk file with all of your image at the beginning of the disk with all of the new space at the end of the disk (beyond the current extent of partition 1)
5. Use fdisk on the drive fdisk /dev/sdb1
6. press <p> to display the disk. You should see something like this: Disk /dev/sdb: 320 GB (or what ever size you extended your vmdk file to)
7. When you are sure you are on the correct drive (note this next part is a bit nerve wracking) we are going to use fdisk to delete the partition and recreate it with the additional space. The keys to doing this right is to NOT change the starting block of the partition and to ALWAYS make the ending block larger than it is currently. If you fail to do this… well that is why you have two sets of backups.
8. delete partition 1, write the changes to disk and exit fdisk.
9. go right back into fdisk with fdisk /dev/sdb
10. Create a new partition 1 with the same start block that we just deleted and choose the default value for the end block. This should default to the end of the vmdk that we extended earlier. Write the changes to disk and then exit fdisk.
11. The next thing we need to check is to see if we broke the file system in any way. Run the following command to check the file system. e2fsck -f /dev/sdb1
12. If everything is still ok with the file system then we can extend it to the partition size with resize2fs /dev/sdb1
13. Now we just need to remount the expanded disk onto the /images mount point with the mount -a command.

We need to unmount the new hard drive from the /mnt/test mount mount point with this command
umount /mnt/test

Run the df -h command to ensure that sdb1 has been unmounted.

Then clean up the test folder (not needed anymore) with rmdir /mnt/test

Now we need to mount our new hard drive over the /images folder. We can do this with the following command
mount -t ext4 /dev/sdb1 /images

Change to the /images folder and you should see all of your host image files.

At this point we are almost done. If we were to reboot the FOG server this manual mount command would not be active after the reboot. So lets make this change permanent.

You will need to edit the fstab in /etc Insert the following line info fstab at the bottom.

/dev/sdb1    /images    ext4    defaults    0    1

Now lets unmount the images folder with all of the host image files.
umount /images

If you show the files in the /images folder it should be blank.
ls -la /images

Now lets mount the /images folder again using the following command
mount -a

Us the df -h command to show that we’ve mounted /dev/sdb1 on /images.

Reboot your fog server and use the df command to make sure your images are remounted over the /images folder.

We know that the new disk is /dev/sdb so use fdisk to create a new partition on the blank disk. If you have questions about fdisk google it.

fdisk /dev/sdb
Create a <n>ew
<p>artition
numbere <1>
<default>
<default>
<w>rite the changes to disk
<q>uit fdisk

Next we need to format the new partition with our linux file system.

mkfs.ext4 /dev/sdb1

With the new disk formatted we need to connect it to our really full FOG server. If your fog server is 100% full you may have a problem with the next command you may need to find and delete an unwanted log file from somewhere to make a little room to create a new directory.

For this step we will create a mount point to connect our new hard drive to
mkdir /mnt/test

Now we’ll attach our new disk to that mount point
mount -t ext4 /dev/sdb1 /mnt/test

if you run the lsblk command again you should now see that sdb has a partition sdb1 <note this instruction probably should appear before the we formatted the filesystem just for the flow of the document>

The command df -h will show both the root file system and the new hard drive mounted on the /mnt/test mount point.

Now we need to move the host image files from the /images directory onto our new empty hard drive.

mv /images/* /mnt/test

After a bit of churning your host image files will be moved to the new disk.

After all of your image files are safely on the new disk we need to unmount the current mount point and remount the new disk over the /images directory.

My recommendation would be for the developers to move the /images folder to another location like /opt/fog/images so that it is along side the snapins folder, so this could be done with one action. With that said, there are ways to get the file system back without loosing or deleting the images stored on the FOG server. This “fix” involves some advanced linux understanding but it isn’t that difficult to do.

First of all if you have your FOG server running in a virtual machine these steps will be easy since you will need to create a new (additional) virtual disk and attach it to your virtual machine. This will create two hard drives attached to your FOG server. If you have a physical FOG server you will need to had a second physical hard drive to make this work.

(the rest of this will assume you have FOG running on a virtual machine) With that second hard drive (vmdk) added to your FOG server run the following command

lsblk

NAME   MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda      8:0    0 298.1G  0 disk 
├─sda1   8:1    0 294.3G  0 part /
├─sda2   8:2    0     1K  0 part 
└─sda5   8:5    0   3.8G  0 part [SWAP]
sdb      9:0    0 100.0G  0 disk 

As you can see above there are now 2 hard drives attached to this FOG server. There is /dev/sda that’s about 300G and /dev/sdb that is 100G in size.

We are going to take that 100G (new) vmdk file and put a partition and file system on it and then finally mount it on a temp location so we can copy the files in /images to it and off the root file system (which is currently 100% full).

This is more of a discussion than a help request.

We had an issue where we ran out of disk space on one of our test environment FOG servers. The problem started when we uploaded more host images than we had space on the FOG server. Searching the FOG server we saw that the host images are stored in /images and the snapins are stored in /opt/fog/snapins. Both of these folders exist on the root file system. Coming from old school unix we would typically create the /opt and /home not on the root file system because these directories typically grow the most. And on the older unix file systems if you fill the file system the OS crashes very badly.

(note for any of this thread to be understandable you need to read from the bottom of this thread to the top since the later posts appear first)

That’s a bit more strong arm than I think would be needed. Typically you “whould” have an enable/disable flag for the image and just turn it on and off via a checkbox. When you delete the image definition you loose the existence that image was ever there instead of just turning it off for a bit.

@Wayne-Workman said:

I would suggest we pool our knowledge to just create some base-level scripts that will sync two DBs based on the exact same rules that the FOGImageReplicator follows. I’ve outlined these rules before in other threads.

I’m not thinking anything drastic. Its more like how pfsense sends http calls to a remote node to sync its configuration data. While its a bit deeper discussion that we should do here. The idea would be for the FOGReplicator to move the files as they do today. When all of the files in the current image directory have been moved, then make a http call to a php page the remote node (it should already know everything it needs to know to do this [i.e. no new database fields]) which adds the image information to the remote database.

@Tom-Elliott

That is pretty much what I have with the bashrc settings. The issue is that the no_proxy points to localhost and the script calls the fog server IP address [from my memory].

BUT I could use bits of what you posted with something like this. (understand I already have the exports for the http_proxy… in my bash rc)

no_proxy="<ip_add_fog_server>" ./installfog.sh -y

That would allow the installer script to run unaltered and still tell wget to no proxy for the fog server IP address. I don’t know how big an issue this is or how many fog servers are behind proxy servers. I just wanted to get it documented for others that may find this post.

I can see for beginners creating a partitions for the image is a bit daunting depending on the distribution. I’m not sure if there is an easy solution for this. I’m sure I could document it for Centos 6 without much pain, but I see the majority of the deployments here are ubuntu.

As for the vmdk stuff that is pretty easy, to add a second vmdk and mount that for just the /images. As for expanding the vmdk, I do have to look it up every time I do it, but it is also pretty easy.

I’ve got the e6440 on my network and they deploy fine with FOG.

Not related to the e5440, make sure for dhcp option 66 you enter the IP ADDRESS of the fog server not the name. Some pxe boot loaders are pretty dumb and do not do dns resolution correctly.

It sounds like you entered the right value for dhcp option 67, that should be undionly.kpxe with no path or spaces on either side.

As for the system it sounds like you have that right too. Legacy mode, secure boot off and you need to enable the legacy rom.

@Wayne-Workman said:

@slovisa After freeing up some space, you’d repair mysql. let me see if I can find some commands for that… I’ve done this before at my site when a past bug caused apache logs to grow out of control…

Just a comment here: standard practice is to move the /home /opt (and a few other) directories to something other than the root partition/filesystem. Having the /images directory in the root partition will lead to the corruption the OP has now. While I’m not shaming here, it would be good practice to move the fog /images folder to something other than the root partition, like /opt/images. In addition recommend during the OS build instructions to put the /opt directory on its own hard drive partition. (on a virtual machine I would even recommend its own vmdk file. The logic here is this is where your deployment images are, if you run out of space in the partition, you can extend the vmdk file then extend the fs to give more room for your images quick and easy.)

OK on the system history. Sometimes you don’t know unless you ask first.

As for the commands it appears that the mysql service is not running and since I’m from rhel land (redhat) I don’t know ubuntu commands very well.

A quick google-fu excersion shows me this command.
sudo /etc/init.d/mysql start

That should start the mysql service.

I need to provide a way to bock an image from being deployed, or being scheduled to be deployed. If you are working in an environment when many techs its possible that an image may be put on hold for several reasons. We need to stop that image from being deployed to the targeted hosts. The request to deploy could come from the GUI interface or via a quick image via the PXE boot menu. In either case we need to block deployment until/if the image is released for use.

One of the areas of concern is about version control. We may need to leave a previous version of an image on the fog server until the new version is fully vetted. That previous version needs to be lock/blocked/disabled from being redeployed. Because the image name is saved in the host record its important to keep the blocked image from being used. It would also be handy to have a nice message to indicate why the image is not allowed to be deployed at this time.

I can tell you through testing I know this so far.

1. You can do this mostly with a storage node and the location plugin.
2. The storage nodes don’t have the bits required for tftp to work

Some caveats to what I just said.

1. The storage nodes are storage nodes only. You can add the tftp service description for xinetd and the tftp files. But the storage node is not a fully deployment node. There is no user interface, all of the techs must access the master node to deploy images to the remote locations.
2. The storage nodes do not have a local mysql database (as far as I can see). They connect back to the master deployment node to access its database. I see this as being an issue with latency when crossing a WAN link.

I have been thinking of ways to map this out to do what ( I ) need it to do, but it would be one off and fragile at best. The best solution is to have this done natively within the program and not use any external hacks.

Well the error is telling me that the mysql service is not running.

Tell us a bit more about this system:
Is this a new install? If not how long has it been in production.?
Have you made any changes to the server since it last worked?

On to the issue if you open an command window and issue the following command.

ps -aux | grep mysql

Does it show the mysqld service running?

If not then issue the following command (sorry its in rhel syntax)

service mysqld start

or change to the /etc/init.d folder and issue

./mysqld start

To start the service will be distribution specific but the idea is to start the mysql server and see if you get the same results.

Are these directories shared via nfs (using the command I provided)?

and to answer your question if you cd to the /images folder then issue a

ls -la *   

command that will show you file ownership and access rights.

While I don’t have an answer for you, I do have a feature request out there for the exact same thing. https://forums.fogproject.org/topic/6014/create-the-concept-of-a-foreignmasterstorage-deployment-node
Right now there the only way to do this is manually or scripting via sql dump and sql import of the images and snapins tables between the deployment nodes.

You can do something similar but with storage nodes (not a master deployment node), and you will need to install the /tftpboot files on the storage node. You will need to use the location plugin and storage groups.