Need ability to unpublish captured image

[george1421]

Issue: When working through a normal (initial) image development process we may capture several reference images before we settle on a golden image. If we have a multisite setup with a storage node in each location, as we capture these intermediate images these images are automatically replicated to the remote storage nodes consuming bandwidth and space on the remote storage nodes. We need a way to capture an image but have it remain on the master node (unpublished) until we determine that the image is golden and then publish the image to the storage nodes.

Some of these images could be several GB in size and may take up to a day to completely arrive at the remote storage node. Currently there isn’t any way to indicate the replication has finished via the GUI. If we try to deploy an image before the sync has fully completed the deployment will of course fail. This could be managed externally via a manual workflow process, but ideally the management interface should block deployment of the image until it completes to the defined location.

[Wayne Workman]

Just create a 2nd storage group on your master server. Don’t set it to replicate to anything. It can reside in the exact same /images directory (as long as image names don’t conflict).

When you’re ready to push the image, just hack the web interface Leave the image data sit in the /images directory, delete the definition, and just re-create it in your regular storage group. It should start replicating.

[Wayne Workman]

I’m pushing for a release… I don’t want any more features lol.

[george1421]

The other folder route might work, but I don’t think I can capture an image to a different folder (??). As for the replication (unless something has changed recently) if a file exists in the /images folder the replicator will copy it dutifully to all nodes in the storage group. If you have a multi site setup the link at HQ is typically pretty utilized already. Adding additional useless traffic to that link is something we want to avoid.

We can proceduralize this today if we setup a development FOG master node where the images are developed. Once we have the golden image, we could rsync it to the production master node and then manually create the image information on the production master node. While it should work it does require host OS access and someone with the authority and skill to do this. I’m trying to avoid diving into the OS for a solution to this if at all possible.

[Wayne Workman]

@george1421 You missed what I was saying.

The replicator doesn’t care what is in the /images directory. It checks with the DB and finds out from the DB what to copy. If the DB says “These images are in this storage group, these are the nodes in this group” then the replicator will copy those images across those nodes.

If you create a 2nd storage group and 2nd storage node in that group (on the same master server), then those images will not replicate - because the DB says so.

[Wayne Workman]

You can have more than one storage node configured on a machine. And as long as image names do not conflict - the storage nodes can even use the same directory.

[george1421]

@Wayne-Workman I’ll have to confirm that. The last time I checked all files in the /images folder was replicated to the storage nodes even if they were not specifically called out in an image or the database. The only exception was if the images were in */dev and */ca.

I can test this today since I just rebuilt the POC system this AM with the latest build. But again, I want to avoid having to dive into the OS to move things around. While I have the necessary skills to do this, not everyone will or should have the access to do this.

[Wayne Workman]

@george1421 said:

But again, I want to avoid having to dive into the OS to move things around.

I don’t think you have to if you use the same images directory that already exists on your master server.

[Tom Elliott]

The current replicators in svn check the individual files/folders, not just simply replicate the folder as is.

That said, I totally can see the need for enabling/disabling items. We already use a method for it in handling storage nodes, why can’t we use that same ideology for images and/or snapins? I suppose we could use it in other areas too, but still.

So without any further ado, I am pleased to state that this is now added. I even made the selectors know about this. Meaning that the drop down lists will display only enabled items if the isEnabled flag is within the class. That also means it could be relatively easy to port to other classes/plugins too.

In either case here you go and I hope it does what you want/need.

[Wayne Workman]

@Tom-Elliott Awesome.

[Tom Elliott]

Replicator flag has now been added as well. This will keep the image or snapin (or whatever) from replicating to other nodes/groups. Paired with the “enabled” element, it means the image can be fully unpublished, in theory. Not meaning you can’t add/edit the items but rather it will do, I hope, what was requested.

[george1421]

@Tom-Elliott said:

So without any further ado, I am pleased to state that this is now added. I even made the selectors know about this. Meaning that the drop down lists will display only enabled items if the isEnabled flag is within the class. That also means it could be relatively easy to port to other classes/plugins too.

In either case here you go and I hope it does what you want/need.

^^ I wish I could up vote this 100 times. Well done!! I’ll check this out to see how it works a bit later. But this will help us by allowing us to keep our FOG environment simple (i.e. without isolated dev FOG server, complex scripts to hide or move images around).

[george1421]

@Tom-Elliott said:

Replicator flag has now been added as well. This will keep the image or snapin (or whatever) from replicating to other nodes/groups. Paired with the “enabled” element, it means the image can be fully unpublished, in theory. Not meaning you can’t add/edit the items but rather it will do, I hope, what was requested.

As long as the replicators honor the unpublished flag then we should be good. This should keep the image on the master FOG server until we have it approved for deployment.

[Wayne Workman]

@george1421 I think the replicator will act as usual - but will check if the replicator flag is set. If it’s not set, no replication.

If the Enabled flag is set determines if the image is selectable (and therefore assignable) to a host.

That’s how I interpreted it anyways.

So for a image dev in your company, I suppose they would initially just disable the replicator flag. Once the image is assigned to their one/two test machines, they would then clear the enable flag also so that nobody else can select the image.

@Tom-Elliott Does the above sound accurate?

I think it wouldn’t be good if the Enable flag prevented image capture/deployment (if the image was previously assigned to a host).

[george1421]

@Tom-Elliott I got sucked into fixing a downed business system tonight, so I haven’t had a chance to test this new build out. But while I was working on the outage I started thinking about this new setting. When you upload an image in the new environment, does it come in published or unpublished (may not be the correct term)? If it comes in published, then depending on the replicator cycle I have no more than 10 minutes (probably less depending on where the replicator is in its cycle) to unpublish it before the replicator. Once the replicator starts sending the file then unpublishing it after the fact is a bit moot. I just want to confirm that when an image is captured it enters the system in as unpublished. It will be up to the FOG admins to publish the image for replication. I can see that we would still want local deployment of the image, we would just want to kill replication of the images and snapins until they are published. (hopefully I didn’t make that too confusing)

[george1421]

@Wayne-Workman I can see it working as you have defined it too. For my specific case, I might want to deploy the image at HQ but not send it to the remote sites until it has been qualified and approved. My specific goal would be to conserve WAN bandwidth. But I also don’t want to make too many exceptions because then programming becomes crazy for every one off condition.

[Wayne Workman]

@george1421 I agree. Default should be “Replication - disabled/unchecked”

FOG Users - when they see that checkbox, they will immediately know what it does. So no worries of confusion.

[Tom Elliott]

@george1421 you have to create the definition first, right? The checkbox for replicate is there and on by default. However you can tell it not to be replicate by Unchecking the box.

[george1421]

@Tom-Elliott Ok that’s right I have to create the definition first. So I’ll have the chance to turn it off. The day has been too long and the brain cells have already gone to bed. This sounds exactly spot on. I’ll give it a go in the AM. Thanks.

[Wayne Workman]

Solved this thread.