SOLVED Uploading to wrong storage group

  • I’m using FOG build 7645. I have two storage nodes configured on the same Ubuntu machine, called Koei and Olifant. Each belongs to its own storage group, so it contains different data.

    2_1467881857259_Screenshot from 2016-07-07 10:44:13.png

    1_1467881857259_Screenshot from 2016-07-07 10:47:19.png

    Then I create a brand new image (not yet captured – protected was unticked):

    3_1467881857259_Screenshot from 2016-07-07 10:50:29.png

    And associate it with the client machine that I use for testing:

    0_1467881857259_Screenshot from 2016-07-07 10:52:09.png

    (I have the location plugin installed, but not configured for this host. Not sure if that makes a difference).

    Now when I capture the image from the host, I expect the upload to appear in /mnt/olifant/images/dev/MAC_ADDRESS and then move to /mnt/olifant/images/IMAGE_NAME, but it actually appears in /images/dev/MAC_ADDRESS, and then the system throws this error:

    Error returned: Type: 2. failed to open stream: No such file or directory.

    A quickfix is to go ssh into the server while the capture process is running and make a symlink from /mnt/olifant/images/dev/MAC_ADDRESS to /images/dev/MAC_ADDRESS. FOG will then move the symlink to /mnt/olifant/images and the task can complete. Then I fixed it by removing the symlink and moving the folder manually.

    What could be causing this?

  • I’m solving for now as I’m fairly sure this is fixed. Of course feel free to test and update this thread as any other information comes forward. I’ve tried to test, but too many other things going on at the moment.

  • Sorry, since I did away with the second storage group, it’s hard to test this. I’ll create a second storage group and test this when I have more time. For now, I’m pushing a deadline to get the lab deployed before Monday.

  • Is there any way we can get more information?

    I know, for a fact, FOG can run two nodes (hell two groups each with the a node that points at the same place) without issue. It’s how I test different things. As for “bandwidth” usage, I’d imagine the items would look, more or less, the same minus the slight delay you might have when reading the “individual” nodes’ charts.

  • @dolf I’ve successfully ran two storage nodes on one interface before just fine. In fact both storage nodes were using the exact same directory, as well. I’m sure this isn’t the problem. There must be something else that was going on.

  • /images can hold everything, for now. I removed the second storage node and storage group. I’ll just move the older images to the extra drive and symlink them to /images/…

    So according to @Quazz , this is not supported. Fine by me. It seemed like a logical use case to me, so maybe we should say somewhere in the Wiki that this is not supported?

  • I’m curious if that 1.5TB drive can hold everything… if so, do away with the other node.

  • Moderator

    @dolf I think you’re on to something there. The problem might be caused by the fact that the nodes are using the same interface.

    To be honest, I’ve not seen this specific use case before, so I’m just guessing.

  • If I add the image to the koei storage group as well, will it be copied (replicated) from /mnt/olifant/images/ to /images/ ? I could try that, remove it from the olifant group, and then add it again?

  • # showmount -e
    Export list for
    /mnt/olifant/images/dev *
    /mnt/olifant/images     *
    /images/dev             *
    /images                 *

    Thanks for the workaround ideas. Although I think we still have to solve the bug… If I apply a workaround, I won’t be able to help you find the bug.

    Are there any success stories of having two nodes on one machine? Is it even supported?

  • Moderator

    @Wayne-Workman TBH I did not read the entire thread so if this has already been asked my apologies,

    What is the results of showmount -e

    OP: If I understand this correctly you only have one FOG server but you have 2 hard drives in this physical server with each not having enough space for all of your images, do I understand this right? If so I think I would take a little different and less complex approach. There is two routes I can think of

    1. Setup LVM and just add those disks to an LVM volume group and then create a LVM logical volume. Let LVM decide where to span the files on those two physical disks. Temp mount that LVM volume over /mnt and then move the content of /images to that LVM volume. Unmount /mnt and remount that new LVM volume over /images and be done with it. (make sure you update fstab). That way you are running a standard FOG configuration without having to deviate from the norm.
    2. On your existing FOG server create 2 directories under /images like olifant and koei. Mount those two drives over those directories. Then when you create your image definitions make sure you pick one of the two drives. So if you create an image name called WIN7X64 make sure the destination path isn’t /images/WIN7X64 but /images/koei/WIN7X64 instead. Its not as clean as doing it at the OS level as with LVM but it should work.

  • @dolf Ok. I’m about out of ideas then.

    I’m going to need to ask for the help of @Tom-Elliott, @Moderators and @Testers for ideas.

  • I have only one physical Ubuntu machine running fog, with two hard drives. There is a storage node on each, belonging to different storage groups, so that I don’t have to have all of the images on one disk (not enough space). /mnt/olifant is simply the mount point for a physical 1.5TB internal SATA drive with one ext4 partition.

  • @dolf Why does the olifant node have the directory /mnt/... ? In it’s description, you say it’s in lab2-server, is that the fog server or another server? Have you mounted a remote directory to the fog server?

  • No change when disabling koei. Also disabled olifant, tried again, enabled one and tried again, then both and tried again. No change.

  • Also - temporarily disable the Koei (root) storage node. It’s a check box in it’s settings.

    Then see what happens.

  • A few posts ago, I did post the output of ls - lha which shows the existence of .hidden files. 🙂 They are present with the right permissions, IMO, but I pasted the output in case I overlook something.

  • @dolf Does tree list hidden files? The command I posted does. You should have these files in these directories:

  • Changed fsids as suggested, and rebooted. No change. To be honest, I have not worked with NFS before. Thanks for the help thus far.

  • @dolf said in Uploading to wrong storage group:

    /images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
    /images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)
    /mnt/olifant/images *(ro,sync,no_wdelay,no_subtree_check,insecure_locks,no_root_squash,insecure,fsid=0)
    /mnt/olifant/images/dev *(rw,async,no_wdelay,no_subtree_check,no_root_squash,insecure,fsid=1)

    That’s the problem I think (or at least one of them).

    fsid cannot have duplicates.

    You have: 0,1,0,1
    They should be: 0,1,2,3