• Hello,

    I wanted to post this in general as I am unsure of the correct area to post issues with Hostname Changer.

    Currently, we just started experiencing issues with Hostname Changer over the past couple of weeks. There has been no maintenance done on the Fog Server, No image changes, and no changes on the back end previous to the issues we started to experience.

    We are running FOG Version 1.5.9 with Debian 10. We have also tried a couple of the trunk versions as well for troubleshooting this issue, including 1.5.9.60. We also tested with the latest Kernel build. None of these seem to have resolved the issues we are experiencing, nor did they have any change in the issue we are experiencing.

    The issue that we have started to see somewhat broad, but I will explain as best I can.

    1. On new images, if the PC doesn’t exist Hostname Changer will no longer create an AD Object for the PC.

    2. Sometimes imaging a computer using the name of a pre-existing AD object will cause additional trust relationship issues, but FOG will act like it’s on the domain.

      Ex: We had a computer named TEST that we are replacing with a new device. The device images and successfully renames to TEST. Initially we are able to access the computer with a domain account, but after a few moments it will receive a trust relationship error. FOG.log will show the device is joined to the domain with code 2691. Using Control Panel, we can rename the computer to TEST-01 and put it into a workgroup. FOG will then rename the device back to TEST and add it to the domain, but we’ll receive the same error. We can rename the computer in FOG to TEST-01 and it’ll add it in AD as a new object. We can then delete the old TEST object and rename it again using FOG. That will typically allow the computer to work correctly under the name TEST.

    3. Renaming a computer does not create a new AD object. However, it is still able to access the domain (no sign issues or trust relationship errors). Adding a new computer with the previous name will cause trust relationship issues on the new computer.

    4. Sometimes renaming a PC WILL create a new AD object, which is different from number 3 above.

    These are just some of the scenarios that we are experiencing, I can list them all but there are a vast number of scenarios that occur.

    One thing I did find on an old forum post, is the following setting:

    Fog Configuration > General Settings > CHANGE HOSTNAME EARLY

    Turning this off allowed new AD objects to be created successfully on new images with no preexisting AD entry. But a plethora of other issues still exist with Hostname Changer that didn’t exist before.

    The strange thing is, this setting has been on and in place for years without issue.

    We have also verified the domain account being utilized is active and has all permissions and attributes necessary to successfully add PCs to the domain. We tested this manually with a few PCs with 100% success.

    Any thoughts or suggestions on where to look or troubleshoot would be appreciated!

    The Fog Log shows the PC as on the domain in most of these cases with no errors.

    Thanks!

    -Lambo


  • @sebastian-roth

    Hi Sebastian,

    No worries at all, i just figured i would ask, as there are multiple domain controllers in our organization and i was hoping there would be a way to point fog to utilize the local DC. No worries though at all!

    After testing with the Server 2019 DC in production, it seems that some of the issues presented have been resolved in some regard.

    Here is what we found:

    If a computer AD account already exists, it seems we can reimage with no issues now. we tested around 15 images and they all worked successfully. We can also reimage a computer where the AD account has been ‘reset’ instead of deleted. These tests were successful as well.

    However, if we delete the computer AD account or image a new PC and have the option to add the computer to the domain, the issue we were previously experiencing exposes itself. The computer will allow AD accounts to log in, but trying to deploy software shows a trust relationship issue as well as having the computer nowhere to be found in Active Directory.

    What would you like to see done here? I am about out of ideas on our end unfortunately, other than standing up a test domain and seeing if the issue persists there and if so, trying to narrow down where the problem is. I think we should be able to set this up and we can continue deeper testing there.

    Thanks!

    Lambo

  • Senior Developer

    @lambo said in Hostname Changer AD Issues:

    Do you also know if it is possible to ‘lock’ the domain controller to a specific domain controller in Fog? This would be helpful so we could lock Fog in to utilizing our local domain controller instead of possibly utilizing another in the network.

    Do you have serveral AD controllers and want to force the client to use a specific one? That’s not something I can answer. From what I know about Windows AD I would guess the answer is No. But that’s definitely beyond my skills.

    My thinking was you would setup a single AD testing server using a complete new domain for testing, not part of the existing domain in any way. Not sure if that’s possibly in your environment.


  • @sebastian-roth

    Hi Sebastian,

    I hope all is well with you!

    Sorry for the delay.

    Ok so we were able to produce the AD issue with the AD team.

    Unfortunately, we didn’t see much happening in terms of event logs.

    There were three separate failures that occurred, but not necessarily related to joining the domain in my experience. I could certainly be mistake though.

    1. Audit Failure - Kerberos Authentication Service - 4786

    2. Audit Failure - Directory Service Access - 4662

    3. Audit Failure - Credential Validation - 4776

    None of these failures utilized our image account for Fog / Hostname Changer, so I don’t see them as being a smoking gun unfortunately.

    The AD Team was planning a migration for our AD server to Windows Server 2019, so they will be completing this ahead of schedule for us. This migration will take place this week, so we are hopeful that it could assist with resolving the issues that we are seeing.

    If that does not work, the AD team will allow us to utilize a test AD server for troubleshooting this issue. We should then be able to fix up and test any settings we think are important.

    Do you also know if it is possible to ‘lock’ the domain controller to a specific domain controller in Fog? This would be helpful so we could lock Fog in to utilizing our local domain controller instead of possibly utilizing another in the network.

    Again, sorry for the delay, and thank you so much. I really appreciate your assistance!

    Thanks!

    Lambo

  • Senior Developer

    @lambo Sounds good. Keeping my fingers crossed that you will find out more soon.

    Don’t think we have an official document on the imaging process. Just a few things that I have in mind:

    • Disable secure boot
    • Set disk controller to AHCI mode instead of RAID mode
    • Disable bitlocker (manage-bde -off c:)
    • Un-join from domain
    • Disable fast boot in Windows 10 and do a clean shutdown

    @george1421 @Tom-Elliott Some essential point I am missing?


  • @sebastian-roth

    I just wanted to update you Sebastian.

    This Wednesday / Thursday I will be working with the AD team to view logs during the imaging processes. They are most likely going to request that I have access to set up a temporary AD Server / get access to their Test AD servers.

    I am hoping to have an update for you after testing this week.

    Please let me know if you need anything else.

    Do you also have a link to the Fog recommended image process so i can ensure it is not an issue with our image as well? Our Windows version is Win 10 Enterprise 20H2.

    Thanks!

    Lambo


  • @sebastian-roth

    That makes sense! I was thinking along the lines of a corruption or similar with this specific issue.

    I can check with our higher ups to see if we could create a local AD server for testing purposes to see if we would be able to work with this server on narrowing down the issues that are presenting themselves. Let me get back to you on this point and let you know if this is possible.

    Thanks!

    Lambo

  • Senior Developer

    @lambo While I do understand the urge to rule out certain things to find out what is causing this I don’t think that setting up a new FOG server will be of any help. This is an issue where the FOG server itself is not involved really. Sure the fog-client polls the AD information from the FOG server DB but after that it’s just the fog-client calling Windows API and talking to your Windows AD server(s). If we would see error codes that tell us credentials are wrong, then we could blame or look into the FOG server but from the evidence we have right now I can’t see the FOG server causing this.

    Would you be able/allowed to setup your own AD server for testing?


  • @sebastian-roth

    Unfortunately, removing the snapin from the process and upgrading the Fog Client had no change in the outcome. We ran images all day yesterday with the same outcome.

    I think I may try deploying a new Fog server and test if it has the same outcome. I don’t have any other ideas on our current server. Do you have any other suggestions?

    Thanks!

    Lambo


  • @sebastian-roth

    Certainly!

    My team an I are working on running without the snapin, as well as updating the Fog client just in case!

    I will be in touch.

    Have a great day!

    Lambo

  • Senior Developer

    @lambo Will be interesting to see if imaging without the snapin does make a difference!

    Other than that I don’t have other ideas right now. Please give that a try and we’ll see from there.


  • @sebastian-roth

    Ok, that is understandable. Is there anything that can verify the installation?

    I will work on the Fog client and seeing if upgrading changes anything, as well as imaging without the snapin enabled.

    Do you have any other suggestions?

    Thanks!

    Lambo

  • Senior Developer

    @lambo said in Hostname Changer AD Issues:

    Do you suggest maybe downgrading the server version to 1.5.8 as well just to test and ensure we didn’t encounter some weird issue when upgrading, perhaps something got corrupted on our end or similar?

    Don’t think a downgrade will be of help in this case. Also be aware that going back is not something we officially support within the FOG software. It’s hard enough to make updating work for as many people as possible without too much trouble. So it’s a manual process of taking an old DB backup and so on. As well there is no auto-downgrade to an older version of the fog-client.


  • @sebastian-roth

    Sorry for the delay, we have a major project that I am currently involved with in my organization.

    I agree! But there must be a cause whether its on our end or something that may have gotten corrupted on our server. This whole issue is quite strange and is very perplexing. I can’t seem to find any rhyme or reason.

    I will have a look at these error codes and see if I can’t find anything that would be causing this issue.

    The only thing that we have is a snapin for our software deployment software that is set to run after the PC is imaged and on the domain. I can certainly try to run some images without this snapin enabled and see if that makes a difference?

    Other than that, i am not seeing anything that could cause issues like we are seeing. I’m not seeing too much in the event logs and I don’t have access to the AD server, but i had the team look at it and they said they haven’t seen anything unusual.

    Let me take a look at the Fog client as well, even though you might not think it could be the issue, we can try to ensure that it isn’t causing some funky issue.

    Do you suggest maybe downgrading the server version to 1.5.8 as well just to test and ensure we didn’t encounter some weird issue when upgrading, perhaps something got corrupted on our end or similar?

    Thanks, sorry again for the delay!

    Lambo

  • Senior Developer

    @lambo Well done. Looking through your notes and logs I find it very awkward that we have different results for tests that should end up the same! The fog-client simply calls official MS API to do rename, AD join/leave and so on. While it’s still possible that it’s something we do wrong in the code I find it very strange that it would work one time and fail the next few tries.

    What I found in the fog-client logs is a few unknown return codes:

    FOG Test 1.log: 2/25/2021 9:10:45 AM HostnameChanger Unknown Return Code:  1791
    FOG Test 5.log: 2/26/2021 10:19:55 AM HostnameChanger Unknown Return Code:  1791
    FOG Test 7.log: 2/26/2021 1:34:39 PM HostnameChanger Unknown Return Code:  1791
    FOG Test 8.log: 2/26/2021 2:26:16 PM HostnameChanger Unknown Return Code:  1332
    FOG Test 8.log: 2/26/2021 2:28:48 PM HostnameChanger Unknown Return Code:  1332
    FOG Test 9.log: 2/26/2021 3:49:57 PM HostnameChanger Unknown Return Code:  1791
    
    • 1791: A remote procedure call is already in progress for this thread. (reference)
    • 1332: No mapping between account names and security IDs was done. (reference)

    Possibly you have some other software interfering with the fog-client? Do you have other tools installed that would run straight after the client comes up after imaging?

    Did you find anything obvious in the event logs on the client or the AD server when it fails to create the machine account?

    As a side note: You still have the 0.11.19 fog-client installed. Unfortunately the CA certificate shipped with that binary ran out last year and so the auto updater can’t pull up to the 0.12.0 version coming with FOG 1.5.9. It downloads the SmartInstaller but cannot verify the signature and therefore cannot install it. So at some point you’ll need to update your base image with the newer fog-client.

    Though I don’t think 0.12.0 will make much of a change in this scenario you are facing here. There have not been updates to the HostnameChanger module since 0.11.19.


  • @sebastian-roth

    Here is a link you can try, Let me know if you are able to access the drive.

    https://drive.google.com/drive/folders/1aOwYRK71pVZL68uIrzPzT1ylOmquVzeP?usp=sharing

    Fog logs are listed, Testing Notes.txt are the overall notes showing what tasks have been taken for each fog log.

    Let me know if you need anything else!

    Thanks!

    Lambo


  • This post is deleted!
  • Senior Developer

    @lambo I am not exactly sure what size and file types are allowed in the forums. You can give it a try or upload to a file share/cloud of your choice and post a link here.


  • @sebastian-roth

    Ok i Have the data available!

    Would you like me to put the summary and all 10 Fog logs right here or is there a different way you prefer I upload these files?

    Thanks!

    Lambo


  • @Sebastian-Roth

    Hi Sebastian,

    Just wanted to update you that I am combing through the data that I have on the imaging sequences that we spoke about last.

    I will be able to upload the process and data to you shortly!

    Thanks!

    Lambo

239
Online

8.1k
Users

15.0k
Topics

141.4k
Posts