FOG Client stops reporting & working

[Joe Schmitt]

@Wayne-Workman what is the content of settings.json?

[Wayne Workman]

@Joe-Schmitt It’s full of just a bunch of white spaces.

[Wayne Workman]

I do have a snapshot of our FOG Server that is from RC-10 that I am going to test briefly, I’ll also make a snapshot of the current Server state, and then move to RC-15.

[Wayne Workman]

I rolled back to RC-10 via snapshot, reset encryption data for every host, and then rebooted my computer. The problem persisted. Same exact error in the user log, the fog.log still isn’t being written to.

At this point I doubt it’s a server issue. @Joe-Schmitt

I also believe all computers in my building are affected…
I’ll send out a snapin to do a ‘roll-call’ of sorts and see what happens.

I’m going to go back to my snapshot of RC-15, reset encryption data on all hosts again, re-install the FOG Client on my computer, and move forward.

Here is C:\Program Files (x86)\FOG\fog.log from my computer before I uninstall.

0_1477314930885_fog.log

[Joe Schmitt]

@Wayne-Workman the problem is what I mentioned earlier. Settings.json has been cleared by something.

[Wayne Workman]

@Joe-Schmitt But by what? I didn’t do it clearly. What could possibly do it?

[Wayne Workman]

An update on this, at least 140 out of 450 computers in my building are not affected… The rest, I don’t know. Maybe they were off, maybe they don’t have the fog client installed, or maybe they have the issue.

[Joe Schmitt]

https://github.com/FOGProject/fog-client/issues/76

[Tom Elliott]

Guessing this is still in a “we have no idea yet”, I’m just trying to keep this near the top of the list so it doesn’t get lost.

[Wayne Workman]

Still no idea. I’ve been running the debugger client that Joe gave me. It’s not yet experienced the issue, and therefore has not created the information file yet.

[Tom Elliott]

I’m very strongly thinking this related to a Windows update. Particularly one in regards to the .NET framework.

[Wayne Workman]

@Tom-Elliott Between our summer deployment and now, we’ve ran zero windows updates. We can’t risk downtime, and windows updates are not trustworthy enough, and we have too large a gap of downtime in the summer to not utilize for imaging an updated image with.

[Tom Elliott]

@Wayne-Workman I’m only giving the information as I’m seeing it.

This appears to have started in late July, early August, correct? Which would’ve been around the time (potentially) an image was being updated? I don’t know.

I still am leaning towards some windows update, and particularly a Windows Update in regards to the .NET Framework.

It’s the only guesses I can have at this point and with the “debug client” we should’ve seen something by now if it was server related. I suppose there could still be an issue in the client, but considering the client running has probably usurped the number of cycles before the issue became present, I would imagine this is not the issue either.

[jhuebner]

I don’t know if this is related:

I had an issue where the content of the settings.json got corrupted on half of the computers and I had to reinstall the agent manually. I don’t know what killed the contents of the file. That happened with the stable version of fog (1.2.0). After reinstalling the agent I upgraded to trunk and am now waiting to see what happens next.

[Tom Elliott]

@jhuebner The new client never worked with the version 1.2.0 of FOG. The new client was ONLY introduced during the trunk builds. It’s no wonder why they got corrupted, they had nothing to work from if you really did have the new client and FOG 1.2.0.

[jhuebner]

Thanks for your quick reply.

I was using the Agent 0.11.0 with 1.2.0, I think that’s the one that came with 1.2.0. I am of the opinion that, whatever version, nothing should kill the settings file so that the agent/zazzles does not work anymore. I am in the good position that our production environment is more of a lab-style character (university campus) - so the world does not end when the agent is not working. I just have to do more manual work which I of course try to avoid.

Besides that: I really like the project and appreciate the work you put into this!

[Tom Elliott]

@jhuebner 0.11.0 was NOT by any means the version that shipped with FOG 1.2.0.

The reasoning a file might go corrupt in the new client particularly with 0.11 on 1.2.0 (and potentially others) is because the client get’s configuration information from the Server. This configuration information has the potential to change each cycle, but to ensure the client operates during the time when a server may be unreachable we update information internally. If this data is corrupted the client usually cleans itself up to ensure something doesn’t go wrong, but if it can’t do that and the file is “open” there is a possible layer for the file to become corrupted.

We haven’t figured out WHAT is causing this and have been working to try to replicate the problem. To me, it seems like a windows update, but this is only my opinion.

[jhuebner]

@Tom-Elliott Thanks for the explanation. I am now in the process of upgrading all four fog instances I manage to trunk. This way I might be helpful to advance this further. Keep it rolling!

[Wayne Workman]

Ever since deploying the debug 0.11.5 client that Joe gave me, the issue has not reappeared. It’s been about 20 or so days, I think that’s plenty of time. It could be possible that it is a bug in the client, but at this point and being unable to reproduce, I think it’s a non issue.

For all those that experience the settings.json file going blank, just uninstall and reinstall the fog client. You might do this manually or with a very carefully crafted startup script via GPO, that checks for the presence of a dummy file to indicate if it should uninstall/reinstall or not.

[Hanz]

@Wayne-Workman I also had this occur, oddly enough after running a snapin that forces the client to update via WSUS. I recently added .NET 4.6.1 to WSUS.

yup. here’s event viewer from that client
The performance counter name string value in the registry is not formatted correctly. The malformed string is 뢭傞碻�鬋쏄军⍮휒㾲籍봯禵ᦂ瞣㛂⯦⿟闚椂↛抋篾귩恫嗋᜸⸑蓣톽�쾐൐⨨莐潋㌍胡빌폍伤밧縒�鬋웘趲ප᥀巽뼉랥躎衙┱禌㾬됯㏨궒外㈦慙ﭯ潘ﯦ랢ꤎ懨䄩賩ꗼ�唹훐⩦�ẏ⾟疽褎⌼凴䞛枢ﴙᚅ옝햌Ш↊ᱰ貼Ӗ醑᥆㔥㎦Ὢ苻☍▩臄ꮨ僭烻�﫿釶⧐첨륗ꃕ⽷㢭�ᬓ욄䤘똎ꖷ뜕뷲훴洬勳꓅뛰“ⵏ敺铨苌哅쇇寒楣ᗳ땦샄쿆ﳸᚅᄽ鱔�⬈잃䯧浯厃ݖ�麳䋺꺋릎揍쟏ކꞯ봮蒵慻ኋ풭聕慻ꛓ젭ฦ崘ᆶ໰㾾ّ腳㗢⧹籛䖩蹯ඝ䎈﨨旑䮋굯䗙阪ꍢ佦빏桟᧑禛捲굃쥏�쑚胬�좮圠ߺ쭷䳭瓕㐕꺯墽䣣씳鳎綸ø쿹淗팅嫵䛕髠텊떪㠷醛ୟꯊ핚隂➼㋳쏔晶럤ﷲ顝༷㛲ㇿ灳ᰲ䷯矂퉷₾�䎡똪胉ᜬ馿ꗶ㏏켗ꔙ柄⽱៟ꉬ蟊贘톙鷤留閸ꗯ䀡膒릹㣩﫛㮃廹죘쓈⛛曬夐뙶빖摷帛룒䚄�됯㍍땫ᢦ戼㑦灰ﻻ�㻩䌲㔔뎤틕慻쪃鈐�㝚ิ뿮⯷浨リ齃鹸⿧ྵ᩽抃똭�ﹹ辗켟䛶渊㝛躴㙎匔ꋅ့畹⅟䅆汐攳섡증뿋蕖멥ꉘᏎ휻︇栿媼楔ꜣ転亥ി医�⿗뽟﷼꼆鹣㣎ᶉꌾ轭끿뙲師߿꼔檑ꢖꪶ羚緓晴埻욽得⡇넃¶鲱鶤᯾ﱘ뤗鈋裂煉⒑貵뎸כּ뉗욻푍붆杍촛깺㛯揝뮳ꄠꍢ饋㛎䳅輧뭩롫㳄䪄☙c豤ᖳ澷督㉁⶝る鞀멒곮綦묭驼浡⩐䘣᧣㿷묶䠗쩆ꎄл㻶ﻟ뉃ڻ耼萐葽쉅탺⻬顈譍钕Ć펰�员䰙⊰벹꨷槚糈줣搗끘贵�㇒ꮃ壕║㫇럐瓥摮˷猉ત榢懺뷵쏝銻㗝舂㊤๶咧썛嶳붯�अ劵庛╘杼ꩥ�㒽죉褓瀍䏨脖죜׮볚⒉案랴´뉗槻ʏ䑈㮜㍯쩏齣ௌ᪒嫖ᵤ솭挖錝쯝໰嚡燀⶚ꪌ뙼ྥ눷腻斤틠ᩆꊥ�껙⮇巙紼糭현䨙ᩰ涏஻⠒츞掕幪ᒻ찄晅⿨Ῐ�₼ⷩ�봔揗䑦౫덇쟞쨘춣ힼ⛏룛�䤅◩ᑐ宺ꦝ긵睤�薨䭆딐ﰶ췘ք唀劯텆辑䳙㉘骓篙帯㦛菇䑧ላ鳂ʢ͉糨賐൨僔蔱㇢䴝쎄㝣ᘓ찤䯕漣枾拂泇갦᧱な驒懑磷嫬苕➄뤦浧뷚ẋ嚳蚵ꡥ塅₌鈝㾒⁞껡ᤦ䝅握贚夑䁱ᣏꖶ込短␗咎꫁䢖랣䱾뮿愆衰↛赣刬㬞幦沐躻㔨쌷챴禘붍곈樝⠍툚띏瘶␗棐�ⶑ淯↮䤗邷骪톕힪뛇ם飉簯찍끑蕋�᫮遱ꉆᱭⓂþ们虈뛩䧏씂�襾焢ꆪꔔ썍괽寊氥瞭京�ਖ਼쟴྇茶�翯竽岽䡽ꢎ樳䓰砠શ뗳베宪㚷鹲懹戲�꥙囹鞾㭚ଫꈍ쵓╏系跿塀㋐봏뎲揝�붨䅃�㙥ῼ. The first DWORD in the Data section contains the index value to the malformed string while the second and third DWORDs in the Data section contain the last valid index values.

I got a bunch of “chinese” in my client log when this occurred

also had another warning saying something about registry being open and in use as well

Hope it helps guys.