Imaging stops after client boots up
-
Rouge dhcp. Well, frist thought was this, as not on fog, but we had some same situation, so we go and rested it. Not this kind of problem here (as far as tests went). And, for the fun of it, now we make tons of tests with different hardwares to see any pattern if we may see, and latest test machines produced this: if ever occure, nothing can change it (well, 10-12 reboot, no change on same box). BUT, cos always there is a but… if it stucks after.xz file load, i get the root@146 like prompt i can ANYTIME go on with imaging commencing ./linuxrc. Anytime!. So, if network issue anyhow, it is only in the boot process of the ipxe downloaded OS. It can be a clue
Now we collect motherboard info for listing working and not working machines and still there are machines we find with working and not working status.
I even replaced the fog server machine with other one (not the best solution, but I relocated the disk to other hw). Same issues: sometimes works, sometimes not. Pretty frustrating.
I will go through the suggestions you made above as soon as i am able, but as always, we have to do many things separately. And, as we were after testing phase, this system got into production, so, not the best to fall back to previous versions or go backup; but I think I will not be able to avoid that…
-
What version of FOG are you running? What OS distribution?
Also, before you do any more of these labor-intense tests, can you try to re-run your installer?
-
I run the troubleshoot.sh now, made a little change on host ip to be less… well, open
. Here is the log it generated. -
[quote=“Foglalt, post: 46977, member: 26236”]I run the troubleshoot.sh now, made a little change on host ip to be less… well, open
. Here is the log it generated.[url]http://pastebin.com/1PKpRzmN[/url][/quote]
The first thing I noticed is that a ton of your services aren’t even running/reporting (unless you deleted this from the output??):
[CODE]----------------------RPCBind status below
----------------------NFS status below
----------------------Firewall status below
----------------------FOGMulticastManager status below
----------------------FOGSnapinReplicator status below
----------------------FOGImageReplicator status below
----------------------FOGScheduler status below
----------------------Installation log below
[/CODE]Can you check your firewall?
Can you make sure NFS is running?
Can you make sure RPCBind is running?Also, the FOG services should be running, too.
and, your ownership for things in /images is not uniform, however it shouldn’t matter considering everything has 777 permissions.
-
Ask anything! All is welcome and maybe closer to solution. Btw, you said things strange to me (and I was puzzled to see them before posting log to you, but I thought i may be underinformed what is normal and what is not.
- Firewall iptables is totally empty (iptables -L, all default output)
- NSF can be reached (as I said linuxrc can read from it as manually started)
- rpcbind ha processes (well, I am not a pro of all, so, I olny tested processes; what should look for)
- Fog services. well, it surprised me to see this, I havent noticed. How could I acomplish that? It was a stock installation, so what went rong without notice…!? Well, ok, how to redo them without making a mess?
- ownership of files of the images are not uniform, yes, cos after testing was ok, some images from old storage was copied for further testing with new image vs old image context.
As for asking about removed parts of log: nothing was removed, only some personal info is hidden.
-
Can you take a picture of the error you’re seeing on clients? Or, the screen it sits on? Post the picture here, please.
-
well, actually nothing special (sorry, today i had zero time to test and make photo). after init files downloaded (.xz for example) it gets a normal looking root prompt. (will make photo, btw, next week)
-
Well, I am sure not a debug friendly solution I did, but I had to move on to a working state with my fog server. So, I replaced the machine, do a full reinstall, redo the personalized changes of the server, then tadaaa, all works as charm (btw, I keep the actual fog server and the buggy one to have a bit-by-bit comparison to see what is the difference. Up to that point we will know nothing unfortunatelly.
Wayne, I really appreciate your attention. If I find any detectable and – best case – reproducable reason for my issue, I will post it, send bug report if needed.
Bests
-
UPDATE: sad news, especially for me. Case reopened, as the fresh installation, on different box, with ultra zero modification (only pxe menu access was set) AND SAME ISSUE… means: I smell some bug in the infrastructure what the old version was resistent (or change was after v1.2 installed).
I would welcome all debug methods to detect where the problem lies (sure not in fog basic setups, maybe not in hardware of the server, as 2 totally different instal does same problem)
Any suggestion?
-
Try Trunk? [url]http://jbob.io/wiki/index.php/Upgrade_to_trunk[/url]
Couldn’t hurt at this point.
-
Tom thinks the drive HAD a GPT, but the update made it think it has MBR and that broke it. And Windows doesn’t properly remove GPT fragments from the disk.
You can use fixparts to repair it.
-
Trunk: at first I was not sure what are you talking about, for me, trunk meant a different thing. So, on to the latest version you suggest?
GPT: hm… so you meant if we do fixpart we can do a try to deploy and see if it helps? well, a try always can be ok. My problem is that not ALL deploy fails! I hate random errors as hard to get
too many factors and users dont like this if i say “wait a while, we may solve it. or not” In the past we had mbr issue when “never used disk we got with dealers recovery on it”. We killed mbr and deploy was ok. But, that time it was not same, 0.32 version and the actial deployement died with error. This time 1.2, and after init.xz and the other file loaded, it gives back prompt. From what point we can kick on with ./linuxrc (if gpt issue is in place, do you think it would stuck like it?)
We will do a try tomorrow with fixparts somehow (i havent ever used that as I remember, hope nothing extra is needed to fix it)
-
[quote=“Foglalt, post: 47312, member: 26236”]Trunk: at first I was not sure what are you talking about, for me, trunk meant a different thing. So, on to the latest version you suggest?
GPT: hm… so you meant if we do fixpart we can do a try to deploy and see if it helps? well, a try always can be ok. My problem is that not ALL deploy fails! I hate random errors as hard to get
too many factors and users dont like this if i say “wait a while, we may solve it. or not” In the past we had mbr issue when “never used disk we got with dealers recovery on it”. We killed mbr and deploy was ok. But, that time it was not same, 0.32 version and the actial deployement died with error. This time 1.2, and after init.xz and the other file loaded, it gives back prompt. From what point we can kick on with ./linuxrc (if gpt issue is in place, do you think it would stuck like it?)
We will do a try tomorrow with fixparts somehow (i havent ever used that as I remember, hope nothing extra is needed to fix it)[/quote]
You’d use a debug deployment to try it.
-
Well, it dont let me rest, can you enlighten me a bit, surely you know it better. The boot process is stuck, maybe somehow here is the key. Machine boots up, ipxe configurate the environment, then get the boot.php from the url where it should. As my poor knowledge helps it gets the tasklist and then generate the boot menu. If I understand well if the machine has task to do, that menu is an instant deployment. As no deployment starts, only I get a prompt, can it be because somehow the tasklist or the “bootmenu” generation fails?
Ofc I see active tasks, so not on this end (server). When that boot.php generates the “menu” what is the exact next step of booting? (is there a step by step list for me to understand?)
-
the easiest way to understand what the menu is telling you client is to go to the boot menu in a browser. it’s generated as a plain text file accessable via url
-
I forgot to post a screen of the “stop point” which I am hanging at after imaging stops. As I said, no error at all, only it stops here. And, I ran a bit around with mbr, gpt issue possibility. Put some old hdd into failing machine to see what it has (no gpt even touched those disks, a lot faster than remove what I have never saw).
What do you think?
-
Would you be willing to upgrade to the Trunk version?
read more here:
https://wiki.fogproject.org/wiki/index.php/Upgrade_to_trunkthis method (IMHO) is the simplist.
https://wiki.fogproject.org/wiki/index.php/SVN -
It sort of looks like your tasks start in “Debug” mode every time…
-
Yes. I havent looked it this way, but… well, yes. all can be done, but have to be done manually. Well, partially, as I only have to kickstart it with linuxrc command, not all stepts it does 1by1, but yes.
Ok, svn trunk updated to revision: 3482 (in the fog logo it writes this number I mean). We do some tests and see what happens.
UPDATE: svn version was up, and we have a bit worse scenarion… up to know, kicking it mad it work. now, no upload, no imaging. and i am going crazy…
stage1, imaging: after the ipxe boot, we see some testing before imaging. then at a point (erasing mbr/gpt info is done correctly it says, and here comes ->) initializing /dev/sda with NTFS partition…error: /dev/sda1: no such file or directory. some more error on same and udev. Then comes: using hard disk /dev/sda, and write cash check ok. And here error again (i guess it is cos of the first hdd error) Preparing hard disks (stage1): error. no such file or dir (sda1 missing again).
Fun fact: preparing hard disk (stage3) done properly (status written as “done”).Here comes the blue screen of partclone. It says: starting to restore /dev/sda1). Calculating bitmap… please wait, then come the final error: Target partition size (105 MB) is smaller than 35717 MB). Use option -C to disable size checking (dangerous).
Here we stand now. Any guess?
We at first thought it is about some new settings on fog image settings panel, so we went there. Some new option (like the wanted partition selection, or mbr backup, which is really nice to see coming!), but nothing obvious. So, we gave a try to create and image then redo it on an other disk to see what happens. Fail. At this point I was, you may understand :D, at a point of kicking something, so I go to eat for safety of my office
(i will go through the video we made about this failure to see if there is any clue I can tell about imaging, but it was some /dev/ thing again. Like if some hardware driver go wrong or so (2 disk we tried).As I do many tests to solve this problem I have a backup of things, so I went back to official last release to let my collegaue do some imaging with the “kicking it manually” way, to let him release some steam and do his work till we find a solution)
Ah yes, btw, This time we are over that “debug-like” prompt and it wants to start imaging. Only it cant cos of device error. Am I right?
-
Final case: The debugging of this state closed. No solution found with those machines in testing. I ended up with a fully new hardware architecture, a lot newer machine. Fully reinstall again, now on Ubuntu 14.04.2 LTS, 64bit. Fun fact, issue never again happened. Maybe candid camera was removed from room… damn, it was a bit frustrating haveing such insane malfunction with a normally working software. A week passed by, no issue anymore. Strange.
Thx for replies and tries in helping!
