Final case: The debugging of this state closed. No solution found with those machines in testing. I ended up with a fully new hardware architecture, a lot newer machine. Fully reinstall again, now on Ubuntu 14.04.2 LTS, 64bit. Fun fact, issue never again happened. Maybe candid camera was removed from room… damn, it was a bit frustrating haveing such insane malfunction with a normally working software. A week passed by, no issue anymore. Strange.
Thx for replies and tries in helping!
Yes. I havent looked it this way, but… well, yes. all can be done, but have to be done manually. Well, partially, as I only have to kickstart it with linuxrc command, not all stepts it does 1by1, but yes.
Ok, svn trunk updated to revision: 3482 (in the fog logo it writes this number I mean). We do some tests and see what happens.
UPDATE: svn version was up, and we have a bit worse scenarion… up to know, kicking it mad it work. now, no upload, no imaging. and i am going crazy…
stage1, imaging: after the ipxe boot, we see some testing before imaging. then at a point (erasing mbr/gpt info is done correctly it says, and here comes ->) initializing /dev/sda with NTFS partition…error: /dev/sda1: no such file or directory. some more error on same and udev. Then comes: using hard disk /dev/sda, and write cash check ok. And here error again (i guess it is cos of the first hdd error) Preparing hard disks (stage1): error. no such file or dir (sda1 missing again).
Fun fact: preparing hard disk (stage3) done properly (status written as “done”).
Here comes the blue screen of partclone. It says: starting to restore /dev/sda1). Calculating bitmap… please wait, then come the final error: Target partition size (105 MB) is smaller than 35717 MB). Use option -C to disable size checking (dangerous).
Here we stand now. Any guess?
We at first thought it is about some new settings on fog image settings panel, so we went there. Some new option (like the wanted partition selection, or mbr backup, which is really nice to see coming!), but nothing obvious. So, we gave a try to create and image then redo it on an other disk to see what happens. Fail. At this point I was, you may understand :D, at a point of kicking something, so I go to eat for safety of my office 
(i will go through the video we made about this failure to see if there is any clue I can tell about imaging, but it was some /dev/ thing again. Like if some hardware driver go wrong or so (2 disk we tried).
As I do many tests to solve this problem I have a backup of things, so I went back to official last release to let my collegaue do some imaging with the “kicking it manually” way, to let him release some steam and do his work till we find a solution)
Ah yes, btw, This time we are over that “debug-like” prompt and it wants to start imaging. Only it cant cos of device error. Am I right?
I forgot to post a screen of the “stop point” which I am hanging at after imaging stops. As I said, no error at all, only it stops here. And, I ran a bit around with mbr, gpt issue possibility. Put some old hdd into failing machine to see what it has (no gpt even touched those disks, a lot faster than remove what I have never saw).
What do you think?
Well, it dont let me rest, can you enlighten me a bit, surely you know it better. The boot process is stuck, maybe somehow here is the key. Machine boots up, ipxe configurate the environment, then get the boot.php from the url where it should. As my poor knowledge helps it gets the tasklist and then generate the boot menu. If I understand well if the machine has task to do, that menu is an instant deployment. As no deployment starts, only I get a prompt, can it be because somehow the tasklist or the “bootmenu” generation fails?
Ofc I see active tasks, so not on this end (server). When that boot.php generates the “menu” what is the exact next step of booting? (is there a step by step list for me to understand?)
Trunk: at first I was not sure what are you talking about, for me, trunk meant a different thing. So, on to the latest version you suggest?
GPT: hm… so you meant if we do fixpart we can do a try to deploy and see if it helps? well, a try always can be ok. My problem is that not ALL deploy fails! I hate random errors as hard to get 
too many factors and users dont like this if i say “wait a while, we may solve it. or not”
In the past we had mbr issue when “never used disk we got with dealers recovery on it”. We killed mbr and deploy was ok. But, that time it was not same, 0.32 version and the actial deployement died with error. This time 1.2, and after init.xz and the other file loaded, it gives back prompt. From what point we can kick on with ./linuxrc (if gpt issue is in place, do you think it would stuck like it?)
We will do a try tomorrow with fixparts somehow (i havent ever used that as I remember, hope nothing extra is needed to fix it)
UPDATE: sad news, especially for me. Case reopened, as the fresh installation, on different box, with ultra zero modification (only pxe menu access was set) AND SAME ISSUE… means: I smell some bug in the infrastructure what the old version was resistent (or change was after v1.2 installed).
I would welcome all debug methods to detect where the problem lies (sure not in fog basic setups, maybe not in hardware of the server, as 2 totally different instal does same problem)
Any suggestion?
Well, I am sure not a debug friendly solution I did, but I had to move on to a working state with my fog server. So, I replaced the machine, do a full reinstall, redo the personalized changes of the server, then tadaaa, all works as charm (btw, I keep the actual fog server and the buggy one to have a bit-by-bit comparison to see what is the difference. Up to that point we will know nothing unfortunatelly.
Wayne, I really appreciate your attention. If I find any detectable and – best case – reproducable reason for my issue, I will post it, send bug report if needed.
Bests
well, actually nothing special (sorry, today i had zero time to test and make photo). after init files downloaded (.xz for example) it gets a normal looking root prompt. (will make photo, btw, next week)
Ask anything! All is welcome and maybe closer to solution. Btw, you said things strange to me (and I was puzzled to see them before posting log to you, but I thought i may be underinformed what is normal and what is not.
As for asking about removed parts of log: nothing was removed, only some personal info is hidden.
I run the troubleshoot.sh now, made a little change on host ip to be less… well, open . Here is the log it generated.
Rouge dhcp. Well, frist thought was this, as not on fog, but we had some same situation, so we go and rested it. Not this kind of problem here (as far as tests went). And, for the fun of it, now we make tons of tests with different hardwares to see any pattern if we may see, and latest test machines produced this: if ever occure, nothing can change it (well, 10-12 reboot, no change on same box). BUT, cos always there is a but… if it stucks after.xz file load, i get the root@146 like prompt i can ANYTIME go on with imaging commencing ./linuxrc. Anytime!. So, if network issue anyhow, it is only in the boot process of the ipxe downloaded OS. It can be a clue
Now we collect motherboard info for listing working and not working machines and still there are machines we find with working and not working status.
I even replaced the fog server machine with other one (not the best solution, but I relocated the disk to other hw). Same issues: sometimes works, sometimes not. Pretty frustrating.
I will go through the suggestions you made above as soon as i am able, but as always, we have to do many things separately. And, as we were after testing phase, this system got into production, so, not the best to fall back to previous versions or go backup; but I think I will not be able to avoid that… 
I still have no clue on why imaging stops. Up to this point we take notes to find a pattern in hardware, etc, but still find no solution. And after a LOT of machines cloned it is a really frustrating to see that now it doesnt work more than 1 out of 10 (!) cases. Even worked-before hardwares stop working. Still no error messages at all!
It has error massage at the 7th step. error says 2 things:
for me it is a bit strange at this point, especially knowing that if it has a valid task to deploy, gets stuck at a prompt, THEN commint ./linuxrc command it goes into imaging process, and finishes it without any further error. additional, maybe useful info: the point where it stucks has a prompt:
“root@146#”
where the 146 is the first number from the cliens or server ip. does it have any additional meaning for you?
[quote=“Wayne Workman, post: 46356, member: 28155”]Can you look through this WiKi article and give these steps a try?
[url]http://fogproject.org/wiki/index.php/Troubleshooting_an_image_push_to_a_client[/url]
Lets see exactly where the problem is.[/quote]
Hi!
Yes, I checked the device list, not on badboys list. And, that same hardware worked before (2-3 days before). We have plenty of practically clone hw, and the one we happen to have problem, the exact one, worked even before. That is the most frustrating part of the story. We use fog for years and are happy to use it. And now this happens (well, we use 1.2 only for a few weeks, and problem only occures this version, not the previously used 0.32).
Hi!
I have a strange issue. Task for deployment is given, pc restarted for imaging and it stops with a root prompt. Like if something the scripts failed to start their job on clientside. If i test a linuxrc script, it goes on and finishes without error, image deployed properly. But why it stucks right before doing imaging?
Fog 1.2, stock install. Problem was not seen till a point which cannot be interfering, but let it be mentioned: an extra iso boot image option was added to the boot menu for ipxe.
Any suggestion?
(3-4 times it happened and kinda frustrating, no same visuals noticed. Last occurance was accompanied with a DHCP/BOOTP error displayed, as follows
“DHCP/BOOTP: Reply not for us, op[2] xid[45d6eedc].”
As I recall this is not a good simptom as not all same issue was followed by this dhcp message.
I know that I disappeared a bit with this issue, but as I wanted to make a reproducable version of this problem I created many virtual box instances to make a closer look on the problem. And now, for reason unknown, I cannot reproduce the issue. Hosts are created as I want them to be, etc. Really, really dont get the problem, what happened. If I again can reproduce the issue maybe I will be aware of the reasons.
hi!
Sorry for not properly detailed question, I try to fill in some holes, and do more test. I did the approximate following to produce the issue: installed a stock, unmodified ubuntu 14.04 (I know about supported OS versions, but foregoing tests are needed for future), downloaded latest fog (1.2.0). Installation went ok on both sides. For my tests I needeed 1 host and 1 image, so I did preparation. Created the image (webui, put in file and os type details), was ok I could see it after page refresh, too. Created the host item (mac, os, image selection, etc). It said in the hovering info, host is added normally.
I did some other preparation with the host itself, then I checked the host list. There came the surprise, no host is list. I checked the image, there it was. From that on I made test of creating hosts. Import csv, import from .sql files, all were ok, but manual host creation was still missing. I know that most of my hosts are registered from host itself or mass import as csv, but was completly shocked that i cannot create host from webui.
As I have spare hw shortage now for test, I go back to virtual machines and do more test, get more information, too. As I am finished, i will fill in maybe more holes in informations. And ofc I did digging on forum for mentioning this before, but all I could find was not totally same issues, this is why I posted mine (basically no error in any logs and host is not craeated, with no warnings or so)
more on it: I had no entry in host db at all. i can only see the image entry. Like i created host to /dev/nul
[quote=“Tom Elliott, post: 36921, member: 7271”]Um, what?
Do the hosts that you “register” exist in the DB? Yes you may have to actually peer into the real database to find out. Do you have ID Entries of 0?
Maybe you could verify that the database for the hosts table is in working order running a repair on it?[/quote]
Hi!
I am making tests of 1.2 version as soon I will do an upgrade of working 0.32 version. Up to now some of my problems are solved, but a “funny” one I cannot figure out. On the web management page i cannot create hosts! Isnt it strange? I add details of host, select image, etc. click add, then page says up there host is added. But I cannot see no host in list. Even more, i can redo again and again, getting no error messages, so seems database is intact (same mac and host all time) and registers no actual hosts.
Any suggestion where I can find solution?
system: ubuntu 14.04 lts (stock install), and fog 1.2. Btw on ubutu 12.04 lts it works, that is not an option for understanding the problem to skip OS update. Once I will have to, so better be prepared.