Posts made by Foglalt

@george1421

The actual hardware part was not identified, we had not enough time and the computer was not available to disassamble (it is a brand new laptop, which has garantee issue if opened up). The process where the slowness was noticed was various. Some of the ipxe boot process (like downloading bzimage), some adjusting the disk (partition writing, mbr saving, etc). Fun fact that the actual disk loading with image was not hindered with slowness. Considering that the image deploy is a massive amount of data writing it is strange. Compared to the bzimage’s tiny size, it is not clear what caused the actual slowness.

It was during disk io and can be during network traffic. During the process we couldnt detect more error than actual slowness only. All done, but insanely slowly. When we switched back to legacy mode, it was like a charm. Fast and easy.

We did a few more investigations and came to a seemingly working solution. Sorry for the long wait, but in virus situation we have limited hw access and those are only on fixed days.

Part one of the case: high cpu load. This was the easiest. It was a disk issue (smart showed no valid error, but I insisted on a test with a new disk. faster, bigger etc. It was a time to buy and make it running). So, the load became ok (back to zero or 0.1, as was normally).

Part two, the slowness. We had a massive new hw to image, most of them are almost same hw, but we found out the the slow ones has some undocumented hw difference maybe. (Meaning they should be identical, but actually they are not). Solution: we disabled the uefi mode and now it is now properly working (drawback it needs some finishing at the end… but doable). I dont know what is the true hw that gives this error, but legacy mode seems ok atm.

For future investigations, or for the logs, here is the actual hw that we found guilty in a few percentages:

It is a “hp elitebook” laptop. All should be same, but somehow somewhere they are not. The problematic hw fails on many place in speed. Sometimes even the bzimage download is “visible” (normally it is 100% ok at once), sometimes disk partitioning is stuck for a long time, etc).

I think and I hope the case is closed. How can I mark it “solved”?

(oh, forgot to mention: we did changes in kernels, no difference with those some)

@Tom-Elliott
Ah, sorry, I forgot to answer this question of you:

bzImage Version: 4.19.118
bzImage32 Version: 4.19.118

And no, atm we have zero special hosts, so no need special kernel. In old days we had, but atm no, only one.

@Tom-Elliott

Our production network is a strange thing nowadays. Almost all computer has an ip what is considered public address, but ofc it is not true in reality. First of all, our pcs can see the outside world, can communicate. The world outside cant communicate with us only if allowed explicitely. So, the network is theoretically public, but actually it is not public. Traffic is routed and walled if needed. If no anomalies found, no traffic from the outside at all to normal machines or servers. This setup is because of many of our projects need public things with less or no routing.

The imaging worked fine since 1.5.7 version, we had this kinda issue with 1.4.x when i had to change the client kernel for a strange “delay during saving mbr” issue. And now, it “sounds” same. This is why I first changed a few kernels for testing purposes to see what happens. (I still do tests with previosly used kernels)

As for the load. The load is, you are right, not surely the reason for the issue. I just wanted to give as many details as possible. As we had zero tasks running the load was really strange. the same os version with almost same service setups (web, some php, mysql data backend, zero high throughput data, like in a iddle fog server) does 0.1 0.1 0.1 load (even with an lot older machine, less memory).

And I am practically sure that the actual fog machine had a lot less load previously. So something happened, or happening, I still need to discover what it is. During my tests I forgot to start the stopped apache and the load fell to 3.0 somethings from the 7.0 somethings. Normally fog consist of not infinite amount of services but the true shock for me what that it does something but hides it no running or stuck process but load…

One of my thoughts was a failing hdd or some, but smart says it is ok. Not intact and virgin, but is ok.

So, Elliott, slowness is not the result of the load (especially that the gui or ssh is responsive and fast). I agree. And here comes the “but” part Any suggestion? (I will try to upgrade to current version, but less options in covid situation. I dont want to kick the table from under my colleague who have to be in building if I dont have to

@Tom-Elliott

it is not a virtual machive, an actual machine.

Private network or not, it is not my domain to decide. I kicked some ass today seeing that even echo is alloved to that machine from the outside (btw they said, netadmins, other traffic is not allowed. i will need to seel some ass to make sure it is…)

What other information do you suggest to collect?

Hello guys!

We have noticed an issue what we try to investigate but I need a bit of information and help as I am stuck ATM. We found out that some imaging (upload and download, too) became insanely slow. The deploy part seems (ATM!) that is target hw dependent, which is good, as it is not on our side. The problem became more interesting when it seemed that even the “master imaging” became related to the issue. The master image is created regulary basis on a virtual machine (win default one, not virtualbox or such; hyperV). Uploads are slooow.

We have discovered that “preparation stages are slow, too, not the actual imageing” (so not only cloning process, but the scripting ones).

At a point we had an issue about dealing with filesystem, partitioning. Once we had such, that was a kernel issue, now it is not, or a different one, as I tried few versions.

Server officially does practically nothing. Sole job is the foging. As with no task it shows strange issues, I started another investigations. Found out the it has HIGH loads, and that load is practically permanent. With not too much done on the server.

The load is almost all time at 7.0. Very little oscillation. Can we somehow locate the reason fog-wise? (another point is a possible hw failure ofc, but atm I dont really think so I can detect that as virus broke physical bonds with machines so I try to detect software issues at first.

Can I have suggestions? My problem is that I am not a performance tuner type, I have not too much experience in such. All I see that web or ssh side server has zero delay to respond. But the imaging process has delays at points where previously it has zero. I dont know that the high load is the issue, or it is a result. I can collect all data if needed, as I have access remotely, but atm I am stuck atm.

We have 3 issue:

• high and no-reason load permanently, with zero task on server
• in the imaging between actuly data load we have steps where things became “stuck” for 10s of minutes (partitioning, mounting fs, post scripts entry points)
• and I see network traffic on a permanent basis what was not there if i recall well.

I made a few screenshots if it may help.

here it stucks for a long time.

here, the partition deployment is fast, then the start of the next partition is delayed a LOT (win10, 3 small and 1 big data partition, as it creates them). Normally the 3 little ones are only “blinks of an eye”, not we can even drink coffee between them). Data throughput is fast.

Here it does something, again strangely slow.

Here is the truely slow one. (this is the one that i mentioned that previously it was a kernel issue with other fog versions, what I could solve with new kernels).

We currently run the 1.5.7 fog version, on a debian (stretch, 9.12). The machine has 8G of ram, 500G of disk for active things, and 1T disk for “prebackup” (mounted only in case, otherwise inactive)

(UPDATE: I see things in dmesg what I havent seen before:
[456483.853291] rpc-srv/tcp: nfsd: sent only 18600 when sending 32900 bytes - shutting down socket

And many of this… I found a bug that caused such years ago, i dont think that is it, (i have version: ii nfs-kernel-server 1:1.3.4-2.1 )

Any suggestion what does this poor machine does in his free time that need to be killed? ATM smart show no disk error, what I know).

I really appreciate this, really detailed and helpful. Most of this was in my head cos of many reading and usage, but I think I would messed it up with failing memory, forgetting 1-2 things out. You guys, are still in any kind of support!

Hi!

Can I ask for a help? We need a network help to improve our fog possibilities (we had issues, what is not fog based, but network based what prevents some things to happen during fog). The support (outside help) is not a fog specialist, so asked us to get as many information as possible to detect the problem. As a part of it, I would like to give them the info about the fog booting process. From ipxe to partcloning.

Can I ask for the steps of the boot process? I dont need bit depth, but to detect the problem, I would like to give details as much as I can. Like: “network boot, ipxe dhcp starts, then…” and so on.

(we have issues of these steps with some hw, before image starts to deploy the actual hdd, but it is strange, and not fully global. maybe some of our hw is not properly working in our network, or idk).

I would appreciate any details and helps. Thanks in advance

@george1421 I was thinking about what is best. New thread or ask in between. I chose the later cos I only need a step-by-step “what happens in stages of fog imaging”, cos network support needs it to find the guilty part. My test revealed that the problem is not the fog itself in this case, but was the network devices in our network somewhere. Still suggets to open a new thread? If so, I will do to make things clear.

We has problems of such, but was not able to filter out a proper method that worked. We had to find other methods (if possible, even sdd out, and into other machine, etc). Now we have a very few hw what has such problem. At the moment we are scheduled to use network support time to figure out some other problem, what caused almost same issue (separated fog is working with stupid switch, but built in wired system it fails with exact same “errormessages”. To make a solution closer, may I ask the Fog experts here?

the detailed steps of fogging can be named 1 by 1? since ipxe boot to partclone section. Somewhere there is a solution for our hw, but our network support leaks somehow and misses the solution. Now we have outsourced methods, but we need to feed them with a many details, as possible.)

Can I ask a humble favour to have the detailed information I need? step by step at fog vesion 1.4 to actual versions if we have differences at all. My knowledge is farm from hw or firmware based details of the process

@george1421 I will skip this model if I can. I hope I will not have them back soon. Lets close it. ATM I dont that that hw at hand, and have other problems that needs solution (not fog related).

As for the fog versions. I would upgrade it when i will have time for it. BTW can it br upgraded directly from this “gap of versions”?

@george1421 sorry, definitely my bad that i disappeared. we had to dismiss those hw and my boss killed my time that i can experiment.

bzImage: Linux kernel x86 boot executable bzImage, version 4.11.0 (root@debian64) #1 SMP Thu May 11 15:44:36 EDT 2017, RO-rootFS, swap_dev 0x7, Normal VGA

@george1421 said in ASRock B360M Pro4 imaging issue:

We need a bit more information to help you.

What version of FOG are you using?
What version of FOS Linux (kernel) are you using?

If you manually register this target computer, setup a debug capture/deploy then pxe boot the device. After a few screens of text that you need to clear using the enter key, you will be dropped to a linux command prompt. At the FOS Linux command prompt key in lspci -nn | grep -i net and post the results here.

Stay at the FOS Linux command prompt in case we need to have you type in a few more commands.

the fog is 1.4.4. but the kernel… atm i dont know. will check later. We had to urgently finish that machine (not from our hw pool), so for next detail I have to wait a bit, but will gather what you requested. S soon I come with info I hope.

hello guys!

We ha a new hw at company, and we are having strange issue with it. During the imaging the stucks at a point telling cant get IP via dhcp. Actually it can, cos it is after it got already ip in the sooner step of pxe boot. Anyone maybe have such motherboard maybe? Looks like it is a driver issue or something, but as even the newest windows 10 knows nothing about it without personal attendance I think it is something really new, or having issues. It has an intel on board lan (well, on the specification page is confusing as says not too much about exact model number or i saw 1-2 different).

Oh… I didnt even think about it that way. Especially that we have the “locked” state for the same purpose, prevent accidental deletion. Actually it is a double of the double protection. Not truly bad, but… strange design.

Hello guys!

I found a strange thing during image deletion. My colleague reported that deleted images still are on disk, so I investigated and found the following. If I want to delete more than one image, I do the following. List images, put the checks on the selected ones, then I press “delete selected ones”. If I do so, images are deleted from database properly, but files are intact, stay on disk.

If I want to delete one image only, I go to image data page, on the left i click on delete. THEN it asks if I want to delete the files, too. check put, database entry and files are gone with the clicks.

What is the difference? I mean why the “delete files, too” question is not asked on mass deletion? It is not a huge issue, but when we delete images, we delete more at once (version upgrade on production machines with different setups).

Fog version: 1.4.4, OS Debian.

As time goes, time to time i turn back to my previous things for refreshing memories, getting new ideas, whatever. So it happened to this topic.

I am again thinking about fully offline cases. We here discussed that zstd compression is not compatible with clonezilla so i have to recapture image to use tools like clonezilla which uses same image taking tool. But… as now i have read, clonezilla now capable using zstd compression (er… whatever it means exactly btw).

Let me ask if it is still true or not to use already taken images (by for) deploy from a live clonezilla-like tool without recapturing? Anyone tested maybe?

@Quazz ok, then i keep my already downloaded kernels/inits, and wait for next release for further tests.

@Sebastian-Roth Hi! I am aware that it is an old post above mine, i was just curious if the state of this issue was finally “merged into patches”. I mean that i used the kernel of it since this post, and I somehow missed the mention of it in a later release.

Well, I wanted to know if the current release (1.5.5) contains the proper kernel, or should i use the one i already have downloaded.

@george1421 Yes, seems like it. But worth a question and a few googling thx anyway (still open for other solution, if anyone reads this topic later)