I know I’ve been out of this for a little bit. I check in here or there, but just been extremely busy.

I don’t want to stop contributing, I just am taking time for myself after my workly duties.

I have to give a big gratitude and thanks for everyone here trying to help out whether by code, by helping the rest of the community, or documentation.

@Sebastian-Roth I know you’re busy but you’ve kept the project rolling even with the minimal availability you have. Thank you.
@george1421 I’m sure you’re busy, but I still see you posting and helping where possible and amenible. Thank you.
@Wayne-Workman I know you’re helping where you can as well. (Of course I can’t exactly post everybody because I’ve been busy and honestly not keeping up with the forums as much as I probably should.)

@everyone Thank you. Thank you for still believing in this project. We’re doing the best with what we have. Please understand in we’re lacking, it’s most likely unintentional. I know I’m just busy.

https://news.fogproject.org/fog-1-3-5-and-client-0-11-11-officially-released/

https://news.fogproject.org/fog-1-5-0-rc-11/

TLDR; Rerun the fog installer if you have lost “Database Connectivity” to your fog server, or run the ALTER USER syntax shown below.

So Ubuntu 16, among others I suppose, enable a “security updates” to be applied automatically as a “default” to things. Why, well it makes it simpler to ensure your Ubuntu systems are in compliance and patched for any potential exploits. This causes unknown and unexpected issues.

I figured it’d be a safe thing to express that there could be problems (as many of you have already experienced) that when these updates go up (with or without your knowledge) it can break functionality in unexpected and inopportune ways.

The quickest fix is to simply rerun the fog installer which should correct the problem.

As a note, it seems this problem is specific only when the mysql account is the 'root' user AND the password is blank.

The “fix” if you must do it manually is to open a terminal and obtain root:
Super (Windows Key) + T then sudo -i (in most cases).

From there, open mysql with mysql -u root

NOTE: MySQL MUST be run with ROOT.

Run:

ALTER USER 'root'@'127.0.0.1' IDENTIFIED WITH mysql_native_password BY ''; AND
ALTER USER 'root'@'localhost' IDENTIFIED WITH mysql_native_password BY '';

It’s okay if one of them fails. This is going to fix Most people’s issues.

I would highly recommend removing the unattended-upgrades as many of these “sudden” issues came as a security patch ubuntu pushed out. By default Ubuntu typically set’s this for you as enabled and it can cause havoc on you as you (the admin) may not have “done” anything.

To prevent this problem from happening in the future you could run:

apt-get -y remove unattended-upgrades (AS Root again).

FOG is still actively being developed. It’s not necessarily readily apparent, but we can assure you things are still being worked on. These updates may not be communicated in a way that everybody just knows, but can easily be seen if one were to look at our repository site.

Between our own schedules and lives, we can get very busy. We try to keep things updated and help out on the forums even during lull periods. This might mean we aren’t pushing an RC or release as frequently. It may mean we’re working on other things for the project, such as can be seen if looking at our github site.

Our forums are heavily active, and this should point as an indicator to our “status” as well.

If anybody would like to see an increase in developers donating their time to making this free software, consider donating either with monetary support or by spending personal time to help with development.

FOG is an open source project - it’s even in the name. It is driven by people donating their time and resources. The releases of FOG revolve around when developers can spare a few hours throughout the week. Sometimes that will mean releases will be further, sometimes that will mean releases will be faster. That’s just the nature of our project, and many other open source projects.

Hey everybody,

I know you see me here on occasion from time to time. Life decisions have made it more difficult for me to do things I would normally be doing. Rest assured, I am still around, and while I’m not quite as active as I was in the past, it’s not because I don’t want to be.

I had to move, and as part of that I have none of my normal development stuff readily available. Part of the move made me not have a laptop, until today.

I need to setup my dev environment again, so it may take a little bit, but I will be back up.

https://news.fogproject.org/fog-1-5-10-41-officially-released/

While this maybe spur of the moment, it should officially release 1.5.10 with all relevant bug/security fixes encapsulated (among a few other features.)

This has brought a new methodology of releases in that bug/security releases should be done much more regularly.

dev-branch, historically, was a place where new development occurred (hence its name) but over the last couple of years or so it’s mainly been a bug/security thing, not really a true development approach.

This is OKAY, in my head. Why:

Well we have the master branch which is the ‘baseline’ of a verions. We have the dev-branch, which allows us to work on bugs/security issues. We have the working branches for what will eventually become master. Working is our “dev branch” but keeps proper seperation of things in my opinion.

Basically:
master -> basis for dev-branch (which merges into stable on a regular cadence - still being worked out)
working -> basis of forward development

So we effectively have:
production
staging
development well seperated.

THis may mean on the regular automated releases, no announcement or news article will be created, and I think that’s okay.

Hopefully exciting we have a release after 1.25 years

Thank you!

That’s correct. The main reason fog is constantly moving forward is because the codebase is improved upon. Major bugs tend to be addressed for the next release. We don’t do an LTS because there’s really two main people working on fog in a consistent manor. Those two are @Joe-Schmitt and myself. Debian and Libreoffice have the team too be able to perform such a feat. Their product is Opensource but they have an employment team which can afford them that luxury. FOG has a team but we make no money and as such are required to work full time jobs. We work on FOG in our free time. I’ve had the ability to even work on it from work because we used the software.

Maintaining many different versions is difficult. And we don’t have a support team. WYSIWYG and I think we’ve done pretty well on support, even if we don’t have the ability to do dedicated support for our product. 1.5 was a major step toward modernizing the GUI. 1.6 will vastly improve on this. It was only recently we kind of came up with a road map on how best to proceed. Of note, 1.5 will be maintained until 1.6 is released. 1.6 is focused on making he GUI much more modern. 1.7 will be focused mostly toward fixing and refactoring the FOG client. 1.8 will focus on making the FOS system more modular and usable. I don’t know yet for 1.9. 2.0 will bridge the gap for our rewrite based on the work from 1.5 and up. While we do plan to try to do backports where possible, it’s much easier to ask people to update to the latest version than it is to try to maintain many different versions with backports in mind. At least for what FOG does.

I doubt this will appease anybody, but it’s what I think needs to be said. We are working hard and provide support for our product as best we can. The community makes fogs support system, I think, one of the best around. Add to that and you can almost always have a developer working side by side to help and fix issues as they come up, I don’t think it’s unfair to ask users to update to a specific version. Even if there are bugs, we will always try to correct what we can, when we can. (And normally it’s a pretty quick turn around).

I’m not perfect and I’ll give you that. We don’t even have a test suite to know if things are working as intended. We have to rely on the community and suggestions are great, just understand our answers won’t always be what people want to hear.

https://news.fogproject.org/fog-1-4-0-officially-released/

https://news.fogproject.org/fog-1-4-4-officially-released/

@FireGun679 Would you be able to update your fog to dev-branch or even better if you can update all nodes to working-1.6? potentially?

worst case at least upate to dev-branch or even the current stable and I think you’ll see things working.

@FireGun679 What version of FOG are you running? (Bottom right corner)

@rpycroft Please attempt to update to 2231.

@sgennadi It already is possible You just have to do the MOK or (use the new task type you see in working-1.6 to do all the work)

Kernels are already signed automatically.

@Matthiman, @sgennadi We are workign to a processed method of allowing secureboot which (in my test env’s) has proven successful.

The nicer part is this does not require Secure boot to be enabled to work.

@sgennadi SWEET! Thank you!

@Matthiman Your issue appears similar to an issue that’s being described here:
https://forums.fogproject.org/topic/18213/version-1-6-0-beta-3187-ipxe-2-0-0-x-problems

Though they seem to indicate changing to an older bootfile (snponly.efi/ipxe.efi whatever) worked so it’s not a kernel/init issue that’s happening causing this to occur.

This may be what’s happening with everyone in this thread as well, but the first image that I see from Fog_Newb looks more like what I’d expect where there’s not enough ram available to load.

@sgennadi Can you try a little test for me please:

Set your bootfile from snponly.efi -> secureboot/snponly.efi please.

This may not fix the keyboad issue you’re seeing but i’m taking a guess.

I’m unable to replicate the error but all I have are VMs.

@rpycroft That’s the confirmation I needed, thank you. 1.2 to 6.5 GB/min is
the fix working. The change is merged now, so it’ll be in the next kernel
release rather than something you have to keep hand-installing.

Two corrections from my end.

First, tx checksumming: ko was never broken. That line is about jumbo frames –
it’s the driver saying this chip can’t do TX checksum offload if you enable
jumbo, and at a normal 1500 MTU it never applies. It’s printed based on the
chip model alone, so you’ll still see it on the new kernel. Don’t let it worry
you when you spot it again. Your ethtool output was very likely the same before
the swap; there was nothing to fix there.

Second, I got two details wrong in the setpci commands I posted. My apologies.
The write should be masked so it only touches the two ASPM bits – as I wrote
it, it would have cleared the whole Link Control register, including the common
clock configuration, which you don’t want to disturb on a live link. And the
order was backwards: the spec wants L1 switched off at the NIC before the root
port, not the other way round.

Reading the current state (this part was fine):

setpci -s 02:00.0 CAP_EXP+10.w
setpci -s 00:1c.5 CAP_EXP+10.w

Only the last hex digit matters:

0 4 8 C  ->  ASPM off
1 5 9 D  ->  L0s
2 6 A E  ->  L1
3 7 B F  ->  L0s + L1

And clearing it, NIC first, masked to the ASPM bits:

setpci -s 02:00.0 CAP_EXP+10.w=0000:0003
setpci -s 00:1c.5 CAP_EXP+10.w=0000:0003

If you did already run the originals, nothing you’ve reported suggests any harm
came of it, and a reboot clears the lot either way.

No pressure on this next bit at all – the fix is confirmed and merged, and
this doesn’t gate anything. But if you ever have one of those 3070s spare, the
read-only pair above run once under UEFI and once under legacy would tell us
why the two differ. My theory is the Dell firmware enables L1 on the UEFI
path and leaves it off under CSM, which would explain everything you saw, but
that’s still me inferring rather than knowing. Genuinely optional – if it’s
easy, great, and if not, don’t spend your time on it.

Thanks again for testing this so quickly.

@rpycroft Thank you for the writeup. The elimination you did up front — cables,
link speed, compression, RAID vs AHCI, all three storage nodes, BIOS, iPXE
files — saved me a lot of guessing, and the commit you linked is the right one.

You’re right that it’s the r8169 switch, but not quite in the way it looks.
The in-kernel driver isn’t worse than the vendor one here. It exposed something
we’d had broken for nine years without knowing.

Two kernel options that are on by default upstream have been off in the FOS
kernel config since 2016:

# CONFIG_PCIEASPM is not set
# CONFIG_PCI_MMCONFIG is not set

ASPM is PCIe link power management — it lets the link drop into a low power
state when idle and wake when there’s traffic. Waking up is the expensive part.

Nobody noticed for nine years because the Realtek vendor drivers we used to
ship handled ASPM themselves. Our r8168 build had dynamic ASPM compiled in,
which effectively switched ASPM off any time packets were moving. It didn’t
matter that the kernel couldn’t manage it, because the driver never asked the
kernel to.

r8169 does it the normal way: it asks the PCI core to disable L1 once, when the
card is probed. That’s where this bites. With CONFIG_PCIEASPM off, the function
it calls isn’t a function at all — it’s a stub that returns “success” without
doing anything. So r8169 is told the OS disabled L1, marks ASPM as being under
OS control, and then goes on to enable ASPM and L1.2 on the card — on a
kernel with no ASPM code in it that disabled nothing. Not one line in the logs
to say so.

That “falling back to CSI” line you included is the second option. Without
CONFIG_PCI_MMCONFIG the kernel only sees the first 256 bytes of each device’s
config space, and the registers controlling the L1 sub-states live past that.
So even a kernel that wanted to fix this couldn’t reach them. Including that
line is what let me tie the two together, so thank you for pasting it.

That also explains the two things in your report that looked strangest:

Why UEFI and not legacy — ASPM is programmed by the firmware, and Dell’s UEFI
path turns L1 on where the legacy/CSM path leaves it off. Our kernel couldn’t
change it either way, so whatever the firmware picked stuck. Legacy was never
actually faster; you were getting a machine where ASPM was already off.

Why deploys and not captures — a deploy is your client receiving. The link
goes quiet between bursts from the server, drops into L1.2, and pays the wake
cost over and over. A capture is the client sending, which keeps the link
busy so it never gets the chance to sleep. That’s your 15 GB/min upload sitting
next to a 1.2 GB/min download on the same cable.

Worth saying before anyone suggests it: adding pcie_aspm=off to the kernel
arguments does nothing on a FOS kernel. The code registering that argument is
inside the same #ifdef, so it’s compiled out too. It would look like you tried
the standard fix and it didn’t help.

I’ve built you an experimental x64 kernel with both options turned back on:

https://github.com/FOGProject/fos/releases/tag/EXP_20260805-123232

sha256  6fadc7204889bc76fe943520319044940c24654365a523d5df7adf8327a88d25

It’s the kernel only — your init is untouched and doesn’t need changing. Back
up your current /var/www/html/fog/service/ipxe/bzImage, drop this one in its
place, and deploy to one of the 3070s. Putting the old file back is the whole
rollback if it doesn’t help.

If you want to confirm the diagnosis yourself first, it takes about a minute
and doesn’t need my kernel at all. Boot a 3070 to a shell in debug mode:

lspci -nn | grep -i ethernet     # note the Realtek address, e.g. 02:00.0
lspci -t                         # note the root port above it, e.g. 00:1c.5

setpci -s 02:00.0 CAP_EXP+10.w
setpci -s 00:1c.5 CAP_EXP+10.w

The bottom two bits of each value are the ASPM setting — 0 is off, 2 is L1,
3 is L0s+L1. Run it once booted UEFI and once booted legacy. If I have this
right, UEFI shows a 2 or a 3 and legacy shows a 0.

And to watch it fix itself, clear both (root port first) and rerun the deploy
without rebooting:

setpci -s 00:1c.5 CAP_EXP+10.w=0000
setpci -s 02:00.0 CAP_EXP+10.w=0000

That’s not a fix you can keep — it’s gone on the next boot — but if the speed
jumps back to 6-10 GB/min, that confirms it before you swap any files.

One loose end: could you post the raw output of ethtool -k <interface>? You
mentioned tx checksumming being off and I want to be sure I’m not waving away a
second, separate problem. It shouldn’t be able to cause a download-only
slowdown, but I’d rather look than assume.

Thanks for the report and for offering to test — I’m taking you up on it.