Massive CPU usage from a service

[Tom Elliott]

@LLamaPie I am pretty sure this isn’t something FOG is doing. I don’t know of a command named .systmd and seems questionable at best. The fact that it’s being used and spawned by www-data is real hmmm if you ask me.

[LLamaPie]

@Tom-Elliott Yep, that is what I was worried about. Worst case I need to nuke the server and rebuild.

[Tom Elliott]

@LLamaPie If you can get the backup db and images of course. I don’t think they’re affected, but whatever is relaying is using a ton of CPU. so at least not a full startover I think.

[LLamaPie]

@Tom-Elliott Well Sophos found this:

[george1421]

@LLamaPie

/var/www/html/fog/management/.sys

Is not something that FOG creates. ref: https://github.com/FOGProject/fogproject/tree/stable/packages/web/management

.sys file / directory name means the directory is hidden unless you use the command ls -la /var/www/html/fog/management also .systmd is a hidden file made to represent systemd application.

I did find this article: https://sarperavci.com/ironshade-writeup-tryhackme/

So the question is how was this server compromised and if we don’t know it will probably happen again. What version of FOG did you have installed?

[Tom Elliott]

@george1421 Running Fog Version: 1.5.10.15

[george1421]

@Tom-Elliott The key is that its post the security update for FOG.

The question I have is:

1. How did that file/malware get onto the server
2. Why did it pick that specific path to hide in.
3. When was the server compromised. The date on the files in that directory may give us a clue.
4. Could it happen again? We don’t know because we don’t know how it was installed.

It almost seems intentional and deliberate to pick that specific path. I don’t think apache normally has write access to that path.

@OP is your fog server exposed directly to the internet?

[LLamaPie]

@george1421 Nope, that is what is baffling us as well. The server is local only and locked down. No one outside the network should be able to access it.

It’s hard to say when it was compromised but we did notice the sudden spike in resource usage 1-2 weeks ago. The server is largely left alone as it does what it needs to do. Beyond running updates on occasion, no changes are made. I will keep an eye on things. I just know after Sophos cleaned up the issue yesterday it has been fine. It’s too soon to say for sure.

What I can possibly do is scan some of our long-term back ups to figure out how long it’s been infected. We will want to discard them anyway. I’ll see what I can do.

[Tom Elliott]

@LLamaPie Based on what I see of the issue is it possible the server is this service too?
https://sarperavci.com/Froxlor-Authenticated-RCE/

I don’t know what Froxlor is, but the lol.php and .systmd that you saw seems to point that somebody was trying to do some bitcoin mining possibly?

[LLamaPie]

@Tom-Elliott Yea the “coinminer config” that Sophos nuked + the 400% CPU usage makes me think it was being used to do some sort of mining.

[george1421]

@LLamaPie Well keep an eye on it, my concern is that there is a bug in php that allowed this to happen some how. Its in the http path and not the nfs path. Hopefully you will scan and or keep an eye on the cpu usage. When you detect a change surely review the apache logs both error and access as well as login logs to abnormal activity.

[LLamaPie]

@george1421 Yep, can do. I’ll keep you guys posted. So far it’s been fine since Sophos cleaned it.

[LLamaPie]

@LLamaPie Everything has been clean now for about a week. I would consider this at least resolved on our end. Still no answer about when it became compromised exactly. Our hyper-paranoid theory is it may have been a “time bomb”. This could have been on the server for months before popping up. Our long-term solution is keeping endpoint protection in place. I have nothing else to add but if I discover anything I will let everyone know.