Disappearing hosts from host list

[Wayne Workman]

@developers I have found a pattern - not sure what it means but I wanted to share to get your eyes on it.

These next two code blocks are from the first log the script provided:

Date & time: 2018. febr. 28., szerda, 09:29:01 CET
Found hostID '5' without a primary MAC.

From the history table, there are some events about one hour earlier with the same hostID.

[2018-02-28,08:28:10],Host,ID: 5 NAME: laci1 has been successfully updated.	laci	2018-02-28 08:28:10	10.10.36.124
[2018-02-28,08:28:10],MACAddressAssociation,ID: 131 has been successfully updated.	laci	2018-02-28 08:28:10	10.10.36.124
[2018-02-28,08:27:45],Host,ID: 5 NAME: laci1 has been successfully updated.	laci	2018-02-28 08:27:45	10.10.36.124

These next two blocks are from the recent log the script provided.

Date & time: 2018. ápr. 10., kedd, 09:52:01 CEST
Found hostID '43' without a primary MAC.

Again, we have an event in the history table from about one hour earlier with the same hostID.

[2018-04-05,08:53:10],Host,ID: 43 NAME: laci6 has been successfully updated.	laci	2018-04-05 08:53:10	146.110.36.124

Could it be that the timestamps from the history table are just wrong and these events actually happened at the same time? If so, I’m betting that changing the name of a host can somehow - sometimes - cause this primary MAC missing issue.

[Tom Elliott]

@wayne-workman The timestamps are accurate. The event you’re showing is from February 28th, vs April 5th. I don’t know if the timestamps are in UTC or correct to the timezone the user is in.

[Wayne Workman]

@tom-elliott The timestamps are from whatever is set on his server. You see the 1-hour pattern though in both occasions? Makes no sense to me. Only thing I can think of is a problem in an hourly ran service. This is the script he’s running every minute via cron: https://github.com/FOGProject/fog-community-scripts/blob/master/troubleshootingTools/monitor-missing-primary-mac.sh

[Foglalt]

Tomorrow or later tonight i send you another. Yes, again it happened. Maybe it van be same pattern at least. Btw host is not renamed. But mac is changed. All time!

[Wayne Workman]

@foglalt When you say the MAC is changed, are you changing it via the GUI or is this something that’s happening that is not supposed to happen? Please elaborate.

[Foglalt]

@wayne-workman It is simply the following:

• pc1 comes in for reinstallation/installation, its mac is registered in a “dummylikehost” (for example “laci1”).
• image selected, task set to deploy, pc1 finishes, turned off
• pc2 comes in for same purpose, host “laci1” got a mac overwrite (mac gui field selected, typed in the new mac, update button).

the missing host is detected by your script, or previously it was detected when pc3 comes in for processing.

this is why i asked before how others do massive cloning. colleagues do this method cos with it you dont need to remake cloning groups (no image update? so you dont need to change image name). it is like another 5 pc came, you put them on the table, put cables in, register new macs and launch process of cloning. we normally never keep hosts in database, as you may saw in our database before. we only have a few in them. few dedicated ones (like image creators machine and some other).

[Foglalt]

One more thing about yesterdays instance of the problem: during the host update the new mac was registered on the host (this time laci2, to name it as in log you may see) and when the task start was prepared (those hosts are in a group for mass deploy) colleague ralised that group has less member than needed. voila, missing host. so it was disappeared after host update (or during)

[Wayne Workman]

@foglalt So is it accurate to say “When an existing host’s MAC address is changed to something else, sometimes the primary MAC address is lost.” ?

Also, you know you can image without registering, right?

[Foglalt]

@wayne-workman what do you mean by this? we do this cos of easily possible mass actions (multicast is done with registered hosts, etc or did i miss something?)

[Wayne Workman]

@foglalt said in Disappearing hosts from host list:

multicast is done with registered hosts, etc or did i miss something?

This thread suggests it’s possible:
https://forums.fogproject.org/topic/9669/image-multicast-without-host-registration-ipxe-input-output-error

[Foglalt]

@wayne-workman but how? Anyway it doesnt change the mysterious disappearing sadly

[Wayne Workman]

@developers the pattern I thought I found earlier no longer holds with this 3rd log.

[Foglalt]

As i am a regular user for detecting and killing hosts with missing primary mac, unfortunatelly, may i use stored procedures for this? does it have any problem with fog database? will it be wiped with fog upgrade? i am trying to find a proper way for colleague to kill issue without me if i am out of office, and for this i am thinking about using stored procedures (well, same for other database commands we may use to maintenance in fog maybe nice)

is it a source of problem or i may use them?

[Tom Elliott]

@foglalt fog doesn’t do any removing or adding or triggers or procedures so if you can think of one to write that can help fix the issue for you then I say go for it. Please, if you’re feeling kind, post it here and maybe what you write might be able to allow me to apply a more direct fix to the core of fog. I’m hoping with 1.6, however, that this may be fixed due to how we’re approaching Mac addresses.

[Foglalt]

@Tom-Elliott nothing special i would do, just like looking up missig primary mac host and killing the orphans so practically i would move my garbadge collection scripts to database. Maybe i put some authentication before it and let others use it, colleaguea i mean. So nothing new at all.

[Wayne Workman]

@foglalt You could alter the script you have for detecting them to fix them. FOG wouldn’t touch cron or the script.

[Foglalt]

The plan is that i will make some extra scripts for my colleague and for some logging, and if all the goes to database as stored procs, maybe script to detect ajd delete disabled ones go there too. But deleting is what i like to have more control, dont want make fixing with deletion with script.

But… As a second thought, what if we check if missing ones have actual mac address but flag is missing only for primary mac. If i set primary mac back to 1 nstead of simply deleting the host as before, will they come back “online” maybe?

[Wayne Workman]

@foglalt said in Disappearing hosts from host list:

But… As a second thought, what if we check if missing ones have actual mac address but flag is missing only for primary mac. If i set primary mac back to 1 nstead of simply deleting the host as before, will they come back “online” maybe?

Sounds reasonable to me. Try it.

[Foglalt]

Normally the issue happens when a host mac is updated (primary mac field is selected with ctrl+a, then entered the new mac). we never use intentionaly more mac on a host (and they dont have any more interface, not like laptops with wifi, eth, whatever simultaneously). How can a host have secondary mac in database? When i first saw such i asked colleagues how they did, and some faint memory i have about it whuch tells when updated, the previouly overwritten mac becames secondary. But it is far from intentional, those are pc-s, with 1 mac only.

Can it be a clue to the investigation…?

like this:

MariaDB [fog]> select * from hostMAC where hmHostID=37;
+------+----------+-------------------+--------+-----------+-----------+----------------+-----------------+
| hmID | hmHostID | hmMAC             | hmDesc | hmPrimary | hmPending | hmIgnoreClient | hmIgnoreImaging |
+------+----------+-------------------+--------+-----------+-----------+----------------+-----------------+
|  170 |       37 | 8c:89:a5:53:1b:f5 |        | 0         | 0         | 0              | 0               |
|  298 |       37 | 6c:4b:90:4e:53:9e |        | 1         |           | 0              | 0               |
+------+----------+-------------------+--------+-----------+-----------+----------------+-----------------+

As you may see, the primary mac has a strange difference from additional (298 is the primary, 170 is the additional mac id). why it has no zero at pending and why has the other a zero? That field is not manually edited, so the software logic updates that field. But why differently? I still have a feeling that this issue is connected with “pending mac”…

[Tom Elliott]

@foglalt no the pending Mac is not the issue. It looks up pending macs based on a 1 on the pending.

The issue, it seems, is a primary Mac is changing for a host. The primary should move to additional macs and the new primary should become primary. However the move is probably happening before setting the new primary Mac causing fog to think the host is no longer valid.