Best posts made by Quazz

@zaboxmaster said in Need to create a While loop on deployment for when an image has not been assigned:

while the $OSID = null then sleep 200

That’s because you’re setting the $OSID to null in your while loop, so it will never end.

A single = means set value to

Double = (==) means compare strings

In bash, if you want to check if a variable is ‘null’, you want to do something like (not sure if syntax is 100% correct) (-z checks if the variable is null)

while [[ -z "$OSID" ]] 
do 
        sleep 200
done

@Tom-Elliott As far as I understand, this problem is occurring specifically on NVME SSDs

@loutrage Looks the resize test fails, which automatically marks the partition as fixed size.

Possibly the filesystem or disk is damaged causing the failure.

Try and fix the filesystem by running chkdsk C: /f on the windows client you want to capture. (you’ll then need to confirm to schedule on reboot and then ofc reboot)

@george1421 Regardless, partclone is much slower for him than even this dd test. In the github issue thread I linked, they hypothesize that dd uses O_DIRECT and partclone does not, hence the difference.

It’s also worth noting that they had better speeds when the target was mounted, which live Ubuntu does automatically when it can.

Of course this is working under the assumption that it’s the exact same issue.

@tech49 Please make a new thread for the new issue so we can help you there. (since it’s unrelated to the thread)

@Duncan This suggests that it is indeed a kernel issue. Interesting that it ran at all with the newer inits since they’re only slated for backwards compatibility to 4.14 I believe.

My best guess at the moment is that the APST feature introduced in 4.10 is either the problem in its entirety or related to it somehow.

It’s still building, but when it’s done, there will be an init available at https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/107/artifacts

EDIT: That build failed due to unrelated error, here is a different link https://drive.google.com/open?id=1u_HuN5NSpzb7YmQBAsrzDELteNmlWUWU

This will include an NVME cli utility that will give some info and allow some management over the NVME device.

I’d be interested in seeing a debug deploy on this init (use kernel 4.19 as well). If you could schedule one for a problematic host and run the following commands that would help a ton.

sudo nvme get-feature -f 0x0c -H /dev/nvme0

That will list out some info, the one I’m interested in is whether APST is enabled or not.

If it’s enabled you can disable it by doing

sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

Then type

fog

Press enter (you’ll have to do this a couple more times until it starts partclone and such)

I’m hoping that this will resolve the issue entirely and if so we can add to the inits if an NVME device is detected. APST is unneeded in FOS environment since we don’t care about power consumption of the storage device since it just needs to get captured or deployed and then the system takes it from there.

To elaborate, I recall acpi=off being useful for rather old machines (who’d often have buggy implementations), but the exact opposite for newer machines who rely upon ACPI.

@Sebastian-Roth Yes, glibc is generally the problem in kernel mismatches such as these, the question is where exactly? I’m not sure, but could it also say broken pipe if it gets interrupted unexpectedly by its piped data throwing a critical error such as say… kernel too old?

edit: Looking through build logs, ZSTD seems to be getting compiled against system gcc atm, at least from what I can tell. But OP is using pigz, so uhhhh

edit2: On the note of pigz, Buildroot provides a version these days, so we could remove the manually specified package unless we are attached to version 2.3.4 for some reason

edit3: Perhaps the workspace wasn’t cleaned properly for the build? Or alternatively, perhaps the GCC version has to be lowered.

To clarify -a0 disables the checksum; junkhacker in the original thread made a small mistake with -aX0 as the X was a placeholder that needed to be replaced with either a 0 or 1 for it to work. It shouldn’t cause any issues and removing it will break compatibility with images made under older versions of partclone.

partclone.dd not having a in its short options is actually a bug in my opinion as per

https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L352 clearly listing several options that are for some reason not included in the sopt list for dd.

That said; for dd specifically it turns out we can ignore that potential bug and remove a0 as per https://github.com/Thomas-Tsai/partclone/blob/master/src/partclone.c#L402 setting the checksum_mode to 0 anyway.

And you are right that c is not an option for dd (anymore); I’m guesssing it was removed since a raw copy is always a clone so it makes no sense to specify it.

All that said, I think proper handling of LVM as per what Tom said should be the way forward.

@cmachado The integer is overflowing on the node because 32bit has a 2GB limit for integers, but the filesizes exceed that number.

If you get a 64 bit node your problems will disappear I reckon.

@ty900000 That build does not yet include the partclone fixes, though. But we are getting close to that.

@Sebastian-Roth He’ll need newer inits that include the NVME package since it wasn’t included in the 1.5.7 package iirc.

@banana123 Looks like your first partition is incorrectly identified as being resizable, which is likely at the root of your problems.

Though, it looks like that partition has been altered in some way.

Currently, FOS uses partition flags to attempt to identify “non-resizable” partitions.

As far as I can tell, your first partition is missing essential flags (namely System and Active), causing FOS to misidentify the partition as a normal NTFS data partition that should be resized.

@kafluke You may have to defragment the drive and/or run chkdsk on it if you’ve deleted a significant amount of data.

@scosta You can rearrange partitions. I would use something like macrium reflect to create a clone to an identical disk where I manually arrange the partitions, you can then use that disk for capture in FOG, once that is done, you can deploy it to the original disk and see if everything works.

There’s probably other ways or even faster ways, but messing with partition tables can be scary.

@mp12 Would it be possible to test the binaries between 1.5.3 and 1.5.7 (so 1.5.4, 1.5.5, 1.5.6)?

This will help us track down roughly when the problem was introduced. (as there is about 2 years between 1.5.3 and current dev-branch I believe)

@george1421 Relevant github issue for further discussion and thoughts on MyISAM -> InnoDB specifically: https://github.com/FOGProject/fogproject/issues/370

@Sebastian-Roth I mean, this seems to be the very same problematic code (or closely related to) that has already been given us various issues, primarily low speeds, which was subsequentially partially (or fully for some disks) addressed by setting the latency kernel parameter to 0 by default.

That said, as we then discovered, in some cases it is not sufficient and we had to disable ASPM using the NVME cli utility for them to work normally. I don’t believe that was ever integrated into FOS since we don’t fully know if this could cause issues in otherwise properaly working drives.

For NVME disks specifically it’s APST, a subset of ASPM.

sudo nvme set-feature -f 0x0c -v=0 /dev/nvme0

That line should disable it (assuming disk name is nvme0)

@Sebastian-Roth Perhaps not for the MBR issue; but certainly for the resize issue.

@Sebastian-Roth Been very busy and don’t really have a great test setup right now.

Ran a quick and dirty test and get roughly 20% speed bump from the provided init over the one on the dev-branch. (ignore the exact numbers, it’s not very scientific, it’s a significant enough difference at least)

Seems to be roughly what was reported in here by OP from what I can tell.

I don’t have 2.89 images to test to see if they do or don’t work, however. That will be key.