Error on image upload

[Zourous]

Hi Sebastian,

Many thanks for your speedy reply at such short notice. Much appreciated as I’m on a short timescale to get a task completed. I think I might have solved it, but your post gave me a clue. Firstly I tried the fix in your follow up post about updating the init files and ramdisk size but this didn’t solve it. I then had a look at the partitions with the debug task. It showed up a USB stick I had connected as well as the fixed drive I wanted to image and this is where the issue is. I tried again with the USB out and it worked fine and as expected. I even ran the task again with the USB inserted just to prove it and yes the issue was back again. This is most strange as I swear I’ve uploaded before with a USB stick in and assumed the process would see it and just ignore it. I quite often use the USB stick just before upload to run my sysprep answer file from. Anyhow I’m happy I know the answer. Thanks again.

[Sebastian Roth]

@Zourous Thanks heaps for letting us know. Definitely something I wouldn’t have thought of!

Would you do me a favor? Please run another debug upload task with the USB key plugged in and when you get to the shell run the following commands, take pictures and post here:

parted -l /dev/sda
parted -l /dev/sdb

My guess is that the USB key is first in the list (sda) and therefor we run into an issue.

@Quazz Maybe we should add a check for removable drives and warn the user about this.

[Quazz]

@Sebastian-Roth Good idea in general, though, not entirely sure how to go about that at the moment.

[Sebastian Roth]

@Quazz I think there is a flag for removable devices in /proc or /sys… Will have to look that up though.

[Zourous]

@Sebastian-Roth said in Error on image upload:

@Zourous Thanks heaps for letting us know. Definitely something I wouldn’t have thought of!

Would you do me a favor? Please run another debug upload task with the USB key plugged in and when you get to the shell run the following commands, take pictures and post here:

parted -l /dev/sda
parted -l /dev/sdb

My guess is that the USB key is first in the list (sda) and therefor we run into an issue.

@Quazz Maybe we should add a check for removable drives and warn the user about this.

This is after the command parted -l /dev/sda. You see the USB stick at the bottom. If I run parted-l /dev/sdb I just get the USB information on it’s own.

It looks like the USB stick was bootable at some point in a previous use (I didn’t realise this before) so maybe that it where the confusion is and Fog automatcially adds it to the image to upload?

[Sebastian Roth]

@Zourous I am not exactly sure I understand this yet. My guess was that maybe the USB key turns up as sda. Maybe that happens randomly but it’s very unlikely (now that I think a bit more about it) as SCSI/ATA disk subsystem devices usually get enumerated before the USB stuff. But if you have a couple of minutes you could do the whole test for maybe 10 times (reboot into debug task again and run both parted commands to see if at some point the USB drive shows up as sda).

It looks like the USB stick was bootable at some point in a previous use (I didn’t realise this before) so maybe that it where the confusion is and Fog automatcially adds it to the image to upload?

If the image is set to “single disk…” then FOG simply grabs the first disk. So if it’s randomly changing (tests mentioned above) that might explain things. The boot flag shouldn’t play a role as far as I know.

Looking at the first picture you posted again I see that it tries sda1 and sda2 and so it’s very unlikely the USB key. But still we are very sure this was causing it. So possibly some command we use in the scripts is just being confused by the USB key.

[Quazz]

@Sebastian-Roth Perhaps it’s not so much that the USB stick is first, but rather that the command that check the partition flags fails when it’s plugged in for some reason.

Although, I’d have no clue how that would even possible since the parted command explicitly is ran against a single disk, which doesn’t appear to be the USB stick anyway.

And the command shouldn’t fail on it regardless as far as I can tell.

edit: Given the line throwing the error, I can only assume he’s using older init files (since there’s no test command at line 109 at latest)

I can see this line throwing such an error under certain conditions:

if [[ $(parted -l "$hd" | grep boot | awk '{print $1}') -eq $part_number || $(parted -l "$hd" | grep msftres | awk '{print $1}') -eq $part_number || $(parted -l "$hd" | grep hidden | awk '{print $1}') -eq $part_number ]]; then

which was changed at https://github.com/FOGProject/fos/commit/693a4f74d81fad5eba09799768c53f25b7c50e3c#diff-9c3fc2c5bb45d13976e29f477b8e67d4

[Quazz]

@Zourous What FOG version are you running?

[Zourous]

@Quazz said in Error on image upload:

@Zourous What FOG version are you running?

I am running FOG version 1.5.5

Let me know if there is anything else you want me to test?

[Quazz]

@Zourous Try updating to 1.5.6, it should include the newer init files which I believe should not run into this error, but should be tested of course.

[Sebastian Roth]

@Zourous Would you mind doing the test I suggested (10 times in row to see if USB key is always sdb)?

[Zourous]

@Sebastian-Roth said in Error on image upload:

@Zourous Would you mind doing the test I suggested (10 times in row to see if USB key is always sdb)?

I tried it 10 times and it was the same each time. I even tried an 11th time with another stick and it was the same. I’m guessing you could probably replicate this on a similar set-up.

[Zourous]

@Quazz said in Error on image upload:

@Zourous Try updating to 1.5.6, it should include the newer init files which I believe should not run into this error, but should be tested of course.

Sorry Quazz, it’s not something I have time for or want to do at the moment. I did test some newer init files that were posted earlier in this thread and I assumed these were the newest ones.

[Quazz]

@Zourous Did anything change with the newer init? Different line number or slightly different error?

[Sebastian Roth]

@Zourous said in Error on image upload:

I tried it 10 times and it was the same each time. I even tried an 11th time with another stick and it was the same. I’m guessing you could probably replicate this on a similar set-up.

Thanks! I’ll definitely look into this when I have a bit more time.

[Zourous]

@Quazz said in Error on image upload:

@Zourous Did anything change with the newer init? Different line number or slightly different error?

I will try and test when I have a chance maybe next week

[Zourous]

Also I have just noticed a new problem off the back of this problem. The USB stick got uploaded into the image at some point. I noticed this and used a tool to take the extra USB partition away again from the image, but after this the images uploads and shows the size as the full space of the fixed disk and also doesn’t resize to a bigger disk when it is deployed to a bigger disk. I’m not sure how I can correct this issue now.

[Sebastian Roth]

@Zourous I am fairly sure I found why you ran into this issue. I was absolutely sure I had seen something very similar some weeks ago but just have not had the time in the last days to sit down and play with this stuff to figure it out.

@Quazz You might remember this last fix. We had a very similar if not the same error message in situations with single drives that have the same flag set on different partitions. In both cases the problem is caused because a variable is being set with a multiline result and that causes the if clause to fail with syntax error.

The problem here is that parted -l /dev/sda does not return the information for sda only but sda and sdb as stated in the parted man page: “-l, --list lists partition layout on all block devices”.

So I have pushed out a fix for this. Inits are building. You can manually download the latest init files when they are done in 3-4 hours: https://dev.fogproject.org/blue/organizations/jenkins/fos/detail/master/90/artifacts/ (init.xz and init_32.xz)

Put those in /var/www/html/fog/service/ipxe/ - rename the original ones to have a backup copy of those. As well go to the FOG web UI -> FOG Configuration -> FOG Settings -> FTP Server and increase KERNEL RAMDISK SIZE from 127000 to 275000 if you haven’t done that already.

@Zourous said:

I noticed this and used a tool to take the extra USB partition away again from the image

I am not sure I get this. Which tool did you use to “take away” the USB partition? Do you mean you removed the partition from the USB key before actually uploading or after upload finished?

About the “doesn’t resize”… We need more information on this! Partition layout (d1.partitions from the image directory) and so on.

[Quazz]

@Sebastian-Roth Ahhh, that makes sense, good find!

[Zourous]

@Zourous said:

I noticed this and used a tool to take the extra USB partition away again from the image

I am not sure I get this. Which tool did you use to “take away” the USB partition? Do you mean you removed the partition from the USB key before actually uploading or after upload finished?

About the “doesn’t resize”… We need more information on this! Partition layout (d1.partitions from the image directory) and so on.

I just used a tool outside of Windows to remove the inserted USB partition that had made it’s way onto the main image. By doing this I assume it has messed something up and that the partitions are not recalculated when I try to upload the image again because it seems to not shrink the image to the size of the data before upload or then resize it after deployment. Here is the d1.partitions file

label: dos
label-id: 0x86308630
device: /dev/sda
unit: sectors

/dev/sda1 : start=        2048, size=     1024000, type=7, bootable
/dev/sda2 : start=     1026048, size=   249043631, type=7

I noticed there is a d1.fixed_size_partitions file and I tried taking the :2 out of this, but this didn’t solve it.