Not displaying images

Cire3

Ok, long story short… Had fog 1.2.0 in Ubuntu 13.10 server, working well under ESXI as a VM with 2 vdisk. I had a 16Gb along with 4Tb running on 4 X 3Tb RAIOD0 drives that I used for images ( /dev/sdb1 mounted to /images )

Before anyone freaks out seeing RAID0, I also backup to 4Tb running on another 4 X 3Tb RAID0. To me it’s all about the speed if you can afford the hardware. I would risk losing one image in worst case if I had a new image during the day with a disk failure after, and I’m totally good with that. I backup nightly to a separate DAS

Started having issues with an SVN upgrade, just wouldn’t work… Cleanup SVN issues, etc… So I tried an upgrade with Ubuntu 13.10 to 14.04, as I was reading about issues with earlier versions of fog that had been upgraded. I started with 1.0.0 and moved my way up slowly as I don’t touch it unless I have an issue of a feature is new that I need. FIgured Windows 10 may change things, so lets get updated.

Anyway, upgrade went to hell in a hand bag, php5 failed to update locking up the server. Forced to restart it ( Crash it basically in an upgrade ) This caused it to not boot, block error’s etc… Followed everything with these block error’s and just decided to reload new.

I formatted the 16Gb vdisk and just installed Ubuntu 14.04 lts x64 server. Now I connected the second vdisk of 4Tb, mounted it to images after renaming it to images.old. Copied everything from images.old to /images after I mounted it.

Steps to add the disk are as followed.

sudo mv /images /images.old
sudo mkdir /images
sudo mount -t ext4 /dev/sdb1 /images
mount
sudo cp -av /images.old/* /images/
sudo touch /images/.mntcheck
sudo nano /etc/fstab
sdb1 /images ext4 defaults 0 0
sudo chmod 777 -R /images

Now it rebooted and mounted fine, I can go to images and see then in there ( 1.8Tb’s of images ) This was the drive I used with my earlier version of fog 1.2.0. From my understanding fog just list what it see’s in the /images folder if I’m not mistaken ?

Any way to force fog to index this location and allow my images to be added ?

My images page show’s no images uploaded, but again I can see them listed in /images ?

Any help would be great, and again I’m still loving Fog !

Restarted the server just to make sure it mounted.

Joseph Hales

You can recreate each image manually thru the web interface or you can import you mysql database for the same info and it would restore your hosts and images.

Cire3

Now I remember, thanks a bunch ! Forgot I been here before…

However my original issue reared it’s ugly head, and I’m shocked on the new build to see it.

Error on deployment : gunzip: invalid magic (in black) then offset (in blue) This is not a partclone image ?

Umm, it is a partclone image ? If I select the old style it still doesn’t work.

I’m not understanding why all of a sudden I would have this issue ? Can’t imaging something with my images ?

I did change my drive size from 2Tb to 4Tb, would that change the structure of the image somehow with the way fog see’s it ?

Tom Elliott

I think we need to know exactly what version of fog you’re running first. There a couple issues that I’m sure have been fixed. This also depends on what’s happening on the disks. 2tb worth of images takes quite a while to copy between disks and even other folders. Is it possible the copy/transfer got interrupted?

Cire3

@Tom-Elliott

Sorry, version is 4353 as of a little bit ago. bzImage Version: bzImage32 Version: 4.1.3

Ubuntu 14.04 Server x64, updated.

While waiting ( I know slap my hands now ) I upgraded using SVN. As well as upgrading the Kernel hoping it would fix it.

I have issue " pigz: skipping: <stdin> is not compressed This is not a partclone image.

The thing is I just expanded the second disk, it’s the same vdisk as what was running my install on 13.10. So I wouldn’t assume anything with the images. So no transferring, that would have been my first thought as well.

I just dropped the 2nd disk in ESXi, reloaded on my 16Gb 1st disk. After install I reconnected the second drive. As I said, I did expand the drive on the 13.10 install. And that’s now the disk I’m using on the 14.04 install.

Figured if I just reinstalled it fresh on 14.04 I would be up to date and kicking. Now just kicking myself…lol

Thanks !!

Wayne Workman

@Cire3 So is your issue solved? What’s the problem?

Cire3

@Wayne-Workman

Wow, that’s strange… I know I posted… Anyway I thought I did

While I waited I upgraded using SVN, upgraded Kernel as well ( I know slap my hands now )

version: 4535 bzImage32 Version: 4.1.3 Ubuntu 14.04 x64 Server

The disk is the same as before, I only expanded it. It was a 4Tb vmdk, but I ran it as 2Tb because I thought it would be forever till I filled it. Well, forever came…

I just dropped Disk2 when I reloaded Ubuntu on Disk1 ( 16Gb ). After updates, upgrades, installed Fog 1.2.0

Added my 4Tb drive and mounted it to /images. Now my issue is I’m not sure if my expansion using resize2 messed something up ? I can’t imaging, but I seen stranger things… So I transferred no images, same vmdk as 13.10

Current error is now pigz: skipping: <stdin> is not compressed This is not partclone image.

Do you think expanding the disk could change the way it’s seeing the images ?

Wayne Workman

@Cire3 said:

Do you think expanding the disk could change the way it’s seeing the images ?

It’s possible if you didn’t start the partition exactly (and I do mean absolutely exactly) at the same sector as it was before. But normally, when the start sector changes, no data is accessible at all from the partition. So this makes me think that the start sector is the same… but something must have happened obviously between working and non working. Tom’s idea below is very likely.

I’m hoping you have some sort of backup still on that “DAS” that you spoke of…

I think we should start over from scratch with this storage expansion. Confirm your backups are good. After that, just unmount this virtual hdd and just delete it totally, start over clean with the actual size that you want to have from the git-go. Then copy your images from backup at the FILE LEVEL - I don’t know what that will entail for you, but you need to move the files themselves to the new partition, not restore partitions or snapshots or other such things that overly complicate the process more than it needs to be and is likely the very thing that broke your images.

Following a more basic method like I’ve described will almost guarantee a successful storage expansion. Naturally, after all the above is done, you still need permissions, the dev folder, the mntcheck files and so on.

I’m willing to give assistance in the PM hours after 5:00 PM (UTC -06:00), or on Sunday, just chat me up.

Cire3

@Wayne-Workman

I have Veeam as well as vSphere VDR, I should be able to use Veeam for files only. I was hoping something stupid I was overlooking ( I can hope ) However I would have to agree at this point something has to be off. I thought when I first set it up I would be golden, not realizing that fdisk and mbr has the 2Tb limit. And that was fine for a while, but I seem to fill that rather quick. Now with Windows 10 my images will likely double, so I’m shooting for 6Tb I’m thinking.

Or maybe better yet a couple 2Tb vdisk ? So that way if one messes up I can just move 2Tb worse case, because I know 2Tb alone will take most of a day just to move. Even though ESXi is all in one box, it still acts like it’s transferring through a 1Gb switch. Not disk to disk.

Anyway, many thanks for your time and effort ! I’ll pay better attention and make sure I’m using my full disk before I ever start loading images

Again thanks for your time, and Fog !

All the best !

Wayne Workman

@Cire3 Please follow up when you can, we want to know if you were successful.

SaxxAppeal

This post is deleted!

SaxxAppeal

I am experiencing the same issue!

Just moved 2 images from my FOG 0.32 server to my 1.2.0 server, changed the image type to PartImage in the web GUI.

“gunzip: invalid magic”

I verified permissions on my images folder and subfolders. Ideas, anyone?

Ubuntu 14 Server, FOG 1.2.0

Wayne Workman

@SaxxAppeal said:

I am experiencing the same issue!

Just moved 2 images from my FOG 0.32 server to my 1.2.0 server, changed the image type to PartImage in the web GUI.

“gunzip: invalid magic”

I verified permissions on my images folder and subfolders. Ideas, anyone?

Ubuntu 14 Server, FOG 1.2.0

Is your .32 server still running?

SaxxAppeal

@Wayne-Workman

my .32 server is still running, but not well. Once I got the 1.2.0 server running properly, I experimented with a few things and broke it…

I can get the FOG GUI up, though. Worst case scenario, I’ll try to undo the damage on my .32 server and deploy, then upload to the new server… But I didn’t want to do that if I didn’t have to.

Wayne Workman

@SaxxAppeal can you compare the file sizes of the image on your .32 server to the size of the image on the 1.2.0 server?

Can you post the output from both servers here?

The command will be something like this on both:

ls -lahRt /images

SaxxAppeal

@Wayne-Workman said:

ls -lahRt /images

The file sizes look to be the same. I have many more images on the old server than I do on the new server, but the particular one I am having trouble with is “760AuditModeSummer15”

OLD SERVER:

/images/760AuditModeSummer15:
total 13G
drwxrwxrwx 62 fog root 4.0K 2015-07-01 13:35 …
-rwxrwxrwx 1 root root 13G 2015-06-23 13:30 d1p2.img
drwxrwxrwx 2 root root 4.0K 2015-06-23 13:08 .
-rwxrwxrwx 1 root root 8.5M 2015-06-23 13:08 d1p1.img
-rwxrwxrwx 1 root root 512 2015-06-23 13:08 d1.mbr

NEW SERVER:

/images/760AuditModeSummer15:
total 13G
-rwxrwxrwx 1 root root 512 Aug 10 16:48 d1.mbr
drwxrwxrwx 2 root root 4.0K Aug 10 16:48 .
-rwxrwxrwx 1 root root 8.5M Aug 10 16:48 d1p1.img
-rwxrwxrwx 1 root root 13G Aug 10 16:48 d1p2.img
drwxrwxrwx 24 root root 4.0K Aug 10 16:42 …

Thanks for your help!

Cire3

I just hate when someone doesn’t post what they ended up doing… Oh wait ! That was me !

Anyway, I went the route easiest to flow… I can be lazy like that.

I has an issue with the drive. For whatever reason it mounted, expanded, did what it should. However something was goofed in the expansion so it was of no use. Partclone didn’t see any images when it came time to deploy ( Not at all a fog issue )

I had a backup clone of Fog with the same 1.8Tb of data. One of the many reasons I LOVE ESXI.

Changed the IP address, fired it up ( Both VM’s at once ) My images are on a separate disk ( separate vmdk, basically like using 2 hard drives )

So I just dropped the bad disk on the new build in vSphere, created a new disk of the size I needed this time. Created the partition with the correct tools for GPT. Formatted it ext4, and then was time for some Linux magic

Now with the clone and new build running, I just ran rsync keeping the permissions on everything. Went home because even on a DAS to DAS , 1.8Tb is a lot of freaking images

Woke up and all is well. Updated to Trunk 4443. Now it was time to test.

What I was shocked at was the speed… I would have done this a long time ago if I knew how freakishly fast this was going to be !

Client : HP Elite 8200 Desktop CMT, i7, 8Gb, 120 SSD. I build my master images on SSD’s

I start @ 16GB/min and settle down at about 14.5GB/min download. ( Single Disk Resizable / Compression @ 6 )

I upload starting @ 5GB/min and soon hold about 6.5GB/min

On ESXi I’m a happy camper, I thought the numbers were wrong at first.

Again big thanks to the Fog project and all that help !

Wayne Workman

@Cire3 said:

What I was shocked at was the speed… I would have done this a long time ago if I knew how freakishly fast this was going to be !

I hit on this point all the time. People never believe just how large of an improvement there is.

I’m happy that you got it going, and pleased that I helped make that happen.

-----OP reported success - thread solved.

Not displaying images

287

11.7k

17.1k

154.7k