Posts made by dolf

dolf

OK so while the imaging runs, I’m reading code.

CloneZilla saves the partition like this (from the log file clonezilla-img

partclone.ntfs -z 10485760 -N -L /var/log/partclone.log -c -s /dev/sda2 --output - | pigz -c --fast -b 1024 -p 16 --rsyncable | split -a 2 -b 4096MB - /home/partimag/2016-07-12-21-img-eelabtest/sda2.ntfs-ptcl-img.gz. 2> /tmp/split_error.TMVF3J

Fog does it like this (from savePartition() in src/buildroot/package/fog/scripts/usr/share/fog/lib/funcs.sh)

partclone.$fstype -fsck-src-part -c -s $part -O $fifoname -N -f 1

I’m not really sure how the FIFO buffer works, but it’s called /tmp/pigz1, so I guess it’s the same as piping the output of partclone through pigz, which is exactly what CloneZilla does. Therefore, maybe this would be sufficient to “convert from CloneZilla format to Fog format”:

cat sda2.ntfs-ptcl-img.gz.* > d1p2.img

In other words, no recompression is needed. Am I missing something here?

dolf

WOW thanks for the advice. I’ll try that soon.

I think the 1MB space between the partitions is just for alignment.

dolf

CloneZilla files, excluding the large ones.

dolf

Is the partition table manipulated in any way when capturing a non-resizable multiple partition image? If not, it probably works just like CloneZilla, which I’m using now. And that works.

I’ll test your idea as soon as I have time. For now, I’m trying to work around the issue to save time. Working through the night to get the image ready. 200 PCs to deploy soon…

Where can I find the exact ntfsresize command used by FOS? I looked at the fog.upload script while in FOS, and there was a call to shrinkPartition or something like that, but I couldn’t find where the call to ntfsresize happens. I would like to type that exact command on a terminal and see what happens.

dolf

Say I have a few images made using CloneZilla. The folder structure looks like this:

user@myserver:~/partimag$ tree -ash
.
├── [4.0K]  image-name
│   ├── [ 677]  blkdev.list
│   ├── [ 249]  blkid.list
│   ├── [6.3K]  clonezilla-img
│   ├── [ 159]  dev-fs.list
│   ├── [   4]  disk
│   ├── [ 20K]  Info-dmi.txt
│   ├── [ 15K]  Info-lshw.txt
│   ├── [2.0K]  Info-lspci.txt
│   ├── [ 216]  Info-packages.txt
│   ├── [  90]  Info-saved-by-cmd.txt
│   ├── [  10]  parts
│   ├── [  33]  sda1.info
│   ├── [8.8M]  sda1.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.aa
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ab
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ac
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ad
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ae
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.af
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ag
│   ├── [3.8G]  sda2.ntfs-ptcl-img.gz.ah
│   ├── [1.5G]  sda2.ntfs-ptcl-img.gz.ai
│   ├── [  37]  sda-chs.sf
│   ├── [1024K]  sda-hidden-data-after-mbr
│   ├── [ 512]  sda-mbr
│   ├── [ 333]  sda-pt.parted
│   ├── [ 295]  sda-pt.parted.compact
│   └── [ 190]  sda-pt.sf
etc...

FOG Images, however, look like this:

user@myserver:/images$ tree
.
├── an-old-image
│   ├── d1.fixed_size_partitions
│   ├── d1.original.fstypes
│   ├── d1.original.partitions
│   ├── d1.original.swapuuids
│   ├── rec.img.000
│   └── sys.img.000
├── a-newer-image
│   ├── d1.fixed_size_partitions
│   ├── d1.mbr
│   ├── d1.minimum.partitions
│   ├── d1.original.fstypes
│   ├── d1.original.swapuuids
│   ├── d1p1.img
│   ├── d1p2.img
│   └── d1.partitions
etc...

Could I somehow import the CloneZilla images into FOG, without deploying and recapturing the image?

dolf

Just to verify that this is not a hardware issue, I restored the image to another PC using CloneZilla, and tried to capture using FOG. Same results! The resize step totally corrupts the MFT, leaving the PC unbootable.

dolf

@Wayne-Workman What do you mean? It’s right after the previous images, same computer, same everything. The output of lsblk and fdisk -l was taken right after the Volume is corrupt. You should run chkdsk. error.

dolf

dolf

There’s the problem!

The partclone log shows the same thing which I snapped here:

Is the “ntfs flag” the same thing as the “dirty flag”? I could clear it using ntfsfix --clear-dirty /dev/sda2, but I would like to see why it failed in the first place…

But I tried it in any case. First a dry run:

Then the real deal:

With no luck.

dolf

I scanned for bad sectors the day before yesterday, and there were none, so it should be fine.

Resizing the file system in debug mode now. There is no indication of progress, so I’m guessing that is uses ntfsresize, but the output is piped elsewhere? Is the output of ntfsresize logged somewhere?

dolf

I’m on Ubuntu 14.04 though…
Anyway, it’s working with php5.

dolf

Thanks. Will do!
I’m running Fog 8515 on Ubuntu 14.04

dolf

OK, this time I backed everything up using CloneZilla. It failed again, so I’m restoring what I had with CloneZilla as I write this.

I should add that I did a full chkdsk /F /V /X on reboot, followed by a proper shutdown (not hibernate or restart) of Windows, before trying to capture with FOG.

Starting the capture:

I wasn’t fast enough to take a picture of the error in partclone, but I think it was “not an NTFS partition” or something like that. This is what I did get:

I tried my best to stop it, but ye olde computer would not listen.

I’m guessing that /var/log/partclone.log is on the Fog Operating System running on the client, because I don’t find it on the server. I’m just waiting for CloneZilla to restore the disk, so I can try to capture again in debug mode.

Could someone guide me through the basics of debug mode? The wiki only has a tutorial for Windows XP https://wiki.fogproject.org/wiki/index.php?title=Debug_Mode#capture_Debug and even that isn’t very descriptive.

It would be awesome if the entire output of FOS could be logged on the server somehow. Possibly over FTP or NFS? Consider that a feature request. Overall, I find that FOG is good at logging things that went well, but when things go south, it’s quite hard to find good error messages.

dolf

Hello again!

This is what I’m doing:

Deploy a working Win7 image to my test PC
Uninstall some apps. Install others. Delete some files.
Create a new image and capture it.

It takes a while to resize partitions, etc., so I walk away and do other work. When I came back, I saw an error, but I was just to late to read it before it rebooted. There were no files in /images/dev/<MAC>, but the folder was created. This makes me think the capture failed before partclone even started.

It automatically tries again on reboot, because the task still exists in the FOG system. Then I get this error:

Failed to load $MFT: Input/output error NTFS is inconsistent.

I can’t run chkdsk /f on Windows, because Windows doesn’t boot after this. It just reboots forever. No F8 menu, nothing.

I checked folder permissions, NFS and FTP. I also tried installing TinyCoreLinux on another test PC and capturing that. It worked. Deploying also works.

I’m deploying my previous Win7 image once again in the hope that I don’t lose my work a third time.

Questions:

Are the errors from the capture/deploy tasks logged somewhere? Or are they forever lost if I miss the 1 minute gap?
Can I make the 1 minute gap longer?
Could we somehow make the system safer, so that capturing the image does not destroy it? I’ve never had problems with CloneZilla destroying my images. I’d make backups using CloneZilla, but it takes so much time…
Is there any way to fix the partition without booting Windows? I tried with a GParted live CD, and it gives me the exact same error:

dolf

AHA! After purging all forms of PHP and Apache from my server, I re-ran the fog installer. (Discovered some apt issues with old and obsolete packages. Killed those by uninstalling everything that I don’t need. Ran the fog installer again. Then it worked.) Still got errors about mysqliroot and curl_init() being missing, but then I had this idea:

Apache2 is using the php5 module, so it always invokes php5. php5-curl and php5-mysql were installed. But the fog installer also caused php7.0 to be installed. php7.0-curl and php7.0-mysql were not installed. So I checked which version was being used on the command line:

# update-alternatives --config php
There are 2 choices for the alternative php (providing /usr/bin/php).

  Selection    Path             Priority   Status
------------------------------------------------------------
* 0            /usr/bin/php7.0   70        auto mode
  1            /usr/bin/php5     50        manual mode
  2            /usr/bin/php7.0   70        manual mode

Press enter to keep the current choice[*], or type selection number: 1
update-alternatives: using /usr/bin/php5 to provide /usr/bin/php (php) in manual mode

Sure enough, it started working after changing from php7.0 to php5. All services are running, even after a reboot.

I am a happy bloke.

dolf

My bad. I didn’t enable the apache php5.6 module after uninstalling php5.

# a2enmod php5.6
# service apache2 restart

Still didn’t work. It seems like I broke my php and/or webserver. After reading this, I noticed that the php mime config was missing. So I did the brave thing and purges all versions of php and apache from my server. Starting over…

dolf

And then:

# rm /var/run/fog/*

# service FOGImageReplicator status
 * FOGImageReplicator is not running
# service FOGImageReplicator start
 * Starting FOG Computer Imaging Solution: FOGImageReplicator [ OK ] 
# service FOGImageReplicator status
 * FOGImageReplicator is running

# service FOGMulticastManager status
 * FOGMulticastManager is not running
# service FOGMulticastManager start 
 * Starting FOG Computer Imaging Solution: FOGMulticastManager [ OK ] 
# service FOGMulticastManager status
 * FOGMulticastManager is running

# service FOGPingHosts status
 * FOGPingHosts is not running
# service FOGPingHosts start
 * Starting FOG Computer Imaging Solution: FOGPingHosts [ OK ] 
# service FOGPingHosts status
 * FOGPingHosts is running

# service FOGScheduler status 
 * FOGScheduler is not running
# service FOGScheduler start
 * Starting FOG Computer Imaging Solution: FOGScheduler [ OK ] 
# service FOGScheduler status 
 * FOGScheduler is running

# service FOGSnapinReplicator status
 * FOGSnapinReplicator is not running
# service FOGSnapinReplicator start
 * Starting FOG Computer Imaging Solution: FOGSnapinReplicator [ OK ] 
# service FOGSnapinReplicator status
 * FOGSnapinReplicator is running

But after a reboot, they are not running any more:

# service FOGImageReplicator status
 * FOGImageReplicator is not running
# service FOGMulticastManager status
 * FOGMulticastManager is not running
# service FOGPingHosts status
 * FOGPingHosts is not running
# service FOGScheduler status 
 * FOGScheduler is not running
# service FOGSnapinReplicator status
 * FOGSnapinReplicator is not running

All of the logs in /opt/fog/log contain the same thing:

[07-12-16 9:42:11 am] 
        ___           ___           ___      
       /\  \         /\  \         /\  \     
      /::\  \       /::\  \       /::\  \    
     /:/\:\  \     /:/\:\  \     /:/\:\  \   
    /::\-\:\  \   /:/  \:\  \   /:/  \:\  \  
   /:/\:\ \:\__\ /:/__/ \:\__\ /:/__/_\:\__\ 
   \/__\:\ \/__/ \:\  \ /:/  / \:\  /\ \/__/ 
        \:\__\    \:\  /:/  /   \:\ \:\__\   
         \/__/     \:\/:/  /     \:\/:/  /   
                    \::/  /       \::/  /    
                     \/__/         \/__/     

  ###########################################
  #     Free Computer Imaging Solution      #
  #     Credits:                            #
  #     http://fogproject.org/credits       #
  #     GNU GPL Version 3                   #
  ###########################################

Except the service master:

[07-12-16 9:39:16 am] FOGImageReplicator Start
[07-12-16 9:39:16 am] FOGImageReplicator forked child process (23817).
[07-12-16 9:39:16 am] FOGImageReplicator child process (23817) is running.
[07-12-16 9:39:48 am] service_signal_handler (23816) received signal 2.
[07-12-16 9:39:48 am] service_signal_handler (23816) killing child (23817).
[07-12-16 9:39:48 am] service_signal_handler (23816) exiting.
[07-12-16 9:42:11 am] FOGImageReplicator Start
[07-12-16 9:42:11 am] FOGImageReplicator forked child process (24066).
[07-12-16 9:42:11 am] FOGImageReplicator child process (24066) is running.
[07-12-16 9:42:30 am] FOGMulticastManager Start
[07-12-16 9:42:30 am] FOGMulticastManager forked child process (24139).
[07-12-16 9:42:30 am] FOGMulticastManager child process (24139) is running.
[07-12-16 9:42:42 am] FOGPingHosts Start
[07-12-16 9:42:42 am] FOGPingHosts forked child process (24184).
[07-12-16 9:42:42 am] FOGPingHosts child process (24184) is running.
[07-12-16 9:42:54 am] FOGTaskScheduler Start
[07-12-16 9:42:54 am] FOGTaskScheduler forked child process (24241).
[07-12-16 9:42:54 am] FOGTaskScheduler child process (24241) is running.
[07-12-16 9:43:03 am] FOGImageReplicator Start
[07-12-16 9:43:03 am] FOGImageReplicator forked child process (24300).
[07-12-16 9:43:03 am] FOGImageReplicator child process (24300) is running.
[07-12-16 9:46:11 am] service_signal_handler (24064) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24064) killing child (24066).
[07-12-16 9:46:11 am] service_signal_handler (24064) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24136) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24136) killing child (24139).
[07-12-16 9:46:11 am] service_signal_handler (24136) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24181) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24181) killing child (24184).
[07-12-16 9:46:11 am] service_signal_handler (24181) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24238) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24238) killing child (24241).
[07-12-16 9:46:11 am] service_signal_handler (24238) exiting.
[07-12-16 9:46:11 am] service_signal_handler (24298) received signal 15.
[07-12-16 9:46:11 am] service_signal_handler (24298) killing child (24300).
[07-12-16 9:46:11 am] service_signal_handler (24298) exiting.

They fail to start using service FOGxxx start, since the pids still exist in /var/run/fog. Deleting those and starting manually as in the beginning of this post works.

dolf

I took the liberty of first:

Correcting the command: find / -xdev -name "php.ini*" -exec grep "extension" {} /dev/null \;
Uninstalling php5 (since I have php5.6 installed, too)

Here is the output: php_extensions.txt

Then I tried this:

# apt-get install php5.6-mysql
# phpenmod mysqli

It runs! BUT there is another error:

PHP Fatal error:  Call to undefined function curl_init() in /var/www/html/fog/lib/fog/fogurlrequests.class.php on line 49

But it didn’t stop running, and a log file appeared at /opt/fog/log!

dolf

Eureka! It says:

Missing Extensions: mysqliroot

dolf

/images can hold everything, for now. I removed the second storage node and storage group. I’ll just move the older images to the extra drive and symlink them to /images/…

So according to @Quazz , this is not supported. Fine by me. It seemed like a logical use case to me, so maybe we should say somewhere in the Wiki that this is not supported?