@Tom-Elliott

I haven’t got to the imaging portion yet, but I do get this error now:

new_error.txt

2 Nodes, 1 Master. Both are set at 10 connections.

@Tom-Elliott said:

If you don’t mind opening a separate thread please regarding this now new issue. With the new thread can you post the fog servers Apache error logs?

No problem: https://forums.fogproject.org/topic/5822/checking-in-error-on-git-7428

Thanks for your help again!

When attempting to image, we get an error that simply states “Checking In…” with a counting timer that just keeps going up.

Per Tom’s request, I’ve included the last few entries of log files from httpd’s error_log .

Hey Tom! Thanks for helping with this. We dont get that error anymore and now get a “Checking in… that just keeps counting” Any ideas on that? I rebooted FOG as well.

Using github’s 4724 version, we are no longer able to image. PXE boots, we choose quick image, and the next screen tells us: no network interfaces found

This worked up until the upgrade. Just curious what may have caused this. My snapshot didnt save properly somehow, so I don’t have the ability to go back unfortunately.

Just curious if anyone has any ideas on where to start.

EDIT:
Forgot to include that it is CentOS7 and worked fine prior to the upgrade. I’m trying to figure out which version we had before.

@Wayne-Workman said:

Sorry, I’m a touch confused now. My logs are empty in the /var/log/httpd directory. I even cleaned them out too just to be safe, still only about 10MB.

Or, were you just mentioning about the log files in general?

@Marcus-Allen said:

proc/kcore

Got it

@Wayne-Workman

15773912        ./var/www/html/fog/service/ipxe/init.xz
15775828        ./var/www/html/fog.prev/service/ipxe/init.xz
18102074        ./boot/initramfs-3.10.0-229.el7.x86_64.img
18179668        ./boot/initramfs-3.10.0-229.el7.x86_64kdump.img
18874368        ./var/lib/mysql/ibdata1
24281088        ./var/lib/rpm/Packages
39835970        ./boot/initramfs-0-rescue-e248d8e9ddcb47a99cdece8ee62fabc5.img
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1
67108864        ./sys/devices/pci0000:00/0000:00:0f.0/resource1_wc
140737486266368 ./proc/kcore

@Wayne-Workman

I’m bumping up to 64gb. This is a FOG Only server, no images are stored here.

Thats correct. Its a VM, I can increase the disk space if needed, however last week I was using about 11GB, so I didnt bump it up yet.

Filesystem               Size  Used Avail Use% Mounted on
/dev/mapper/centos-root   30G   30G  303M 100% /
devtmpfs                 911M     0  911M   0% /dev
tmpfs                    921M     0  921M   0% /dev/shm
tmpfs                    921M  8.5M  912M   1% /run
tmpfs                    921M     0  921M   0% /sys/fs/cgroup
/dev/sda1                497M  120M  378M  24% /boot

@Wayne-Workman said:

du -a -h / | sort -n -r -h | head -n 10

sort: options '-hn' are incompatible

@Wayne-Workman

I dont think that is working right:

du -a -h / | sort -n -r | head -n 10
1020K   /usr/lib/kbd/consolefonts
1016K   /usr/share/doc/sudo-1.8.6p7
1016K   /usr/lib64/python2.7/site-packages/gi
1012K   /usr/share/pki/ca-trust-source
1012K   /usr/share/pki
1012K   /usr/lib/udev/hwdb.d/20-usb-vendor-model.hwdb
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x/bnx2x.ko
1012K   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers/net/ethernet/broadcom/bnx2x
1008K   /usr/lib/firmware/cxgb4
1004K   /var/lib/mysql/mysql

Is that re right spot to put that switch in?

@Wayne-Workman

Here is the output:

1621736 /
975920  /usr
404516  /var
296068  /usr/share
268040  /var/lib
225724  /usr/lib
199072  /var/lib/php/session
199072  /var/lib/php
167288  /usr/lib64
155356  /usr/bin

Here is the /usr folder

975920  /usr
296068  /usr/share
225724  /usr/lib
167288  /usr/lib64
155356  /usr/bin
113044  /usr/lib/modules/3.10.0-229.el7.x86_64
113044  /usr/lib/modules
110424  /usr/lib/modules/3.10.0-229.el7.x86_64/kernel
99992   /usr/share/locale
78648   /usr/lib/modules/3.10.0-229.el7.x86_64/kernel/drivers

Unfortunately I have tried that and the apache logs are VERY small right now.

total 16K
-rw-r--r-- 1 root root 8.8K Jul 20 14:31 access_log
-rw-r--r-- 1 root root  997 Jul 20 14:30 error_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_access_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_error_log
-rw-r--r-- 1 root root    0 Jul 20 14:30 ssl_request_log

I did go ahead and delete them, then reboot and still no luck.

So, I came into work today and the fog service was stopped. Rebooted the machine and saw the message that it couldnt write to the disk.

Result of df

Filesystem              1K-blocks     Used Available Use% Mounted on
/dev/mapper/centos-root  31340544 31015780    324764  99% /
devtmpfs                   932704        0    932704   0% /dev
tmpfs                      942348        0    942348   0% /dev/shm
tmpfs                      942348     8608    933740   1% /run
tmpfs                      942348        0    942348   0% /sys/fs/cgroup
/dev/sda1                  508588   122016    386572  24% /boot

Result of du -sh

du: cannot access ‘./proc/12278/task/12278/fd/4’: No such file or directory
du: cannot access ‘./proc/12278/task/12278/fdinfo/4’: No such file or directory
du: cannot access ‘./proc/12278/fd/4’: No such file or directory
du: cannot access ‘./proc/12278/fdinfo/4’: No such file or directory
1.6G    .

I cannot figure out where these “files” are.

@Tom-Elliott Gotcha. We’re just imaging like crazy and that is creating a lot of these errors which filled about 12GB of logspace up. I’ve adjusted my logrotation (which is probably the better answer anyways).

I noticed my Apache error_log was getting large so I checked into it. I’m getting a large amount of these same errors from every machine that is imaged.

Here is the last few lines:

[Thu Jul 16 08:47:11.413272 2015] [:error] [pid 2828] [client 10.1.33.158:37609]                             PHP Warning:  in_array() expects parameter 2 to be array, null given in /var/ww                            w/html/fog/service/auto.register.php on line 73
[Thu Jul 16 08:53:13.914835 2015] [:error] [pid 2883] [client 10.1.33.159:56712]                             PHP Warning:  in_array() expects parameter 2 to be array, null given in /var/ww                            w/html/fog/service/locationcheck.php on line 3
[Thu Jul 16 08:53:28.964064 2015] [:error] [pid 2864] [client 10.1.33.159:56714]                             PHP Warning:  in_array() expects parameter 2 to be array, null given in /var/ww                            w/html/fog/service/auto.register.php on line 73

CentOS7, SVN 3791

The ONLY issue I’ve ever had was with Spanning Tree. I enabled Rapid Spanning Tree Protocol, and the issue went away.

I ended up actually disabling STP on the ports used by fog. I really would just disable STP on those affected ports and see if that fixes it. I bet it does.