• Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login
  • Recent
  • Unsolved
  • Tags
  • Popular
  • Users
  • Groups
  • Search
  • Register
  • Login

No startable FOG processes and empty DB backups after FOG upgrade

Scheduled Pinned Locked Moved Solved
FOG Problems
3
16
2.7k
Loading More Posts
  • Oldest to Newest
  • Newest to Oldest
  • Most Votes
Reply
  • Reply as topic
Log in to reply
This topic has been deleted. Only users with topic management privileges can see it.
  • U
    utopia
    last edited by utopia Nov 12, 2016, 9:00 PM Nov 13, 2016, 2:58 AM

    Server
    • FOG Version: 1.3.0-RC-18 (svn5991)
    • OS: Ubuntu 14.04.5 LTS (running as a VMware guest)
    Client
    • Service Version: NA
    • OS: NA
    Description

    I am attempting to upgrade my FOG server beyond its current RC-18 (to RC-22). While the installfog.sh script appears to run and complete as it normally does, it has resulted in two odd problems.

    First, it is no longer possible to start any processes for the five FOG services (FOGSnapinReplicator, FOGScheduler, FOGMulticastManager, FOGImageReplicator, FOGPingHosts). The processes can not be started from either 1) running the installfog.sh script, 2) rebooting the OS, or 3) manual attempts to stop/start or restart the services via cli (the listing below would also show the FOG processes if they were running).

    root@FogU1404T01:~# ps -ef | grep php
    root      15633      1  0 15:03 ?        00:00:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)                    
    www-data  15636  15633  0 15:03 ?        00:00:00 php-fpm: pool www
    www-data  15637  15633  0 15:03 ?        00:00:00 php-fpm: pool www
    root      19272  19258  0 15:38 pts/14   00:00:00 grep --color=auto php
    

    Second, I have my fogsettings config file set to enable automated DB backups during FOG upgrades. This has been working fine for many months. However, the upgrade to 1.3.0-RC-22 produces an empty (zero byte) file for the automated DB backup. I can successfully create a good DB backup manually from the cli both before and after the step which updates the DB schema. This second issue is less of a concern since I know I can work around it manually as needed. So my main focus is on the first issue.

    I have attached the three files from FOG installer/updater …/bin/error_logs folder.
    The foginstall.log shows that the installfog.sh script appears to have run normally.
    0_1478999699458_foginstall.log

    But the presence of fog_err_1.3.0-RC-22.log is new to me (I haven’t seen it prior to RC-13).
    0_1478999647595_fog_err_1.3.0-RC-22.log
    In particular, that log contains:

    Considering dependency proxy for proxy_fcgi:
    Module proxy already enabled
    Module proxy_fcgi already enabled
    Module setenvif already enabled
    ERROR: Module php5-fpm does not exist!
    

    Here is the file fog_error_1.3.0-RC-22.log.
    0_1479000379589_fog_error_1.3.0-RC-22.log
    In particular, the tail of that log ends with:

     * Starting FOG Computer Imaging Solution: FOGMulticastManager
       ...done.
     * Starting FOG Computer Imaging Solution: FOGImageReplicator
       ...done.
     * Starting FOG Computer Imaging Solution: FOGSnapinReplicator
       ...done.
     * Starting FOG Computer Imaging Solution: FOGScheduler
       ...done.
     * Starting FOG Computer Imaging Solution: FOGPingHosts
       ...done.
     * Stopping NFS kernel daemon
       ...done.
     * Unexporting directories for NFS kernel daemon...
       ...done.
     * Exporting directories for NFS kernel daemon...
       ...done.
     * Starting NFS kernel daemon
       ...done.
    chmod: cannot access ‘/var/log/php*fpm’: No such file or directory
    

    It appears that there may be a problem with the script trying to modify non-existent log files. The relevant contents of the /var/log directory listing shows the following (Which would seem to explain the chmod failure above - “No such file or directory”. Perhaps correctable by changing to /var/log/php*fpm* or /var/log/php*fpm.log ?):

    -rw-r-----  1 mysql             adm         20 Oct 24 15:12 mysql.log.7.gz
    -rw-------  1 root              root         0 Nov 11 15:06 php5-fpm.log
    -rw-------  1 root              root       388 Nov 11 15:03 php5-fpm.log.1
    -rw-------  1 root              root       111 Sep  6 06:36 php5-fpm.log.10.gz
    -rw-------  1 root              root       146 Aug 31 06:45 php5-fpm.log.11.gz
    -rw-------  1 root              root       124 Aug 23 06:56 php5-fpm.log.12.gz
    -rw-------  1 root              root       248 Nov  1 06:45 php5-fpm.log.2.gz
    -rw-------  1 root              root       137 Oct 23 14:41 php5-fpm.log.3.gz
    -rw-------  1 root              root       153 Oct 18 06:38 php5-fpm.log.4.gz
    -rw-------  1 root              root       166 Oct  9 17:01 php5-fpm.log.5.gz
    -rw-------  1 root              root       131 Oct  3 15:18 php5-fpm.log.6.gz
    -rw-------  1 root              root       111 Sep 29 06:38 php5-fpm.log.7.gz
    -rw-------  1 root              root       134 Sep 21 06:34 php5-fpm.log.8.gz
    -rw-------  1 root              root       123 Sep 13 06:49 php5-fpm.log.9.gz
    -rw-r--r--  1 root              root     10005 Nov 11 14:28 pm-powersave.log
    

    From what I can see, everything seems to point to php5-fpm (the PHP FastCGI Process Manager) as the potential cause of the problem. It doesn’t matter whether I try the FOG upgrade from a fully updated OS or not, or whether I try it from a snapshot prior to when I first see the fog_err_1.3.0-RC-22.log (going back to RC-13). The FOG upgrade always results in the same two problems of no startable FOG processes and an empty automated DB backup file.

    So for now, I am unable to upgrade FOG. And I am stuck functioning at 1.3.0-RC-18 which I am able to get back to using VMware snapshots. I’m hoping that the info I’ve provided can help identify a solution. And I appreciate any insight and assistance in finding one.

    1 Reply Last reply Reply Quote 0
    • W
      Wayne Workman
      last edited by Nov 13, 2016, 4:34 PM

      Have you tried installing php5-fpm ?

      Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG!
      Daily Clean Installation Results:
      https://fogtesting.fogproject.us/
      FOG Reporting:
      https://fog-external-reporting-results.fogproject.us/

      T 1 Reply Last reply Nov 13, 2016, 4:36 PM Reply Quote 0
      • T
        Tom Elliott @Wayne Workman
        last edited by Nov 13, 2016, 4:36 PM

        @Wayne-Workman ondrej changed put the repo so it is looking for php5.6-fpm

        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

        1 Reply Last reply Reply Quote 0
        • T
          Tom Elliott
          last edited by Nov 13, 2016, 4:41 PM

          https://github.com/FOGProject/fogproject/commit/45a468ffae9a3aabd2e145b957c02f00eb4d9f92

          Hopefully this commit will address that.

          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

          1 Reply Last reply Reply Quote 0
          • T
            Tom Elliott
            last edited by Nov 13, 2016, 4:45 PM

            Please try these steps:

            Edit the /opt/fog/.fogsettings file and remove the packages line.

            Edit the php_ver and php_verAdds lines to read as:

            php_ver="7.0"
            php_verAdds="-7.0"
            

            Remove the apache and php items from your installation:

            rm -rf /etc/{php,apache}*
            apt-get purge php5* libapache*php* apache*
            

            Rerun the fog installer.

            This should take a bit longer than you’re likely used to, but it should also make things MUCH easier for you.

            Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

            Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

            Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

            U 1 Reply Last reply Nov 13, 2016, 6:37 PM Reply Quote 0
            • U
              utopia @Tom Elliott
              last edited by Nov 13, 2016, 6:37 PM

              @Tom-Elliott OK, this is pretty much what I was expecting. So I’ve tried those steps and it looks good up until the point where installfog.sh fails at “Stopping web service”. Here is the foginstall.log which ends with:
              0_1479061190246_foginstall.log

              * Setting up fog user.........................................Already setup
              * Setting up fog password.....................................OK
              * Stopping FOGMulticastManager Service........................OK
              * Stopping FOGImageReplicator Service.........................OK
              * Stopping FOGSnapinReplicator Service........................OK
              * Stopping FOGScheduler Service...............................OK
              * Stopping FOGPingHosts Service...............................OK
              * Setting up and starting MySQL...............................OK
              * Backing up user reports.....................................Done
              * Stopping web service........................................Failed!
              

              Here is the fog_error_1.3.0-RC-22.log which ends with:
              0_1479061317524_fog_error_1.3.0-RC-22.log

              * Stopping FOG Computer Imaging Solution: FOGScheduler
                ...done.
              * Stopping FOG Computer Imaging Solution: FOGPingHosts
                ...done.
              mysql stop/waiting
              mysql start/running, process 25774
              /etc/init.d/apache2: 64: .: Can't open /etc/apache2/envvars
              /etc/init.d/apache2: 76: .: Can't open /etc/apache2/envvars
              ERROR: APACHE_PID_FILE needs to be defined in /etc/apache2/envvars
              

              /etc/apache2/envvars does not exist. Here is a listing of the /etc/apache2 directory:

              root@FogU1404T01:/etc/apache2# lsa
              total 24
              drwxr-xr-x   4 root root  4096 Nov 13 09:53 .
              drwxr-xr-x 141 root root 12288 Nov 13 10:04 ..
              drwxr-xr-x   2 root root  4096 Nov 13 09:54 conf-available
              drwxr-xr-x   2 root root  4096 Nov 13 09:51 mods-available
              root@FogU1404T01:/etc/apache2# 
              

              I think this may be caused by old php5 processes still running as shown here:

              root@FogU1404T01:~# ps -ef | grep php
              root       1717      1  0 09:26 ?        00:00:00 php-fpm: master process (/etc/php5/fpm/php-fpm.conf)
              www-data   1719   1717  0 09:26 ?        00:00:00 php-fpm: pool www
              www-data   1720   1717  0 09:26 ?        00:00:00 php-fpm: pool www
              root      25547      1  0 10:04 ?        00:00:00 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
              www-data  25550  25547  0 10:04 ?        00:00:00 php-fpm: pool www
              www-data  25551  25547  0 10:04 ?        00:00:00 php-fpm: pool www
              root      26194   3422  0 10:28 pts/4    00:00:00 grep --color=auto php
              

              I could restore my snapshot and try again after making sure there are no php5 processes running. What do you think?

              1 Reply Last reply Reply Quote 0
              • U
                utopia
                last edited by Nov 13, 2016, 7:16 PM

                After rebooting the server I now have the FOG processes and the new php7 processes running as shown here.

                root@FogU1404T01:~# ps -ef | grep php
                root       1146      1  0 10:59 ?        00:00:00 php-fpm: master process (/etc/php/7.0/fpm/php-fpm.conf)
                www-data   1557   1146  0 10:59 ?        00:00:00 php-fpm: pool www
                www-data   1558   1146  0 10:59 ?        00:00:00 php-fpm: pool www
                root       1576      1  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
                root       1600      1  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
                root       1623      1  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
                root       1646      1  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
                root       1670      1  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
                root       2167   1600  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGTaskScheduler/FOGTaskScheduler
                root       2168   1646  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGImageReplicator/FOGImageReplicator
                root       2169   1670  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGPingHosts/FOGPingHosts
                root       2170   1576  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGSnapinReplicator/FOGSnapinReplicator
                root       2171   1623  0 10:59 ?        00:00:00 /usr/bin/php -q /opt/fog/service/FOGMulticastManager/FOGMulticastManager
                root       3570   3526  0 11:01 pts/4    00:00:00 grep --color=auto php
                

                But I don’t have any web GUI, obviously since the installfog.sh script never completed. So I’m going to first try rerunning the installer. And if that doesn’t work, I’ll restore my snapshot and try the steps again - this time making sure I don’t have any old php5 processes running.

                1 Reply Last reply Reply Quote 0
                • U
                  utopia
                  last edited by Nov 13, 2016, 9:48 PM

                  @Tom-Elliott Rerunning the installer fails with the same results as before. Restoring the snapshot and trying the steps again after killing the three old php5 running processes also fails with the same results. Here are the logs from the last attempt with no php5 processes running.
                  1_1479073227092_foginstall.log
                  0_1479073227091_fog_error_1.3.0-RC-22.log

                  As with the other attempts, the directory /etc/apache2 only contains the subdirectories conf-available and mods-available. /etc/apache2/envvars does not exist.

                  T 1 Reply Last reply Nov 13, 2016, 10:16 PM Reply Quote 0
                  • T
                    Tom Elliott @utopia
                    last edited by Nov 13, 2016, 10:16 PM

                    @utopia can you run:
                    a2dismod envvars

                    Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                    Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                    Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                    1 Reply Last reply Reply Quote 0
                    • U
                      utopia
                      last edited by Nov 13, 2016, 10:57 PM

                      root@FogU1404T01:~# a2dismod envvars
                      ERROR: Module envvars does not exist!
                      root@FogU1404T01:~#
                      
                      T 1 Reply Last reply Nov 13, 2016, 10:58 PM Reply Quote 0
                      • T
                        Tom Elliott @utopia
                        last edited by Tom Elliott Nov 13, 2016, 4:59 PM Nov 13, 2016, 10:58 PM

                        @utopia Try:

                        rm -rf /etc/apache*; apt-get purge apache2

                        Then rerun installer.

                        Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                        Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                        Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                        T 1 Reply Last reply Nov 13, 2016, 11:05 PM Reply Quote 0
                        • T
                          Tom Elliott @Tom Elliott
                          last edited by Nov 13, 2016, 11:05 PM

                          @Tom-Elliott I suppose because it’s stopping this is the issue.

                          Maybe try:

                          touch /etc/apache2/envvars
                          

                          Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                          Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                          Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                          1 Reply Last reply Reply Quote 0
                          • U
                            utopia
                            last edited by Nov 13, 2016, 11:32 PM

                            root@FogU1404T01:~# rm -rf /etc/apache*; apt-get purge apache2
                            Reading package lists... Done
                            Building dependency tree       
                            Reading state information... Done
                            The following packages were automatically installed and are no longer required:
                              libntdb1 python-ntdb
                            Use 'apt-get autoremove' to remove them.
                            The following packages will be REMOVED:
                              apache2* libapache2-mod-php5*
                            0 upgraded, 0 newly installed, 2 to remove and 3 not upgraded.
                            After this operation, 10.2 MB disk space will be freed.
                            Do you want to continue? [Y/n] y
                            (Reading database ... 173250 files and directories currently installed.)
                            Removing libapache2-mod-php5 (5.6.23+dfsg-1+deprecated+dontuse+deb.sury.org~trusty+1) ...
                            apache2_invoke php5 prerm: No action required
                            Purging configuration files for libapache2-mod-php5 (5.6.23+dfsg-1+deprecated+dontuse+deb.sury.org~trusty+1) ...
                            apache2_invoke php5 postrm: No action required
                            Removing apache2 (2.4.18-1+deb.sury.org~trusty+2) ...
                            /etc/init.d/apache2: 64: .: Can't open /etc/apache2/envvars
                            /etc/init.d/apache2: 76: .: Can't open /etc/apache2/envvars
                            ERROR: APACHE_PID_FILE needs to be defined in /etc/apache2/envvars
                            invoke-rc.d: initscript apache2, action "stop" failed.
                            Purging configuration files for apache2 (2.4.18-1+deb.sury.org~trusty+2) ...
                            dpkg: warning: while removing apache2, directory '/var/www/html' not empty so not removed
                            Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
                            root@FogU1404T01:~# 
                            

                            Then reran script

                             * Adding needed repository....................................OK
                             * Preparing Package Manager...................................OK
                             * Packages to be installed:
                            
                            apache2 bc build-essential cpp curl g++ gawk gcc gzip htmldoc lftp libapache2-mod-php7.0 libc6 libcurl3 m4 mysql-client mysql-server net-tools nfs-kernel-server openssh-server php7.0 php7.0-bcmath php7.0-cli php7.0-curl php7.0-fpm php7.0-gd php7.0-json php7.0-ldap php7.0-mbstring php7.0-mcrypt php7.0-mysqlnd php-gettext sysv-rc-conf tar tftpd-hpa tftp-hpa vsftpd wget xinetd zlib1g 
                            
                            
                             * Installing package: apache2.................................Failed!
                            

                            I assumed that I would touch the envvars file after the apache2 package was reinstalled, then run the installer again after that if necessary. But since the package installation fails, I can’t get that far. Do you want the two log files from this?

                            T 1 Reply Last reply Nov 14, 2016, 12:16 AM Reply Quote 0
                            • T
                              Tom Elliott @utopia
                              last edited by Nov 14, 2016, 12:16 AM

                              @utopia I suppose you could do the manual installation of apache.

                              Or it just failed out because of the warning (the /var/www/html issue).

                              If you need to test it, I suppose you could do:

                              apt-get install apache2 -f
                              service apache2 start
                              service apache2 stop
                              

                              Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                              Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                              Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                              1 Reply Last reply Reply Quote 0
                              • U
                                utopia
                                last edited by Nov 14, 2016, 6:40 AM

                                @Tom-Elliott “apt-get install apache2 -f” fails. However, aptitude was able to untangle the dependency trees. For your reference, the attached file shows a rough summary of the steps I took to sort it out.
                                0_1479102893333_Fog_troubleshooting_161113d.txt

                                Based upon the new php 7.0, I was able to successfully upgrade FOG from RC-18 to RC-22. This gave me running FOG processes and a functional GUI with the new apache2. Then, to test it out further, I upgraded again from RC-22 to RC-23. That successfully showed that I once again get automated DB backup files created during the FOG upgrade process which now do contain data (no longer zero byte size files). There are still some stray php5 processes which try to start and get killed during server bootup - which I will look into cleaning up. And I will be monitoring this server a bit more closely over the next few upgrades to make sure it remains stable.

                                I do have a final related follow-up question. As the end of the attached file shows, the database backup files have different sizes depending upon whether they are created automatically during the FOG upgrade or created manually via the command line, such as “mysqldump --allow-keywords -x -v fog > fogbackup.sql”. I assume that both types of DB backup are equally useful for restoring or transferring the DB to either the same or a different FOG server - assuming of course that the target server is running the proper FOG version to match the DB backup file. Would that be a correct assumption that either method of creating the DB backup files is equally useful for restore or transfer?

                                Lastly, I most certainly do THANK YOU for your insight in helping sort this out. Your assistance is VERY MUCH APPRECIATED!

                                T 1 Reply Last reply Nov 14, 2016, 11:18 AM Reply Quote 0
                                • T
                                  Tom Elliott @utopia
                                  last edited by Nov 14, 2016, 11:18 AM

                                  @utopia There is some variants in what’s going on for SQL backup. While the version that “automatically” creates a new backup is slightly larger, this is mostly just comments.

                                  I’ve seen some strange things in regards to Ubuntu in the past. The unfortunate part is that not all systems are the same. What you experienced does not mean everybody experiences. The only other thing I can think is ondrej changed how the repo is used relatively recently. Of note, php5-fpm is no longer proper, it’s supposed to be php5.6-fpm as described earlier in the posting. I’ve added this change for the working branch now so hopefully this will no longer be an issue.

                                  I don’t know what php5 items are still lingering, but I’d imagine you could remove them with aptitude just as well?

                                  Please help us build the FOG community with everyone involved. It's not just about coding - way more we need people to test things, update documentation and most importantly work on uniting the community of people enjoying and working on FOG! Get in contact with me (chat bubble in the top right corner) if you want to join in.

                                  Web GUI issue? Please check apache error (debian/ubuntu: /var/log/apache2/error.log, centos/fedora/rhel: /var/log/httpd/error_log) and php-fpm log (/var/log/php*-fpm.log)

                                  Please support FOG if you like it: https://wiki.fogproject.org/wiki/index.php/Support_FOG

                                  1 Reply Last reply Reply Quote 0
                                  • 1 / 1
                                  1 / 1
                                  • First post
                                    6/16
                                    Last post

                                  196

                                  Online

                                  12.0k

                                  Users

                                  17.3k

                                  Topics

                                  155.2k

                                  Posts
                                  Copyright © 2012-2024 FOG Project