[Solved]Email notifications suck in queuednotifications

BW~Merlin · 2024-01-11 05:01:42

Since updating to the latest version of GLPI automatic emails are no longer being sent.

I have confirmed that the test email notification function works and that the test email was received and nothing has changed configuration wise other than upgrading from an earlier version of GLPI and email notifications had been working for nearly two years prior to upgrading GLPI.

I dug into notifications queue under Administration and can see that I have about 100 emails in the queue. I then found queuednotification under Automatic actions and can see that the schedule is "running" but last ran on 2023-12-13 05:27.
When I look in the logs under queuednotification the last log entry is for the 2023-12-08 03:58 and is listed as fully processed.
I tried to manually execute any pending automatic actions but queuednotification is still listed as "Running".

I am unsure on what else to try to get these emails to send again.

Last edited by BW~Merlin (2024-02-12 03:31:40)

LaDenrée · 2024-01-11 09:46:31

sometimes some actions are stuck in running state. click on the little x to stop it. then it will run again.

you should activate watcher action : it alerts when any action is stuck

BW~Merlin · 2024-01-12 01:39:13

LaDenrée wrote:

sometimes some actions are stuck in running state. click on the little x to stop it. then it will run again.
you should activate watcher action : it alerts when any action is stuck

Thanks for this. I have tried pressing the little x next to the words running inside of the queuednotification automatic action and waiting a while for it to run again by itself and nothing has come through.

I have checked and the watcher action was already enabled and running so it does not appear to be doing what I would expect of it if it is suppose to notify you of a stuck automatic action.

cconard96 · 2024-01-12 03:30:40

If notifications are not working, the automatic action watcher isn't going to be much help.
If you want a better idea about the "health" of your GLPI, there is the "/status.php" script and the "bin/console system:status" CLI command.

These tools both provide information which can be checked manually or monitored using a system like Nagios, Zabbix, icinga, etc.

For example:
bin/console system:status --format=json --private

{
    "glpi": {
        "status": "PROBLEM"
    },
    "db": {
        "status": "OK",
        "main": {
            "status": "OK"
        },
        "replicas": {
            "status": "NO_DATA",
            "servers": []
        }
    },
    "cas": {
        "status": "NO_DATA"
    },
    "ldap": {
        "status": "PROBLEM",
        "servers": {
            "ADCJDS": {
                "status": "PROBLEM",
                "status_msg": "Unable to connect to the LDAP server"
            }
        },
        "status_msg": "OK: 0, WARNING: 0, PROBLEM: 1, TOTAL: 1"
    },
    "imap": {
        "status": "NO_DATA",
        "servers": []
    },
    "mail_collectors": {
        "status": "NO_DATA",
        "servers": []
    },
    "crontasks": {
        "status": "PROBLEM",
        "stuck": [
            "watcher",
            "queuednotification",
            "syncJamf",
            "queuedwebhookclean"
        ],
        "status_msg": "RUNNING: 4, STUCK: 4, TOTAL: 65"
    },
    "filesystem": {
        "status": "OK",
        "session_dir": {
            "status": "OK"
        }
    },
    "plugins": {
        "status": "NO_DATA"
    }
}

Adding the "--private" option allows a more complete display of data.

You can also restrict the output to a single "service". For example, "bin/console system:status --format=json --private --service crontasks" will output only the data for automatic actions.

BW~Merlin · 2024-01-12 04:44:31

Running

/status.php

I got the following

GLPI_DB_OK
GLPI_SESSION_DIR_OK
Check LDAP servers: MyDomain - Domain - Staff_OK
 MyDomain - Domain - Student_OK
 MyDomain - Domain - Groups_OK
 MyDomain - Domain_OK
No IMAP server
No CAS server
No mail collector
Check crontasks: queuednotification_PROBLEM

GLPI_PROBLEM

So does appear that there is something wrong with crontasks.

Running

sudo /var/www/html/glpi/bin/console system:status --format=json --private

I get the following

{
    "glpi": {
        "status": "PROBLEM"
    },
    "db": {
        "status": "OK",
        "master": {
            "status": "OK"
        },
        "slaves": {
            "status": "NO_DATA",
            "servers": []
        },
        "main": {
            "status": "OK"
        },
        "replicas": {
            "status": "NO_DATA",
            "servers": []
        }
    },
    "cas": {
        "status": "NO_DATA"
    },
    "ldap": {
        "status": "OK",
        "servers": {
            "MyDomain - Domain - Staff": {
                "status": "OK"
            },
            "MyDomain - Domain - Student": {
                "status": "OK"
            },
            "MyDomain - Domain - Groups": {
                "status": "OK"
            },
            "MyDomain - Domain": {
                "status": "OK"
            }
        }
    },
    "imap": {
        "status": "NO_DATA",
        "servers": []
    },
    "mail_collectors": {
        "status": "NO_DATA",
        "servers": []
    },
    "crontasks": {
        "status": "PROBLEM",
        "stuck": [
            "queuednotification"
        ],
        "status_msg": "RUNNING: 1, STUCK: 1, TOTAL: 50"
    },
    "filesystem": {
        "status": "OK",
        "session_dir": {
            "status": "OK"
        }
    },
    "plugins": {
        "status": "NO_DATA"
    }
}

How do I find what is "stuck" and unstick it?

cconard96 · 2024-01-12 15:18:28

"Stuck" just means that the status of the task was changed to "Running" in the DB and the last time they were run was either over 2 hours ago or over 2 times their configured run frequency.
There may have been a PHP or SQL error that occurred when the task was running and caused it to be left in the "running" state.

You can check the GLPI logs (files/_log) for more information.

cconard96 · 2024-01-12 15:59:50

To "unstick" as was said before, there is a "x" button next to where it says the status is "Running" on the automatic action's form which can reset the state to "Waiting". There is another "x" next to the last run date to reset that as well.

BW~Merlin · 2024-01-30 00:42:48

Sorry for the very late reply.

When I look in the

php-errors.log

I see this same error message over and over (edit: looks like text wrapping didn't work correctly/as expected in nano so I have used cat to gather the corrected output of the error log).

[2023-12-20 02:31:00] glpiphplog.WARNING:   *** PHP Warning (2): mkdir(): No such file or directory in /var/www/html/glpi/src/Toolbox.php at line 1222
  Backtrace :
  src/Toolbox.php:1222                               mkdir()
  ...System/Requirement/DirectoryWriteAccess.php:113 Toolbox::testWriteAccessToDirectory()
  src/System/Requirement/AbstractRequirement.php:117 Glpi\System\Requirement\DirectoryWriteAccess->check()
  src/System/Requirement/AbstractRequirement.php:138 Glpi\System\Requirement\AbstractRequirement->doCheck()
  ...ystem/Requirement/DirectoriesWriteAccess.php:70 Glpi\System\Requirement\AbstractRequirement->getValidationMessages()
  src/System/Requirement/AbstractRequirement.php:117 Glpi\System\Requirement\DirectoriesWriteAccess->check()
  src/System/Requirement/AbstractRequirement.php:166 Glpi\System\Requirement\AbstractRequirement->doCheck()
  .../twig/twig/src/Extension/CoreExtension.php:1637 Glpi\System\Requirement\AbstractRequirement->isOutOfContext()
  ...ates/1d/1dd3fec4619d531def7fa122d855dc50.php:59 twig_get_attribute()
  vendor/twig/twig/src/Template.php:394              __TwigTemplate_7d25644d3aada0804b0aaaa0bc2d35ee->doDisplay()
  vendor/twig/twig/src/Template.php:367              Twig\Template->displayWithErrorHandling()
  vendor/twig/twig/src/TemplateWrapper.php:47        Twig\Template->display()
  src/Application/View/TemplateRenderer.php:184      Twig\TemplateWrapper->display()
  inc/config.php:225                                 Glpi\Application\View\TemplateRenderer->display()
  inc/includes.php:58                                include_once()
  index.php:87                                       include()

Last edited by BW~Merlin (2024-02-05 05:36:59)

BW~Merlin · 2024-02-05 06:43:53

When I run

 sed -n 1219,1229p /var/www/html/glpi/src/Toolbox.php

to try and see what is relating to I get the following.

       // Check directory creation which can be denied by SElinux
        $sdir = sprintf("%s/test_glpi_%08x", $dir, $rand);

        if (!mkdir($sdir)) {
            return 4;
        }

        if (!rmdir($sdir)) {
            return 3;
        }

if I run

which mkdir

I get

/usr/bin/mkdir

I am unsure what the next step is or if I am even on the right track.

BW~Merlin · 2024-02-06 06:49:45

As recommended in Discord chat I added

trigger_error('Failed to create file: ' . $sdir, E_USER_WARNING);

so when I run

 sed -n 1219,1230p /var/www/html/glpi/src/Toolbox.php

it looks like this (I hope I added it to the correct location and format)

       // Check directory creation which can be denied by SElinux
        $sdir = sprintf("%s/test_glpi_%08x", $dir, $rand);

        if (!mkdir($sdir)) {
            //added this part to see what directory is causing the issue related to ticket emails not being sent
           trigger_error('Failed to create file: ' . $sdir, E_USER_WARNING);
            return 4;
        }

        if (!rmdir($sdir)) {
            return 3;
        }

When I

cat /var/www/html/glpi/files/_log/php-errors.log

I am still not seeing any entries newer than 2023-12-20

[2023-12-20 02:31:00] glpiphplog.WARNING:   *** PHP Warning (2): mkdir(): No such file or directory in /var/www/html/glpi/src/Toolbox.php at line 1222
  Backtrace :
  src/Toolbox.php:1222                               mkdir()
  ...System/Requirement/DirectoryWriteAccess.php:113 Toolbox::testWriteAccessToDirectory()
  src/System/Requirement/AbstractRequirement.php:117 Glpi\System\Requirement\DirectoryWriteAccess->check()
  src/System/Requirement/AbstractRequirement.php:138 Glpi\System\Requirement\AbstractRequirement->doCheck()
  ...ystem/Requirement/DirectoriesWriteAccess.php:70 Glpi\System\Requirement\AbstractRequirement->getValidationMessages()
  src/System/Requirement/AbstractRequirement.php:117 Glpi\System\Requirement\DirectoriesWriteAccess->check()
  src/System/Requirement/AbstractRequirement.php:166 Glpi\System\Requirement\AbstractRequirement->doCheck()
  .../twig/twig/src/Extension/CoreExtension.php:1637 Glpi\System\Requirement\AbstractRequirement->isOutOfContext()
  ...ates/1d/1dd3fec4619d531def7fa122d855dc50.php:59 twig_get_attribute()
  vendor/twig/twig/src/Template.php:394              __TwigTemplate_7d25644d3aada0804b0aaaa0bc2d35ee->doDisplay()
  vendor/twig/twig/src/Template.php:367              Twig\Template->displayWithErrorHandling()
  vendor/twig/twig/src/TemplateWrapper.php:47        Twig\Template->display()
  src/Application/View/TemplateRenderer.php:184      Twig\TemplateWrapper->display()
  inc/config.php:225                                 Glpi\Application\View\TemplateRenderer->display()
  inc/includes.php:58                                include_once()
  index.php:87                                       include()

LaDenrée · 2024-02-06 11:10:19

glpi version ?
server type windows/linux ?

check setup>general>system

do you see :

Database engine version (10.6.16) is supported.
No files from previous GLPI version detected.
The log file has been created successfully.
Write access to /var/lib/glpi/_cache has been validated.
Write access to /etc/glpi/ has been validated.
Write access to /var/lib/glpi/_cron has been validated.
Write access to /var/lib/glpi has been validated.
Write access to /var/lib/glpi/_dumps has been validated.
Write access to /var/lib/glpi/_graphs has been validated.
Write access to /var/lib/glpi/_lock has been validated.
Write access to /var/lib/glpi/_pictures has been validated.
Write access to /var/lib/glpi/_plugins has been validated.
Write access to /var/lib/glpi/_rss has been validated.
Write access to /var/lib/glpi/_sessions has been validated.
Write access to /var/lib/glpi/_tmp has been validated.
Write access to /var/lib/glpi/_uploads has been validated.

check if log path is /var/log/glpi or /var/www/glpi/files/_log

check if apache is owner and permissions

BW~Merlin · 2024-02-07 02:09:34

Other than the issue with the web server root directory and timezone (that is next on my list to fix) it would appear that things are okay.

Operating system: Linux <orgcode>-helpdesk-01 5.15.0-92-generic #102-Ubuntu SMP Wed Jan 10 09:33:48 UTC 2024 x86_64
PHP 8.1.2-1ubuntu2.14 apache2handler (Core, FFI, PDO, Phar, Reflection, SPL, SimpleXML, Zend OPcache, apache2handler, apcu, bz2,
	calendar, ctype, curl, date, dom, exif, fileinfo, filter, ftp, gd, gettext, hash, iconv, imagick, intl, json, ldap, libxml,
	mbstring, memcache, mysqli, mysqlnd, openssl, pcre, pdo_mysql, posix, readline, session, shmop, sockets, sodium, standard,
	sysvmsg, sysvsem, sysvshm, tokenizer, xml, xmlreader, xmlwriter, xsl, zip, zlib)
Setup: max_execution_time="30" memory_limit="128M" post_max_size="8M" safe_mode="" session.save_handler="files"
	upload_max_filesize="2M" disable_functions="" 
Software: Apache/2.4.52 (Ubuntu) (Apache/2.4.52 (Ubuntu) Server at <IP address> Port 80
)
	Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:122.0) Gecko/20100101 Firefox/122.0
Server Software: Ubuntu 22.04
	Server Version: 10.6.16-MariaDB-0ubuntu0.22.04.1
	Server SQL Mode: STRICT_TRANS_TABLES,ERROR_FOR_DIVISION_BY_ZERO,NO_AUTO_CREATE_USER,NO_ENGINE_SUBSTITUTION
	Parameters: glpiuser@localhost/glpidb
	Host info: Localhost via UNIX socket
	
PHP version (8.1.2-1ubuntu2.14) is supported.PHP version (8.1.2-1ubuntu2.14) is supported.
Sessions configuration is OK.Sessions configuration is OK.
Allocated memory is sufficient.Allocated memory is sufficient.
mysqli extension is installed.mysqli extension is installed.
Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter.Following extensions are installed: dom, fileinfo, filter, libxml, json, simplexml, xmlreader, xmlwriter.
curl extension is installed.curl extension is installed.
gd extension is installed.gd extension is installed.
intl extension is installed.intl extension is installed.
zlib extension is installed.zlib extension is installed.
The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.The constant SODIUM_CRYPTO_AEAD_XCHACHA20POLY1305_IETF_NPUBBYTES is present.
Database engine version (10.6.16) is supported.Database engine version (10.6.16) is supported.
No files from previous GLPI version detected.No files from previous GLPI version detected.
The log file has been created successfully.The log file has been created successfully.
Write access to /var/lib/glpi/_cache has been validated. Write access to /var/lib/glpi/_cron has been validated. Write access to /var/lib/glpi has been validated. Write access to /var/lib/glpi/_dumps has been validated. Write access to /var/lib/glpi/_graphs has been validated. Write access to /var/lib/glpi/_lock has been validated. Write access to /var/lib/glpi/_pictures has been validated. Write access to /var/lib/glpi/_plugins has been validated. Write access to /var/lib/glpi/_rss has been validated. Write access to /var/lib/glpi/_sessions has been validated. Write access to /var/lib/glpi/_tmp has been validated. Write access to /var/lib/glpi/_uploads has been validated.Write access to /var/lib/glpi/_cache has been validated.
Write access to /var/lib/glpi/_cron has been validated.
Write access to /var/lib/glpi has been validated.
Write access to /var/lib/glpi/_dumps has been validated.
Write access to /var/lib/glpi/_graphs has been validated.
Write access to /var/lib/glpi/_lock has been validated.
Write access to /var/lib/glpi/_pictures has been validated.
Write access to /var/lib/glpi/_plugins has been validated.
Write access to /var/lib/glpi/_rss has been validated.
Write access to /var/lib/glpi/_sessions has been validated.
Write access to /var/lib/glpi/_tmp has been validated.
Write access to /var/lib/glpi/_uploads has been validated.

Web server root directory configuration is not safe as it permits access to non-public files. See installation documentation for more details.Web server root directory configuration is not safe as it permits access to non-public files. See installation documentation for more details.
GLPI data directories are located in a secured path.GLPI data directories are located in a secured path.
Sessions configuration is secured.Sessions configuration is secured.
OS and PHP are relying on 64 bits integers.OS and PHP are relying on 64 bits integers.
exif extension is installed.exif extension is installed.
ldap extension is installed.ldap extension is installed.
openssl extension is installed.openssl extension is installed.
Following extensions are installed: bz2, Phar, zip.Following extensions are installed: bz2, Phar, zip.
Zend OPcache extension is installed.Zend OPcache extension is installed.
Following extensions are installed: ctype, iconv, mbstring, sodium.Following extensions are installed: ctype, iconv, mbstring, sodium.
Write access to /var/www/html/glpi/marketplace has been validated.Write access to /var/www/html/glpi/marketplace has been validated.
Timezones seems not loaded, see https://glpi-install.readthedocs.io/en/latest/timezones.html.Timezones seems not loaded, see https://glpi-install.readthedocs.io/en/latest/timezones.html.

Log files are in

/var/log/glpi

and I can see entries being made into them.

Looking in

/var/log/glpi/php-errors.log

I can see this error message

[2024-01-12 02:39:26] glpiphplog.WARNING: Test logger
[2024-01-12 02:39:26] glpiphplog.NOTICE:   *** PHP User deprecated function (16384): Plain-text status output is deprecated please use the JSON format instead by specifically using the "--format json" parameter. In the future, JSON output will be the default. in /var/www/html/glpi/src/Toolbox.php at line 582
  Backtrace :
  src/Toolbox.php:582                                trigger_error()
  src/Console/System/CheckStatusCommand.php:86       Toolbox::deprecated()
  vendor/symfony/console/Command/Command.php:298     Glpi\Console\System\CheckStatusCommand->execute()
  vendor/symfony/console/Application.php:1040        Symfony\Component\Console\Command\Command->run()
  src/Console/Application.php:286                    Symfony\Component\Console\Application->doRunCommand()
  vendor/symfony/console/Application.php:301         Glpi\Console\Application->doRunCommand()
  vendor/symfony/console/Application.php:171         Symfony\Component\Console\Application->doRun()
  bin/console:122                                    Symfony\Component\Console\Application->run()

[2024-01-12 02:40:15] glpiphplog.WARNING: Test logger
[2024-01-12 02:42:07] glpiphplog.WARNING: Test logger
[2024-02-01 22:21:36] glpiphplog.CRITICAL:   *** PHP Error (1): Allowed memory size of 134217728 bytes exhausted (tried to allocate 49203500 bytes) in /var/www/html/glpi/vendor/phpmailer/phpmailer/src/PHPMailer.php at line 3430
[2024-02-06 23:20:09] glpiphplog.WARNING: Test logger

When you say

check if apache is owner and permissions

what am I checking apache is the owner of?

Last edited by BW~Merlin (2024-02-07 02:10:41)

LaDenrée · 2024-02-07 09:59:21

apcahe should be owner of
/var/lib/glpi (recursively)
and
/var/log/glpi (recursively)

so that glpi can create files in these 2 folders.

in your previous post you refer to "cat /var/www/html/glpi/files/_log/php-errors.log" => GLPI doesn't write here anymore but in var/log/glpi

BW~Merlin · 2024-02-08 02:02:09

When I run

ls -la /var/log/

I get the following

drwxrwxr-x   2 www-data  www-data           4096 Jan 11 02:43 glpi

which (in my limited understanding) shows that apache is the owner and has the required permissions.

When I run

ls -la /var/lib/

I get the following

drwxrwxr-x 27 www-data  www-data  4096 Feb  6 23:20 glpi

which once again (as I understand) apache is already the owner.

I have rerun the following

sudo chmod 775 -Rf /path/here

and

sudo chown -R www-data:www-data /path/goes/here

on both

/var/lib/glpi

and

/var/log/glpi

BW~Merlin · 2024-02-12 03:28:31

Okay I took another run at this and have resolved the mailing issue.
When I run

cat /var/log/glpi/php-errors.log

and look through the logs this error was of concern

[2024-02-08 03:40:50] glpiphplog.CRITICAL:   *** PHP Error (1): Allowed memory size of 134217728 bytes exhausted (tried to allocate 49203500 bytes) in /var/www/html/glpi/vendor/phpmailer/phpmailer/src/PHPMailer.php at line 3430

so I did some searching online (really wish I could post links to help people out) and found that I needed to edit

/etc/php/8.1/apache2/php.ini

and change

memory_limit=128M

to

memory_limit=1024M

then restart apache

sudo systemctl restart apache2

after which I back into "Automatic actions", opened up "queuednotification" clicked both of the "x" and waited a bit and emails started coming through again. Due to the back log of emails (only 50 emails are set to send at a time) I went into "notifications queue" and selected them all the remaining notifications and used the bulk action to send the emails out.

Going forward, when php is updated to say 8.2 am I going to have to edit

/etc/php/8.x/apache2/php.ini

and change "memory_limit"?

Forum GLPI-Project

Announcement

#1 2024-01-11 05:01:42

[Solved]Email notifications suck in queuednotifications

#2 2024-01-11 09:46:31

Re: [Solved]Email notifications suck in queuednotifications

#3 2024-01-12 01:39:13

Re: [Solved]Email notifications suck in queuednotifications

#4 2024-01-12 03:30:40

Re: [Solved]Email notifications suck in queuednotifications

#5 2024-01-12 04:44:31

Re: [Solved]Email notifications suck in queuednotifications

#6 2024-01-12 15:18:28

Re: [Solved]Email notifications suck in queuednotifications

#7 2024-01-12 15:59:50

Re: [Solved]Email notifications suck in queuednotifications

#8 2024-01-30 00:42:48

Re: [Solved]Email notifications suck in queuednotifications

#9 2024-02-05 06:43:53

Re: [Solved]Email notifications suck in queuednotifications

#10 2024-02-06 06:49:45

Re: [Solved]Email notifications suck in queuednotifications

#11 2024-02-06 11:10:19

Re: [Solved]Email notifications suck in queuednotifications

#12 2024-02-07 02:09:34

Re: [Solved]Email notifications suck in queuednotifications

#13 2024-02-07 09:59:21

Re: [Solved]Email notifications suck in queuednotifications

#14 2024-02-08 02:02:09

Re: [Solved]Email notifications suck in queuednotifications

#15 2024-02-12 03:28:31

Re: [Solved]Email notifications suck in queuednotifications

Board footer