Forum GLPI-Project

quitaiskiluisf

Member

From: Brazil

Registered: 2014-08-31

Posts: 2

Wrong charset on tickets open by mailgate when connecting to IMAP acco

Hi!

We started using GLPI to manage the IT workflow at the company I work on and we've found an issue with the mail gateway (probably a bug). We're using Google Apps as our e-mail provider and we've set up an e-mail account to use GLPI's mail gateway feature (this very same account is used for sending ticket notifications to our users). GLPI is connecting to that account through IMAP and it can both fetch and send e-mails.

The problem is that some of the tickets opened through the mail gateway show garbled characters, just as if the encoding was wrong somewhere. Below, I've attached a picture to better illustrate the problem (the title of the ticket is correct; the description of the ticket should contain the same characters as the title):

It's important to notice that this issue only affects the body of the ticket (the subject is ok, even though it contains all those same characters that, in the body, got mangled). Another important aspect of the issue is that it seems to affect only mails fetched through IMAP (the characters are decoded properly if GLPI fetches the same message from the same server using POP3).

I've started debugging the code and found some interesting facts:

- In the MailCollector class, in the buildTicket() method: after retrieving the body of the message, the charset property is checked and, if not UTF-8, the body is converted to this character encoding;
- By looking at the IMAP messages transferred from/to the mail server, it seems the subject of the message shows up twice: first encoded as ISO-8859-1 and later using UTF-8 encoding; the body itself is UTF-8 base64 encoded (I've also attached the IMAP conversation, as it might help).
- The charset property (set by the decodeMimeString() method) is set to "ISO-8859-1".

The problem seems to be that the charset property is set to ISO-8859-1 and then, the body (which is assumed, incorrectly, to be in this character set), is converted to UTF-8, messing up with non-ascii characters.

I'm stuck at this point. I very little about the specifics of the IMAP protocol and even less on the PHP API to connect to an IMAP server (my programming background is on Python and C#, but, in neither case, I've worked with reading e-mails, just sending them). I think I've found where the problem is in code, but, I don't know exactly how to fix it; I'm afraid of making changes and end up messing some other part of GLPI.

This tests were done using GLPI 0.84.5, but I've tested 0.84.7 on a separate environment and it showed the same behaviour (in fact, the MailCollector class hadn't suffered any change from 0.84.5 to 0.84.7). The 0.84.5 tests were made on a CentOS 6.4 x64 Linux server, while the 0.84.7 tests were done in a CentOS 6.5 x64 Linux VM (PHP 5.3.3).

Has anyone already faced the same issue? Should this be reported as a GLPI bug? Or is this some misconfigured library?

quitaiskiluisf

Member

From: Brazil

Registered: 2014-08-31

Posts: 2

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

IMAP mail conversation below:

<= - data received from server
=> - data sent to server

<=  * OK Gimap ready for requests from 200.203.69.102 dc5mb96465122vdd
 => 00000000 CAPABILITY
<=  * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 XYZZY SASL-IR AUTH=XOAUTH AUTH=XOAUTH2 AUTH=PLAIN AUTH=PLAIN-CLIENTTOKEN
<=  00000000 OK Thats all she wrote! dc5mb96465122vdd
 => 00000001 AUTHENTICATE PLAIN
<=  +
 => [USER AND PASSWORD REMOVED]
<=  * CAPABILITY IMAP4rev1 UNSELECT IDLE NAMESPACE QUOTA ID XLIST CHILDREN X-GM-EXT-1 UIDPLUS COMPRESS=DEFLATE ENABLE MOVE CONDSTORE ESEARCH
<=  00000001 OK luis.helpdesk.ci@gmail.com Luís Fernando Quitaiski authenticated (Success)
 => 00000002 SELECT INBOX
<=  * FLAGS (\Answered \Flagged \Draft \Deleted \Seen $Phishing $NotPhishing)
<=  * OK [PERMANENTFLAGS (\Answered \Flagged \Draft \Deleted \Seen $Phishing $NotPhishing \*)] Flags permitted.
<=  * OK [UIDVALIDITY 1] UIDs valid.
<=  * 1 EXISTS
<=  * 0 RECENT
<=  * OK [UIDNEXT 382] Predicted next UID.
<=  * OK [HIGHESTMODSEQ 257049]
<=  00000002 OK [READ-WRITE] INBOX selected. (Success)
 => 00000003 FETCH 1 (UID ENVELOPE BODY.PEEK[HEADER.FIELDS (Newsgroups Content-MD5 Content-Disposition Content-Language Content-Location Followup-To References)] INTERNALDATE RFC822.SIZE FLAGS)
<=  * 1 FETCH (UID 381 RFC822.SIZE 3109 INTERNALDATE "30-Aug-2014 22:47:54 +0000" FLAGS () ENVELOPE ("Sat, 30 Aug 2014 19:47:34 -0300" "=?ISO-8859-1?B?VGVzdGUg4+Lg4ero6e7s7fX08vP7+frk6+8=?= =?ISO-8859-1?B?9vwgLSDDwsDBysjJzszN1dTS09vZ2sTLz9bc?=" (("=?ISO-8859-1?Q?Lu=EDs_Fernando_Quitaiski?=" NIL "lfernandoq" "gmail.com")) (("=?ISO-8859-1?Q?Lu=EDs_Fernando_Quitaiski?=" NIL "lfernandoq" "gmail.com")) (("=?ISO-8859-1?Q?Lu=EDs_Fernando_Quitaiski?=" NIL "lfernandoq" "gmail.com")) (("=?ISO-8859-1?Q?Lu=EDs_Fernando_Quitaiski?=" NIL "luis.helpdesk.ci" "gmail.com")) NIL NIL NIL "<CAKT6oy6MDe9KeV6KZaNmCwnXHwRWUytovKeiHv1QRe9+xMLJVg@mail.gmail.com>") BODY[HEADER.FIELDS (Newsgroups Content-MD5 Content-Disposition Content-Language Content-Location Followup-To References)] {2}
<=
<=  )
<=  00000003 OK Success
 => 00000004 FETCH 1 BODY.PEEK[HEADER]
<=  * 1 FETCH (BODY[HEADER] {1975}
<=  Delivered-To: luis.helpdesk.ci@gmail.com
<=  Received: by 10.114.23.202 with SMTP id o10csp363960ldf; Sat, 30 Aug 2014
<=   15:47:54 -0700 (PDT)
<=  Return-Path: <lfernandoq@gmail.com>
<=  Received-SPF: pass (google.com: domain of lfernandoq@gmail.com designates
<=   10.50.171.199 as permitted sender) client-ip=10.50.171.199
<=  Authentication-Results: mr.google.com; spf=pass (google.com: domain of
<=   lfernandoq@gmail.com designates 10.50.171.199 as permitted sender)
<=   smtp.mail=lfernandoq@gmail.com; dkim=pass header.i=@gmail.com
<=  X-Received: from mr.google.com ([10.50.171.199]) by 10.50.171.199 with SMTP id
<=   aw7mr12846175igc.27.1409438874105 (num_hops = 1); Sat, 30 Aug 2014 15:47:54
<=   -0700 (PDT)
<=  DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113;
<=   h=mime-version:from:date:message-id:subject:to:content-type;
<=   bh=9CVkKPW/Lc7tM9BivonuKDVZPHM3voBRFd+APn+ZFHs=;
<=   b=pgKZYMYfeab2pCccUGyGKnfaSRyUwaTi0KSkzlvnWnTkAZEy9nsPh56pgLntSKAHxt
<=   ebnj89X6NtH8aN/vflTlNYjZpYmLI9IvkjcVe4hU9HyzDzKBSlqeKAmWoBl2WBh9PyYa
<=   JqzdCNg6d8HV4/x4bsTmgi5tZQdw8a97u1kStkUYZ4yL55g4JUyeFEwKQptjZgAZSfKt
<=   aBMO5O/YKU2e/loD9X6U8sS9qqt2WsjBfhw7ss2G8H1SBU4EC42kIfO/DwjthjugsFWt
<=   yHQy6D0n78ZD283RnxYYs2X97mU72NAU4jHVvTbKne8KEvsba8GDA2fy17DK8heEkg5/ UKJg==
<=  X-Received: by 10.50.171.199 with SMTP id aw7mr12846175igc.27.1409438874101;
<=   Sat, 30 Aug 2014 15:47:54 -0700 (PDT)
<=  MIME-Version: 1.0
<=  Received: by 10.64.173.134 with HTTP; Sat, 30 Aug 2014 15:47:34 -0700 (PDT)
<=  From: =?UTF-8?Q?Lu=C3=ADs_Fernando_Quitaiski?= <lfernandoq@gmail.com>
<=  Date: Sat, 30 Aug 2014 19:47:34 -0300
<=  Message-ID: <CAKT6oy6MDe9KeV6KZaNmCwnXHwRWUytovKeiHv1QRe9+xMLJVg@mail.gmail.com>
<=  Subject: =?UTF-8?B?VGVzdGUgw6PDosOgw6HDqsOow6nDrsOsw63DtcO0w7LDs8O7w7nDusOkw6vDr8O2w7wgLQ==?=
<=   =?UTF-8?B?IMODw4LDgMOBw4rDiMOJw47DjMONw5XDlMOSw5PDm8OZw5rDhMOLw4/DlsOc?=
<=  To: =?UTF-8?Q?Lu=C3=ADs_Fernando_Quitaiski?= <luis.helpdesk.ci@gmail.com>
<=  Content-Type: multipart/alternative; boundary=089e0111d4206c6dc60501e08f3f
<=
<=  )
<=  00000004 OK Success
 => 00000005 FETCH 1 (BODYSTRUCTURE FLAGS)
<=  * 1 FETCH (FLAGS () BODYSTRUCTURE (("TEXT" "PLAIN" ("CHARSET" "UTF-8") NIL NIL "BASE64" 216 2 NIL NIL NIL)("TEXT" "HTML" ("CHARSET" "UTF-8") NIL NIL "BASE64" 558 7 NIL NIL NIL) "ALTERNATIVE" ("BOUNDARY" "089e0111d4206c6dc60501e08f3f") NIL NIL))
<=  00000005 OK Success
 => 00000006 FETCH 1 BODY[2]
<=  * 1 FETCH (FLAGS (\Seen) BODY[2] {558}
<=  PGRpdiBkaXI9Imx0ciI+PHNwYW4gc3R5bGU9ImZvbnQtZmFtaWx5OmFyaWFsLHNhbnMtc2VyaWY7
<=  Zm9udC1zaXplOjEzcHgiPlRlc3RlIMOjw6LDoMOhw6rDqMOpw67DrMOtw7XDtMOyw7PDu8O5w7rD
<=  pMOrw6/DtsO8IC0gw4PDgsOAw4HDisOIw4nDjsOMw43DlcOUw5LDk8Obw5nDmsOEw4vDj8OWw5w8
<=  L3NwYW4+PGRpdj48Zm9udCBmYWNlPSJhcmlhbCwgc2Fucy1zZXJpZiI+PGJyPjwvZm9udD48L2Rp
<=  dj48ZGl2Pjxmb250IGZhY2U9ImFyaWFsLCBzYW5zLXNlcmlmIj5Db250ZcO6ZG8gZGEgbWVuc2Fn
<=  ZW0uPGJyIGNsZWFyPSJhbGwiPg0KDQo8L2ZvbnQ+PGRpdj48YnI+PC9kaXY+LS0gPGJyPjxkaXY+
<=  THXDrXMgRmVybmFuZG8gUXVpdGFpc2tpPGJyPjwvZGl2PjxkaXY+PGJyPjwvZGl2Pg0KPC9kaXY+
<=  PC9kaXY+DQo=)
<=  00000006 OK Success
 => 00000007 STORE 1 +Flags (\DELETED)
<=  * 1 FETCH (FLAGS (\Seen \Deleted))
<=  00000007 OK Success
 => 00000008 EXPUNGE
<=  * 1 EXPUNGE
<=  * 0 EXISTS
<=  00000008 OK Success
 => 00000009 CLOSE
<=  00000009 OK Returned to authenticated state. (Success)
 => 0000000a LOGOUT
<=  * BYE LOGOUT Requested
<=  0000000a OK 73 good day (Success)

kpcomp

Member

Registered: 2008-07-12

Posts: 57

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Having similiar problem with every update (now with 0.85) - this fix helped:
https://forge.glpi-project.org/issues/4875

Jordi Esquius

Member

Registered: 2015-01-19

Posts: 2

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Hello,

We have exactly the same problem but I would add that the problem only happens when a file is attached to the email. If you send only text the conversion works perfect but not when a file is attached, at least in 0.85.1 distribution.

This is the only reason to avoid the go live of the application in the company. We would thank you if anyone could find a solution or if the bug is solved in the next release.

Thanks,
Jordi.

jsf

Member

From: Lisboa, Portugal

Registered: 2015-01-20

Posts: 1

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Hello,

I had the same problem and fix it this way.

• File: ./inc/ticket.class.php
  Function: convertContentForTicket
  Line: 5976

The error occurs when the ticket with an attached file is created, using the function "convertContentForTicket".
This function uses PHP DOMDocument to parse the e-mail content before create the ticket.
If I understand well, the problem is due to the absence of the tag "<head>" in some messages, so:

   change
$html = str_replace('<head>', '<head>'.$contentType, Html::entity_decode_deep($content_html));

   by
// bySTI: JSF - Test if "<head>" tag exists
$sp = strpos($html, '<head>');
if ($sp === false){
   $html = '<head>'.$contentType.Html::entity_decode_deep($content_html);
} else {
   $html = str_replace('<head>', '<head>'.$contentType, Html::entity_decode_deep($content_html));
}

Jordi Esquius

Member

Registered: 2015-01-19

Posts: 2

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Great!.. It has worked!..

Thanks a lot!
Jordi.

pgerardo

Member

Registered: 2015-01-31

Posts: 4

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Hi all,

I have the exact same problem that quitaiskiluisf, with all imported mail from gmail using IMAP.
Although I have made this

Hello,
I had the same problem and fix it this way.
File: ./inc/ticket.class.php
Function: convertContentForTicket
Line: 5976
The error occurs when the ticket with an attached file is created, using the function "convertContentForTicket".
This function uses PHP DOMDocument to parse the e-mail content before create the ticket.
If I understand well, the problem is due to the absence of the tag "<head>" in some messages, so:
   change
$html = str_replace('<head>', '<head>'.$contentType, Html::entity_decode_deep($content_html));
   by
// bySTI: JSF - Test if "<head>" tag exists
$sp = strpos($html, '<head>');
if ($sp === false){
   $html = '<head>'.$contentType.Html::entity_decode_deep($content_html);
} else {
   $html = str_replace('<head>', '<head>'.$contentType, Html::entity_decode_deep($content_html));
}

The change didn't produce any alteration in the final result. I'm doing something wrong? Thanks in advance for some help someone could provide.

stagger

Member

Registered: 2009-02-18

Posts: 1

Re: Wrong charset on tickets open by mailgate when connecting to IMAP acco

Hi,

I had the same problem but after update to 0.85.3 it's ok.