Looking for more information on how to do PHP the right way? Check out PHP: The Right Way

Ahmed Shreef's Blog:
iconv misunderstands UTF-16 strings with no BOM
Aug 27, 2010 @ 18:36:56

Ahmed Shreef has a recent post to his blog about an issue he had converting UTF-16 strings over to UTF-8 with the iconv functionality in PHP. Specifically, he ended up with "rubbish unreadable characters" after the conversion.

I had a problem last week with converting UTF-16 encoded strings to UTF-8 using PHP's iconv library on a Linux server. my code worked fine on my machine but the same code resulted in a rubbish unreadable characters on our production server.

In his example (a basic "Hello World" in Arabic) he notes that there's no byte order mark on the string and, because of this, the iconv feature tries to guess if it's big-endian or little-endian. This guessing varies from machine to machine resulting in the inconsistencies he saw. The solution is to define the "to" and "from" for the conversion manually rather than letting it just guess.

tagged: byteordermark bom iconv utf16 utf8 convert

Link:

Danne Lundqvist's Blog:
Detecting UTF BOM - byte order mark
Apr 29, 2010 @ 16:47:03

In a new post to his blog Danne Lundqvist looks at a common pitfall that could trip you up if you're not careful with your UTF-8 data - not looking for the UTF byte order mark that tells the application if it needs to be handled as UTF content.

One such thing is the occurrence of the UTF byte order mark, or BOM. [...] For UTF-8, especially on Windows, it has become more and more common to use it to indicate that the file is indeed UTF. Most text editors handle this well and you won’t ever see these bytes. As it should be.

He points out what could cause an issue - using strcmp or substr but it can be prevented by looking at and removing those first three bytes if needed. He includes a snippet of code that does just that.

tagged: byteordermark utf utf8 utf16 detect

Link:


Trending Topics: