Rails：尽管使用UTF8编码序列化哈希的困境(Rails: encoding woes with serialized hashes despite UTF8)

 我刚刚从ruby 1.9.2更新到ruby 1.9.3p0（2011-10-30修订版33570）。 我的Rails应用程序使用postgresql作为其数据库后端。 与数据库编码一样，系统区域设置为UTF8。 rails应用程序的默认编码也是UTF8。 我有中文用户输入中文字符和英文字符。 字符串存储为UTF8编码的字符串。 
 
 Rails版本：3.0.9 
 
 由于更新数据库中的某些现有中文字符串不再正确显示。 这不会影响所有字符串，但只会影响序列化哈希的一部分。 所有其他以纯字符串形式存储的字符串仍然显示为正确。 
 
 
 例： 
 
 这是一个序列化的散列，它以UTF8字符串形式存储在数据库中： 
 
broken = "--- !map:ActiveSupport::HashWithIndifferentAccess \ncheckbox: \"1\"\nchoice: \"Round Paper Clips  \\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\"\ninfo: \"10\\xE7\\x9B\\x92\"\n"
 
 为了将这个字符串转换为红宝石哈希，我使用YAML.load反序列化它： 
 
broken_hash = YAML.load(broken)
 
 这会返回一个包含乱码的内容： 
 
{"checkbox"=>"1", "choice"=>"Round Paper Clips  ï¼\u0088å\u009B\u009Eå½¢é\u0092\u0088ï¼\u0089\r\n", "info"=>"10ç\u009B\u0092"}
 
 乱码的东西应该是UTF8编码的中文。 broken_hash['info'].encoding告诉我，ruby认为这是#<Encoding:UTF-8> 。 我不同意。 
 
 有趣的是，所有其他没有序列化的字符串看起来不错，但是。 在同一个记录中，不同的字段包含看起来恰到好处的中文字符 - 在rails控制台，psql控制台和浏览器中。 每一个字符串---不管是序列化的散列还是普通的字符串---由于更新看起来都很好，所以保存到数据库中。 
 
 
 尽管ruby声称这已经是UTF-8，我试图将可能出错的编码（如GB2312或ANSI）中的乱码文本转换为UTF-8，当然我失败了。 这是我使用的代码： 
 
require 'iconv'
Iconv.conv('UTF-8', 'GB2312', broken_hash['info'])
 
 这个失败是因为ruby不知道如何处理字符串中的非法序列。 
 
 我真的只想运行一个脚本来修复所有旧的，大概破碎的序列化哈希字符串，并完成它。 有没有办法将这些断开的字符串转换成类似中文的东西？ 
 
 
 我刚刚在原始字符串中编码了UTF-8字符串（在上面的例子中称为“破碎”）。 这是以序列化字符串编码的中文字符串： 
 
 chinese = "\\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\" 
 
 我注意到，很容易通过转义它来转换为真正的UTF-8编码字符串（删除转义反斜杠）。 
 
 chinese_ok = "\xEF\xBC\x88\xE5\x9B\x9E\xE5\xBD\xA2\xE9\x92\x88\xEF\xBC\x89\r\n" 
 
 这将返回一个正确的UTF-8编码的中文字符串： "（回形针）\r\n" 
 
 只有当我使用YAML.load(...)将字符串转换为红宝石哈希时，事情才会分崩离析。 也许我应该在输入YAML.load之前处理原始字符串。 只是让我想知道为什么这样... 
 
 
 有趣！ 这很可能是由于YAML引擎“心灵”在1.9.3中默认使用的。 我使用YAML::ENGINE.yamler = 'syck'切换到“syck”引擎，并且正确解析了断开的字符串。 

I've just updated from ruby 1.9.2 to ruby 1.9.3p0 (2011-10-30 revision 33570). My rails application uses postgresql as its database backend. The system locale is UTF8, as is the database encoding. The default encoding of the rails application is also UTF8. I have Chinese users who input Chinese characters as well as English characters. The strings are stored as UTF8 encoded strings.
 
Rails version: 3.0.9
 
Since the update some of the existing Chinese strings in the database are no longer displayed correctly. This does not affect all strings, but only those that are part of a serialized hash. All other strings that are stored as plain strings still appear to be correct.
 
 
Example:
 
This is a serialized hash that is stored as a UTF8 string in the database:
 
broken = "--- !map:ActiveSupport::HashWithIndifferentAccess \ncheckbox: \"1\"\nchoice: \"Round Paper Clips  \\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\"\ninfo: \"10\\xE7\\x9B\\x92\"\n"
 
In order to convert this string to a ruby hash, I deserialize it with YAML.load:
 
broken_hash = YAML.load(broken)
 
This returns a hash with garbled contents:
 
{"checkbox"=>"1", "choice"=>"Round Paper Clips  ï¼\u0088å\u009B\u009Eå½¢é\u0092\u0088ï¼\u0089\r\n", "info"=>"10ç\u009B\u0092"}
 
The garbled stuff is supposed to be UTF8-encoded Chinese. broken_hash['info'].encoding tells me that ruby thinks this is #<Encoding:UTF-8>. I disagree.
 
Interestingly, all other strings that were not serialized before look fine, however. In the same record a different field contains Chinese characters that look just right---in the rails console, the psql console, and the browser. Every string---no matter if serialized hash or plain string---saved to the database since the update looks fine, too.
 
 
I tried to convert the garbled text from a possible wrong encoding (like GB2312 or ANSI) to UTF-8 despite ruby's claim that this was already UTF-8 and of course I failed. This is the code I used:
 
require 'iconv'
Iconv.conv('UTF-8', 'GB2312', broken_hash['info'])
 
This fails because ruby doesn't know what to do with illegal sequences in the string.
 
I really just want to run a script to fix all the old, presumably broken serialized hash strings and be done with it. Is there a way to convert these broken strings to something resembling Chinese again?
 
 
I just played with the encoded UTF-8 string in the raw string (called "broken" in the above example). This is the Chinese string that is encoded in the serialized string:
 
chinese = "\\xEF\\xBC\\x88\\xE5\\x9B\\x9E\\xE5\\xBD\\xA2\\xE9\\x92\\x88\\xEF\\xBC\\x89\\r\\n\"
 
I noticed that it is easy to convert this to a real UTF-8 encoded string by unescaping it (removing the escape backslashes).
 
chinese_ok = "\xEF\xBC\x88\xE5\x9B\x9E\xE5\xBD\xA2\xE9\x92\x88\xEF\xBC\x89\r\n"
 
This returns a proper UTF-8-encoded Chinese string: "（回形针）\r\n"
 
The thing falls apart only when I use YAML.load(...) to convert the string to a ruby hash. Maybe I should process the raw string before it is fed to YAML.load. Just makes me wonder why this is so...
 
 
Interesting! This is likely due to the YAML engine "psych" that's used by default now in 1.9.3. I switched to the "syck" engine with YAML::ENGINE.yamler = 'syck' and the broken strings are correctly parsed.

原文：https://stackoverflow.com/questions/8558101