内存高效的替代Python字典(Memory Efficient Alternatives to Python Dictionaries)

 在我目前的一个项目中，我正在扫描一些文字，查看单词三元组的频率。 在我第一次去，我使用默认字典三级深。 换句话说， topDict[word1][word2][word3]返回这些单词出现在文本中的次数， topDict[word1][word2]返回一个字典，其中包含单词1和2之后出现的所有单词等。 
 
 这功能正常，但是它非常内存密集。 在我的初始测试中，它使用像存储三进制存储在文本文件中的内存的20倍，这似乎是一个过多的内存开销。 
 
 我的怀疑是，许多这些字典正在创造出比实际使用的更多的插槽，所以我想用这种方式使用更高内存效率的其他内容替换字典。 我强烈希望有一个解决方案，允许按照字典的关键查找。 
 
 从我所知道的数据结构，使用红黑或AVL的平衡二叉搜索树可能是理想的，但我真的不愿意自己实现。 如果可能的话，我更喜欢坚持使用标准的python库，但如果他们能最好的工作，我肯定会接受其他替代方案。 
 
 那么有没有人对我有任何建议？ 
 
 编辑添加： 
 
 感谢您的答复。 到目前为止，有几个答案建议使用元组，当我将前两个词汇集成一个元组时，这对我来说并没有太大的帮助。 我犹豫使用这三个键作为关键，因为我希望能够容易地查找前两者的所有第三个字。 （即我想要像topDict[word1, word2].keys() ）的结果。 
 
 我正在玩的当前数据集是Wikipedia For Schools的最新版本。 例如，解析一千页的结果对于一个文本文件是像11MB，其中每行是三个单词，并且count个tab分隔。 以字典格式存储文本我现在使用大约需要185MB。 我知道指针会有一些额外的开销，但是差异似乎是过分的。 

In one of my current side projects, I am scanning through some text looking at the frequency of word triplets. In my first go at it, I used the default dictionary three levels deep. In other words, topDict[word1][word2][word3] returns the number of times these words appear in the text, topDict[word1][word2] returns a dictionary with all the words that appeared following words 1 and 2, etc.
 
This functions correctly, but it is very memory intensive. In my initial tests it used something like 20 times the memory of just storing the triplets in a text file, which seems like an overly large amount of memory overhead.
 
My suspicion is that many of these dictionaries are being created with many more slots than are actually being used, so I want to replace the dictionaries with something else that is more memory efficient when used in this manner. I would strongly prefer a solution that allows key lookups along the lines of the dictionaries.
 
From what I know of data structures, a balanced binary search tree using something like red-black or AVL would probably be ideal, but I would really prefer not to implement them myself. If possible, I'd prefer to stick with standard python libraries, but I'm definitely open to other alternatives if they would work best.
 
So, does anyone have any suggestions for me?
 
Edited to add:
 
Thanks for the responses so far. A few of the answers so far have suggested using tuples, which didn't really do much for me when I condensed the first two words into a tuple. I am hesitant to use all three as a key since I want it to be easy to look up all third words given the first two. (i.e. I want something like the result of topDict[word1, word2].keys()). 
 
The current dataset I am playing around with is the most recent version of Wikipedia For Schools. The results of parsing the first thousand pages, for example, is something like 11MB for a text file where each line is the three words and the count all tab separated. Storing the text in the dictionary format I am now using takes around 185MB. I know that there will be some additional overhead for pointers and whatnot, but the difference seems excessive.

原文：https://stackoverflow.com/questions/327223