首页 \ 问答 \ python中内存高效的2d可扩展数组？(Memory-efficient 2d growable array in python?)

python中内存高效的2d可扩展数组？(Memory-efficient 2d growable array in python?)

 我正在开发一个处理大量数据的应用程序。  
 ....并继续运行我的计算机内存。 :(  
 Python在变量上有大量的内存开销（根据sys.getsizeof（））。 例如，一个带有一个整数的基本元组占用56个字节。 一个空列表，64个字节。 严重的开销。  
 Numpy数组非常适合减少开销。 但它们并不是为了有效增长而设计的（参见最快的方法来增长一个数字阵列 ）。 数组（ https://docs.python.org/3/library/array.html ）似乎很有希望，但它是1d。 我的数据是2d，一个数组的行数任意，列宽度为3个浮点数（理想情况下为float32），另一个数据的列宽为2个整数（理想情况下为uint32）。 显然，使用~80字节的python结构来存储每行12或8个字节的数据将占用我的内存消耗总量。  
 在Python中将内存使用率保持为“假”2d的唯一现实方法，即将数组作为arr [row * WIDTH + column]并将行计为len（arr）/ WIDTH？ 

I'm working on an app that processes a lot of data. 
.... and keeps running my computer out of memory. :( 
Python has a huge amount of memory overhead on variables (as per sys.getsizeof()). A basic tuple with one integer in it takes up 56 bytes, for example. An empty list, 64 bytes. Serious overhead. 
Numpy arrays are great for reducing overhead. But they're not designed to grow efficiently (see Fastest way to grow a numpy numeric array). Array (https://docs.python.org/3/library/array.html) seems promising, but it's 1d. My data is 2d, with an arbitrary number of rows and a column width of 3 floats (ideally float32) for one array, and a column width of two ints (ideally uint32) for the other. Obviously, using ~80 bytes of python structure to store 12 or 8 bytes of data per row is going to total my memory consumption. 
Is the only realistic way to keep memory usage down in Python to "fake" 2d, aka by addressing the array as arr[row*WIDTH+column] and counting rows as len(arr)/WIDTH?

原文：https://stackoverflow.com/questions/45250182

更新时间：2023-07-28 16:07

最满意答案

 通过执行解析到shell cron-launcher.sh的第二个参数解决了该问题  
 fac是命令从cron echo "toto" >> /tmp/shell.log  
 SHELL=/bin/cron-launcher.sh * * * * * test echo "toto" >> /tmp/shell.log  
 在shell cron-launcher被解析为$2 。  
 因此我必须修改我的代码，如下所示：  
#!/bin/bash

#Script use to launch cron tasks

SHELL="/bin/bash"
LOGDIR="/var/log/cronlauncher/"

#echo "$USER" >> /tmp/test.log
COMMAND="$2"
eval $COMMAND

The issue was solved by executing the 2nd argument parsed to the shell cron-launcher.sh 
The fac is that the command echo "toto" >> /tmp/shell.log from the cron  
SHELL=/bin/cron-launcher.sh * * * * * test echo "toto" >> /tmp/shell.log 
is parsed as $2 in the shell cron-launcher. 
Hence I juste had to modify my code as follows: 
#!/bin/bash

#Script use to launch cron tasks

SHELL="/bin/bash"
LOGDIR="/var/log/cronlauncher/"

#echo "$USER" >> /tmp/test.log
COMMAND="$2"
eval $COMMAND

python中内存高效的2d可扩展数组？(Memory-efficient 2d growable array in python?)

最满意答案

相关问答

执行Django shell命令作为cron(Execute Django shell command as cron)[2022-02-01]

Crontab自定义shell执行cron(Crontab custom shell execute cron)[2022-03-18]

如何在不使用crontab -e的情况下更新crontab(how to update crontab without using crontab -e)[2024-02-01]

Cron作业与shell脚本(Cron job with shell script)[2023-07-26]

Cron和Crontab文件不在Docker中执行(Cron and Crontab files not executed in Docker)[2022-08-03]

在虚拟环境中使用cron安排django自定义命令(schedule django custom commands with cron inside a virtualenvironment)[2022-08-13]

ubuntu 10.10中的Php Cron工作不执行lampp - 需要帮助(Php Cron job in ubuntu 10.10 does not execute for lampp - Help needed)[2022-01-16]

通过shell添加CRON来运行Python(Add CRON via shell to run Python)[2024-01-19]

无法通过OpsWorks找到由Chef自定义配方创建的cron作业(Can't find cron job created by Chef custom recipe through OpsWorks)[2021-12-30]

Cron不会执行ruby脚本(Cron won't execute ruby script)[2021-08-05]

相关文章

最新问答