首页 \ 问答 \ 在Matlab中读取和处理大型文本文件(Reading and processing a large text file in Matlab)

在Matlab中读取和处理大型文本文件(Reading and processing a large text file in Matlab)

我正在尝试将大文本文件(几百万行)读入Matlab。 最初我使用的是importdata(file_name),这似乎是一个简洁的解决方案。 但是我需要使用Matlab 7(是的,我知道它的旧版本),似乎不支持importdata。 因此,我尝试了以下内容:

while ~feof(fid)    
    fline = fgetl(fid);
    fdata{1,lno} =  fline ;
    lno = lno + 1;
end

但这真的很慢。 我猜它是因为它在每次迭代时调整数组大小。 有没有更好的方法来做到这一点。 请记住,输入数据的前20行是字符串类型数据,其余数据是3到6列十六进制值。


I'm trying to read a large text file (a few million lines) into Matlab. Initially I was using importdata(file_name), which seemed like a concise solution. However I need to use Matlab 7 (yeah I know its old) and it seems importdata isn't supported. As such I tried the following:

while ~feof(fid)    
    fline = fgetl(fid);
    fdata{1,lno} =  fline ;
    lno = lno + 1;
end

But this is really slow. I'm guessing its because its resizing the array on each iteration. Is there a better way of doing this. Bearing in mind the first 20 lines of the input data are string type data and the remainder of the data is 3 to 6 columns of hexadecimal values.


原文:https://stackoverflow.com/questions/5953182
更新时间:2023-08-08 18:08

最满意答案

你永远不会得到一个.NET应用程序立即显示的启动画面。 即使你已经使用程序集来消除JIT编译时间,你仍然需要等待所有的.NET Framework DLL被加载到内存中。 这是一个非常大的框架,并且需要花费不少的时间来加载冷启动。 没有什么可以真正做到的。

微软试图尽可能地缓解痛苦。 WindowsFormsApplicationBase (它在Microsoft.VisualBasic命名空间中定义,但不要让它吓跑你;它完全可以从C#应用程序中使用)提供了一个用于显示启动画面的内置机制。 您所要做的就是将其SplashScreen属性设置为适当的形式,其他所有内容都在幕后处理。 即使在冷启动的情况下,它已经针对最大响应时间进行了大量优化,但仍不会立即发生

您唯一的选择是在非托管代码中编写小封装,其唯一目的是尽快抛出初始屏幕,然后调用.NET应用程序开始启动自身。 当然,这里的打火机越好越好。 C ++是一个选项,但C可能是更好的选择。 您需要最大限度地减少必须链接的外部库的数量,因此像MFC或Qt这样的框架绝对是不可用的:您需要直接定位Windows API。

Visual Studio团队在VS 2010中做了类似的事情。他们对他们的博客上提供的流程有一个非常有趣的解释:

尽管Visual Studio 2010使用WPF作为其主窗口,但使用WPF作为启动屏幕需要等待CLR和WPF初始化,然后才能在屏幕上绘制单个像素。 尽管我们在.Net 4.0中对CLR和WPF启动速度做了一些巨大的改进,但仍然不能完全匹配原始Win32的性能。 因此,我们选择了原生C ++代码和Win32作为启动画面。

但我不会在这里花太多时间。 由于您通常应该给用户启用和关闭启动屏幕的选项,并且大多数用户会选择关闭启动屏幕,因此很多人不太可能首先看到它。 任何好的优化分析器都会告诉你,这不值得优化。


You're never going to get a splash screen for a .NET application to show instantly. Even if you've NGen'ed the assemblies to eliminate the JIT-compile time, you still have to wait while all of the .NET Framework DLLs are loaded into memory. It's a pretty large framework, and it takes a non-trivial amount of time to load upon a cold start. Nothing you can really do about that.

Microsoft has tried to ease the pain as much as possible. The WindowsFormsApplicationBase class (it's defined in the Microsoft.VisualBasic namespace, but don't let that scare you off; it's perfectly usable from a C# application) provides a built-in mechanism for showing a splash screen. All you have to do is set its SplashScreen property to the appropriate form, and everything else is handled behind the scenes. It's been heavily optimized for maximum response time, even in a cold-start situation, but it's still not going to be instant.

The only other option that you have is to write small wrapper in unmanaged code, whose only purpose is to throw the splash screen up as quickly as possible, then call your .NET application to start launching itself. The lighter the better here, of course. C++ is an option, but C is probably a better option. You need to minimize the number of external libraries that you have to link, so a framework like MFC or Qt is definitely out: you need to target the Windows API directly.

The Visual Studio team did something similar in VS 2010. They have a pretty interesting explanation of the process available on their blog:

Even though Visual Studio 2010 uses WPF for its main window, using WPF for the splash screen would require that we wait for the CLR and WPF to initialize before we could paint a single pixel on the screen. While we’ve made some tremendous improvements in CLR and WPF startup speed in .Net 4.0, it still can’t quite match the performance of raw Win32. So, the choice was made to stay with native C++ code and Win32 for the splash screen.

But I wouldn't spend too much time here. Since you should generally give users the option to turn splash screens on and off, and most users will opt to turn it off, it's unlikely that very many people will ever see it in the first place. Any good optimization profiler would tell you that it isn't worth optimizing.

相关问答

更多
  • 请参阅WPF 3.5 SP1:启动屏幕 或者在VS2010中点击Solution Explorer Add -> New Item ,从已安装的模板列表中选择WPF ,并且Splash Screen应位于中间列表的底部。 注意:启动画面在构造函数之后和主窗口Window_Loaded回调之前/之后被移除。 我将所有的初始化都移到了主窗口构造函数中,并且它很有用,而且非常简单。 See WPF 3.5 SP1: Splash Screen Or within VS2010 click on the Solut ...
  • 我以前为独立的JavaFX 2.0应用程序创建了一个启动页面示例 。 我更新了示例,演示了通过启动页面上的进度条和进度文本监视加载进度。 要使代码适应监视应用程序的初始化进度,而不是将ProgressBar绑定到WebEngine的loadWorker.workDone属性,请创建一个执行昂贵的初始化工作的JavaFX 任务 ,并通过任务的progressProperty和messageProperty监视任务的进度 。 这里是基于前面段落中概述的任务方法的基于飞溅的样本的链接。 对于WebStart或浏览 ...
  • 有几种方法可以做,您可以在如何创建启动画面中更详细地看到(Java教程>使用JFC / Swing创建GUI>使用其他Swing功能) 使用java -splash: 在manifest.mf文件中使用Main-Class: 和SplashScreen-Image: 特别是对于NetBeans,您可以看到以下教程: Splash Screen Beginner Tutorial - NetBeans Wiki ...
  • 必须刷新ApplicationUnderTest作为启动屏幕的元素是试图搜索的元素。 Had to refresh the ApplicationUnderTest as the elements for the splash screen were the ones attempting to search.
  • 内部项目文件: program Project1; uses Forms, Unit1 in 'Unit1.pas' {Form1}, uSplashScreen in 'uSplashScreen.pas' {frmSplashScreen}; {$R *.res} begin Application.Initialize; Application.MainFormOnTaskbar := True; frmSplashScreen := TfrmSplashScreen. ...
  • 你永远不会得到一个.NET应用程序立即显示的启动画面。 即使你已经使用程序集来消除JIT编译时间,你仍然需要等待所有的.NET Framework DLL被加载到内存中。 这是一个非常大的框架,并且需要花费不少的时间来加载冷启动。 没有什么可以真正做到的。 微软试图尽可能地缓解痛苦。 WindowsFormsApplicationBase类 (它在Microsoft.VisualBasic命名空间中定义,但不要让它吓跑你;它完全可以从C#应用程序中使用)提供了一个用于显示启动画面的内置机制。 您所要做的就是 ...
  • 您可以禁用应用程序的多任务处理功能,并在每次从主屏幕重新打开应用程序时从头开始。 @implementation MyAppDelegate - (void) applicationDidEnterBackground:(UIApplication *)application { exit(0); } @end 或者请参阅“ iOS应用程序编程指南 ”中的“ 退出后台执行 ”: “...您可以通过将UIApplicationExitsOnSuspend键添加到应用程序的Info.plist ...
  • StackOverflow上的答案已经有问题可以帮助您实现这一目标: 如何制作闪屏? 将视频文件集成到Android应用程序中作为应用程序背景 (参见接受的答案 There are already questions with answers on StackOverflow that can help you achieve this: How do I make a splash screen? Integrating video file in android app as app backgroun ...
  • 在mainView构造函数中 public MainView() { SplashScreen splashScreen = new SplashScreen(); splashScreen.Show(); ... } 然后 Action emptyDelegate = delegate { }; bool IsContentRendered = false; private void WindowViewBase_Loaded(object sender, RoutedEventArgs e ...
  • 在启动时显示启动画面很容易。 在您的应用程序的Main()方法(在Program.cs中)中,在Application.Run(...)行之前放置这样的内容: SplashForm splashy = new SplashForm(); splashy.Show(); Application.Run(new MainForm(splashy)); 修改主窗体的代码和构造函数,使其看起来像这样: private SplashForm _splashy; public MainForm(SplashForm ...

相关文章

更多

最新问答

更多
  • 您如何使用git diff文件,并将其应用于同一存储库的副本的本地分支?(How do you take a git diff file, and apply it to a local branch that is a copy of the same repository?)
  • 将长浮点值剪切为2个小数点并复制到字符数组(Cut Long Float Value to 2 decimal points and copy to Character Array)
  • OctoberCMS侧边栏不呈现(OctoberCMS Sidebar not rendering)
  • 页面加载后对象是否有资格进行垃圾回收?(Are objects eligible for garbage collection after the page loads?)
  • codeigniter中的语言不能按预期工作(language in codeigniter doesn' t work as expected)
  • 在计算机拍照在哪里进入
  • 使用cin.get()从c ++中的输入流中丢弃不需要的字符(Using cin.get() to discard unwanted characters from the input stream in c++)
  • No for循环将在for循环中运行。(No for loop will run inside for loop. Testing for primes)
  • 单页应用程序:页面重新加载(Single Page Application: page reload)
  • 在循环中选择具有相似模式的列名称(Selecting Column Name With Similar Pattern in a Loop)
  • System.StackOverflow错误(System.StackOverflow error)
  • KnockoutJS未在嵌套模板上应用beforeRemove和afterAdd(KnockoutJS not applying beforeRemove and afterAdd on nested templates)
  • 散列包括方法和/或嵌套属性(Hash include methods and/or nested attributes)
  • android - 如何避免使用Samsung RFS文件系统延迟/冻结?(android - how to avoid lag/freezes with Samsung RFS filesystem?)
  • TensorFlow:基于索引列表创建新张量(TensorFlow: Create a new tensor based on list of indices)
  • 企业安全培训的各项内容
  • 错误:RPC失败;(error: RPC failed; curl transfer closed with outstanding read data remaining)
  • C#类名中允许哪些字符?(What characters are allowed in C# class name?)
  • NumPy:将int64值存储在np.array中并使用dtype float64并将其转换回整数是否安全?(NumPy: Is it safe to store an int64 value in an np.array with dtype float64 and later convert it back to integer?)
  • 注销后如何隐藏导航portlet?(How to hide navigation portlet after logout?)
  • 将多个行和可变行移动到列(moving multiple and variable rows to columns)
  • 提交表单时忽略基础href,而不使用Javascript(ignore base href when submitting form, without using Javascript)
  • 对setOnInfoWindowClickListener的意图(Intent on setOnInfoWindowClickListener)
  • Angular $资源不会改变方法(Angular $resource doesn't change method)
  • 在Angular 5中不是一个函数(is not a function in Angular 5)
  • 如何配置Composite C1以将.m和桌面作为同一站点提供服务(How to configure Composite C1 to serve .m and desktop as the same site)
  • 不适用:悬停在悬停时:在元素之前[复制](Don't apply :hover when hovering on :before element [duplicate])
  • 常见的python rpc和cli接口(Common python rpc and cli interface)
  • Mysql DB单个字段匹配多个其他字段(Mysql DB single field matching to multiple other fields)
  • 产品页面上的Magento Up出售对齐问题(Magento Up sell alignment issue on the products page)