Big files

Topics: Developer Forum, User Forum
Mar 26, 2010 at 6:20 PM

I'm new to this component so maybe someone can give me an idea on how to use this great control to handle big files more then 1G.

Basically I read a chunk of bytes from the file stream in the loop add asynchronously Append() it to the control. At some point around 201512960 bytes it excepts with "External component has thrown an exception." with the following backtrace:

 

   at System.Windows.Forms.UnsafeNativeMethods.CallWindowProc(IntPtr wndProc, IntPtr hWnd, Int32 msg, IntPtr wParam, IntPtr lParam)
   at System.Windows.Forms.NativeWindow.DefWndProc(Message& m)
   at System.Windows.Forms.Control.DefWndProc(Message& m)
   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.SendMessageDirect(UInt32 msg, IntPtr wParam, IntPtr lParam) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 34
   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.SendMessageDirect(UInt32 msg, Int32 wParam, String lParam) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 263
   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.AppendText(Int32 length, String s) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 447
   at ScintillaNet.Scintilla.AppendText(String text) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.cs:line 3060
   at RADScillaTest.RADViewTest.backgroundWorkerLoadFile_ProgressChanged(Object sender, ProgressChangedEventArgs e) in C:\Work\RADVISION\sources\RadLogAnalizer\RADScillaTest\RADViewTest.cs:line 115

   at System.Windows.Forms.UnsafeNativeMethods.CallWindowProc(IntPtr wndProc, IntPtr hWnd, Int32 msg, IntPtr wParam, IntPtr lParam)

   at System.Windows.Forms.NativeWindow.DefWndProc(Message& m)

   at System.Windows.Forms.Control.DefWndProc(Message& m)

   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.SendMessageDirect(UInt32 msg, IntPtr wParam, IntPtr lParam) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 34

   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.SendMessageDirect(UInt32 msg, Int32 wParam, String lParam) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 263

   at ScintillaNet.Scintilla.ScintillaNet.INativeScintilla.AppendText(Int32 length, String s) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.INativeScintilla.cs:line 447

   at ScintillaNet.Scintilla.AppendText(String text) in D:\ScintillaNET\Branches\2.0\ScintillaNET\Scintilla.cs:line 3060

   at RADScillaTest.RADViewTest.backgroundWorkerLoadFile_ProgressChanged(Object sender, ProgressChangedEventArgs e) in C:\Work\RADVISION\sources\RadLogAnalizer\RADScillaTest\RADViewTest.cs:line 115

I know that the latest Notepad++ is using the same C Scintilla library and it's able to open 2G files.

Any help is appreciated.

 

Coordinator
Mar 26, 2010 at 8:57 PM

It's not easy as Scintilla was never meant to handle files that big. Even in Notepad++ general performance slows to a crawl when opening a plain text file > 1GB (I dare not try with a lexer like XML). With ScintillaNet you have the additional overhead of the .NET runtime and depending upon how you go about loading it you might be copying data around 2-3 times even converting it from ASCII to UTF-16 back to ASCII. Considering there's a 2GB memory limit per process on 32bit Windows you're almost certainly going to run out of memory before the whole file is loaded. Scintilla keeps the entire document in memory.

You could try and do some trickery that only loads portions of the file and dynamically swaps it out whenever the region is scrolled to. But if you were to do this it would probably be easier and less overhead to do it with a regular text box or some other control that maybe has this functionality built in. What are you trying to do?

Mar 26, 2010 at 10:05 PM

Thank you Chris for the quick response!

Turns out that the latest version of SciLexer.dll does the trick and eats up 1GB file without problem in 2 minutes. No editing thou but I don't need it.

I'm trying to build a special log analyzer with simple highlight feature that can work with such a huge files.

I was building a control that dynamically swaps data and it can handle 2**64-1 bytes size of files but it takes to much time to develop feature like "select" and "highlight" so I was looking for a 3rd party .NET control and ScintillaNET looks promising. 

Do you have near future plans to officially update scintilla library support from 1.7.6 to 2.0.3?

Coordinator
Mar 26, 2010 at 10:11 PM

I think Jacob is planning on using the latest Scintilla in his release. But I've never encountered any problems using a newer version of Scintilla with existing ScintillaNet code so you should be fine. Good luck and thanks for pointing out the new behavior, I'll have to go check out the new version (it's been a while).

Coordinator
Mar 27, 2010 at 5:55 AM

As Chris said there is a planned ScintillaNET release in the works that will update the SciLexer.dll to the newest version but there's really no reason you couldn't drop in a new version with the current ScintillaNET release.

I also echo what Chris said about Scintilla loading entire documents into memory and add a word of caution. Even though newer SciLexer builds appear to handle large files better, a file that large may work on your machine given the amount of memory/virtual memory you have available but could fail or become unstable on a user's machine that does not have as many resources.

 

Jacob