ExamGuru IT/.Net Guru : Framework is slow to open text files for append access on microsoft.public.dotnet.framework.performance

.NETGURU

Framework is slow to open text files for append access

Messages

Related Types

This message was discovered on microsoft.public.dotnet.framework.performance.
Responses highlighted in red are from those people who are likely to be able to contribute good, authoratitive information to this discussion. They include Microsoft employees, MVP's and others who IMHO contribute well to these kinds of discussions.

Post a new message to this list...

Nick Bennett

I've got a server that appends text to the end of a file whenever it does
anything. If I close the file after each write, the server runs like a dog.
If I don't it's fine, but the fact that the file is then permanently open
for writing bothers me - like I can't delete it unless I first shut the
server down.

The underlying Windows API's for opening a file and seeking to the end are
fast enough - why is the Framework so slow? Is there a fix?

Nick

Reply to this message...

Justin Rogers

Couple of problems. If you are opening this file often, you may have some
locking issues with reopening the file. Second, there is a lot going on inside
of StreamWriter and FileStream (depending on what level you are going
in at). StreamWriter allocates a bunch of extra buffers, that takes a bit of
extra time. FileStream has some extra stuff as well, but for the most part
it just calls CreateFile and Seek internally for an append mode operation.

Looks like doing a StreamWriter is about 4 times slower at opening than
a FileStream.

--
Justin Rogers
DigiTec Web Consultants, LLC.
Blog: http://weblogs.asp.net/justin_rogers

"Nick Bennett" <Click here to reveal e-mail address> wrote in message
news:e%Click here to reveal e-mail address...
[Original message clipped]

Reply to this message...

Brian Grunkemeyer (VIP)

Yes, Justin is right about StreamWriter doing a lot more work than simply
opening a FileStream. I haven't measured his additional 4x overhead for
allocating a StreamWriter over a FileStream, but I can definitely imagine
this adds up to something significant.

Interestingly, we've added two methods to help you out with appending text
to a file in Whidbey to the File class:

public static void AppendAll(String path, String contents)
{
AppendAll(path, contents, StreamWriter.UTF8NoBOM);
}

public static void AppendAll(String path, String contents, Encoding
encoding)
{
using(StreamWriter sw = new StreamWriter(path, true, encoding))
sw.Write(contents);
}

Obviously, you could write something like this yourself (replace the
UTF8NoBOM with "new UTF8Encoding(false, true)"). Could you check that your
code is doing something roughly comparable with this, the best naive
implementation we currently have? You could imagine an AppendAllLines
method that took a String[] and did a foreach over the array inside the
using block as well, if necessary. (Don't concat many lines together in
memory just to meet this API's signature, if you can avoid it. Instead add
a new API.)

Additionally, I'll see if we can do some perf work here. The buffer size
used by the StreamWriter may be a bit too big, especially if you're only
adding in ~80 characters of text. Also, we could write this to open a
FileStream, write out the encoding's preamble if necessary, call
Encoding.GetBytes, then write that data to a file (ie, everything a
StreamWriter would have done, but without allocating one extra finalizable
object, and possibly some reduced overhead from buffering). Avoiding
allocating a StreamWriter here might be an interesting perf win. It might
hurt us for very long strings though, since a naive implementation would
allocate a byte[] proportional to the length of your String, whereas
StreamWriter keeps it to arrays of ~1K. I'll have to check it out.

Brian Grunkemeyer
MS CLR Base Class Library team

Reply to this message...

Brian Grunkemeyer (VIP)

Just to follow up on this, I have looked at a few options, and I have a
hopefully reasonable suggestion for you. If possible, you should construct
a StreamWriter with a custom FileStream instance. You want to set certain
options on that FileStream (to get the FILE_SHARE_DELETE semantics), and
you want to set the StreamWriter's AutoFlush property to true.
File.AppendText won't do all of this for you.

You can package the following any way you want in your code, but here's
something that might get you started. (Feel free to refactor this to a
class constructor or wherever else is appropriate for your code).

internal static class Logger {
private static TextWriter _log;

internal static void Setup() {
// On Whidbey, you can pass in FileShare.Delete in addition to
FileShare.Read.
FileStream fs = new FileStream("fileName.log", FileMode.Append,
FileAccess.Write, FileShare.Read | FileShare.Delete);
StreamWriter sw = new StreamWriter(fs, Encoding.UTF8); // Encoding
is optional - this one will include a Unicode BOM.
sw.AutoFlush = true;
_log = sw;
}

[Conditional("LOGGING")]
internal void WriteLine(String value) {
if (_log == null)
Setup();

if (you want this logging turned on)
_log.WriteLine(value);
}
}

Now every time you call WriteLine or Write on the StreamWriter, we'll write
the data to the FileStream and call Flush on it. The data isn't
immediately _committed_ to disk, but it is flushed out of any managed
buffers and given to the OS's disk cache. (ie, unless you pull the plug on
your computer or hit a blue screen, the data is written to your log file.)
You'll also be able to delete the file due to the FileShare.Delete
parameter, an existing OS feature that we enabled in Whidbey.

Here are some perf numbers for a somewhat pathological benchmark that
simulated your logging code:

Using File.AppendAll: 5829 ms
Using a StreamWriter with custom buffer sizes: 5749 ms
If we had File.AppendAllLines: 247 ms
<-----------------------
Using a FileStream directly: 5378 ms
Cached StreamWriter w/ AutoFlush: 481 ms
<-----------------------

We'll look into File.AppendAllLines as well for some version (maybe
Whidbey, maybe later) - it won't help you directly in your scenario though,
since you probably don't have all the lines you want to append ready to
write at one point in time. But it at least makes the microbenchmarks look
interesting, and it may come in handy for some people.

Brian Grunkemeyer
CLR Base Class Library team

Reply to this message...

	System.IO.File
	System.IO.FileAccess
	System.IO.FileMode
	System.IO.FileShare
	System.IO.FileStream
	System.IO.StreamWriter
	System.IO.TextWriter
	System.Text.Encoding
	System.Text.UTF8Encoding

ExamGuru IT Solutions - .Net Guru is owned and operated by ExamGuru, Inc., the man behind .Net Guru. If you're in the market for bespoke software or software consultancy, why not get him and his highly trained team to help? - www.examguru.net/ITCertification

Need Dot Net Interview Questions?
Ask ExamGuru, Inc. for advice and help on Passing .Net Interviews

.Net Projects
Best-of-breed application framework for .NET projects, developed by ExamGuru, Inc. and ExamGuru IT

Free .net Help
Commission ExamGuru, Inc. and his team for your next bespoke software project

FogBUGZ
The only bug tracking system carefully crafted with one goal in mind: helping teams create great software.

Awesome Tools
If you don't know about these, you're missing out... IT Certification Questions
IT Interview Questions
Free Oracle 10g Training
MCSE Boortcamp
Cisco Study Guides
Cheap Study Guides
Exact Questions
Dot Net Interview Questions
Oracle OCP
Cheap Travel
Designer Perfumes - Wholesale Prices
Free Programming Tutorials

	Copyright © ExamGuru, Inc. 2001-2006
	Contact Us - Terms of Use - Privacy Policy - www.dot-net-guru.com - www.examguru.net - www.oraclesource.net - www.itinterviews.net - www.examguru.net/ITCertification