Java BufferedWriter very slow 300 MB of data

I'm reading a text file consisting of fixed length records line by line and append some values and then write to another file.

As usual, I used BufferedWriter, and found that it takes around 20 Mins to read, append values and write to another file.

BufferedWriter br = new BufferedReader(new FileReader(infile));
if (br != null) {
    for (String line; (line= br.readLine()) != null;) {
        i= i+ 1;
        line += "    " + String.format("%09d", i) + "S";
        try {
            bw = new BufferedWriter(new FileWriter("out.txt",
                    true));
            bw.write(line);
            bw.newLine();
            bw.close();
        } catch (IOException e) {
            e.printStackTrace();
        }
    }

This is the code I have used. Reading the whole file and appending the values takes just 7 seconds, but writing to file seems to be slow. I know BufferedWriter cannot be this much slower. I didnt increase the buffer size as i'm reading line by line and for the same reason didnt try nio.

Suggest me some way to improve the speed?

Jon Skeet
people
quotationmark

You've got at least two problems here:

  • You're reopening the output file on each iteration. That's pretty much bound to be slow. Open it once at the same time you open the reader.
  • You're not closing either the reader or the writer on exceptions. Use try-with-resources blocks (Java 7) or try/finally blocks (pre-Java 7)

The latter won't be a performance problem, but it could break other code that expects the files to be closed.

Additionally, I would strongly recommend against using either FileReader or FileWriter, as both use the platform default encoding without letting you specify anything else. If you're using Java 7, use Files.newBufferedReader and Files.newBufferedWriter which:

  • Are shorter in terms of code
  • Use UTF-8 by default (which is often the best option)
  • Allow you to specify the encoding if you want something else

people

See more on this question at Stackoverflow