I've been trying to tune this program for better memory usage, and I
seem to be hitting a brick wall. The basic idea is that we open up
the input file, read it in as a CharBuffer, then apply regexes to the
text in the file (the regexes require a CharBuffer, String, or
StringBuffer as input). It works for most files, but when the file
gets to be large, I start getting OutOfMemory errors. In theory, I
should be reading in just a readable portion of the file at any one
time, but that doesn't seem to be actually happening. Instead, it
allocates a character buffer large enough to hold the entire input
file all at once, and if that overflows memory, it errors out. Does
anyone have any suggestions? Thanks! Chris

public void getFileSetup(File inputFile, File outputGoodFile,
File outputBadFile){
setFileIn(inputFile);

if (outputGoodFile.exists()){outputGoodFile.delete();}
setGood(outputGoodFile);
if (outputBadFile.exists()){outputBadFile.delete();}
setBad(outputBadFile);

try{
inSt = new FileInputStream(getFileIn());
setInCh(inSt.getChannel());
int sz = (int)(getInCh().size());
ByteBuffer bb = getInCh().map
(FileChannel.MapMode.READ_ONLY, 0, sz);
Charset cs = Charset.forName("UTF-8");
CharsetDecoder cd = cs.newDecoder();
//here is the new code
System.out.println("Ready to decode input
buffer.");
System.out.println(bb.remaining());
int n = (int)(bb.remaining() *
0.25);//figuring 4 bytes per char
CharBuffer cb = CharBuffer.allocate(n);
System.out.println("Set up new CharBuffer");

if(n==0){
setInBuf(cb);
}
//bb.reset();
for(;;){
CoderResult cr;
if(bb.hasRemaining()){
cr=cd.decode(bb,cb,true);
}
else{
cr=cd.flush(cb);
}
if(cr.isUnderflow()){
break;
}
if(cr.isOverflow()){
n*=2;
CharBuffer o =
CharBuffer.allocate(n);
cb.flip();
o.put(cb);
cb=o;
continue;
}
cr.throwException();
}
cb.flip();
setInBuf(cb);

//end of new code following commented line is
old code
//setInBuf(cd.decode(bb));
//System.out.println("Here is the char buffer
produced: "+getInBuf().toString());
System.out.println("Decoded input buffer");

goodSt = new FileOutputStream(getGood(),true);
setGoodCh(goodSt.getChannel());

badSt = new FileOutputStream(getBad(),true);
setBadCh(badSt.getChannel());
}
catch(FileNotFoundException fnf){
System.out.println("File specified not
found: " + fnf);
fnf.printStackTrace(System.out);
}
catch(IOException ioe){
System.out.println("I/O Exception thrown: " +
ioe);
ioe.printStackTrace(System.out);
}
}