Subject:Processing Large Numbers of Files

Posted by: ALewisohn

Date:5/31/2006 6:35:27 AM

I have a process that performs the following steps:

1. Reads a file containing absolute folder paths
2. Loops through those folders and their subfolders and finds any files of type SWA and puts them in a Queue. (About 96,000)
3. Dequeues each file, checks to see if it exists, Resamples it, and converts it to an MP3.

The problem is that the process runs fine for the first 1200 files or so, taking about .3 seconds to process each one. After that it gets progressively slower. I let it run for about 8 hours and it had slowed down to 38 seconds a file. The box it's running on is very fast.

Intel Xeon Dual 3.06 GHz w/ 3.75 GB of Ram

Subject:RE: Processing Large Numbers of Files

Reply by: _TJ

Date:5/31/2006 4:17:54 PM

That sounds like ISfFileHost objects arent getting closed or released.

Make sure that you ISfFileHost.Close(...) your files when you are done with them.

Also, files opened by script are held partially open by the script for as long as their is a variable that references the object. This is an artifact of the way the .NET runtime works. You can influence garbage collection by explicitly setting a variable to null after it's done processing. The earlier you do that, the more quick the object will get collected. So I would suggest that you look closely at your code for ISfFileHost objects that are getting abandoned rather than released.

If that doesnt' work, you may want to pause every 1000 files or in an DoEvents() loop and give the app and/or the .NET runtime a chance to finish closing out objects. If Sound Forge never gets a change to process messages, then it never gets a change to release certain objects. calling ISfFileHost.WaitForDoneOrCancel() is the normal way you give Sound Forge a chance to process messages, so make sure that you script it doing that at least once for every file.

tj

Subject:RE: Processing Large Numbers of Files

Reply by: ALewisohn

Date:6/1/2006 8:39:09 AM

I was already doing a Close on most files, and I implemented a DoEvents on the ActiveWindow. The only place where I'm still having a problem is that when Sound Forge fails to open a file all subsequent attempts to open a file fail as well. Here is my code to open a file:

ISfFileHost currentFile = null;

// Open the Current SWA File
try
{
currentFile = app.OpenFile(file, true, false);
currentFile.WaitForDoneOrCancel();
}
catch
{
currentFile = null;
app.DoEvents(app.Win32Window.Handle);
return;
}

I added the DoEvents in there in the hope that redrawing the application would fix it. But no go.

Subject:RE: Processing Large Numbers of Files

Reply by: _TJ

Date:6/1/2006 3:46:02 PM

the catch body only executes if there is an error in the try body,

try {} catch {}

is the construct for catching errors and preventing them from terminating your script, I think you wan to use the

try { } finally {}

construct here instead. finally is for necessary cleanup, it will execute after the try body if the try body fails OR if it succeeds.

tj