Serious HDV memory flaw in Vegas?

Sunfox wrote on 1/21/2005, 9:06 PM

First, you can read about my HDV exploits here: http://mediasoftware.sonypictures.com/Forums/ShowMessage.asp?MessageID=350531&Replies=16&Page=1

...where it was basically determined that I was running out of memory.

So, at any rate today I had this brilliant idea of replacing the SD clips in my finished 7 minute video with their HD counterparts - since I know exactly what clips I need, etc.

So, at first this went swimmingly, but when I reached the 5 minute mark replacing clips I started having the exact same problem - my 1gb memory was full and the pagefile had grown to near 2gb and caused playback to stop.

So I added another gigabyte of memory, upping the system to 2gb real memory. Loaded the file back up... and I continue to have **exactly** the same problem, except that now my entire 2gb of memory is used *and* a near 2gb pagefile is created.

What the - exactly what was working in 3gb before is now consuming 4gb and allowing no further expansion?

At this point I am severely annoyed with the whole Vegas/CineForm/HDV thing, especially since it appears that buying the most powerful PC on the planet will NOT help the situation.

Am I expected to only want to create 30 second HDV clips?

What I don't understand is why merely having the HDV clips on the timeline is consuming so much more memory than their SD counterparts. We're not talking rendering or even playing the clips back at this point, I just mean having the clips *exist*. Sure, they're higher resolution and bigger in file size, but what kind of information is Vegas trying to keep in active memory that needs so much more space for HDV versus DV videos?

Comments

farss wrote on 1/21/2005, 9:42 PM

Not that I can really help here but simply dropping HiRes still into an SD project does gobble something up so I'd imagine there's some correlation with HDV. However it sounds to me like your problem mught be related to memory leakage. It doesn't take much, a dodgy address line driver or whatever somewhere on the mobo might be your problem.

Maybe try some tests, try putting HiRes still or generated media onto the time line. Try pulling things out to narrow the problem down a bit more. If you still get this problem then that clears the Cineform code.
Bob.

TheHappyFriar wrote on 1/21/2005, 9:49 PM

Did you try telling your swap file to be a max of, say, 20gb? See how much it sucks up then? And did you end up calling that chad guy that spot recomended? They might be able tosolve your problem.

I can't test out this codec (I've got Win2k) but I've loaded HD TGA sequences running several minutes in length into Vegas w/o memory problems. Even ran different effects on them no problem (besides what would be expected from an XP 1800 CPU, performance wise) From looking at the cineform site, it says that this is a WMV/MPEG-2 codec. I don't think you said how the system worked when you turned the timeline thumbnails off. did you try that? Also, do you have any other HD codec's installed, or any other mpeg-2 codec's at all (besides mainconcept)?

Sunfox wrote on 1/21/2005, 10:03 PM

Well, regarding memory leakage all I can say is that the memory seems to be reported by the system as in use by vegas50.exe, and clears up as soon as its shut down...

The file has about 28 1920x1080 stills on it, and they do take memory yes, but seemingly nothing like these HDV clips.

I just did a test: if I delete a pile of the HDV clips off the timeline and then do a "remove unused files" from the media pool, I get memory back. If I delete all the HDV clips I get all my memory back. So it seems to be something with how Vegas is initializing these clips.

Sunfox wrote on 1/21/2005, 10:26 PM

Actually I'm pretty sure Windows XP has a maximum of 4gb for its pagefile, and yes that's what it's set to. But Vegas still stops shy of 2gb before giving errors. I haven't spoken to the Cineform folks, since I'm feeling less and less like this is actually a codec issue.

I have the "GSpot" codec appliance installed, and it only reports the Cineform codec as usable on the system by those AVI files, so I don't think there's any interferance there. There's probably a pile of MPEG2 codecs on here, but I don't think they come into play with Connect HD.

At any rate, as I reported in my other response to this thread, deleting HDV clips off the timeline and then doing a "remove unused media from project" DOES give me the memory back, so it seems to be something with how Vegas initializes these clips.

I just tried turning off timeline thumbnails (as well as media pool thumbnails) and simply loading the project:

With thumbnails: 15mb free, 1.26gb pagefile
Without thumbnails: 786mb free, 603mb pagefile

...So there's a big savings to not having thumbnails. However, the savings is short lived as playing back 60 seconds of the video sucks memory down to 116mb free with a 1.15gb pagefile. The memory usage remains when I stop playing the file. Interestingly, if I then delete those clips and remove them from the project I get most of the memory back.

mdopp wrote on 1/21/2005, 11:10 PM

Hm, I have a P4 3,06 GHz and 1 GB of RAM running XP-SP2 here.
I have just created a timeline of over 1 hour (!) using a total of 450 CFHD coded HDV Clips - no problem whatsoever. Of course thumbnails are turned on both in the media-pool and in the timeline.
My pagefile is used up to 1,5 GB and I still have some 200 MB RAM left over.
I suggest you take your problem to Cineform's support. They are very helpful.
Or you post at www..sonyhdvinfo.com. A staff member of Cineform (David Newman) is constantly monitoring the forum and gives good advice on any question regarding their products.
I take it for granted you use the latest update of Vegas (5.0d) and Cineform HD.
One final thought: If you haven't upgraded to SP2 this might be a good time to try.
Martin

Spot|DSE wrote on 1/22/2005, 2:35 AM

Same here, we're doing longform timelines all day long, and moreover, putting them on, stripping them off, after showing color correction, filters, etc.
All of that is nasty on the memory, and it's not hiccuping at all. It's got to be something more than just Cineform and Vegas.
I"m not understanding why you're not willing to talk to Thad at Cineform. That's exactly what he's there for.

JJKizak wrote on 1/22/2005, 5:36 AM

I have the same problem and I have the same motherboard. I have to render all the stills out to avi first then re-insert the avi into the timeline,
then remove those stills from the project to allow for the memory usuage. That means all of the stills in the timeline as even a total of (6) stills will cause V5 to run out of memory. After I do this then everything works ok. I also have to use the same method for rolling titles over a still picture (one 1.5 minute rolling text over 6 still pictures used almost 2 gig of memory). The resultant avi file uses nothing. These are jpeg stills. It's definitely not a Cineform problem. I think its the motherboard (875) memory combination. I use hyperthreading and the dual buss memory in automatic.

JJK

SonyEPM wrote on 1/22/2005, 6:37 AM

FYI- files that are present in the media pool, even if not used on the timeline, are "open" and as such do consume memory.

p@mast3rs wrote on 1/22/2005, 1:18 PM

Found that out the hard way today. Was using Recode to convert some files to AVC and my system came to a complete halt. Went through the same work flow the second time with the same result. Third time, left Vegas closed and every thing was smooth as silk.

p@mast3rs wrote on 1/22/2005, 1:20 PM

I hate to hijack this thread, however I have a question about the HDV Avi intermediate rendering. Is the final avi output what gets put back into the Cineform converter to go back out to a FX1 deck? Because the only files Cineform will let me import/export are .TS or .M2T files.

Sunfox wrote on 1/22/2005, 3:08 PM

OK, but I guess the question is "how much more memory for HDV than DV files". I can open up 246 standard DV files plus still shots, etc, and have plenty of memory left. Do the same with the HDV counterparts... and no go.

In fact, I can go from 792mb free / 283mb pagefile to 8mb free / 1.38gb pagefile just by adding 45 clips (22gb worth) to the media pool. And yet adding 246 clips (34gb worth) of DV media only drops me to 737mb free / 334mb pagefile.

So if the HDV memory usage I'm seeing is NOT normal or as expected than obviously I'm going to have to find what the problem is with my PC...

JJKizak wrote on 1/22/2005, 3:10 PM

The final avi output does not get put back into the Cineform converter.
You render it out first in V5 to an mpg file then change the file ending to m2t then into Cineform then PTT.

JJK

JJKizak wrote on 1/22/2005, 3:24 PM

I generally try to keep the stills below 1 meg each, 500k works pretty well. Of course high-def is about 2200 x 2200. which comes out to about 1.5 meg. or more. I can put on 550 stills about 550k each without a problem with standard DV. A ten to one zoom does stretch it a bit though. When you find out what the problem is let me know.

JJK

TheHappyFriar wrote on 1/22/2005, 3:32 PM

don't forget the HD stuff is a compress mpeg-2 file & the DV isn't compressed as much. Vegas (or any app) need to uncompress it more to use it.

Someone in the other thread (and this one) said it's the motherboard (because they have the exact same problem). I'd just do what spot said & call that guy monday morning. They've probley tested lots of MB's & would be able to tell you if it's your MB or something else.

filmy wrote on 1/22/2005, 5:31 PM

Sort of on topic, but off topic, comment here - one of the "selling" points that people love to point out about Vegas is the ability to open up more than one instance of Vegas. However based on SonyEPM's comment this is not an option in the HD/HDV reality. I did not relize this memory issue but the other day I needed to cut and paste some things from a rough cut to a locked cut...so I opened up another instance of Vegas and my entire system crashed. Now it makes perfect sense why...I probably have over 50 hours of footage in my media pool and I tired to open this project twice. (Rough cut version / fine cut version) And if I was editing HD/HDV and tried to do this - seems like I couldn't even get that far. So going back to the "bragging" of those who love to open up more than one instance of Vegas and tell about it - seems to me if I was a newbie and went out and tried to open up several HD/HDV projects and everything kept crashing it would not sell me on Vegas.

Maybe multi instances of Vegas being open should be kept quiet yeah?

TheHappyFriar wrote on 1/22/2005, 5:48 PM

Well, no other program will let you open more then one instance. And, it's not a Vegas flaw, it's a system limitation (kinda like complaining that Vegas doesn't support duel CPU's on a one CPU system).

If you had a LOT of memory (like a 64-bit comp would let you) then this wouldn't happen as often.

Then again, I've never ran out of memory with Vegas. I normally break everything up into smaller chunks- easier for me to manage.

filmy wrote on 1/22/2005, 7:04 PM

>>>it's a system limitation<<<

Well, yes but using that one could say that *any* software that crashes is due to system limitations. For example Vegas has *always* been open ended however if Vegas Video 2 came out saying it could edit all formats of video and encode all formats of video - real time - with no hardware - and than when people complained just say "It is a system limitaiton"...well, you see what I am getting at.

Yes one needs a "monster" system to edit HD material. However unless I am misunderstanding a few things the idea with the Cineform "Connect HD" option it is supposed to lower that overhead a great deal. And the thing is that if everything in the media pool is considered "open", thus considered using memory, I go back to my point about how realistic is it for people to go on about how you can open more than one instance of Vegas? I mean yes one can drive a porsche at 135 MPH into a wall and live however most people wouldn't knowingly do it...unless it was some sort of selling point. (And it doesn't seem to be SoFo/Sony who ever 'sold' the multi instance thing - it is the users)

epirb wrote on 1/22/2005, 7:19 PM

But keep in mind Filmy,the multi instance thing came into play long before HDV was around. and for most cases its a good thing. With others such as high res stills and HDV it has its limitations. But in time, this too will change I'm sure.
I'm just trying to put it all in perspective.....

filmy wrote on 1/22/2005, 7:29 PM

>>> But keep in mind Filmy,the multi instance thing came into play long before HDV was around. <<<

I know - I didn't say that but I sort of implied it. Vegas Video was/is an open ended software - it does not relay on a certian type of hardware to make it run. (So to speak - yes you need windows (and associated hardware) and yes you now need .NET and an internet connection (with associated hardware) but you can use it with a simple video capture card or a complex one. Any OCHI firewire card will work, and so on) The multi instance thing would work fine for most cases - such as web streaming and so on - and even more so with today's high end (or even low end compared to the norm 5 years ago). But as the recent DV mag review showed people love to push the fact that Vegas can open multi instances. So what I am trying to ask/say is that with this memory issue brought up in this thread, aside from the main point of this thread, there is another, more subtle, point - that of multi instances of Vegas. I think *that* is the perspective one needs to think of in this case. There are not really any sort of specs out there that say "To open multi instances of Vegas while editing HD/HDV material your system needs..." (And it really doesn't say "To open multi instances of Vegas while editing [Insert type here] material your system needs..." anywhere either)

Spot|DSE wrote on 1/22/2005, 7:57 PM

It's pretty suggestive the problem is the PC. Since you'd posted this, this morning I captured 3 HDV streams that are an hour in length each, no scene splits. (turned off scene detect in Cineform)
I then placed all three on the timeline, deleted 2 of them, and played the file out. No probs. Then I dragged one from the media pool, made it be a masked split over the first one. Still no problem.
then I placed the third beneath the other two, and reduced the upper track opacities by 60% so all three tracks are visible. Looped it, added a title track, and let it run.

Then I opened another iteration of Vegas. I put a series of short HDV events on the timeline. Played that out.
Then opened still one more iteration of Vegas, and did the same thing.
Now I've got 3 iterations opened, 2 of them are PAL and one is NTSC.
All 3 running reasonably well. (23-28fps)

Did the same thing on another system. It worked perfectly, no dropped frames but it's a dual Xeon 3.62, so I expect great performance.

Then to round out the test, I put the 3 1-hour files on a timeline in my laptop, and played those out. Playback was only 12 fps, but it ran all 3 just fine.

Jeffrey Fisher, who is manning the Technology Center at the Sundance Film Fest also tried this, and had no issues.
I'd be looking at RAM or Bios.

Sunfox wrote on 1/22/2005, 8:12 PM

Thanks for the testing - it's really what I needed to hear. Right now I'm leaning towards an operating system / this particular installation problem (I've been through different memory and have the latest Intel BIOS).

Since I'm planning on upgrading to a new system within the next few months at any rate (I really didn't want to have to do a clean reinstall of Windows until then), and speaking of dual Xeon systems, are you aware of any desktop-based (normal ATX) Xeon motherboards? Seems all I can find are server boards plus one Asus board a bit lacking in frills...

Otherwise I was just looking to getting the fastest P4 I can get my hands on and pack it with as much memory (4gb DDR2) and hard drive space (two 400 or 500gb drives in RAID) as I can muster.

farss wrote on 1/22/2005, 11:13 PM

If you're going for dual Xeons go with the latest from SuperMicro. From the tests I've seen they're much better than the offering from Tynan. But they do COST.

I'm wondering about this problem some more. If I remember Sunfox has hires stills on the T/L as well as HDV, has anyone tried that combination?

Bob.

Sunfox wrote on 1/23/2005, 12:12 AM

...Except that I don't see one standard ATX board from Supermicro. They're all E-ATX. (EDIT: Oh wait, now I see them... for the E7525 chipset.)

Well, I should note that merely loading HDV clips into the media pool with a blank timeline is enough to chew up all my memory, never mind stills...

Just for the interest's sake, I loaded 148 3264x2448 stills into the media pool with no problem (that's 493mb of JPEG photos). I then dragged all 148 onto the timeline (which created a 12 minute timeline) - with thumbnail previews enabled - and although this was a very intensive process, I didn't run out of memory. Far from it - I still had 560mb real memory left and just a 500mb pagefile.

So, on a new timeline I loaded in the 20 smallest HDV clips I had - under 1 gigabytes and 2 minutes total - and saw free memory drop to 260mb and the pagefile increase to 770mb. If I then try to play the 2 minute timeline, free memory drops to nil and the pagefile goes to 1.3gb

Something's definitely funky, but I'm just going to have to live with it for now.

Grazie wrote on 1/23/2005, 1:01 AM

Spot, thanks for the tests and sharing them. Very useful.

So, time to strip down unused Media Pool media. I tend to copy Vegies for different parts of a project. But along with this approach comes ALL the stuff from the other parts of the "other" veg projects too! I shall heed SonyEPM's one line feedback too.

Not wanting to "brag", I guess housekeeping will allow for stable "iterations" of Vegas too.

Grazie