Recently, I have been using this software to edit my friend wedding video. But unable to compile to wmv / avi as it keeps on prompting out of memory. Sometimes it will just stuck at 72% during compiling. Please assist me as I hoping to use this software for other video editing as well.
Yes, but you are running software that was designed for Win XP 32-bit so you only have 2GB of memory to work with regardless of how much physical memory your computer has.
I would download a trial copy of Movie Studio 13.0 and open your project with that and see if it solves your problem. If it does, the solution is to upgrade your software to the 64-bit version of Movie Studio.
Movie Studio 8 is pretty old software and may not support a lot of the modern formats -- or at least not support them well. Particularly AVCHD video and video from phones, etc.
What model of camcorder is your video coming from? How long is your video project?
If you're using photos in your project, the program may also be choking on the size of the photos. Have you optimized the photos for video by sizing them to no larger than 2000x1500 pixels (at 72 ppi)?
Meantime, which processor are you running and with how much RAM? Remember that version 8 is a 32-bit program so, even if your Windows 10 is 64-bit, the program won't be able to take advantage of more than 2-3 gigs of RAM.
Finally, how much free space is on your hard drive? If your space is getting tight (under 80 gigs, say) fragmentation may also be an issue since the program often needs large chunks of free hard drive space in order to work.
I assume you've run Disk Cleanup and Disk Defragment since upgrading or installing Windows 10? As I've said, stray fragments on your hard drive (and installing an operating system creates thousands!) can keep the program from using large chunks of memory, which could interfere with relatively long render.
Also, you may well be bumping up against the limitations of version 8. As we've all noted, it's a 32-bit program, so it can only take advantage of a fraction of your RAM. Also, you're editing AVCHD -- which most editors today can do without even working up a sweat, but which pre-2009 video editors really struggled with.
At least download the trial of version 13 Platinum and give it a test run. (Or find version 12, if you don't like version 13's "big button" interface.) You'll be amazed how much smoother the process will go and how far the software has come!
I will mention though that disk fragmentation has nothing to do with memory usage. While defragmenting doesn't do much of anything to help any problems these days, it certainly won't make any difference with memory issues at all.
The only time anything related to the disk can affect memory is if you don't have enough hard drive space to increase the size of the swap file. If that's the case, you don't have enough hard drive space to do much of anything and you probably wouldn't even get programs launching, so it's very unlikely that this is an issue. And no, defragmenting won't help anything in this case either.