Thursday, March 20, 2008

Windows Vista SP1 and "Unable to load DLL 'VistaDb20.dll': Invalid access to memory location."

I recently updated my "Vista box" to SP1. I ran my current project without recompiling and everything was fine. Then I checked out a file, made some changes, and recompiled my project and found that my application would no longer run.

The error message I got was this:
Unable to load DLL 'VistaDb20.dll': Invalid access to memory location.

What an untimely error. We are trying to release our latest patch of our product and this takes the wind right out of our sails!

After some vigorous exploration we figure out that this has something to do with the DEP. The DEP is Data Excecution Prevention.

We figured out how to completely disable the DEP and sure enough the product could now run.

So now we had to figure out how to disable it pro grammatically (or at least we thought).

We are doing C# development. So I wrapped up a call into the Kernel32.dll to call SetProcessDEPPolicy. This had no effect. While I was doing this Jerry (one of the members of the team) was looking into why the recompile caused things to break.

I did a build on my "XP box" and copied it to my "Vista box" and sure enough it would run just fine. So we knew it had to be something with the compile.

Some Googling reveals some interesting information:
I'm Just Saying

Ed Maurer nails it right down. Thanks Ed.

Jerry added the following to our Post-build event command line:

call "$(DevEnvDir)..\tools\vsvars32.bat"
editbin.exe /NXCOMPAT:NO "$(TargetPath)"
mt.exe -manifest "$(ProjectDir)$(TargetName).exe.manifest" -outputresource:"$(TargetPath);#1"

I just wanted to share this with those that may be having the same problems.

3 comments:

Anonymous said...

Damn!
You don't know how big of a problem you've fixed for me...

It took me a whole day of trying to figure out this problem, and when nothing worked, a whole another day trying to find alternatives against this problem.
I'm surprised that I didn't find your blog earlier.

Thanks you so much for sharing your findings...

Anonymous said...

Idem - Muito Obrigado !!!!!!

Anonymous said...

digerati-illuminatus.blogspot.com; You saved my day again.