Debugging Tools for Windows is now part of the WDK

Yesterday a new release of “Debugging Tools For Windows” was released. Until now, it was possible to download the package as a single download of about 17 MB in size.

Starting with the new release, it seems that the Debugging Tools For Windows is now integrated in the WDK which means to download a 620 MB file.

Also it seems so, that the WDK does not install the Debugging Tools For Windows, it just installs a link to the setup…
So the question is: Why not provide the setup as a separate download as in previous version?

Here is a quote from the Debugging Tools For Windows download page:

This current version of Debugging Tools for Windows is available as part of the Windows Driver Kit (WDK). To download the WDK and install Debugging Tools for Windows:
1. Download and install the WDK.
2. Find the debugging tools link for Windows x86 version on the screen that appears and click to install the debuggers to a location of your choice.
3. After the installation is complete, you can find the debugger shortcuts by clicking Start, pointing to All Programs, and then pointing to Debugging Tools for Windows.

Conclusion: We will never understand marketing…

The Shim Database

The Shim database is a mystic area inside windows… For example it will display an UAC (admin) prompt if your application-name contains the word “*instal*”.
You can display all available shims with the Application Compatibility Administrator by using the /x command line switch.
Also there is a tool from Heath Steward which dumps the database into an XML file.
Of course, he failed to prvide the source-code of his sample project.
Also, Alex Ionescu wrote a small dump tool, but also has never published it…

So I decided to dig into this almost not documented world and write a small Shim-Dumper and Exe-matching tool (whitch source code 😉 ).

Shim Database Tool (sdb) v1.0
Copyright (C) 2010 Jochen Kalmbach

Usage:  sdb.exe [-noids] [-match] [PathToShimDatabse] [PathToFileName]
 -noids  Will prevent the output of the TagIds
 -match  Will match the provided file with the installed databases
         and displays the activated shims
         In this case 'PathToFileName' is required

NOTE: If no shim database path is provided,
      the default database will be used.

You can use it either for dumping the shimdatabase like:

sdb.exe >ShimDatabase.xml

This will redirect the output to an xml-file and will look something like:

  
   setup32.exe
   WordPerfect Office 2000
   Corel
   
   
   
   
    *
    Corel Corporation
    Corel Setup Wizard
   
   
    programs\wpwin9.exe
    Corel Corporation Limited
    WordPerfect® 9
   
   
    appman\tools\cset90.exe
   
   
    WinXPSP1VersionLie
    0x284e0
    
     
     $
    
    
     *

(be aware: the current Win7 database is about 17 MB!)

You also can use this tool to find out, if an application has a shim applied:

C:>sdb -match MyInstaller.exe
Shim found for file: MyInstaller.exe
Flags: 0x0:
Exe-Shim: 0x35472, Flags: 0x0:
Layer-Flags: 0x0:
Shim-Database: 11111111-1111-1111-1111111111111111

Currently it just displays the TagId of the Shim. You can use this to search the xml-file for the corresponding id.

Have fun, using this tool 😉

The project (VS2008) can be found here:
http://blog.kalmbachnet.de/files/sdb_v1.zip
It will compile for x86 and x64.

The mystic variable “$I” during for each

A poster in the german C/C++ forum asked if there is an index available while using a for each loop. He accidently saw in the debug-window a variable called “$I”.
And indeed: There is a “hidden” variable “$I” which can be used inside the for each loop. This variable is the number of the loop-iteration.

Here is a simple example:

int main()
{
  array<int> ^MyArray = { 100, 200, 300, 400 };
  for each( int v in MyArray )
  {
    System::Console::WriteLine(v.ToString() + " " + $I);
  }
  return 0;
}

As you can see, it uses a variable “$I” which was never decalred!
And it will output the following:

100 0
200 1
300 2
400 3

If you debug a normal for each loop and take a look in the “local-watch-window”, you will see the following variables, while you are inside the for each loop:

You can see two “hidden” variables “$I” and “$S1”. And if you step through the loop, you will see that the “$I” variable is incremented for each iteration. “$S1” is a reference to the array.

If you dig further into this issue, you will find out, that for each and the normal “for” loop will result in the identical IL code! This is also true for C#!
See also:
FOREACH Vs. FOR (C#)
To foreach or not to foreach that is the question.

As we can see, the “$I” variable is just a side-effect of the for each loop. It is a compiler generated variable which is used to transform the “for each” into a normal for-loop!

Of course, this is only true in special cases like arrays.
If you have a list which only implements the “IEnumerable” interface and is not an array (like “System::Collections::Generic::List”), then the mistic variable “$I” is gone, because now the compiler uses the “IEnumerable” interface to gbuild a “real” “for each” loop:

In this case “$I” is gone but “$S1” is still there. And “S1” is the enumerator of the list (in this case 

System::Collections::Generic::List::Enumerator<int>

).

The conclusion is:
Do not rely on the compiler generated variable “$I”, and do not use “for each” if you need a index-variable, just use a normal for loop.

Upgrade to VC20xx: Problems with Exception Handling

If you upgrade from VC6 to VC2008, then your project will automatically converted to the new format.
As a side-effect, it will also default to the new exception handling which breaks compatibilty to VC6.

Therefor this post 😉

In VC6, by default the “/EHa” exception model ist activ.
In VC200x and later, by default, the “/EHsc” exception model is active. This means that if you did not explicit specify the “/EHa” model, you will now automatically use the “/EHsc” model, which only catches C++ exceptions!

For example, the following code will work as expected and will crash in VS2008:

#include <stdio.h>
#include <tchar.h>

int _tmain()
{
  try
  {
    printf("Now doing an AV...\n");
    char *c = NULL;
    strcpy(c, "Hello");
    printf(c);
  }
  catch(...)
  {
    printf("Catched....");
  }
}

So be aware, that if you rely on asynchonus exceptions, you need to switch the expetion model in the C/C++ project settings under “Code generation | Enabled C++ Exceptions: Yes with SEH Exceptions (/EHa)“

TFS: Automatically insert Check-In comments into source-code

TFS is in most cases very wonderfull. But if you have projects, which will also be used “outside” of the TFS environment (for example in an other subsidary without TFS access), then it has one disadvantage:
The comments for every check-in is only stored in the TFS version history.

While this is enough, if you always have access to the TFS. But if you work without TFS, then it would be helpfull if these comments are also included in the source-file (like in the old days, when we were using cvs (a feature called keyword-substitution)

I searched the web for any plugin for TFS which has almost the same features… but could not find any plugin….

So I wrote my own plugin, which at least puts the check-in comments into the source-file.
You just need to install this plugin and activate it in the “Check-In policy”.

Then your source-file will contai the comment, after you checked it in:

/*
 * $log$
 * 
 * Comment: And another test
 *          with a multiline
 *          comment...
 * User: xyz
 * DateTime: 2009-07-23 22:05:24
 * Change: edit
 * 
 * Comment: This is an test
 * User: xyz
 * DateTime: 2009-07-23 21:01:22
 * Change: add, edit
 *
 */

I can also customize (like templates) the position and contents of the comments…

If you are interested in this plugin-in, please let me know…

Just a small addition: Here is a post about “keyword expansion” and TFS; and possible side-effects:
http://blogs.msdn.com/buckh/archive/2007/07/07/keyword-expansion-in-tfs.aspx

EDIT: 2009-08-08:
Today I published my first release of the LogSubstPol on CodePlex.

Custom installer without warning dialog… (TrustedInstaller.exe)

If you start an custom install program which is not digital signed, then Vista and later (at least Win7RC1) warns you, that your installer is not digital signed and “do you really wnt to do that”…

This feature is implemented with an shim (in a separate article I will explain shims later).
In short it works like: if an application is started, it checks the shim-database (in general %SYSTEMROOT%\AppPath\sysmain.sdb) if this application matches specific conditions.

For “Installer detection”, there are several conditons. Here is a list, what is checked (Vista and Win7RC1):

  • Productname contains “*instal*” or “*setup*” or “*update*”
  • Companyname contains “*instal*” or “*setup*” or “*update*”
  • Internalname contains “*instal*” and app is not named “TrustedInstaller.exe”
  • Originalfilename contains “*instal*” and app is not named “TrustedInstaller.exe”
  • Filedescription contains “*instal*” and app is not named “TrustedInstaller.exe”
  • Filename contains “*instal*” and is not named “TrustedInstaller.exe”
  • Exportname contains “setup.exe” or “install.exe” or “stub32” or “stub32.exe” or “signstub.exe”
  • Filename contains “*patch*” or “*setup*” or “*uninst*” or “*update*” or “lhaca*.exe”
  • Filedescription contains “*instal*” or “*setup*” or “*update*” or “*uninst*”
  • Originalfilename contains “*setup*” or “*update*”
  • Originalfilename contains “*setup*” or “*update*”

What is the conclusion: If you want to prevent the message box, just name your installer “TrustedInstaller.exe” 😉

You can simple test this by renaming any EXE to “abcinstaldef.exe” and try to execute it 😉 and the rename it to “TrustedInstaller.exe” 😉

On the other hand: Never name your app anything of the above!

Deployment of VC2008 apps without installing anything

If you create a default CRT/MFC application with VS2008, this application will not run on other computers. You application will complain with
“This application has failed to start because the application configuration is incorrect”.

The problem is that by default VC2008 uses the DLL-version of the CRT/MFC. These DLLs are not pre-installed on any OS.
To overcome this problem, you have tree options:

  1. Statically link to the CRT/MFC
  2. Use vcredist_x86.exe / vcredist_x64.exe to install the DLLs in the system
  3. Deploy the CRT/MFC DLLs with your application (in the same directory)

In the following I will explain the last option. Especially in conjunction with VS2008 service pack 1 (SP1). Because this leads to a little bit more compications in ApppLocal deployment.

In general, it is very easy to deploy your application with the needed CRT/MFC DLLs.
Just copy the contents of the directory

  • C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86\Microsoft.VC90.CRT
  • C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\redist\x86\Microsoft.VC90.MFC

into the same directory as your application. Then you should have the following files in your app-folder:

Microsoft.VC90.CRT.manifest
msvcr90.dll
msvcp90.dll
msvcm90.dll
Microsoft.VC90.MFC.manifest
mfc90.dll
mfc90u.dll
mfcm90.dll
mfcm90u.dll

Then your application works on an other computer!

BUT: This does not work, if you installed VS2008-SP1 🙁

The problem with VS2008 SP1 is: It overwrites all files in the “VC\redist” directory with the new version. This is not really bad. The problem is: It has a newer version number inside the manifest files. But if you compile your application with VS2008-SP1 it write the RTM-version into the application manifest! Now the version number in your application manifest does not match the version in the “new” CRT/MFC manifest. Therefore it will refuse to load these DLLs and your application cannot start.

The simplest way to overcome this problem is by changing the “Microsoft.VC90.CRT.manifest” and “Microsoft.VC90.MFC.manifest” files. Replace the version attribute in both Microfot.*.manifest files from “9.0.30729.1” (or whatever version/SP you have) to the version number in your applications-manifest (which is normally “9.0.21022.8” (RTM)).
Old: 

version="9.0.30729.1"

New: 

version="9.0.21022.8"

Then your application will work on an OS without installing anything.

Alternatively, you can change your applications manifest, so that it uses the new version number. This can easily done by defining

#define _BIND_TO_CURRENT_VCLIBS_VERSION 1

in stdafx.h (at the top) or in your project settings. This will embed the actual CRT/MFC-version into your applications manifest (works starting with VS2008-SP1).
Also, if you use new features from the MFC-feature-pack, you should always define this!

Just a small note: You should be aware, that this “AppLocal” installation is not really “AppLocal”… it is only AppLocal, if the vcredist_*.exe was not installed. If the vcredist_*.exe is installed, then the DLLs from the WinSxS directory will be used. If you want to prevent this, you can do a really AppLocal deployment.

A small addition: If you write .NET-apps (/clr) you still must install the .NET Framework redistributable.

But the goood news is: Starting with VC2010, the manifest (WinSxS) is gone 😉

Breaking changes in VS2010 (Beta)

The (beta9 documentation for VS2010 is online, and therefor also the Breaking changes in VS2010 C/C++:

  • “auto” is now a keyword
  • “static_assert” is now a keyword
  • Lambda support excludes support for unquoted GUID in an IDL (it uses the same […] syntax)
  • Some “.NET-Exceptions” can now only be caught with __try __except
  • /GS checks might decrease performance
  • Project files are automatically converted to the new .vcxproj
  • CRT/MFC/ATL/OpenMP-DLLs no longer use fusion (huray!); it uses plan old dll-hell
  • Support for environmentvariable __MSVCRT_HEAP_SELECT was removed
  • New _ITERATOR_DEBUG_LEVEL macro (replaces _SECURE_SCL and _HAS_ITERATOR_DEBUGGING)
  • hash_map and hash_set are now in the std namespace (previously in stdext)
  • SafeInt is included

C++/CLI quiz: Answer

In one of my last post, I asked the question, how does the callstack look like.
It was is simple unmanaged class with a virtual function. If you had executed the sample program, and set a breakpoint into the “Foo” method, you can see in the callstack-window of VS, the following callstack:
Callstack of (native) virtual function in managed code

As you can see, there are two transitions which we had not expected. One transition to native code, and one transition back to managed code.

And if you put a “printf” statement into the copy-constructor of the V-struct, you will see that the copy-constructor is called tiwce! This can have very strong performance effects…

So what might be the course of this “unnecessary” transitions?

The problem is: We have written an unmanaged virtual function as a managed function (the whole file is compiled with /clr). This means, that all unmanaged virtual functions will have two entry points. One entry point is for the “native” v-table call (thiscall); and the other is for the managed entry (clrcall).

If you call a unmanaged virtual function in a managed world, the compiler does not know which function must be callled. This is only examined at runtime through the v-table. Therefor the compiler needs to call the native function which uses the v-table. Then this “proxy”-function is calling the managed entry point. And because there are two functiions involved, the copy-constructor the the V-struct is called twice.

Conclusion:
Be aware that virtual functions in unmanaged classes which are compiled with /clr might lead to performance decrease.
If you have a big codebase, you should only enable /clr for specific files which implements the “wrapper”.
Be aware that every managed/unmanaged transition has additional execution-costs on your application. So prevent this transitions as much as possible.

C++/CLI quiz: What is the exact callstack?

I have a small quiz:
Here is a small code, compiled with “/clr” (this is important).
Can you tell me, what is the exact callstack inside the method “Foo”?
(And how can you prove this)

struct V
{
  V() {}
  V(const V &v) 
  {
    this->i = v.i;
  }
  int i;
};

class C
{
public:
  void CallFoo()
  {
    V v;
    Foo(v);
  }
  virtual void Foo(V v)
  {
    // TODO: What is the callstack!?
  }
};

int main()
{
  C c;
  c.CallFoo();
}

Any hints can be posted as comments.
By the way: VS208 does not show you the exact callstack, but it gives you a hint, that there is something that you do not see 😉

WebService in native Code (WS-*)

After the support for the SOAP-Toolkit was retired, there was no library from Microsoft, to write or consume Web-Services in native code. The suggested solution was to move to the .NET-SOAP classes. Of course, this is not always a solution, for big native code-bases.

It seems that MS has reflected this situation and also has found, that it is a “must have” to have a native WS-* library.

Starting with Windows 7 and Windoes Server 2008 R2, the OSwill have build-in support for native Web-Services.

The APIis called WWS-API (Windows Web Service API). To get an overview and also some examples, you can take a look at the code.msdn-Site.

Also, if you have an existing application and want to move to WWSAPI, you can join a virtual lab, to get help in implementing your web-service (either consuming or exposing).

EDIT:
Nikola added an comment, that WWSAPI will be also available on XP-SP2 and later! These are really great news!
You can already download the beta-bits, see:
Release of WWSAPI beta for Windows XP, Vista, Server 2003 and Server 2008

IE8 smashes Visual Studio 2005 / 2008 Class-Wizard

It seems that the new IE8 has some conflicts with Visual Studio 2008 (incl. SP1). After you installed IE8, you are not able to use the call-wizard for MFC projects anymore. It always gives you an error that there is a bug on the page:
Visual Studio 2008 bug after installing IE8

Currently there is a workaround, until a “real” fix is present:

  • Open regedit
  • Under HKEY_CURRENT_USER \ Software \ Microsoft \ Windows \ CurrentVersion \ Internet Settings \ Zones
    create a new key called 1000 (if it isn’t already there)
  • Under 1000, create a DWORD entry with:
    • Name = 1207
    • Type = REG_DWORD
    • Data = 0x000000

Visual Studio 2008 hotfix for IE8

You can also download the zipped reg-file.

Just for completeness: There is a connect entry since 9 days…

Addition: The problem also occurs with VS2005 (and maybe earlier versions 😉 )

More infos is available from the VC-Teamblog.

Change of Win-API semantics via application manifest!

Starting with Windows 7, the application manifest is becoming more and more important.

In Windows Vista, you need a manifest to bypass the application virtualization. In Window 7, you need a specific application compatibility-manifest to get correct API function behaviour 😉
For example, a race condition in GetOverlappedResult is only solved, if you explicit specify in your manifest, that you want a correct behaviour of this function.
Also a bug in CreateFileEx is only solved if you specify this in your manifest.

Of course, this is in general not a bad idea. But the manifest only allows to enable all new features or nothing. There is no way to exlicit enable only one of the bugfixes.

Here is just a small document of the “Windows 7 – Application Manifests – Compatibility”.

Windows Communication Protocols

Microsoft is publishing many of their “closed” protocols on the internet.
The starting site is: Microsoft Protocol Programs and Open Specifications.

Here is just a small list of available protocol downloads:

Building vcproj-Files with msbuild

If you want to batch-build VC++ projects you can either use devenv or msbuild.
msbuild has many option in which you can change and log the build process.

Here is just a example of a simple .proj-template for starting of batch-builds with msbuild:

You need just to save this file with the extension .proj” and call “msbuild” from the same directory. It will then do a “Rebuild” (see “DefaultTargets”).
Or you can do a “Clean” with “msbuild /t:Clean”.
You can also log to a file with “msbuild /v:d /fileLogger”.
Also, if you do not need a specific build-order of your projects, you can just include every vcproj-File in your subfolders by changing

to

Also, msbuild returns 0 if the build was sucessfull, otherwise “1”. So you can determine the result of the build in your batch-files.

So you can see: msbuild is very powerfull…

All-in-One Code Framework

If you need examples of some of the current Microsoft technologies, you have many technologies and programming languages available. For each of these technologies/languages you need to find a fitting example.

On CodePlex there is now an example-collection for most of the available technologies (COM, ActiveX-Control, ActiveX-Host, ADO, ADO.NET, Outlook-Customizing, DLL, DLL-delay-loaded, LIB, P/Invoke, CLR-hosting / named-pipes, mailslots, shared-memory, remoting) and programming languages (C/C++ (native / MFC / ATL) / C# / VB.NET):

Check out: All-In-One Code Framework