MiniDump support in .NET4

After reading this post (CLR 4.0 advancements in diagnostics) I was happy that now the CLR team is investing resources in better debugging of minidumps.
I made a small example in C#4 and wanted to try this new feature in the VS2010CTP version. You just need to double-click on the dmp-file. It will start VS2010CTP. Then you will need to “start debugging” (for example press F10), and then you will get the following result-screen:
VS20010 CTP Mini-Dump debugging
So it seems that it works “mostly”. But the debugger seems to forget to switch to the exception callstack and instead displays the “minidump-writing callstack”. Hopefully this will be fixed in one of the next (beta) releases.
Here is the link to my connect-feedback: Debugging (managed) MiniDumps displays the writing-callstack and not the exception-callstack

VS2008 Service Pack 1 is available!

Today, the first service pack for Visual Studio 2008 has shipped:
http://www.microsoft.com/downloads/details.aspx?FamilyID=27673c47-b3b5-4c67-bd99-84e525b5ce61&displaylang=en

But it seems that the installer is not very user-friendly:

If you previously installed a Visual Studio 2008 Hotfix, you must run the Hotfix Cleanup Utility before installing Visual Studio 2008 SP1. For more information, see Visual Studio 2008 Hotfix Cleanup Utility for Installing Visual Studio 2008 SP1.

Also, the KB article with the list of all fixes is not yet updated:
http://support.microsoft.com/kb/945140/en-us

Hopefully this SP will fix many bugs ūüėČ

Main disadvantage of (really) AppLocal deployment

In my last post I described how to (really) deploy CRT/MFC DLLs into the same directory as the application (xcopy deployment).

“Really” means, that it will never load any DLLs installed in the WinSxS directory. This is archived by removing the “publicKeyToken” from the manifest. If you leave the “publicKeyToken” inside the manifest, then it will force loading of (possibly) installed WinSxS versions (and updates/redirections).

The option of really AppLocal installations are not explaind on any Microsoft side (or at least I could not find it). Only the “normal” (semi) AppLocal installation is (a little bit) explaind, for example here:
Visual C++ Libraries DLL Deployment.

One of the main reasons for this is the new focus from Mircosoft to Security.

The main disadvantage of really AppLocal installations is, that Microsoft is not able to update or patch your installation (or better the CRT/MFC components of your installation). And this sees to be anathema to Microsoft.
The same is also true for static linked applications.

So I fear, that Microsoft will

  • remove or deprecate static linking in the next version of VS
  • remove the possibility to really AppLocal installations in the next OS /MFC/CRT DLLs

Hopefully, this will not come true…

Howto: Deploy VC2008 apps without installing vcredist_x86.exe

There are several reasons for xcopy deployment of an application (also known as application local). One main reason is that you are independent of what the target computer has installed.
Also your application always uses the “correct” (or better: tested) version of DLLs, regardless of what MS installed or updated (see: .NET2 SP1 update breaks old apps!?).

The easiest way to overcome the problem is to link static against the CRT/MFC. But in some scenarios this is not an option and not possible.

But to be independent from OS updates or from vcredist_x86.exe installations of other apps, you need to do the following steps:

  • Remove the auto-generation of manifests (from all DLLs) and change the manifest generation form your EXE to an external manifest; you can do this by choosing: Project|Properties|Configuration Properties|Manifest Tool|Input and Output|Embed Manifest: No
  • Recompile your application and modify the external manifest as follows:
  • Copy your application and the external manifest to your deployment directory
  • Open the manifest-file (appname.exe.manifest) and remove the “publicKeyToken” from all MFC/CRT/ATL/OpenMP entries. Please leave the publicKeyToken to the “Microsoft.Windows.Common-Controls” entry.
  • Copy all neccessary directories under %ProgramFiles%\Microsoft Visual Studio 9.0\VC\redist\x86 to your deployment directory
  • In all sub-directories (Microsoft.VC90.CRT, optional: Microsoft.VC90.MFC, Microsoft.VC90.ATL, Microsoft.VC90.OPENMP, Microsoft.VC90.MFCLOC) open the manifest-file and also remove the publicKeyToken
  • Also lookup the version info from these manifest files and correct the version-info of the corresponding entries in your application manifest file
  • Save all manifests and let your program run; it should now run on all supported OS without installing anything…

After doing all these manifest stuff you can also embed the manifest into your application (EXE). And of course: The same can be done with x64 and IA64 apps.

I have made an example of the default MFC app (4.6 MB) for reference.

The (simple) manifests for the new MFC feature pack and the application looks like:

Application.exe.manifest



  
    
      
        
      
    
  
  
    
      
    
  
  
    
      
    
  
  
    
      
    
  

Microsoft.VC90.CRT.manifest



    
    
     
     
    

Microsoft.VC90.MFC.manifest



    
    
     
     
     
    

Instead of putting the “Microsoft.VC90.MFC” and “Microsoft.VC90.CRT” directories into the application folder, you can also just put the files from these folders into the application directory. The main advantage is, that your app will also work on W2k-SP4.

Here is also my older post for VS2005.

Available hotfixes for VC2005-SP1 / VC2008 RTM / VC2008-SP1

There are already several hotfixes available for VC2005 SP1 and VC2008 RTM. Here is a short overview:

Updated: 2008-11-06
Updated: 2009-03-09
Updated: 2009-03-13
Updated: 2009-03-18
Updated: 2009-05-08
Updated: 2009-05-13
Updated: 2009-05-23
Updated: 2009-10-06

MFCNext / VS2008 VC++ Feature Pack is RTM

The feature Pack for VC++ 2008 is now available for download.

It is some kind of confusing:
You can download the feature Pack for the englisch version only (ENU)!

If you have installed a localized version of VS (e.g. german), you must wait until the release of SP1. Hopefully the ENU-SP1 will also include the Feature Pack…

If you have a non-english OS the installation will fail! You need to change the system-language and formats to “en-us” and re-run the installation… this is a typically US bug…

Also one note: If you want to use the new Ribbon UI (Office 2007 UI, also known as Fluent UI), you must agree to the Office 2007 license!

Here are the links:
Download-Site
Direct download link
Redistributable package (x86)
Docu: MFC Feature Pack
Docu: TR1 Extensions

Just upgrade to VS2008! Even without an existing license!

According to the Microsoft Website Visual Studio 2008 Pricing there is no need to buy the full product. You can always buy the upgrade for VS2008 Standard or Professional if you have:

  • An earlier version of Microsoft Visual Studio
  • Any other developer tool (including free developer tools, such as Visual Studio Express Editions or Eclipse)

That looks interesting! So, download the express edition, then you can buy the upgrade version of VS2008!

I just made a copy of the page to be sure, that if MS removes this sentense in the future, I have a copy to witness:
VisualStudio2008Pricing_2008-04-04.xps

Unhandled exceptions in VC8 and above… for x86 and x64

Starting with VC8 (VS2005) it is not possible to catch all unhandled exceptions with your installed exception-filter (via SetUnhandledExceptionFilter). In some situations, the CRT forces the call of WER (Windows Error Reporting). One situation (/GS buffer overruns) can never be catched in your filter. If you want to catch all exception inside your filter, you need to intercept calls to SetUnhandledExceptionFilter to prevent the removing of all previous installed filters.

See also my previous post about this problem:
“SetUnhandledExceptionFilter” and VC8″

Here is the code to “hook” the SetUnhandledExceptionFilter which works now for x86 and x64:

#include 
#include 
#include 

#if defined _M_X64 || defined _M_IX86
LPTOP_LEVEL_EXCEPTION_FILTER WINAPI 
  MyDummySetUnhandledExceptionFilter(
  LPTOP_LEVEL_EXCEPTION_FILTER lpTopLevelExceptionFilter)
{
  return NULL;
}
#else
#error "This code works only for x86 and x64!"
#endif

BOOL PreventSetUnhandledExceptionFilter()
{
  HMODULE hKernel32 = LoadLibrary(_T("kernel32.dll"));
  if (hKernel32 == NULL) return FALSE;
  void *pOrgEntry = GetProcAddress(hKernel32, 
    "SetUnhandledExceptionFilter");
  if(pOrgEntry == NULL) return FALSE;

  DWORD dwOldProtect = 0;
  SIZE_T jmpSize = 5;
#ifdef _M_X64
  jmpSize = 13;
#endif
  BOOL bProt = VirtualProtect(pOrgEntry, jmpSize, 
    PAGE_EXECUTE_READWRITE, &dwOldProtect);
  BYTE newJump[20];
  void *pNewFunc = &MyDummySetUnhandledExceptionFilter;
#ifdef _M_IX86
  DWORD dwOrgEntryAddr = (DWORD) pOrgEntry;
  dwOrgEntryAddr += jmpSize; // add 5 for 5 op-codes for jmp rel32
  DWORD dwNewEntryAddr = (DWORD) pNewFunc;
  DWORD dwRelativeAddr = dwNewEntryAddr - dwOrgEntryAddr;
  // JMP rel32: Jump near, relative, displacement relative to next instruction.
  newJump[0] = 0xE9;  // JMP rel32
  memcpy(&newJump[1], &dwRelativeAddr, sizeof(pNewFunc));
#elif _M_X64
  // We must use R10 or R11, because these are "scratch" registers 
  // which need not to be preserved accross function calls
  // For more info see: Register Usage for x64 64-Bit
  // http://msdn.microsoft.com/en-us/library/ms794547.aspx
  // Thanks to Matthew Smith!!!
  newJump[0] = 0x49;  // MOV R11, ...
  newJump[1] = 0xBB;  // ...
  memcpy(&newJump[2], &pNewFunc, sizeof (pNewFunc));
  //pCur += sizeof (ULONG_PTR);
  newJump[10] = 0x41;  // JMP R11, ...
  newJump[11] = 0xFF;  // ...
  newJump[12] = 0xE3;  // ...
#endif
  SIZE_T bytesWritten;
  BOOL bRet = WriteProcessMemory(GetCurrentProcess(),
    pOrgEntry, newJump, jmpSize, &bytesWritten);

  if (bProt != FALSE)
  {
    DWORD dwBuf;
    VirtualProtect(pOrgEntry, jmpSize, dwOldProtect, &dwBuf);
  }
  return bRet;
}

LONG WINAPI MyUnhandledExceptionFilter(
struct _EXCEPTION_POINTERS *lpTopLevelExceptionFilter)
{
  // TODO: MiniDumpWriteDump
  FatalAppExit(0, _T("Unhandled Exception occured!"));
  return EXCEPTION_CONTINUE_SEARCH;
}

int _tmain()
{
  SetUnhandledExceptionFilter(MyUnhandledExceptionFilter);
  BOOL bRet = PreventSetUnhandledExceptionFilter();
  _tprintf(_T("Prevented: %d"), bRet);

  abort();  // force Dr.Watson in release!
}

Screencast: API Logging with the Application Compatibility Layer

Hi all!

Today I did a screencast on how to use the application compatibility layer (and especially the APILogger shim) to do API logging.

This is a very interesting screencast if you want to log and display API calls of other programs in a very nice and simple way.

So enjoy (14:59 min):
http://www.kalmbach-software.de/screencasts/UsingAPILogger

Here are also the links which are shown in the presentation:

Screencast: Static link to the C-Runtime to prevent vcredist and overcome “Application configuration” problems

Here is a short screencast which shows how to adjust the project settings to link against the static C-runtime (CRT).

This prevents that you must ship the CRT-DLLs with your application. Your application just works after copying the exe to the target computer.

This is a “german” screencast:

http://www.kalmbach-software.de/screencasts/VC2008EE-StaticLinkCRT/

HowTo: Correctly read reparse data in Vista

In many newsgroups/forums you will see the following approach to read the reparse data (for example the original directory of a junction):

This works very well under XP (but only if you are an admin). It does not work under Vista if UAC is enabled (even if you are admin). It only works if the process was started evelated (as real admin)!

But if you go to the command prompt and type “dir c:\ /AL”, you will see that in the command prompt the display is always possible, even if you are not an admin:

C:\>dir /AL 
 Volume in drive C has no label. 
 Volume Serial Number is B424-9F82              

 Directory of C:\ 
11/02/2006  02:00 PM         Documents and Settings [C:\Users] 
               0 File(s)              0 bytes 
               1 Dir(s)  57,193,873,408 bytes free

So there must be a better way to read the reparse data.

First I tried to find an other function to get the data. But this had no success… then I remembered my research about “Shims” which are part of XP and later. These shims can be used to “fix” application compatibilities in news OSs. To activate a shim for an appliction you can use the “Application Compatibility Toolkit” from Microsoft. And also I remembered an Shim namaned “APILogger”. Ok, then I copied “cmd.exe” to a temp-dir and renamned it; created and installed the sdb-File with the APILogger for this new “testcmd.exe”; and executed the “dir c:\ /AL” command. Afterwards I could easily find the correct approach to get the reparse data (in a later blogpost I will explain how to use the APILogger-shim).

The key point is: You must open the directory only with “FILE_READ_EA” access!

Here is now the full working source-code for reading reparse-data:

#include "stdafx.h"
#include 

typedef struct _REPARSE_DATA_BUFFER {
  ULONG  ReparseTag;
  USHORT  ReparseDataLength;
  USHORT  Reserved;
  union {
    struct {
      USHORT  SubstituteNameOffset;
      USHORT  SubstituteNameLength;
      USHORT  PrintNameOffset;
      USHORT  PrintNameLength;
      ULONG   Flags; // it seems that the docu is missing this entry (at least 2008-03-07)
      WCHAR  PathBuffer[1];
      } SymbolicLinkReparseBuffer;
    struct {
      USHORT  SubstituteNameOffset;
      USHORT  SubstituteNameLength;
      USHORT  PrintNameOffset;
      USHORT  PrintNameLength;
      WCHAR  PathBuffer[1];
      } MountPointReparseBuffer;
    struct {
      UCHAR  DataBuffer[1];
    } GenericReparseBuffer;
  };
} REPARSE_DATA_BUFFER, *PREPARSE_DATA_BUFFER;

#define REPARSE_DATA_BUFFER_HEADER_SIZE  FIELD_OFFSET(REPARSE_DATA_BUFFER, GenericReparseBuffer)

#define MAXIMUM_REPARSE_DATA_BUFFER_SIZE  ( 16 * 1024 )

int _tmain(int argc, _TCHAR* argv[])
{
  HANDLE hFile;
  LPCTSTR szMyFile = _T("C:\\Documents and Settings");  // Mount-Point (JUNCTION)
  //LPCTSTR szMyFile = _T("C:\\Users\\All Users");  // Symbolic-Link (SYMLINKD)

  hFile = CreateFile(szMyFile, FILE_READ_EA, FILE_SHARE_READ|FILE_SHARE_WRITE|FILE_SHARE_DELETE, NULL, OPEN_EXISTING, FILE_FLAG_BACKUP_SEMANTICS | FILE_FLAG_OPEN_REPARSE_POINT, NULL);
  if (hFile == INVALID_HANDLE_VALUE)
  {
    _tprintf(_T("Could not open dir '%s'; error: %d\n"), szMyFile, GetLastError());
    return 1;
  }

  // Allocate the reparse data structure
  DWORD dwBufSize = MAXIMUM_REPARSE_DATA_BUFFER_SIZE;
  REPARSE_DATA_BUFFER* rdata;
  rdata = (REPARSE_DATA_BUFFER*) malloc(dwBufSize);

  // Query the reparse data
  DWORD dwRetLen;
  BOOL bRet = DeviceIoControl(hFile, FSCTL_GET_REPARSE_POINT, NULL, 0, rdata, dwBufSize, &dwRetLen, NULL);
  if (bRet == FALSE)
  {
    _tprintf(_T("DeviceIoControl failed with error: %d\n"), GetLastError());
    CloseHandle(hFile);
    return 1;
  }
  CloseHandle(hFile);

  if (IsReparseTagMicrosoft(rdata->ReparseTag))
  {
    if (rdata->ReparseTag == IO_REPARSE_TAG_SYMLINK)
    {
      printf("Symbolic-Link\n");
      size_t slen = rdata->SymbolicLinkReparseBuffer.SubstituteNameLength / sizeof(WCHAR);
      WCHAR *szSubName = new WCHAR[slen+1];
      wcsncpy_s(szSubName, slen+1, &rdata->SymbolicLinkReparseBuffer.PathBuffer[rdata->SymbolicLinkReparseBuffer.SubstituteNameOffset / sizeof(WCHAR)], slen);
      szSubName[slen] = 0;
      printf("SubstitutionName (len: %d): '%S'\n", rdata->SymbolicLinkReparseBuffer.SubstituteNameLength, szSubName);
      delete [] szSubName;

      size_t plen = rdata->SymbolicLinkReparseBuffer.PrintNameLength / sizeof(WCHAR);
      WCHAR *szPrintName = new WCHAR[plen+1];
      wcsncpy_s(szPrintName, plen+1, &rdata->SymbolicLinkReparseBuffer.PathBuffer[rdata->SymbolicLinkReparseBuffer.PrintNameOffset / sizeof(WCHAR)], plen);
      szPrintName[plen] = 0;
      printf("PrintName (len: %d): '%S'\n", rdata->SymbolicLinkReparseBuffer.PrintNameLength, szPrintName);
      delete [] szPrintName;
    }
    else if (rdata->ReparseTag == IO_REPARSE_TAG_MOUNT_POINT)
    {
      printf("Mount-Point\n");
      size_t slen = rdata->MountPointReparseBuffer.SubstituteNameLength / sizeof(WCHAR);
      WCHAR *szSubName = new WCHAR[slen+1];
      wcsncpy_s(szSubName, slen+1, &rdata->MountPointReparseBuffer.PathBuffer[rdata->MountPointReparseBuffer.SubstituteNameOffset / sizeof(WCHAR)], slen);
      szSubName[slen] = 0;
      printf("SubstitutionName (len: %d): '%S'\n", rdata->MountPointReparseBuffer.SubstituteNameLength, szSubName);
      delete [] szSubName;

      size_t plen = rdata->MountPointReparseBuffer.PrintNameLength / sizeof(WCHAR);
      WCHAR *szPrintName = new WCHAR[plen+1];
      wcsncpy_s(szPrintName, plen+1, &rdata->MountPointReparseBuffer.PathBuffer[rdata->MountPointReparseBuffer.PrintNameOffset / sizeof(WCHAR)], plen);
      szPrintName[plen] = 0;
      printf("PrintName (len: %d): '%S'\n", rdata->MountPointReparseBuffer.PrintNameLength, szPrintName);
      delete [] szPrintName;
    }
    else
    {
      printf("No Mount-Point or Symblic-Link...\n");
    }
  }
  else
  {
    _tprintf(_T("Not a Microsoft-reparse point - could not query data!\n"));
  }
  free(rdata);
  return 0;
}

ADD (2008-03-07): Thanks to the comment of Sergey, I now changed the sample code to correctly display JUNCTIONs (mount points) and SYMLINKD (symbolic-links). It seems that the documentation of REPARSE_DATA_BUFFER is incorrect, because the missed a field named “Flags”. I changed the structure to the correct definition which can be found in the DDK header file “ntifs.h”.

marshal_as library in VC2008

The VC++ team added a (simple) marshal_as library in VC2008. But this library only supports simple datatypes like

  • String^¬†to char* / wchar_t* / BSTR / bstr_t / CComBSTR / std::string / std::wstring / CString<char> / CString<wchar_t>¬† and vice versa
  • IntPtr¬†to HANDLE¬† and vice/versa

So, the support only contains “Strings” and “Handle”. But this are the most commonly needed scenarios for marshaling.
The usage of some of the string marshaling is shown here:
http://www.c-plusplus.de/forum/viewtopic-var-p-is-1455833.html#1455833

An overview of the build-in marshaling can be found here: Overview of Marshaling in C++

There is a website (http://www.marshal-as.net/), in which you can look-up additional marshaling implementations like:

Hopefully many, more will follow…
If you have an implementation of some kind of “marshal_as”, just put a comment to Kate Gregory

Local user-mode dumps on Vista SP1 and W2k8

Starting with Vista, MS introduced the new Windows Error Reporting API (WER) and removed Dr. Watson from the system. The WER-System now directly sends by default the data to WinQual. They also changed the behaviour, that a minidump is only generated if the WinQual-Server requests this. So by default there is no way to get to a local minidump if you are not registered at WinQual (which costs 400$/year).

Several ISVs complained about this issue. And now MS added a new feature in VIsta SP1 and W2k8 to allow the creationg of local minidump regardless of the WER setting. FOr more info see:

Collecting User-Mode Dumps

Want to download the complete .NET source code?

In one of my last posts I said that the .NET source code is available. While this is still true, you might have seens some trouble in getting the source, because the source is only available on demand while debugging a component. Also you must setup VS to retrive the source/PDBs.

But if you just want to look at the source, there was no easy way to do this.

On CodePlex there is now a tool called NetMassDownloader, which allows to download the complete source (or only from indivudual DLLs) to a directory.

Have fun!

Smallest application size for win32 console application

Several times someone asked the question: “Why is a simple console application 52 KB (CRT statically linked) or 7 KB (CRT as DLL) in size?”

The answer is: Because it uses the CRT ūüėČ

Just to show you, that you can have smaller applications, here is (from my point of view) the smallest “usefull” application (Hello world):

// Filename: smallest.cpp 
#include 
#pragma comment(linker, "/entry:entry") 

void entry() 
{ 
  TCHAR szText[] = TEXT("Hello world\n"); 
  DWORD dwWritten; 
  WriteConsole(GetStdHandle(STD_OUTPUT_HANDLE), szText, lstrlen(szText), &dwWritten, NULL); 
}

You must compile and link this file with:
cl /c /O1 /GS- smallest.cpp
link /subsystem:console smallest.obj kernel32.lib

Then you get an EXE file with 2048 Bytes.

You can even reduce it more if you merge the sections together and specify an alignment for the linker (but then program will then not run on Win98;) ):
cl /c /O1 /GS- smallest.cpp
link /subsystem:console /MERGE:.rdata=.text /ALIGN:16 smallest.obj kernel32.lib

Then you get an EXE file with 688 Bytes (VS2008) / 768 Bytes (VS2010).

So the question is: why does the CRT need so much space?

Most of the space is needed for “security reasons”. That is also the reason why I needed the specify the “/GS-” compiler switch to reduce the exe size. If you enable the buffer security checks, it triggers the compiler to use several functions and structures, which must be provided by the CRT.
And if you use the CRT, it also provides the default entry point and initializes all kind of stuff like calling constructors of static/global classes and others.
These are the main reasons why the CRT uses so much memory.

Another think is: if you want to use floating-point values, you also need the CRT. The reason is not very obvious for me… but the compiler is triggering the symbol “_fltused” to be linked with the image. If you provide your own “implementation” of this variable, it seems to work; but I don’t understand why the compiler is needing this variable. This variable must be set to a “magic” value (0x9875); I don’t know where this value comse from (expect it is defined in the CRT sources).

If you want to compile these samples from within the VS (2008) IDE, then you need to change the following setings from the default “Win32 Console Application”:

  • Switch to “Release” configuration
  • C/C++|Code Generation|Buffer Security Check: No (/GS-)
  • Linker|Debugging|Generate Debug Info: No¬†
  • Linker|Advanced|Entry Point: entry
  • Linker|Advanced|Randomize Base Address: Disable Image Randomization (/DYNAMICBASE:NO)
  • Linker|Advanced|Fixed Base Address: Image must be loaded at a fixed address (/FIXED)
  • Manifest Tool|Input and Output|Embed Manifest: No

And then you have your 2KB application from within the IDE ūüėČ

A new team is now improving MSDN Help!

Since VS2002 the new help system was really bad. And it got even worser in every release of visual studio. The slowest version was starting with VS2005 and VS2008. Now MS has heard us!

April announced on her weblog, that MS has now set up a new team to “redesign” the MSDN Help system:

Help is Getting Help – The Long Overdue Makeover for Visual Studio and MSDN Help

Really great news! Hopefully this project will be successful!