VisualStudio 2010 registry chaos

Actually, not a chaos, but it is a catchy word..

The case started with a short note I've received from EmperorXLII:

I wanted to let you know that the new .vsix does not support
command-line execution through MSTest.exe of test assemblies.

In other words, from a VS command prompt,
    mstest /testcontainer:MyTest.dll
reports "No tests to execute".

I still had the .reg file from the previous version; adding
that to the registry fixes the mstest issue.

I immediatelly thought about two registry entries I intentionally skipped in the newest version, but they were really irrelevant. What really suprised me is that after running old .reg file - it started working. It simply could not.

If it started working when he used the old reg file, it means that he probably accidentially activated an version older than 2.1 (older than the one with 'installer'). The MSTest most probably loaded some old modules from the 1.2 or 2.0 version, not deleted and still sitting in the /PrivateAssemblies.

After what I've traced in the last few days, I'm quite sure about that!

The problem

There's a lot of smoke out there when you try to find out which and how the VisualStudio actually reads the registry keys. If you look at it's modules, almost all of them refer only to HKLM hive, but when you check it at runtime - it turns out that HKCU are read in their stead.

For example, looking at TestTypes registration in HKLM, there are two types:

• {13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b} - UnitTest TIP
• {ec4800e8-40e5-4ab3-8510-b8bf29b1904d} - Ordered AutoSuite TIP

and if cheking in HKCU, assuming the xvsr10 is installed:

• {13cdc9d9-ddb5-4fa4-a97d-d965ccfc6d4b} - UnitTest TIP
• {4d2f9ccb-49d3-4caa-9a29-beffc604075a} - XUnitTestTip
• {ec4800e8-40e5-4ab3-8510-b8bf29b1904d} - Ordered AutoSuite TIP

As probably you guessed from the foreword, the commandline MSTest read the HKLM, while the VisualStudio reads HKCU entries. The funny thing is, that neither the MSTest, nor the VisualStudio cares about that. Both of them delegate all the tasks to the QualityTools package. They both delegate to the very same assembly.

So how does it happen, that the same code from the same package, switches back and forth between registry hives?

VisualStudio named instances

If you have ever tries the VS Extensibility SDK, created a tutorial-ish VSIX and ever tries to debug it, you have probably seen a "Visual Studio Experimental Instance". Shortly: this is the very same VS that you use normally, but it is artificially switched to sibling registry root key, by a commandline switch: /rootsuffix Exp

This switch causes the VisualStudio to stop using:

HKCU\Software\Microsoft\VisualStudio\10.0

and start using:

HKCU\Software\Microsoft\VisualStudio\10.0Exp

as its root registry key for keeping most of its configuration. In terms of VSIX, this allows to temporarily install your extension-under-development without danger of bricking your main VisualStudio instance.

However, if you look there, you will also find:

HKCU\Software\Microsoft\VisualStudio\10.0_Config

and you may also find:

HKCU\Software\Microsoft\VisualStudio\10.0Exp_Config

and if you look inside them, their contents will look, well, quite similar to those without _Config suffix.

And yet there's also a HKLM version. So what all that for?

The 'Chaos'

It turns out that the VisualStudio's packages use a layered registry architecture:

• HKLM\Software\Microsoft\VisualStudio\10.0 is where the original information is installed once, per-machine
• HKCU\Software\Microsoft\VisualStudio\10.0{suffix} is the per-user configuration that shadows the HKLM, so that Alice and Bob can have thei own settings, and moreover, so they both can run their 'experimental' instances
• HKCU\Software\Microsoft\VisualStudio\10.0{suffix}_Config is the 'current' user's instance configuration, that shadows the original user's settings, and is easily removable in case something badly fails and needs to be reverted

So, if some required value is not found in hkcu-{suffix}_Config, it is searched for in hkcu-{suffix}, and then, finally, in hklm.

Of course, the user configurations are really meant per-user, so when Alice installs an extension - it's setup gets written to her HKCU registry settings, and Bob's registry is untouched. He will not have that extension activated. This is a great thing! User isolation, licensing and other blahs.

But let's look back at the VisualStudio package development. We want to provide an extension to the VS. Do we really have to support this registry chain?

Well, I actually don't have to use the registry at all, at least in terms of the xvsr10 - the [RegisterTestTypeNoEditor] attribute and CreatePkgDef tool from SDK actually have done all the work for me. However, if I wanted to manually keep something in the registry - then yes, I'd suppose I have to implement the chain.

And I'd be completely wrong.

I turns out that the VisualStudio environment provides several set of ways to access the registry, most notably the class Microsoft.VisualStudio.Settings.ExternalSettingsManager, but let's leave it aside for now. The interesting part is that probably almost no package loaded into VS uses the HKCU at all!

For example, the QualityTools package uses TestTypeInfoCollection, TestConfigHelper, and TestConfigKey classes to read the list of TIPs from the registry, and they all very firmly point to the "LocalMachine" keys. Of course, the leaf TestConfigKey is a wrapper for framework's RegistryKey class, so no big deal - the wrapper handles the switching? Not! This wrapper is dumb. It is a little more than lazy-loading support.

Before you start wondering - wait, there's more! If the plugin/extension you wrote tries to execute similar code:

var lm = Microsoft.Win32.Registry.LocalMachine;
var key = lm.OpenSubKey("Software\\Microsoft\\VisualStudio\\10.0\\EnterpriseTools\\QualityTools\\TestTypes");
var count = key.SubKeyCount;

it will actually read the value of Count=3 instead of Count=2 - what you would expect from reading from HKLM (see the list in 'the problem' section above). And those classes were the normal framework's classes, no custom wrappers. If you inspect te objects with a debugger, it will turn out that they store "HKEY_LOCAL_MACHINE" strings and they even call the advapi32.dll:RegOpenKeyEx with 0x8000002 argument, which is, of course, HKEY_LOCAL_MACHINE.

And now, guess what! When run from within MSTest, the QualityTools package just reads directly from HKLM. The magic is gone. No 'registry subsitution' occurs. And probably, any other package would behave the same.

So what is really going on there?

For every feature, there's a bug

And the bugs do form the chaos.

The devenv.exe executable that provides the entry point for VisualStudio's IDE contains some very clever, tricky, and simultaneously nasty thing: a win32api hooks. Or rather - hijacks or 'detours', as the Microsoft like to name them lately. Devenv.exe behaves just like traditional virus: it seems to load advapi32.dll, but then it rewrites the import chunks with its own trampolines pointing to completely other code. This approach fools the whole manager .Net framework layer, as it is in not a few places only a pretty paper wrapper over the native methods. Here, the native methods were replaced with custom implementation, and the framework happily invokes RegOpenKeyExW and even if it cared to check - it would be completely sure it calls it from advapi32.dll, as the import was actually invoked - but was later redirected elsewhere (by the way, you may want to check on MSDN the Detours library - it uses similar methods).

What's the point of it? Maybe they wanted to allow the developers to invent their own extensions, but they did not want the developers to 'contaminate' the precious registry with their temporary trash. Or maybe they wanted to provide the extension developers with an automatic way of handling the layered registry architecture, so they won't have to do it manually? Or maybe rather, they had few millions of lines of existing code, that was using more-or-less hardcoded HKLM references and now they wanted a way to quickly make the whole configuration per-user and per-named-instance?

Well, whatever. They succeeded only partially: the IDE works smoothly. But in the rest - have failed.

The MSTest tool does not contain ANY of those detours. I didn't check, but I'd bet that most of the tools sitting in Common/IDE, or even, most of the tools that don't use the vs shell - does not contain those detours.

This means, that all of those are automatically crippled: they are unable to notice any of the available per-user and per-instance settings and will only be able, and will only read the default configuration from the HKLM hive. No magic, no cookies. But it's only my guessing, for now, I have been playing around only with MSTest.

This may seem a small issue not worth calling a bug and writing an article about it. Of course, simply copying the keys and values to the HKLM hive is sufficient, and the QualityTools package loaded by MSTest immediatelly notices the entries and the extra TIP. But how about the Bob? He did not wanted to have it installed. Or even, maybe he has not paid for the fancy new extension? (by the way, the same problem occurs if you drop your binaries to Common/IDE/PrivateAssemblies or PublicAssemblies - they instantly go global and are visible for everyone).

And I think that the devenv.exe really hacks hard into lower layers is worth a note :)

For every bug, there's a facepalm

Remember the ExternalSettingsManager I mentioned earlier? It is public, feel free to play with!

Create a new project, reference Microsoft.VisualStudio.Settings (is in GAC or C:\Program Files\Microsoft Visual Studio 2010 SDK SP1\VisualStudioIntegration\Common\Assemblies\v4.0\Microsoft.VisualStudio.Settings.dll), and write there:

using Microsoft.VisualStudio.Settings;

var lm = Microsoft.Win32.Registry.LocalMachine;
var key = lm.OpenSubKey("Software\\Microsoft\\VisualStudio\\10.0\\EnterpriseTools\\QualityTools\\TestTypes");
var count1 = key.SubKeyCount;

var devenvRoot = @"C:\Program Files\Microsoft Visual Studio 10.0\Common7\IDE\devenv.exe";
var extmgr = ExternalSettingsManager.CreateForApplication(devenvRoot);
var settings = extmgr.GetReadOnlySettingsStore(SettingsScope.Configuration);
var count2 = settings.GetSubCollectionCount("EnterpriseTools\\QualityTools\\TestTypes");

The result? Count1 is of course 2, as it reads from HKLM, and Count2 is three, as the class seems to perfectly support the registry layered-ness. As you probably noticed, it also can provide write-access, has several 'scopes' you may ask for, and completely abstracts from the "Software\Microsoft\VisualStudio\10.0" part - so probably it handles not only the HKLM/HKCU switching, but also probably handles /rootsuffix instance switching and maybe even VisualStudio versioning. I only wonder why I had to provide it with devenv executable path, I didn't had the time to look at it yet.

By the way, the Microsoft.VisualStudio.Settings seems to contain similar detours to those from the devenv.exe. For a few hours I was even sure that the devenv.exe imports the detours from M.VS.Settings, but as I digged deeper, I'm not so sure now, and unfortunatelly, my free-time pool has depleted.

So why did the MSTest/QualityTools not use that class? The assembly is loaded into devenv.exe from beginning, QualityTools could use it freely. For me, it seems more natural to use its API when reading the layered-registry, than to hijack the winapi layer.

I can only assume that the advapi32 hijacking present in devenv's native loader was added in as a radical attempt to reduce costs of splitting the registry into several shadowed branches. Surely, revisiting all modules/packages of VisualStudio was far more expensive than that. The Microsoft.VisualStudio.Settings assembly was probably created much later and included in the VS2010 SDK only because some people reported issues with chaotic and incoherent registry reads/writes from different GUIs and tools..