Removing unwanted references to .NET 4.6.1 from web applications

  Subscribe
7/24/2018 - (updated on 7/25/2018)

The title is a bit specific for this blog post, but that's the gist of it: we ended up with a bunch of references to an in-between version of .NET (4.6.1) that was falsely advertising itself as a more optimal candidate for satisfying 4.6.2 dependencies. This is a known issue; there are several links to MS GitHub issues below.

In this blog, I will discuss direct vs. transient dependencies as well as internal vs. runtime dependencies.

tl;dr

If you've run into problems with an application targeted to .NET Framework 4.6.2 that does not compile on certain machines, it's possible that the binding redirects Visual Studio has generated for you use versions of assemblies that aren't installed anywhere but on a machine with Visual Studio installed.

How I solved this issue:

  • Remove the C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\ directory
  • Remove all System* binding redirects
  • Clean out all bin/ and obj/ folders
  • Delete the .vs folder (may not be strictly necessary)
  • Build in Visual Studio
  • Observe that a few binding-redirect warnings appear
  • Double-click them to re-add the binding redirects, but this time to actual 4.6.2 versions (you may need to add <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects> to your project)
  • Rebuild and verify that you have no more warnings

The product should now run locally and on other machines.

For more details, background and the story of how I ran into and solved this problem, read on.

Building Software

What do we mean when we say that we "build" an application?

Building is the process of taking a set of inputs and producing an artifact targeted at a certain runtime. Some of these inputs are included directly while others are linked externally.

  • Examples of direct inputs are the binary artifacts produced from the source code that comprises your application
  • Examples of external inputs are OS components and runtime environments

The machine does exactly what you tell it to, so it's up to you to make sure that your instructions are as precise as possible. However, you also want your application to be flexible so that it can run on as wide an array of environments as possible.

Your source code consists of declarations. We've generally got the direct inputs under control. The code compiles and produces artifacts as expected. It's the external-input declarations where things go awry.

What kind of external inputs does our application have?

  • System dependencies in the runtime target (assemblies like System.Runtime, System.Data, etc.), each with a minimum version
  • Third-party dependencies pulled via NuGet, each with a minimum version

How is this stitched together to produce the application that is executed?

  • The output folder contains our application, our own libraries and the assemblies from NuGet dependencies
  • All other dependencies (e.g. system dependencies) are pulled from the environment

The NuGet dependencies are resolved at build time. All resources are pulled and added to the release on the build machine. There are no run-time decisions to make about which versions of which assemblies to use.

Dependencies come in two flavors:

  • Direct: A reference in the project itself
  • Transient: A direct reference inherited from another direct or transient reference

It is with the transient references that we run into issues. The following situations can occur:

  • A transient dependency is referenced one or more times with the same version. This is no problem, as the builder simply uses that version or substitutes a newer version if that version is no longer available (rare, but possible)
  • A transient dependency is referenced in different versions. In this case, the builder tries to substitute a single version for all requirements. This generally works OK since most dependencies require a given version or higher. It may be that one or another library cannot work with all newer versions, but this is also rare. In this case, the top-level assembly (the application) must include a hint (an assembly-binding redirect) that indicates that the substitution is OK. More on these below.
  • A transient dependency requires a lower version than the version that is directly referenced. This is also not a problem, as the transient dependency is satisfied by the direct dependency with the higher version. In this case, the top-level application must also include an assembly-binding redirect to allow the substitution without warning.
  • A transient dependency requires a higher version than the version that is directly referenced. This is an error (no longer just a warning) that must be solved by either downgrading the dependency that leads to the problematic transient dependency or upgrading the direct dependency. Generally, the application will upgrade the direct dependency.

Assembly-Binding Redirects

An application generally includes an app.config (desktop applications or services) or web.config XML file that includes a section where binding redirects are listed. A binding redirect indicates the range of versions that can be mapped (or redirected) to a certain fixed version (which is generally also included as a direct dependency).

A redirect looks like this (a more-complete form is further below):

<bindingRedirect oldVersion="0.0.0.0-4.0.1.0" newVersion="4.0.1.0"/>

When the direct dependency is updated, the binding redirect must be updated as well (generally by updating the maximum version number in the range and the version number of the target of the redirect). NuGet does this for you when you're using package.config. If you're using Package References, you must update these manually. This situation is currently not so good, as it increases the likelihood that your binding redirects remain too restrictive.

NuGet Packages

NuGet packages are resolved at build time. These dependencies are delivered as part of the deployment. If they could be resolved on the build machine, then they are unlikely to cause issues on the deployment machine.

System Dependencies

Where the trouble comes in is with dependencies that are resolved at execution time rather than build time. The .NET Framework assemblies are resolved in this manner. That is, an application that targets .NET Framework expects certain versions of certain assemblies to be available on the deployment machine.

We mentioned above that the algorithm sometimes chooses the desired version or higher. This is not the case for dependencies that are in the assembly-binding redirects. Adding an explicit redirect locks the version that can be used.

This is generally a good idea as it increases the likelihood that the application will only run in a deployment environment that is extremely close or identical to the development, building or testing environment.

Aside: Other Bundling Strategies

How can we avoid these pesky run-time dependencies? There are several ways that people have come up with, in increasing order of flexibility:

  • Deliver hardware and software together. This is common in industrial applications and used to be much more common for businesses, as well. Nearly bulletproof. If it worked in the factory, it will work for the customer.
  • Deliver a VM (virtual machine) as your application. This includes the entire execution environment right down to the hardware. Safe, but inefficient.
  • Use a container (e.g. Docker) to deliver a description of the execution environment. The image is built to match the declaration. This is also quite stable and can avoid many of the substitution errors outlined above. If components are outdated, the machine fails to start and the definition must first be updated (and, presumably, tested). This type of deployment is getting more reliable but is also overkill for many applications.
  • Deliver the runtime with the application instead of describing the runtime you'd like to have. Targeting .NET Core instead of .NET Framework includes the runtime. This seems like a nice alternative and it's not surprising that Microsoft went in this direction with .NET Core. It's a good solution to the external-dependency issues outlined above.

To sum up:

  • A VM delivers the OS, runtime and application.
  • A Container delivers a description of the OS and runtime as well as the application itself.
  • .NET Core includes the runtime and application and is OS-agnostic (within reason).
  • .NET Framework includes only the application and some directives on the remaining components to obtain from the runtime environment.

Our application targets .NET Framework (for now). We're looking into .NET Core, but aren't ready to take that step yet.

Where can the deployment go wrong?

To sum up the information from above, problems arise when the build machine contains components that are not available on the deployment machine.

How can this happen? Won't the deployment machine just use the best match for the directives included in the build?

Ordinarily, it would. However, if you remember our discussion of assembly-binding redirects above, those are set in stone. What if you included binding redirects that required versions of system dependencies that are only available on your build machine ... or even your developer machine?

Special Tip for Web Applications

We actually discovered an issue in our deployment because the API server was running, but the Authentication server was not. The Authentication server was crashing because it couldn't find the runtime it needed in order to compile its Razor views (it has ASP.Net MVC components). We only discovered this issue on the deployment server because the views were only ever compiled on-the-fly.

To catch these errors earlier in the deployment process, you can enable pre-compiling views in release mode so that the build server will fail to compile instead of a producing a build that will sometimes fail to run.

Add the <MvcBuildViews>true</MvcBuildViews> to any MVC projects in the PropertyGroup for the release build, as shown in the example below:

<PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
  <DebugType>pdbonly</DebugType>
  <Optimize>true</Optimize>
  <OutputPath>bin</OutputPath>
  <DefineConstants>TRACE</DefineConstants>
  <ErrorReport>prompt</ErrorReport>
  <WarningLevel>4</WarningLevel>
  <LangVersion>6</LangVersion>
  <MvcBuildViews>true</MvcBuildViews>
</PropertyGroup>

How do I create a redirect?

We mentioned above that NuGet is capable of updating these redirects when the target version changes. An example is shown below. As you can see, they're not very easy to write:

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <runtime>
    <assemblyBinding xmlns="urn:schemas-microsoft-com:asm.v1">
      <dependentAssembly>
        <assemblyIdentity name="System.Reflection.Extensions" publicKeyToken="B03F5F7F11D50A3A" culture="neutral"/>
        <bindingRedirect oldVersion="0.0.0.0-4.0.1.0" newVersion="4.0.1.0"/>
      </dependentAssembly>
      <!-- Other bindings... -->
    </assemblyBinding>
  </runtime>
</configuration>

Most bindings are created automatically when MSBuild emits a warning that one would be required in order to avoid potential runtime errors. If you compile with MSBuild in Visual Studio, the warning indicates that you can double-click the warning to automatically generate a binding.

If the warning doesn't indicate this, then it will tell you that you should add the following to your project file:

<AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects>

After that, you can rebuild to show the new warning, double-click it and generate your assembly-binding redirect.

How did we get the wrong redirects?

When MSBuild generates a redirect, it uses the highest version of the dependency that it found on the build machine. In most cases, this will be the developer machine. A developer machine tends to have more versions of the runtime targets installed than either the build or the deployment machine.

A Visual Studio installation, in particular, includes myriad runtime targets, including many that you're not using or targeting. These are available to MSBuild but are ordinarily ignored in favor of more appropriate ones.

That is, unless there's a bit of a bug in one or more of the assemblies included with one of the SDKs...as there is with the net461 distribution in Visual Studio 2017.

Even if you are targeting .NET Framework 4.6.2, MSBuild will still sometimes reference assemblies from the 461 distribution because the assemblies are incorrectly marked as having a higher version than those in 4.6.2 and are taken first.

I found the following resources somewhat useful in explaining the problem (though none really offer a solution):

How can you fix the problem if you're affected?

You'll generally have a crash on the deployment server that indicates a certain assembly could not be loaded (e.g. System.Runtime). If you show the properties for that reference in your web application, do you see the path C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461 somewhere in there? If so, then your build machine is linking in references to this incorrect version. If you let MSBuild generate binding redirects with those referenced paths, they will refer to versions of runtime components that do not generally exist on a deployment machine.

Tips for cleaning up:

  • Use MSBuild to debug this problem. R# Build is nice, but not as good as MSBuild for this task.
  • Clean and Rebuild to force all warnings
  • Check your output carefully.
    • Do you see warnings related to package conflicts?
    • Ambiguities?
    • Do you see the path C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461 in the output?

A sample warning message:

[ResolvePackageFileConflicts] Encountered conflict between 'Platform:System.Collections.dll' and 'CopyLocal:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\lib\System.Collections.dll'.  Choosing 'CopyLocal:C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\lib\System.Collections.dll' because AssemblyVersion '4.0.11.0' is greater than '4.0.10.0'.

The Solution

As mentioned above, but reiterated here, this what I did to finally stabilize my applications:

  • Remove the C:\Program Files (x86)\Microsoft Visual Studio\2017\BuildTools\MSBuild\Microsoft\Microsoft.NET.Build.Extensions\net461\ directory
  • Remove all System* binding redirects
  • Clean out all bin/ and obj/ folders
  • Delete the .vs folder (may not be strictly necessary)
  • Build in Visual Studio
  • Observe that a few binding-redirect warnings appear
  • Double-click them to re-add the binding redirects, but this time to actual 4.6.2 versions (you may need to add <AutoGenerateBindingRedirects>true</AutoGenerateBindingRedirects> to your project)
  • Rebuild and verify that you have no more warnings
  • Deploy and TADA!

One more thing

When you install any update of Visual Studio, it will silently repair these missing files for you. So be aware and check the folder after any installations or upgrades to make sure that the problem doesn't creep up on you again.

Sign up for our Newsletter