1 2 3 4 5 6 7 8 9 10 11
Encodo's configuration library for Quino: part III

imageThis discussion about configuration spans three articles:

  1. part I discusses the history of the configuration system in Quino as well as a handful of principles we kept in mind while designing the new system
  2. part II discusses the basic architectural changes and compares an example from the old configuration system to the new.
  3. part III takes a look at configuring the "execution order" -- the actions to execute during application startup and shutdown

Introduction

Registering with an IOC is all well and good, but something has to make calls into the IOC to get the ball rolling.

Something has to actually make calls into the IOC to get the ball rolling.

Even service applications -- which start up quickly and wait for requests to do most of their work -- have basic operations to execute before declaring themselves ready.

Things can get complex when starting up registered components and performing basic checks and non-IOC configuration.

  • In which order are the components and configuration elements executed?
  • How do you indicate dependencies?
  • How can an application replace a piece of the standard startup?
  • What kind of startup components are there?

Part of the complexity of configuration and startup is that developers quickly forget all of the things that they've come to expect from a mature product and start from zero again with each application. Encodo and Quino applications take advantage of prior work to include standard behavior for a lot of common situations.

Configuration Patterns

Some components can be configured once and directly by calling a method like UseMetaTranslations(string filePath), which includes all of the configuration options directly in the composition call. This pattern is perfect for options that are used only by one action or that it wouldn't make sense to override in a subsequent action.

So, for simple actions, an application can just replace the existing action with its own, custom action. In the example above, an application for which translations had already been configured would just call UseMetaTranslations() again in order to override that behavior with its own.

Most application will replace standard actions or customize standard settings

Some components, however, will want to expose settings that can be customized by actions before they are used to initialize the component.

For example, there is an action called SetUpLoggingAction, which configures logging for the application. This action uses IFileLogSettings and IEventLogSettings objects from the IOC during execution to determine which types of logging to configure.

An application is, of course, free to replace the entire SetUpLoggingAction action with its own, completely custom behavior. However, an application that just wanted to change the log-file behavior or turn on event-logging could use the Configure<TService>() method1, as shown below.

application.Configure<IFileLogSettings>(
  s => s.Behavior = LogFileBehavior.MultipleFiles
);
application.Configure<IEventLogSettings>(
  s => s.Enabled = true
);

Actions

A Quino application object has a list of StartupActions and a list of ShutdownActions. Most standard middleware methods register objects with the IOC and add one or more actions to configure those objects during application startup.

Actions have existed for quite a while in Quino. In Quino 2, they have been considerably simplified and streamlined to the point where all but a handful are little more than a functional interface2.

The list below will give you an idea of the kind of configuration actions we're talking about.

  • Load configuration data
  • Process command line
  • Set up logging
  • Upgrade settings/configuration (e.g. silent upgrade)
  • Log a header (e.g. user/date/file locations/etc.; for console apps. this might be mirrored to the console)
  • Load plugins
  • Set up standard locations (e.g. file-system locations)

For installed/desktop/mobile applications, there's also:

  • Initialize UI components
  • Provide loading feedback
  • Check/manage multiple running instances
  • Check software update
  • Login/authentication

Quino applications also have actions to configure metadata:

  • Configure expression engine
  • Load metadata
  • Load metadata-overlays
  • Validate metadata
  • Check data-provider connections
  • Check/migrate schema
  • Generate default data

Application shutdown has a smaller set of vital cleanup chores that:

  • dispose of connection managers and other open resources
  • write out to the log, flush it and close it
  • show final feedback to the user

Anatomy of an Action

The following example3 is for the 1.x version of the relatively simple ConfigureDisplayLanguageAction.

public class ConfigureDisplayLanguageAction<TApplication> 
  : ApplicationActionBase<TApplication>
  where TApplication : ICoreApplication
{
  public ConfigureDisplayLanguageAction()
    : base(CoreActionNames.ConfigureDisplayLanguage)
  {
  }

  protected override int DoExecute(
    TApplication application, ConfigurationOptions options, int currentResult)
  {
    // Configuration code...
  }
}

What is wrong with this startup action? The following list illustrates the main points, each of which is addressed in more detail in its own section further below.

  • The ConfigurationOptions parameter introduces an unnecessary layer of complexity
  • The generic parameter TApplication complicates declaration, instantiation and extension methods that use the action
  • The int return type along with the currentResult parameter are a bad way of controlling flow.

The same startup action in Quino 2.x has the following changes from the Quino 1.x version above (legend: additions; deletions).

public class ConfigureDisplayLanguageAction<TApplication>
  : ApplicationActionBase<TApplication>
  where TApplication : ICoreApplication
{
  public ConfigureDisplayLanguageAction()
    : base(CoreActionNames.ConfigureDisplayLanguage)
  {
  }

  publicprotected override void int DoExecute(
    TApplication application, ConfigurationOptions options, int currentResult)
  {
    // Configuration code...
  }
}

As you can see, quite a bit of code and declaration text was removed, all without sacrificing any functionality. The final form is quite simple, inheriting from a simple base class that manages the name of the action and overrides a single parameter-less method. It is now much easier to see what an action does and the barrier to entry for customization is much lower.

public class ConfigureDisplayLanguageAction : ApplicationActionBase
{
  public ConfigureDisplayLanguageAction()
    : base(CoreActionNames.ConfigureDisplayLanguage)
  {
  }

  public override void Execute()
  {
    // Configuration code...
  }
}

In the following sections, we'll take a look at each of the problems indicated above in more detail.

Remove the ConfigurationOptions parameter

These options are a simple enumeration with values like Client, Testing, Service and so on. They were used only by a handful of standard actions.

These options made it more difficult to decide how to implement the action for a given task. If two tasks were completely different, then a developer would know to create two separate actions. However, if two tasks were similar, but could be executed differently depending on application type (e.g. testing vs. client), then the developer could still have used two separate actions, but could also have used the configuration options. Multiple ways of doing the exact same thing is all kinds of bad.

Multiple ways of doing the exact same thing is all kinds of bad.

Parameters like this conflict conceptually with the idea of using composition to build an application. To keep things simple, Quino applications should be configured exclusively by composition. Composing an application with service registrations and startup actions and then passing options to the startup introduced an unneeded level of complexity.

Instead, an application now defines a separate action for each set of options. For example, most applications will need to set up the display language to use -- be it for a GUI, a command-line or just to log messages in the correct language. For that, the application can add a ConfigureDisplayLanguageAction to the startup actions or call the standard method UseCore(). Desktop or single-user applications can use the ConfigureGlobalDisplayLanguageAction or call UseGlobalCore() to make sure that global language resources are also configured.

Remove the TApplication generic parameter

The generic parameter to this interface complicates the IApplication<TApplication> interface and causes no end of trouble in MetaApplication, which actually inherits from IApplication<IMetaApplication> for historical reasons.

There is no need to maintain statelessness for a single-use object.

Originally, this parameter guaranteed that an action could be stateless. However, each action object is attached to exactly one application (in the IApplication<TApplication>.StartupActions list. So the action that is attached to an application is technically stateless, and a completely different application than the one to which the action is attached could be passed to the IApplcationAction.Execute...which makes no sense whatsoever.

Luckily, this never happens, and only the application to which the action is attached is passed to that method. If that's the case, though, why not just create the action with the application as a constructor parameter when the action is added to the StartupActions list? There is no need to maintain statelessness for a single-use object.

This way, there is no generic parameter for the IApplication interface, all of the extension methods are much simpler and applications are free to create custom actions that work with descendants of IApplication simply by requiring that type in the constructor parameter.

Debugging is important

A global exception handler is terrible for debugging

The original startup avoided exceptions, preferring an integer return result instead.

In release mode, a global exception handler is active and is there to help the application exit more or less smoothly -- e.g. by logging the error, closing resources where possible, and so on.

A global exception handler is terrible for debugging, though. For exceptions that are caught, the default behavior of the debugger is to stop where the exception is caught rather than where it is thrown. Instead, you want exceptions raised by your application to to stop the debugger from where they are thrown.

So that's part of the reason why the startup and shutdown in 1.x used return codes rather than exceptions.

Multiple valid code paths

The other reason Quino used result codes is that most non-trivial applications actually have multiple paths through which they could successfully run.

Exactly which path the application should take depends on startup conditions, parameters and so on. Some common examples are:

  • Show command-line help
  • Migrate an application schema
  • Import, export or generate data

To show command-line help, an application execute its startup actions in order. It reaches the action that checks whether the user requested command-line help. This action processes the request, displays that help and then wants to smoothly exit the application. The "main" path -- perhaps showing the user a desktop application -- should no longer be executed.

Non-trivial applications have multiple valid run profiles.

Similarly, the action that checks the database schema determines that the schema in the data provider doesn't match the model. In this case, it would like to offer the user (usually a developer) the option to update the schema. Once the schema is updated, though, startup should be restarted from the beginning, trying again to run the main path.

Use exceptions to indicate errors

Whereas the Quino 1.x startup addressed the design requirements above with return codes, this imposes an undue burden on implementors. There was also confusion as to when it was OK to actually throw an exception rather than returning a special code.

Instead, the Quino 2.x startup always uses exceptions to indicate errors. There are a few special types of exceptions recognized by the startup code that can indicate whether the application should silently -- and successfully -- exit or whether the startup should be attempted again.

Conclusion

There is of course more detail into which we could go on much of what we discussed in these three articles, but that should suffice for an overview of the Quino configuration library.



  1. If C# had them, that it is. See Java 8 for an explanation of what they are.

  2. This pattern is echoed in the latest beta of the ASP.NET libraries, as described in the article Strongly typed routing for ASP.NET MVC 6 with IApplicationModelConvention.

  3. Please note that formatting for the code examples has been adjusted to reduce horizontal space. The formatting does not conform to the Encodo C# Handbook.

Encodo's configuration library for Quino: part II

In this article, we'll continue the discussion about configuration started in part I. We wrapped up that part with the following principles to keep in mind while designing the new system.

  • Consistency
  • Opt-in configuration
  • Inversion of Control
  • Configuration vs. Execution
  • Common Usage

Borrowing from ASP.NET vNext

Quino's configuration inconsistencies and issues have been well-known for several versions -- and years -- but the opportunity to rewrite it comes only now with a major-version break.

Luckily for us, ASP.NET has been going through a similar struggle and evolution. We were able to model some of our terminology on the patterns from their next version. For example, ASP.NET has moved to a pattern where an application-builder object is passed to user code for configuration. The pattern there is to include middleware (what we call "configuration") by calling extension methods starting with "Use".

Quino has had a similar pattern for a while, but the method names varied: "Integrate", "Add", "Include"; these methods have now all been standardized to "Use" to match the prevailing .NET winds.

Begone configuration and feedback

Additionally, Quino used to make a distinction between an application instance and its "configuration" -- the template on which an application is based. No more. Too complicated. This design decision, coupled with the promotion of a platform-specific "Feedback" object to first-level citizen, led to an explosion of generic type parameters.1

The distinction between configuration (template) and application (instance) has been removed. Instead, there is just an application object to configure.

The feedback object is now to be found in the service locator. An application registers a platform-specific feedback to use as it would any other customization.

Hello service locator

ASP.NET vNext has made the service locator a first-class citizen. In ASP.NET, applications receive an IApplicationBuilder in one magic "Configure" method and receive an IServiceCollection in another magic "ConfigureServices" method.

In Quino 2.x, the application is in charge of creating the service container, though Quino provides a method to create and configure a standard one (SimpleInjector). That service locator is passed to the IApplication object and subsequently accessible there.

Services can of course be registered directly or by calling pre-packaged Middleware methods. Unlike ASP.NET vNext, Quino 2.x makes no distinction between configuring middleware and including the services required by that middleware.

Begone configuration hierarchy

Quino's configuration library has its roots in a time before we were using an IOC container. The configuration was defined as a hierarchy of configuration classes that modeled the following layers.

  • A base implementation that makes only the most primitive assumptions about an application. For example, that it has a RunMode ("debug" or "release") or an exit code or that it has a logging mechanism (e.g. IRecorder).
  • The "Core" layer comprises application components that are very common, but do not depend on Quino's metadata.
  • And, finally, the "Meta" layer includes configuration for application components that extend the core with metadata-dependent versions as well as specific components required by Quino applications.

While these layers are still somewhat evident, the move to middleware packages has blurred the distinction between them. Instead of choosing a concrete configuration base class, an application now calls a handful of "Use" methods to indicate what kind of application to build.

There are, of course, still helpful top-level methods -- e.g. UseCore() and UseMeta() methods -- that pull in all of the middleware for the standard application types. But, crucially, the application is free to tweak this configuration with more granular calls to register custom configuration in the service locator.

This is a flexible and transparent improvement over passing esoteric parameters to monolithic configuration methods, as in the previous version.

An example: Configure a software updater

Just as a simple example, whereas a Quino 1.x standalone application would set ICoreConfiguration.UseSoftwareUpdater to true, a Quino 2.x application calls UseSoftwareUpdater(). Where a Quino 1.x Winform application would inherit from the WinformFeedback in order to return a customized ISoftwareUpdateFeedback, a Quino 2.x application calls UseSoftwareUpdateFeedback().

The software-update feedback class is defined below and is used by both versions.

public class CustomSoftwareUpdateFeedback : WinformSoftwareUpdateFeedback<IMetaApplication>
{
  protected override ResponseType DoConfirmUpdate(TApplication application, ...)
  {
    ...
  }
}

That's where the similarities end, though. The code samples below show the stark difference between the old and new configuration systems.

Quino 1.x

As explained above, Quino 1.x did not allow registration of a sub-feedback like the software-updater. Instead, the application had to inherit from the main feedback and override a method to create the desired sub-feedback.

class CustomWinformFeedback : WinformFeedback
{
  public virtual ISoftwareUpdateFeedback<TApplication> GetSoftwareUpdateFeedback<TApplication, TConfiguration, TFeedback>()
    where TApplication : ICoreApplication<TConfiguration, TFeedback>
    where TConfiguration : ICoreConfiguration
    where TFeedback : ICoreFeedback
  {
    return new CustomSoftwareUpdateFeedback(this);
  }
}

var configuration = new CustomConfiguration()
{
  UseSoftwareUpdater = true
}

WinformDxMetaConfigurationTools.Run(configuration, app => new CustomMainForm(app), new CustomWinformFeedback());

The method-override in the feedback was hideous and scared off a good many developers. not only that, the pattern was to use a magical, platform-specific WinformDxMetaConfigurationTools.Run method to create an application, run it and dispose it.

Quino 2.x

Software-update feedback-registration in Quino 2.x adheres to the principles outlined at the top of the article: it is consistent and uses common patterns (functionality is included and customized with methods named "Use"), configuration is opt-in, and the IOC container is used throughout (albeit implicitly with these higher-level configuration methods).

using (var application = new CustomApplication())
{
  application.UseMetaWinformDx();
  application.UseSoftwareUpdater();
  application.UseSoftwareUpdaterFeedback(new CustomSoftwareUpdateFeedback());
  application.Run(app => new CustomMainForm(app));
}

Additionally, the program has complete control over creation, running and disposal of the application. No more magic and implicit after-the-fact configuration.

What comes after configuration?

In the next and (hopefully) final article, we'll take a look at configuring execution -- the actions to execute during startup and shutdown. Registering objects in a service locator is all well and good, but calls into the service locator have to be made in order for anything to actually happen.

Keeping this system flexible and addressing standard application requirements is a challenging but not insurmountable problem. Stay tuned.


  1. The CustomWinformFeedback in the Quino 1.x code at the end of this article provides a glaring example.

Encodos configuration library for Quino: part I

In this article, I'll continue the discussion about configuration improvements mentioned in the release notes for Quino 2.0-beta1. With beta2 development underway, I thought I'd share some more of the thought process behind the forthcoming changes.

Software Libraries

what sort of patterns integrate and customize the functionality of libraries in an application?

An application comprises multiple tasks, only some of which are part of that application's actual domain. For those parts not in the application domain, software developers use libraries. A library captures a pattern or a particular way of doing something, making it available through an abstraction. These simplify and smooth away detail irrelevant to the application.

A runtime and its standard libraries provide many such abstractions: for reading/writing files, connecting to networks and so on. Third-party libraries provide others, like logging, IOC, task-scheduling and more.

Because Encodo's been writing software for a long time, we have a lot of patterns that we've come up with for our applications. These libraries are split into two main groups:

  • Encodo.*: extensions to the .NET framework or third-party libraries that don't depend on Quino metadata.
  • Quino.*: extensions to the .NET framework, third-party libraries or Encodo libraries that depend on Quino metadata.

A sort of "meta" library that lies on top of all of this is configuration and startup of applications that use these libraries. That is, what sort of patterns integrate and customize the functionality of libraries in an application?

Balancing K.I.S.S. and D.R.Y

Almost nowhere in an application is the balance between K.I.S.S. and D.R.Y. more difficult to maintain than in configuration and startup.

So if we already know all of that, why does Quino need a new configuration library?

As mentioned above, there is a lot of commonality between applications in this area. An application will definitely want to incorporate such common configuration from a library. Updates and improvements to that library will then be applied as for any other. This is a good thing.

However, an application will also want to be able to tweak almost any given facet of this shared configuration. That is: just keep the good parts, have those upgraded when they're changed, but apply customization and extend functionality for the application's domain. Easy, right?

It is here that a good configuration library will find just the right level of granularity for customization. Too coarse? Then an application ends up throwing out too much common configuration in order to customize a small part of it. Too fine? Then the configuration system is too verbose or complex and the application avoids using it.

Instead, a configuration system should establish clear patterns -- optimally, just one -- for how to apply customization.

  • The builder of the underlying configuration library has to consider the myriad situations that might face a library developer and distill those requirements to a common pattern.
  • The library developer needs to think about which parts an application might want to customize and think about how to expose them.

So if we already know all of that, then why does Quino need a new configuration library? Well...

History of Quino's Configuration Library

It's really easy to make things over-complicated and muddy. It's really easy to end up growing several different kinds of extension systems over the years. Quino ended up with a generics-heavy API that made declaring new configuration components very wordy.

The core of Quino is the metadata definition for an application domain. That part has barely changed at all since we first wrote it lo so many years ago. We declared it to be our core business -- the part that we are better than others at -- the part we wanted to have under our own control. Our first draft1 has held up remarkably well.

Many of the other components have undergone quite a bit of flux: changes in requirements and the components themselves as well as new development processes and patterns all contributed to change. Over time, various applications had different needs and made adjustments to a different iteration of the configuration library. We moved from supporting only single-threaded, single-user desktop applications to also supporting multi-user, multi-threaded services and web servers.

...we were left with an ugly configuration system that no-one wanted to extend or use -- so yet another would be invented.

For all of these different applications, we naturally wanted to maintain the common configuration where possible -- but customizations for new platforms stretched the capabilities of the configuration library.

Customization would be made to a new version of that library, but applications that couldn't be upgraded immediately forced backwards-compatibility and thus resulted in several different concurrent ways of configuring a particular facet of an application.

In order to keep things in one place, we ended up breaking the interface-separation rule. Dependencies started clumping drastically, but it was OK because nobody was trying to use one thing without the other ten. But it was hard to see what was going on; customization became a black box for all but one or two gurus. On and on it went, until we were left with an ugly configuration system that no-one wanted to extend or use -- so yet another would be invented, ad-hoc. And so it went.

Principles for Quino 2.0 Configuration

With Quino 2.0, we examined the existing system and came up with a list of principles.

  • Consistency: there should be only be one way of customizing settings and components. When a developer asks how to change something, the answer should always be the same pattern. If not, there better be a damned good reason (see "Configuration vs. Execution" below).
  • Opt-in configuration: No more magic methods or base classes that automatically add components and settings in black boxes. Even if the application has to call one or two more methods, it's better to be declarative than clever(tm).
  • Inversion of Control: Standardize configuration to use an IOC container or service locator wherever possible. Instead of clumping settings in configuration or application objects, create discrete settings and put them in the container. Make dependencies explicit (constructor parameters!) and resolved through the container wherever possible.
  • Configuration vs. Execution: Be very aware of the difference between the "configuration" phase and the "execution" phase. During configuration, the service locator is used in write-only mode; during execution, the service locator is in read-only mode. Code executed during configuration must rely only on explicit dependency-injection via constructor.
  • Common Usage: Establish a pattern for calling configuration methods, from least to most specific. E.g. call Quino's base configuration methods before any application-specific customization. Establish patterns for how to configure a single startup action or how to create settings for a larger component that could be further customized in subsequent phases.

In the next part, we'll take a look at some concrete examples and documentation for the new patterns.2



  1. To be fair, it wasn't our first attempt at metadata. In one way or another, we'd been defining metadata structures for generic programming for more years than we'd be comfortable divulging. A h/t of course to Opus Software's Atlas libraries -- 1 and 2 -- where many of us contributed. Also, I had experience with cross-platform, generic libraries in C++ stretching all the way back to the late 90s as well as the generalized/meta elements of the earthli WebCore. So it was more like the fourth or fifth shot at it, if we're going to be honest -- but at least we got it right. :-)

  2. In particular, I'll add more detail about "Common Usage" for those who might feel I've left them hanging a bit in the last bullet point. Sorry 'bout that. The day is only so long. See you next time...

v2.0-beta1: Configuration, services and web

The summary below describes major new features, items of note and breaking changes. The full list of issues is also available for those with access to the Encodo issue tracker.

Highlights

These are the big ones that forced a major-version change.

Some smaller, but important changes:

  • Added support for RunInTransaction attribute. Specify the attribute on any IMetaTestFixture to wrap a test or every test in a fixture in a transaction. (QNO-4682)
  • Shared connection manager is now disposed when an application is disposed. (QNO-4752)

Breaking changes

Oh yeah. You betcha. This is a major release and we've knowingly made a decision not to maintain backwards-compatibility at all costs. Good news, though, the changes to make are relatively straightforward and easy to make if you've got a tool like ReSharper that can update using statements automatically.

Namespace changes

As we saw in part I and part II of the guide to using NDepend, Quino 2.0 has unsnarled quite a few dependency issues. A large number of classes and interfaces have been moved out of the Encodo.Tools namespace. Many have been moved to Encodo.Core but others have been scattered into more appropriate and more specific namespaces.

This is one part of the larger changes, easily addressed by using ReSharper to Alt + Enter your way through the compile errors.

Logging changes

Another large change is in renaming IMessageRecorder to IRecorder and IMessageStore to IInMemoryRecorder. Judicious use of search/replace or just a bit of elbow grease will get you through these as well.

Configuration changes

Finally, probably the most far-reaching change is in merging IConfiguration into IApplication. In previous versions of Quino, applications would create a configuration object and pass that to a platform-dependent Quino Run() method. Some configuration was provided by the application and some by the platform-specific method.

The example for Quino 1.13.0 below comes from the JobVortex Winform application.

var configuration = new JobVortexConfiguration
{
  MainSettings = Settings.Default
};

configuration.Add(new JobVortexClientConfigurationPackage());

if (!string.IsNullOrEmpty(Settings.Default.DisplayLanguage))
{
  configuration.DisplayLanguage = new Language(Settings.Default.DisplayLanguage);
}

WinformDxMetaConfigurationTools.Run(
  configuration, 
  app => new MainForm(app)
);

In Quino 2.0, the code above has been rewritten as shown below.

using (IMetaApplication application = new JobVortexApplication())
{
  application.MainSettings = Settings.Default;
  application.UseJobVortexClient();

  if (!string.IsNullOrEmpty(Settings.Default.DisplayLanguage))
  {
    application.DisplayLanguage = new Language(Settings.Default.DisplayLanguage);
  }

  application.Run(app => new MainForm(app));
}

As you can see, instead of creating a configuration, the program creates an application object. Instead of using configuration packages mixed with extension methods named "Integrate", "Configure" and so on, the new API uses "Use" everywhere. This should be comfortable for people familiar with the OWIN/Katana configuration pattern.

It does, however, mean that the IConfiguration, ICoreConfiguration and IMetaConfiguration don't exist anymore. Instead, use IApplication, ICoreApplication and IMetaApplication Again, a bit of elbow grease will be needed to get through these compile errors, but there's little to no risk or need for high-level decisions.

There are a lot of these prepackaged methods to help you create common kinds of applications:

  • UseCoreConsole() (a non-Quino application that uses the console)
  • UseMetaConsole() (a Quino application that uses the console)
  • UseCoreWinformDx() (a non-Quino application that uses Winform)
  • UseMetaWinformDx() (a Quino application that uses Winform)
  • UseReporting()
  • UseRemotingServer()
  • Etc.

I think you get the idea. Once we have a final release for Quino 2.0, we'll write more about how to use this new pattern.

Looking ahead to 2.0 Final

This is still just an internal beta of the 2.0 final version. More changes are on the way, including but not limited to:

  • Remove IConfigurationPackage and standardize the configuration API to be named "Use" everywhere (QNO-4771)
  • GenericObject improvements (QNO-4761, QNO-4762)
  • Change compile location for all projects (QNO-4756)
  • Move a lot of properties from ICoreApplication and IMetaApplication to configuration objects in the service locator. Also improve use of and configuration of service locator (QNO-4659)
  • More improvements to the recorders and logging (QNO-4688)
  • Changes to how ORM objects and metadata are generated. (QNO-4506)
  • Separate Encodo and Quino assemblies into multiple, smaller assemblies (QNO-4376)

See you there!

Quino Data Driver architecture, Part III: The Pipeline

In part I of these series, we discussed applications, which provide the model and data provider, and sessions, which encapsulate high-level data context. In part II, we covered command types and inputs to the data pipeline.

In this article, we're going to take a look at the data pipeline itself.

  1. Applications & Sessions
  2. Command types & inputs
  3. The Data Pipeline
  4. Builders & Commands
  5. Contexts and Connections
  6. Sessions, resources & objects

Overview

image

The primary goal of the data pipeline is, of course, to correctly execute each query to retrieve data or command to store, delete or refresh data. The diagram to the right shows that the pipeline consists of several data handlers. Some of these refer to data sources, which can be anything: an SQL database or a remote service.1

The name "pipeline" is only somewhat appropriate: A command can jump out anywhere in the pipeline rather than just at the opposite end. A given command will be processed through the various data handlers until one of them pronounces the command to be "complete".

Command context: recap

In the previous parts, we learned that the input to the pipeline is an IDataCommandContext. To briefly recap, this object has the following properties:

  • Session: Defines the context within which to execute the command
  • Handler: Implements an abstraction for reading/writing values and flags to the objects (e.g. SetValue(IMetaProperty)); more detail on this later
  • Objects: The sequence of objects on which to operate (e.g. for save commands) or to return (e.g. for load commands)
  • ExecutableQuery: The query to execute when loading or deleting objects
  • MetaClass: The metadata that describes the root object in this command; more detail on this later as well

Handlers

Where the pipeline metaphor holds up is that the command context will always start at the same end. The ordering of data handlers is intended to reduce the amount of work and time invested in processing a given command.

Analyzers

The first stage of processing is to quickly analyze the command to handle cases where there is nothing to do. For example,

  • The command is to save or delete, but the sequence of Objects is empty
  • The command is to save or reload, but none of the objects in the sequence of Objects has changed
  • The command is to load data but the query restricts to a null value in the primary key or a foreign key that references a non-nullable, unique key.

It is useful to capture these checks in one or more analyzers for the following reasons,

  1. All drivers share a common implementation for efficiency checks
  2. Optimizations are applied independent of the data sources used
  3. Driver code focuses on driver-specifics rather than general optimization

Caches

If the analyzer hasn't categorically handled the command and the command is to load data, the next step is to check caches. For the purposes of this article, there are two things that affect how long data is cached:

  1. If the session is in a transacted state, then only immutable data, data that was loaded before the transaction began or data loaded within that transaction can be used. Data loaded/saved by other sessions -- possibly to global caches -- is not visible to a session in a transaction with an isolationLevel stricter than RepeatableRead.
  2. The metadata associated with the objects can include configuration settings that control maximum caching lifetime as well as an access-timeout. The default settings are good for general use but can be tweaked for specific object types.

Caches currently include the following standard handlers2:

  • The ValueListDataHandler returns immutable data. Since the data is immutable, it can be used independent of the transaction-state of the session in which the command is executed.
  • The SessionCacheDataHandler returns data that's already been loaded or saved in this session, to avoid a call to a possibly high-latency back-end. This data is safe to use within the session with transactions because the cache is rolled back when a transaction is rolled back.

Data sources

If the analyzer and cache haven't handled a command, then we're finally at a point where we can no longer avoid a call to a data source. Data sources can be internal or external.

Databases

The most common type is an external database:

  • PostgreSql 8.x and higher (PostgreSql 9.x for schema migration)
  • Sql Server 2008 and higher (w/schema migration)
  • Mongo (no schema; no migration)
  • SQlite (not yet released)

Remoting

Another standard data source is the Quino remote application server, which provides a classic interface- and method-based service layer as well as mapping nearly the full power of Quino's generalized querying capabilities to an application server. That is, an application can smoothly switch between a direct connection to a database to using the remoting driver to call into a service layer instead.

The remoting driver supports both binary and JSON protocols. Further details are also beyond the scope of this article, but this driver has proven quite useful for scaling smaller client-heavy applications with a single database to thin clients talking to an application server.

Custom/Aspect-based

And finally, there is another way to easily include "mini" data drivers in an application. Any metaclass can include an IDataHandlerAspect that defines its own data driver as well as its capabilities. Most implementations use this technique to bind in immutable lists of data. But this technique has also been used to load/save data from/to external APIs, like REST services. We can take a look at some examples in more detail in another article.

The mini data driver created for use with an aspect can relatively easily be converted to a full-fledged data handler.

Local evaluation

The last step in a command is what Quino calls "local evaluation". Essentially, if a command cannot be handled entirely within the rest of the data pipeline -- either entirely by an analyzer, one or more caches or the data source for that type of object -- then the local analyzer completes the command.

What does this mean? Any orderings or restrictions in a query that cannot be mapped to the data source (e.g. a C# lambda is too complex to map to SQL) are evaluated on the client rather than the server. Therefore, any query that can be formulated in Quino can also be evaluated fully by the data pipeline -- the question is only of how much of it can be executed on the server, where it would (usually) be more efficient to do so.

Please see the article series that starts with Optimizing data access for high-latency networks for specific examples.

In this article, we've learned a bit about the ways in which Quino retrieves and stores data using the data pipeline. In the next part, well cover the topic Builders & Commands.



  1. E.g. Quino uses a ProtoBuf-like protocol to communicate with its standard application server.

  2. There is an open issue to Introduce a global cache for immutable objects or objects used not in a transaction.

Quino Data Driver architecture, Part II: Command types & inputs

In part I, we discussed applications -- which provide the model and data provider -- and sessions -- which encapsulate high-level data context.

In this article, we're going to take a look at the command types & inputs

  1. Applications & Sessions
  2. Command types & inputs1
  3. The Data Pipeline
  4. Builders & Commands
  5. Contexts and Connections
  6. Sessions, resources & objects

Overview

image

Before we can discuss how the pipeline processes a given command, we should discuss what kinds of commands the data driver supports and what kind of inputs the caller can pass to it. As you can well imagine, the data driver can be used for CRUD -- to create, read, update and delete and also to refresh data.

In the top-right corner of the diagram to the right, you can see that the only input to the pipeline is an IDataCommandContext. This object comprises the inputs provided by the caller as well as command-specific state used throughout the driver for the duration of the command.

Command types

A caller initiates a command with either a query or an object graph, depending on the type of command. The following commands and inputs are supported:

  • Load: returns a cursor for the objects that match a query
  • Count: returns the number of objects that match a query
  • Save: saves an object graph
  • Reload: refreshes the data in an object graph
  • Delete: deletes an object graph or the objects that match a query

Queries

A query includes information about the data to return (or delete).

  • Metadata: The meta-class represents the type of the root object for the command. For example, a "person" or "company".
  • Filtering: Filters restrict the objects to return. A filter can address properties of the root object, but also properties of objects related to the root object. A caller can query for people whose first names start with the letter "m" -- FirstName %~ 'm'2 -- or the caller can find all people which belong to a company whose name starts with the letter "e" -- Company.FirstName %~ 'e'. The context for these expressions is naturally the meta-class mentioned above. Additionally, the metadata/model can also include default filters to include.
  • Ordering: Orderings that determine in which order the data is returned. Orderings are also specified with the expression language, but are usually simpler, like ordering first by LastName and then by FirstName. More complex expressions are supported -- for example, you could use the expression "{LastName}, {FirstName}", which sorts by a formatted string3 -- but be aware that many data stores have limited support for complex expressions in orderings. Orderings are ignored in a query when used to delete objects.

Queries are a pretty big topic and we've only really scratched the surface so far. Quino has its own query language -- QQL -- the specification for which weighs in at over 80 pages, but that's a topic for another day.

Object graphs

An object graph consists of a sequence of root objects and the sub-objects available along relations defined in the metadata.

It's actually simpler than it perhaps sounds.

Let's use the example above: a person is related to a single company, so the graph of a single person will include the company as well (if the object is loaded and/or assigned). Additionally, the company defines a relation that describes the list of people that belong to it. The person=>company relationship is complementary to the company=>person relationship. We call person=>company a 1-1 relation, while company=>person is a 1-n relation.

The following code creates two new companies, assigns them to three people and saves everything at once.

var encodo = new Company { Name = "Encodo Systems AG" };
var other = new Company { Name = "Not Encodo" };
var people = new [] 
{
  new Person { FirstName = "John", LastName = "Doe", Company = other },
  new Person { FirstName = "Bob", LastName = "Smith", Company = encodo },
  new Person { FirstName = "Ted", LastName = "Jones", Company = encodo }
};

Session.Save(people);

The variable people above is an object graph. The variables encodo and other are also object graphs, but only to parts of the first one. From people, a caller can look up people[0].Company, which is other. The graph contains cycles, so people[0].Company.People[0].Company is also other. From encodo, the caller can get to other people in the same company, but not to people in the other company, for example, encodo.People[0] gets "Bob Smith" and encodo.People[0].Company.People[1] gets "Ted Jones".

As with queries, object graphs are a big topic and are strongly bound to the kind of metadata available in Quino. Another topic for another day.

Determining Inputs

Phew. We're almost to the point where we can create an IDataCommandContext to send into the data pipeline.

  • We have an IDataSession and know why we need it
  • We know what type of command we want to execute (e.g. "Load")
  • We have either a query or an object graph

With those inputs, Quino has all it needs from the caller. A glance at the top-left corner of the diagram above shows us that Quino will determine an IMetaClass and an IMetaObjectHandler from these inputs and then use them to build the IDataCommandContext.

An IQuery has a MetaClass property, so that's easy. With the meta-class and the requested type of object, the data driver checks a list of registered object-handlers and uses the first one that says it supports that type. If the input is an object graph, though, the object-handler is determined first and then the meta-class is obtained from the object-handler using a root object from the graph.

Most objects will inherit from GenericObject which implements the IPersistable interface required by the standard object handler. However, an application is free to implement an object handler for other base classes -- or no base class at all, using reflection to get/set values on POCOs. That is, however, an exercise left up to the reader.

At this point, we have all of our inputs and can create the IDataCommandContext.

In the next part, we'll take a look at the "Data Pipeline" through which this command context travels.



  1. You'll notice, perhaps, that this topic is new to this article. I'm expanding the series as I go along, trying to provide enough information to understand the process while keeping the individual blog entries to a digestible size.

  2. "%~" is actually the case-insensitive begins-with operator. You can find out more about comparison operators in the Quino documentation. Browse to "Encodo Base Library" and then "Expressions".

  3. For more information on how to use Quino's unique take on interpolated strings, see the documentation in the footnote above.

Quino Data Driver architecture, Part I: Applications & Sessions

One part of Quino that has undergone quite a few changes in the last few versions is the data driver. The data driver is responsible for CRUD: create, read, update and delete operations. One part of this is the ORM -- the object-relational mapper -- that marshals data to and from relational databases like PostgreSql, SQL Server and SQLite.

We're going to cover a few topics in this series:

  1. Applications & Sessions
  2. The Data Pipeline
  3. Builders & Commands
  4. Contexts and Connections
  5. Sessions, resources & objects

But first let's take a look at an example to anchor our investigation.

Introduction

An application makes a request to the data driver using commands like Save() to save data and GetObject() or GetList() to get data. How are these high-level commands executed? Quino does an excellent job of shielding applications from the details but it's still very interesting to know how this is achieved.

The following code snippet creates retrieves some data, deletes part of it and saves a new version.

using (var session = application.CreateSession())
{
  var people = session.GetList<Person>();
  people.Query.WhereEquals(Person.Fields.FirstName, "john");
  session.Delete(people);
  session.Save(new Person { FirstName = "bob", LastName = "doe" });
}

In this series, we're going to answer the following questions...and probably many more.

  • Where does the data come from?
  • What kind of sources are supported? How?
  • Is at least some of the data cached?
  • Can I influence the cache?
  • What is a session? Why do I need one?
  • Wait...what is the application?

Let's tackle the last two questions first.

Application

The application defines common configuration information. The most important bits for the ORM are as follows:

  • Model: The model is the central part of any Quino application. The model defines entities, their properties, relationships between entities and so on. Looking at the example above, the model will include a definition for a Person, which has at least the two properties LastName and FirstName. There is probably an entity named Company as well, with a one-to-many relationship to Person. As you can imagine, Quino uses this information to formulate requests to data stores that contain data in this format.1 For drivers that support it, Quino also uses this information in order to create that underlying data schema.2
  • DataProvider: The data provider encapsulates all of the logic and components needed to map the model to data sources. This is the part of the process on which this series will concentrate.
  • ConfigurationData: The configuration data describes which parts of the model are connected to which parts of the data provider. The default is, of course, that the entire model is mapped to a single data source. However, even in that case, the configuration indicates which data source: Sql Server? PostgreSql? A remote application server (2nd tier)? With a high-level API as described above, all of these decisions can be made in the configuration rather than assumed throughout the application. Yes, this means that you can change your Quino application from a two-tier to a three-tier application with a single configuration change.

Sessions

So that's the application. There is a single shared application for a process.

But in any non-trivial application -- and any non-desktop application -- we will have multiple data requests running, possibly in different threads of execution.

  • Each request in a web application is a separate data context. Changes made in one request should not affect any other request. Each request may be authenticated as a different user.
  • A remote application-server is very similar to a web application. It handles requests from multiple users. Since it's generally the second layer, it will most likely have direct connections to one or more databases. In this case, it will probably be in charge of executing business logic, most likely in a database transaction. In that case, we definitely don't want one request using the transaction context from another request.
  • Even a non-web client-side application may want to execute some logic in the background or in a separate thread. In those cases, we probably want to keep the data used there separate from the data or objects used to render the other parts of the application.

That's where sessions come in. The session encapsulates a data context, which contains the following information:

  • Application: The application will, as described above, tell the session which model and data provider to use.
  • Current user: For those familiar with ASP.NET, this is very similar to the HttpContext.Current.User but generalized to be available in any Quino application. All data requests over a session are made in the context of this user.
  • Access control: The access control provides information about the security context of an application. An application generally uses the access control to perform authorization checks.
  • Cache: Each session also has its own cache. There are global caches, but those are for immutable data. The session's cache is always available, even when using transactions.
  • ConnectionManager: Many external data sources have transactable/shared state in the form of a connection. As with data, connections can sometimes be shared between sessions and sometimes they can't. The connection manager takes care of knowing all of that for you.

If we go back to the original code sample, we now know that creating a new session with CreateSession() creates a new data context, with its own user and its own data cache. Since we didn't pass in any credentials, the session uses the default credentials for the application.3 All data access made on that session is nicely shielded and protected from any data access made in other sessions (where necessary, of course).

So now we're no closer to knowing how Quino works with data on our behalf, but we've taken the first step: we know all about one of the main inputs to the data driver, the session.

In the next part, we'll cover the topic "The Data Pipeline".


var requestCredentials = requestSession.AccessControl.CurrentUser.CreateCredentials();
using (var session = application.CreateSession(requestCredentials))
{
  // Work with session
}

  1. The domain model is used for everything in a Quino application -- not just the ORM and for schema-migration. We use the model to generate C# code like concrete ORM objects, metadata references (e.g. the Person.Fields.FirstName in the example), or view models, DTOs or even client-side TypeScript definitions. We also use the model to generate user interfaces -- both for entire desktop-application interfaces but also for HTML helpers to build MVC views.

  2. See the article Schema migration in Quino 1.13 for more information on how that works.

  3. This is code that you might use in a single-user application. In a server application, you would most likely just use the session that was created for your request by Quino. If an application wants to create a new session, but using the same user as an existing session, it would call:

The Road to Quino 2.0: Maintaining architecture with NDepend (part II)

In the previous article, I explained how we were using NDepend to clean up dependencies and the architecture of our Quino framework. You have to start somewhere, so I started with the two base assemblies: Quino and Encodo. Encodo only has dependencies on standard .NET assemblies, so let's start with that one.

The first step in cleaning up the Encodo assembly is to remove dependencies on the Tools namespace. There seems to be some confusion as to what belongs in the Core namespace versus what belongs in the Tools namespace.

There are too many low-level classes and helpers in the Tools namespace. Just as a few examples, I moved the following classes from Tools to Core:

  • BitTools
  • ByteTools
  • StringTools
  • EnumerableTools

The names kind of speak for themselves: these classes clearly belong in a core component and not in a general collection of tools.

Now, how did I decide which elements to move to core? NDepend helped me visualize which classes are interdependent.

Direct Dependencies

imageWe see that EnumerableTools depends on StringTools. I'd just moved EnumerableTools to Encodo.Core to reduce dependence on Encodo.Tools. However, since StringTools is still in the Tools namespace, the dependency remains. This is how examining dependencies really helps clarify a design: it's now totally obvious that something as low-level as StringTools belongs in the Encodo.Core namespace and not in the Encodo.Tools namespace, which has everything but the kitchen sink in it.

imageAnother example in the same vein is shown to the left, where we examine the dependencies of MessageTools on Encodo.Tools. The diagram explains that the colors correspond to the two dependency directions.1

We would like the Encodo.Messages namespace to be independent of the Encodo.Tools namespace, so we have to consider either (A) removing the references to ExceptionTools and OperatingSystemTools from MessageTools or (B) moving those two dependencies to the Encodo.Core namespace.

Choice (A) is unlikely while choice (B) beckons with the same logic as the example above: it's now obvious that tools like ExceptionTools and OperatingSystemTools belong in Encodo.Core rather than the kitchen-sink namespace.

Indirect Dependencies

Once you're done cleaning up your direct dependencies, you still can't just sit back on your laurels. Now, you're ready to get started looking at indirect dependencies. These are dependencies that involve more than just two namespaces that use each other directly. NDepend displays these as red bounding blocks. The documentation indicates that these are probably good component boundaries, assuming that the dependencies are architecturally valid.

NDepend can only show you information about your code but can't actually make the decisions for you. As we saw above, if you have what appear to be strange or unwanted dependencies, you have to decide how to fix them. In the cases above, it was obvious that certain code was just in the wrong namespace. In other cases, it may simply be a few bits of code are defined at too low a level.

Improper use of namespaces

For example, our standard practice for components is to put high-level concepts for the component at the Encodo.<ComponentName> namespace. Then we would use those elements from sub-namespaces, like Encodo.<ComponentName>.Utils. However, we also ended up placing types that then used that sub-namespace in the upper-level namespace, like ComponentNameTools.SetUpEnvironment() or something like that. The call to SetUpEnvironment() references the Utils namespace which, in turn, references the root namespace. This is a direct dependency, but if another namespace comes between, we have an indirect dependency.

This happens quite quickly for larger components, like Encodo.Security.

The screenshots below show a high-level snapshot of the indirect dependencies in the Encodo assembly and then also a detail view, with all sub-namespaces expanded. The detail view is much larger but shows you much more information about the exact nature of the cycle. When you select a red bounding box, another panel shows the full details and exact nature of the dependency.

imageimageimage

Base Camp Two: base library almost cleaned up

imageimage

After a bunch of work, I've managed to reduce the dependencies to a set of interfaces that are clearly far too dependent on many subsystems.

  • ICoreConfiguration: references configuration options for optional subsystems like the software updater, the login, the incident reporter and more
  • ICoreFeedback: references feedbacks for several optional processes, like software-update, logins and more
  • ICoreApplication: references both the core configuration and feedback

The white books for NDepend claim that "[t]echnically speaking, the task of merging the source code of several assemblies into one is a relatively light one that takes just a few hours." However, this assumes that the code has already been properly separated into non-interdependent namespaces that correspond to components. These components can then relatively easily be extracted to separate assemblies.

The issue that I have above with the Encodo assembly is a thornier one: the interfaces themselves embody a pattern that is inherently non-decoupling. I need to change how the configuration and feedback work completely in order to decouple this code.

Roadmap for startup and configuration

To that end, I've created an issue in the issue-tracker for Quino, QNO-46592, titled "Re-examine how the configuration, feedback and application work together". The design of these components predates our introduction of a service locator, which means it's much more tightly coupled (as you can see above).

After some internal discussion, we've decided to change the design of the Encodo and Quino library support for application-level configuration and state.

Merge the configuration and application

To date, the configuration has contained all of the information necessary to run an application. The configuration was more-or-less stateless and corresponded to the definition of an application, akin to how a class is the underlying stateless definition, while an object is an instance of that definition. In practice, though, we always use a single application per configuration and the distinction is irrelevant, for all practical purposes. This will simplify all referencing code, as we will no longer need to pass around an IApplication<TConfiguration, TFeedback>.

Move the feedback to the service locator

Instead of treating the feedback like a first-class citizen, with a direct reference on the application, make consumers use the service locator to retrieve an instance. This will remove the remaining generic argument in the definition of IApplication, leaving us with a base interface that is free of generic arguments.

Move specific configuration objects to the service locator

The specific sub-interfaces that introduce dependencies are as follows:

 * IncidentReporter
 * SoftwareUpdater
 * CommandSetManager
 * LocationManager
 * ConnectionSettingsManager

Any components that currently reference the properties on the ICoreConfiguration can use the service locator to retrieve an instance instead.

Move specific settings to sub-objects

The configuration object is not only dependent on sub-objects, but is also overloaded with individual settings that are only used by very few specific sub-components. These will also be extracted into interfaces and moved into the service locator.

 * ILoginConfiguration
 * ISoftwareUpdateConfiguration
 * IFileLogConfiguration

As you can see, while NDepend is indispensable for finding dependencies, it can -- along with a good refactoring tool (we use ReSharper) -- really only help you clean up the low-hanging fruit. While I started out trying to split assemblies, I've now been side-tracked into cleaning up an older and less--well-designed component -- and that's a very good thing.

There are some gnarly knots that will feel nearly unsolvable -- but with a good amount of planning, those can be re-designed as well. As I mentioned in the previous article, though, we can do so only because we're making a clean break from the 1.x version of Quino instead of trying to maintain backward compatibility.

It's worth it, though: the new design already looks much cleaner and is much more easily explained to new developers. Once that rewrite is finished, the Encodo assembly should be clean and I'll use NDepend to find good places to split up that rather large assembly into sensible sub-assemblies.



  1. There is a setting to turn off showing the green dependencies -- where the row depends on the column -- to make it easier to read the matrix. If you do that, though, you have to make sure to select the class from which you're trying to remove dependencies in the column. For example, if class A and B are interdependent, but A should not rely on B, you should make sure A is showing in the column. You can then examine dependencies on row B -- and then remove them. This works very nicely with both direct and indirect dependencies.

  2. This link is to the Quino issue tracker, which requires a login.

The Road to Quino 2.0: Maintaining architecture with NDepend (part I)

Full disclosure

A while back -- this last spring, I believe -- I downloaded NDepend to analyze code dependencies. The trial license is fourteen days; needless to say, I got only one afternoon in before I was distracted by other duties. That was enough, however, to convince me that it was worth the $375 to continue to clean up Quino with NDepend.

I decided to wait until I had more time before opening my wallet. In the meantime, however, Patrick Smacchia of NDepend approached me with a free license if I would write about my experiences using NDepend on Encodo's blog. I'm happy to write about how I used the tool and what I think it does and doesn't do.1

History & Background

imageWe started working on Quino in the fall of 2007. As you can see from the first commit, the library was super-small and comprised a single assembly.

Fast-forward seven years and Version 1.13 of Quino has 66 projects/assemblies. That's a lot of code and it was long past time to take a look a more structured look at how we'd managed the architecture over the years.

I'd already opened a branch in our Quino repository called feature/dependencyChanges and checked in some changes at the beginning of July. Those changes had come as a result of the first time I used NDepend to find a bunch of code that was in the wrong namespace or the wrong assembly, architecturally speaking.

I wasn't able to continue using this branch, though, for the following reasons.

  1. I got the hang of NDepend relatively quickly and got a bit carried away. Using ReSharper, I was able to make a lot of changes and fixes in a relatively short amount of time.
  2. I checked in all of these changes in one giant commit.
  3. I did this all five months ago.
  4. There have been hundreds of subsequent commits on the master branch, many of which also include global refactoring and cleanup.
  5. As a result of the above, merging master into feature/dependencyChanges is more trouble than it's worth.

Release Methodology

With each Quino change and release, we try our hardest to balance backward-compatibility with maintainability and effort. If it's easy enough to keep old functionality under an old name or interface, we do so.

We mark members and types obsolete so that users are given a warning in the compiler but can continue using the old code until they have time to upgrade. These obsolete members are removed in the next major or minor upgrade.

Developers who have not removed their references to obsolete members will at this point be greeted with compiler errors. In all cases, the user can find out from Quino's release notes how they should fix a warning or error.

The type of high-level changes that we have planned necessitate that we make a major version-upgrade, to Quino 2.0. In this version, we have decided not to maintain backward-compatibility in the code with Obsolete attributes. However, where we do make a breaking change -- either by moving code to new or different assemblies or by changing namespaces -- we want to maintain a usable change-log for customers who make the upgrade. The giant commit that I'd made previously was not a good start.

Take Two

Since some of these changes will be quite drastic departures in structure, we want to come up with a plan to make merging from the master branch to the feature/dependencyChanges branch safer, quicker and all-around easier.

I want to include many of the changes I started in the feature/dependencyChanges branch, but would like to re-apply those changes in the following manner:

  • Split the giant commit into several individual commits, each of which encapsulates exactly one change; smaller commits are much easier to merge
  • Document breaking changes in the release notes for Quino 2.0
  • Blog about/document the process of using NDepend to clean up Quino2

So, now that I'm ready to start cleaning up Quino for version 2.0, I'll re-apply the changes from the giant commit, but in smaller commits. At the same time, I'll use NDepend to find the architectural breaks that caused me to make those changes in the first place and document a bit of that process.

Setting up the NDepend Project

I created an NDepend project and attached it to my solution. Version 1.13 of Quino has 66 projects/assemblies, of which I chose the following "core" assemblies to analyze.

image

I can change this list at any time. There are a few ways to add assemblies. Unfortunately, the option to "Add Assemblies from VS Solution(s)" showed only 28 of the 66 projects in the Quino solution. I was unable to determine the logic that led to the other 38 projects not being shown. When I did select the projects I wanted from the list, the assemblies were loaded from unexpected directories. For example, it added a bunch of core assemblies (e.g. Encodo.Imaging) from the src/tools/Quino.CodeGenerator/bin/ folder rather than the src/libraries/Encodo.Imaging/bin folder. I ended up just taking the references I was offered by NDepend and added references to Encodo and Quino, which it had not offered to add.3

The NDepend Dashboard

Let's take a look at the initial NDepend Dashboard.

image

There's a lot of detail here. The initial impression of NDepend can be a bit overwhelming, I supposed, but you have to remember the sheer amount of interdependent data that it shows. As you can see on the dashboard, not only are there a ton of metrics, but those metrics are also tracked on a time-axis. I only have one measurement so far.

Any assemblies not included in the NDepend project are considered to be "third-party" assemblies, so you can see external dependencies differently than internal ones. There is also support for importing test-coverage data, but I haven't tried that yet.

There are a ton of measurements in there, some of which interest me and others that don't, or with which I disagree. For example, over 1400 warnings are in the Quino* assemblies because the base namespace -- Encodo.Quino -- doesn't correspond to a file-system folder -- it expects Encodo/Quino, but we use just Quino.

Another 200 warnings are to "Avoid public methods not publicly visible", which generally means that we've declared public methods on internal, protected or private classes. The blog post Internal or public? by Eric Lippert covered this adequately and came to the same conclusion that we have: you actually should make methods public if they are public within their scope.

There are some White Books about namespace and assembly dependencies that are worth reading if you're going to get serious about dependencies. There's a tip in there about turning off "Copy Local" on referenced assemblies to drastically increase compilation speed that we're going to look into.

Dependencies and cycles

One of the white books explains how to use namespaces for components and how to "levelize" an architecture. This means that the dependency graph is acyclic -- that there are no dependency cycles and that there are certainly no direct interdependencies. The initial graphs from the Encodo and Quino libraries show that we have our work cut out for us.

imageimageimage

The first matrix shows the high-level view of dependencies in the Encodo and Quino namespaces. Click the second and third to see some initial dependency issues within the Encodo and Quino assemblies.

That's as far as I've gotten so far. Tune in next time for a look at how we managed to fix some of these dependency issues and how we use NDepend to track improvement over time.



  1. I believe that takes care of full disclosure.

  2. This is something I'd neglected to do before. Documenting this process will help me set up a development process where we use NDepend more regularly -- more than every seven years -- and don't have to clean up so much code at once.

  3. After having read the recommendations in the NDepend White Book -- Partitioning code base through .NET assemblies and Visual Studio projects (PDF) -- it's clear why this happens: NDepend recommends using a single /bin folder for all projects in a solution.

v1.13.0: Schema migration, remoting, services and web apps

The summary below describes major new features, items of note and breaking changes. The full list of issues is also available for those with access to the Encodo issue tracker.

Highlights

Data & Schema

Remoting & services

  • Fixed several issues in the remoting driver (client and server parts). (QNO-4626, QNO-4630, QNO-4631, QNO-4388, QNO-4575, QNO-4629, QNO-4573, QNO-4625, QNO-4633, QNO-4575)
  • Added a runner for Windows services that allows debugging and shows logging output for applications that use the CoreServiceBase, which extends the standard .NET ServiceBase. The runner is available in the Encodo.Service assembly.

Web

  • Improved default and custom authentication in web applications and the remoting server. Also improved support for authorization for remote-method routes as well as MVC controllers.
  • Improved configuration, error-handling and stability of the HttpApplicationBase, especially in situations where the application fails to start. Error-page handling was also improved, including handling for Windows Event Log errors.
  • Improved appearance of the web-based schema migrator. (QNO-4559, QNO-4561, QNO-4563, QNO-4548, QNO-4487, QNO-4486, QNO-4488)

Winform

  • Data-provider statistics: improved the WinForm-based statistics form. (QNO-4231, QNO-4545, QNO-4546)
  • Standard forms: updated the standard WinForm about window and splash screen to use Encodo web-site CI. (QNO-4529)

System & Tools

  • Removed the dependency on the SmartWeakEvents library from Quino. (QNO-4645); the Quino and Encodo assemblies now no longer have any external dependencies.
  • Image handling: the Encodo and Quino libraries now use the Windows Imaging Components instead of System.Drawing. (QNO-4536)
  • Window 8.1: fixed culture-handling for en-US and de-CH that is broken in Windows 8.1. (QNO-4534, QNO-4553)
  • R# annotations have been added to the Encodo assembly. Tell R# to look in the Encodo.Core namespace to use annotations like NotNull and CanBeNull with parameters and results. (QNO-4508)
  • Generated code now includes a property that returns a ValueListObject for each enum property in the metadata. For example, for a property named State of type CoreState, the generated code includes the former properties for the enum and the foreign key backing it, but now also includes the ValueListObject property. This new property provides easy access to the captions.
public CoreState State { ... }
public ValueListObject StateObject { ... }
public int? CoreStateIdId { ... }
```Improved the **nant fix** command in the default build tools to fix the assembly name as well. The build tools are available in bin/tools/build. See the `src/demo/Demo.build` file for an example on how to use the Nant build scripts for your own solutions. To change the company name used by the "fix" command, for example, add the following task override:

``` * Fixed the implementation of `IntegrateRemotableMethods` to avoid a race condition with **remote methods**. Also improved the stability of the `DataProvider` statistics. ([QNO-4599](https://secure.encodo.ch/jira/browse/QNO-4599))

Breaking changes

  • The generic argument TRight has been removed from all classes and interfaces in the Encodo.Security.* namespace. In order to fix this code, just remove the int generic parameter wherever it was used. For example, where before you used the interface IUser<int>, you should now use IUser (QNO-4576).
  • The overridable method MetaAccessControl.DoGetAccessChecker() has been renamed to MetaAccessControl.GetAccessChecker().
  • Renamed the Encodo.ServiceLocator.SimpleInjector.dll to Encodo.Services.SimpleInjector.dll and Quino.ServiceLocator.SimpleInjector.dll to Quino.Services.SimpleInjector.dll Also changed the namespace Quino.ServiceLocator to Encodo.Quino.Services.
  • Renamed HttpApplicationBase.StartMetaApplication() to CreateAndStartUpApplication().
  • Classes may no longer contain properties with names that conflict with properties of IMetaReadable (e.g. Deleted, Persisted). The model will no longer validate until the properties have been renamed and the code regenerated. (QNO-4185)
  • Removed StandardIntRights with integer constants and replaced it with StandardRights with string constants.
  • The IAccessControl.Check() and other related methods now accept a sequence of string rights rather than integers.
  • IMetaConfiguration.ConfigureSession() has been deprecated. The method will still be called but may have undesired side-effects, depending on why it was overridden. The common use was to initialize a custom AccessControl for the session. Continuing to do so may overwrite the current user set by the default Winform startup. Instead, applications should use the IDataSessionAccessControlFactory and IDataSessionFactory to customize the data sessions and access controls returned for an application. In order to attach an access control, take care to only set your custom access control for sessions that correspond to your application model.[^1]
internal class JobVortexDataSessionAccessControlFactory : DataSessionAccessControlFactory
{
  public override IAccessControl CreateAccessControl(IDataSession session)
  {
    if (session.Application.Model.MetaId == JobVortexModelGenerator.ModelGuid)
    {
      return new JobVortexAccessControl(session);
    }

    return base.CreateAccessControl(session);
  }
}

The default length of the UserModule.User.PasswordHash property has been increased from 100 characters to 1000. This default is more sensible for implementations that use much longer validations tokens instead of passwords. To avoid the schema migration, revert the change by setting the property default length back to 0 in your application model, after importing the security module, as shown below.

var securityModule = Builder.Include<SecurityModuleGenerator>();      
securityModule.Elements.Classes.User.Properties[
  Encodo.Quino.Models.Security.Classes.SecurityUser.Fields.PasswordHash
].MaximumSize = 100;
````Application.Credentials` has been removed. To fix references, retrieve the `IUserCredentialsManager` from the service locator. For example, the following code returns the current user:

Session.Application.Configuration.ServiceLocator.GetInstance().Current


If your application uses the `WinformMetaConfigurationTools.IntegrateWinformPackages()` or `WinformDxMetaConfigurationTools.IntegrateWinformDxPackages()`, then the  `IDataSession.AccessControl.CurrentUser` will continue to be set correctly. If not, add the `SingleUserApplicationConfigurationPackage` to your application's configuration. The user in the remoting server will be set up correctly. Add the `WebApplicationConfigurationPackage` to web applications in order to ensure that the current user is set up correctly for each request. ([QNO-4596](https://secure.encodo.ch/jira/browse/QNO-4596))
  * `IDataSession.SyncRoot` has been removed as it was no longer needed or used in Quino itself. Sessions should *not* be used in multiple threads, so there is no need for a `SyncRoot`. Code that uses it should be reworked to use a separate session for each thread.
  * Moved `IMetaApplication.CreateSession()` to an extension method. Add `Encodo.Quino.App` to the using clauses to fix any compile errors.
  * Removed `IMetaApplication.DataProvider`; use `IMetaApplication.Configuration.DataProvider` instead. ([QNO-4604](https://secure.encodo.ch/jira/browse/QNO-4604))
  * The schema migration API has been completely overhauled. `ISchemaChange` and descendents has been completely removed. `ISchemaAction` is no longer part of the external API, although it is still used internally. The `ISchemaChangeFactory` has been renamed to `ISchemaCommandFactory` and, instead of creating change objects, which are then applied directly, returns `ISchemaCommand` objects, which can be either executed or transformed in some other way. `IMigrateToolkit.GetActionFor()` has also been replace with `CreateCommands()`, which mirrors the rest of the API by returning a sequence of commands to address a given `ISchemaDifference`. This release still has some commands that cannot be transformed to pure SQL, but the goal is to be able to generate pure SQL for a schema migration. ([QNO-993](https://secure.encodo.ch/jira/browse/QNO-993), [QNO-4579](https://secure.encodo.ch/jira/browse/QNO-4579), [QNO-4581](https://secure.encodo.ch/jira/browse/QNO-4581), [4588](https://secure.encodo.ch/jira/browse/QNO-4588), [4591](https://secure.encodo.ch/jira/browse/QNO-4591), [QNO-4594](https://secure.encodo.ch/jira/browse/QNO-4594))
  * `IMigrateSchemaAspect.Apply()` has been removed. All aspects will have to be updated to implement `GetCommands()` instead, or to use one of the available base classes, like `UpdateDataAspectBase` or `ConvertPropertyTypeSchemaAspect`. The following example shows how to use the `UpdateDataAspectBase` to customize migration for a renamed property.

internal class ArchivedMigrationAspect : UpdateDataAspectBase { public ArchivedMigrationAspect() : base("ArchivedMigrationAspect", DifferenceType.RenamedProperty, ChangePhase.Instead)

protected override void UpdateData(IMigrateContext context, ISchemaDifference difference) { using (var session = context.CreateSession(difference)) { session.ChangeAndSaveAll(UpdateArchivedFlag); } }

private void UpdateArchivedFlag(Project obj) }


The base aspects should cover most needs; if your functionality is completely customized, you can easily pass your previous implementation of `Apply()` to a `DelegateSchemaCommand` and return that from your implementation of `GetCommands()`. See the implementation of `UpdateDataAspectBase` for more examples. ([QNO-4580](https://secure.encodo.ch/jira/browse/QNO-4580))
  * `MetaObjectIdEqualityComparer<T>` can no longer be constructed directly. Instead, use `MetaObjectIdEqualityComparer<Project>.Default`.
  * Renamed `MetaClipboardControlDx.UpdateColorSkinaware()` to `MetaClipboardControlDx.UpdateSkinAwareColors()`.
  * `IMetaUnique.LogicalParent` has been moved to `IMetaBase`. Since `IMetaUnique` inherits from `IMetaBase`, it is unlikely that code is affected (unless reflection or some other direct means was used to reference the property). ([QNO-4586](https://secure.encodo.ch/jira/browse/QNO-4586))
  * `IUntypedMessage` has been removed; the `AssociatedObject` formerly found there has been moved to `IMessage`.
  * `ITypedMessage.AssociatedObject` has been renamed to `ITypedMessage.TypedAssociatedObject`. ([QNO-4647](https://secure.encodo.ch/jira/browse/QNO-4647))
  * Renamed `MetaObjectTools` to `MetaReadableTools`.
  * Redefined the protected methods `GenericObject.GetAsGuid()` and `GenericObject.GetAsGuidDefault` as extension methods in `MetaWritableTools`.
  * `IMetaFeedback.CreateGlobalContext()` has been removed. Instead the `IGlobalContext` is created using the service locator.

------------------------------------------------------------------------


[^1]: The schema migration creates a metadata model for your model -- meta-metadata -- and uses the Quino ORM to load data when importing a model from a database. If you aren't careful, as shown in the code example, then you'll attach your custom access control to the sessions created for the schema migration's data-access, which will more than likely fail when it tries to load user data from a table that does not exist in that model.