1 2 3 4 5 6 7 8 9 10 11
ABD: Refactoring and refining an API

We've been doing more internal training lately and one topic that we've started to tackle is design for architecture/APIs. Even if you're not officially a software architect -- designing and building entire systems from scratch -- every developer designs code, on some level.

[A]lways [B]e [D]esigning

There are broad guidelines about how to format and style code, about how many lines to put in a method, about how many parameters to use, and so on. We strive for Clean Code(tm).

But the most important part of code is to think about how you're writing it and what you're building. You shouldn't write a single line without thinking of the myriad ways in which it must fit into existing code and the established patterns and practices.

We've written about this before, in the two-part series called "Questions to consider when designing APIs" (Part I and Part II). Those two articles comprise a long list of aspects of a design to consider.

First make a good design, then compromise to fit project constraints.

Your project defines the constraints under which you can design. That is, we should still have our designer caps on, but the options available are much more strictly limited.

But, frustrating as that might be, it doesn't mean you should stop thinking. A good designer figures out what would be optimal, then adjusts the solution to fit the constraints. Otherwise, you'll forget what you were compromising from -- and your design skills either erode or never get better.

We've been calling this concept ABD -- Always Be Designing.1 Let's take a closer, concrete look, using a recent issue in the schema migration for Quino. Hopefully, this example illustrates how even the tiniest detail is important.2

A bug in the schema migrator

We detected the problem when the schema migration generated an invalid SQL statement.

ALTER TABLE "punchclock__timeentry" ALTER COLUMN "personid" SET DEFAULT ;

As you can see, the default value is missing. It seems that there are situations where the code that generates this SQL is unable to correctly determine that a default value could not be calculated.

The code that calculates the default value is below.

result = Builder.GetExpressionPayload(
  null,
  CommandFormatHints.DefaultValue,
  new ExpressionContext(prop),
  prop.DefaultValueGenerator
);

To translate, there is a Builder that produces a payload. We're using that builder to get the payload (SQL, in this case) that corresponds to the DefaultValueGenerator expression for a given property, prop.

This method is an extension method of the IDataCommandBuilder, reproduced below in full, with additional line-breaks for formatting:

public static string GetExpressionPayload<TCommand>(
  this IDataCommandBuilder<TCommand> builder,
  [CanBeNull] TCommand command,
  CommandFormatHints hints, 
  IExpressionContext context,
  params IExpression[] expressions)
{
  if (builder == null) { throw new ArgumentNullException("builder"); }
  if (context == null) { throw new ArgumentNullException("context"); }
  if (expressions == null) { throw new ArgumentNullException("expressions"); }

  return builder.GetExpressionPayload(
    command,
    hints,
    context,
    expressions.Select(
      e => new ExecutableQueryItem<IExecutableExpression>(new ExecutableExpression(e))
    )
  );
}

This method does no more than to package each item in the expressions parameter in an ExecutableQueryItem and call the interface method.

The problem isn't immediately obvious. It stems from the fact that each ExecutableQueryItem can be marked as Handled. The extension method ignores this feature, and always returns a result. The caller is unaware that the result may correspond to an only partially handled expression.

Is there a quick fix?

Our first instinct is, naturally, to try to figure out how we can fix the problem.3 In the code above, we could keep a reference to the executable items and then check if any of them were unhandled, like so:

var executableItems = expressions.Select(
  e => new ExecutableQueryItem<IExecutableExpression>(new ExecutableExpression(e))
);
var result = builder.GetExpressionPayload(command, hints, context, executableItems);

if (executableItems.Unhandled().Any())
{
  // Now what?
}

return result;
}

We can detect if at least one of the input expressions could not be mapped to SQL. But we don't know what to do with that information.

  • Do we throw an exception? No, we can't just do that. None of the callers are expecting an exception, so that's an API change.4
  • Do we return null? What can we return to indicate that the input expressions could not be mapped? Here we have the same problem as with throwing an exception: all callers assume that the result can be mapped.

So there's no quick fix. We have to change an API. We have to design.

Part of the result is missing

As with most bugs, the challenge lies not in knowing how to fix the bug, but in how to fix the underlying design problem that led to the bug. The problem is actually not in the extension method, but in the method signature of the interface method.

Instead of a single result, there are actually two results for this method call:

  • Can the given expressions be mapped to a string (the target representation)?
  • If so, what is that text?

Instead of a Get method, this is a classic TryGet method.

How to Introduce the Change

If this code is already in production, then you have to figure out how to introduce the bug fix without breaking existing code. If you already have consumers of your API, you can't just change the signature and cause a compile error when they upgrade. You have to decorate the existing method with [Obsolete] and make a new interface method.

So we don't change the existing method and instead add the method TryGetExpressionPayload() to IDataCommandBuilder.

What are the parameters?

Now, let's figure out what the parameters are going to be.

The method called by the extension method above has a slightly different signature.5

string GetExpressionPayload(
  [CanBeNull] TCommand command, 
  CommandFormatHints hints,
  [NotNull] IExpressionContext context,
  [NotNull] IEnumerable<ExecutableQueryItem<IExecutableExpression>> expressions
);

That last parameter is a bit of a bear. What does it even mean? The signature of the extension method deals with simple IExpression objects -- I know what those are. But what are ExecutableQueryItems and IExecutableExpressions?

As an author and maintainer of the data driver, I know that these objects are part of the internal representation of a query as it is processed. But as a caller of this method, I'm almost never going to have a list of these objects, am I?

Let's find out.

Me: Hey, ReSharper, how many callers of that method are there in the entire Quino source? ReSharper: Just one, Dave.6

So, we defined an API with a signature that's so hairy no-one calls it except through an extension method that makes the signature more palatable. And it introduces a bug. Lovely.

We've now figured out that our new method should accept a sequence of IExpression objects instead of ExecutableQueryItem objects.

How's the signature looking so far?

bool TryGetExpressionPayload(
  [CanBeNull] TCommand command, 
  CommandFormatHints hints,
  [NotNull] IExpressionContext context,
  [NotNull] IEnumerable<IExpression> expressions,
  out string payload
);

Are We Done?

Not quite. There are two things that are still wrong with this signature, both important.

Fix the Result Type

One problem is that the rest of the IDataCommandBuilder<TCommand> deals with a generic payload type and this method only works for builders where the target representation is a string. The Mongo driver, for example, uses MongoStorePayload and MongoRetrievePayload objects instead of strings and throws a NotSupportedException for this API.

That's not very elegant, but the Mongo driver was forced into that corner by the signature. Can we do better? The API would currently require Mongo to always return false because our Mongo driver doesn't know how to map anything to a string. But it could map to one of the aforementioned object representations.

If we change the out parameter type from a string to an object, then any driver, regardless of payload representation, has at least the possibility of implementing this API correctly.

Fix parameters

Another problem is that the order of parameters does not conform to the code style for Encodo.

  • We prefer to place all non-nullable parameters first. Otherwise, a call that passes null as the first parameter looks strange. The command can be null, so it should move after the two non-nullable parameters. If we move it all the way to the end, we can even make it optional.
  • Also, primitives should come after the references. (So hints should be third.)
  • Also, semantically, the call is getting the payload for the expressions not the context. The first parameter should be the target of the method; the rest of the parameters provide context for that input.
  • The original method accepted params IExpression[]. Using params allows a caller to provide zero or more expressions, but it's only allowed on the terminal parameter. Instead, we'll accept an IEnumerable<IExpression>, which is more standard for the Quino library anyway.

The final method signature is below.

bool TryGetExpressionPayload(
  [NotNull] IEnumerable<IExpression> expressions,
  [NotNull] IExpressionContext context,
  CommandFormatHints hints,
  out object payload,
  [CanBeNull] TCommand command = default(TCommand)
);

Our API in Action

The schema migration called the original API like this:

result = Builder.GetExpressionPayload(
  null,
  CommandFormatHints.DefaultValue,
  new ExpressionContext(prop),
  prop.DefaultValueGenerator
);

return true;

The call with the new API -- and with the bug fixed -- is shown below. The only non-functional addition is that we have to call ToSequence() on the first parameter (highlighted). Happily, though, we've fixed the bug and only include a default value in the field definition if one can actually be calculated.

object payload;
if (Builder.TryGetExpressionPayload(
  prop.DefaultValueGenerator.ToSequence(),
  new ExpressionContext(prop),
  CommandFormatHints.DefaultValue,
  out payload)
)
{
  result = payload as string ?? payload.ToString();

  return true;
}

One More Design Decision...

A good rule of thumb is that if you find yourself explaining something in detail, it might still be too complicated. In that light, the call to ToSequence() is a little distracting.7 It would be nice to be able to map a single expression without having to pack it into a sequence.

So we have one more design decision to make: where do we add that method call? Directly to the interface, right? But the method for a single expression can easily be expressed in terms of the method we already have (as we saw above). It would be a shame if every implementor of the interface was forced to produce this boilerplate.

Since we're using C#, we can instead extend the interface with a static method, as shown below (again, with more line breaks for this article):

public static bool TryGetExpressionPayload<TCommand>(
  [NotNull] this IDataCommandBuilder<TCommand> builder, // Extend the builder
  [NotNull] IExpression expression,
  [NotNull] IExpressionContext context,
  CommandFormatHints hints,
  out object payload,
  [CanBeNull] TCommand command = default(TCommand)
)
{
  return builder.TryGetExpressionPayload(
    expression.ToSequence(),
    context,
    hints,
    out payload,
    command
  );
}

We not only avoided cluttering the interface with another method, but now a caller with a single expression doesn't have to create a sequence for it8, as shown in the final version of the call below.

object payload;
if (Builder.TryGetExpressionPayload(
  prop.DefaultValueGenerator,
  new ExpressionContext(prop),
  CommandFormatHints.DefaultValue,
  out payload)
)
{
  result = payload as string ?? payload.ToString();

  return true;
}

Conclusion

We saw in this post how we always have our designer/architect cap on, even when only fixing bugs. We took a look at a quick-fix and then backed out and realized that we were designing a new solution. Then we covered, in nigh-excruciating detail, our thought process as we came up with a new solution.

Many thanks to Dani for the original design and Sebastian for the review!



  1. This is a bit of a riff on ABC -- Always Be Closing -- as popularized by Alec Baldwin in the movie Glengarry Glen Ross.

  2. Also, understand that it took much longer to write this blog post and itemize each individual step of how we thought about the issue. In reality, we took only a couple of minutes to work through this chain of reasoning and come up with the solution we wanted. It was only after we'd finished designing that I realized that this was a good example of ABD.

  3. Actually, our first instinct is to make sure that there is a failing test for this bug. But, this article deals with how to analyze problems and design fixes, not how to make sure that the code you write is tested. That's super-important, too, though, just so you know. Essential, even.

  4. Even though C# doesn't include the exceptions thrown in the signature of a method, as Java does. Where the Java version is fraught with issues, see the "Recoverable Errors: Type-Directed Exceptions" chapter of Midori: The Error Model by Joe Duffy for a really nice proposal/implementation of a language feature that includes expected exceptions in the signature of a method.

  5. Which is why we defined the extension method in the first place.

  6. I'm fully aware that my name isn't Dave. It's just what ReSharper calls me. Old-school reference.

  7. This was pointed out, by the way, by a reviewer of this blog post and escaped the notice of both designers and the code-reviewer. API design is neither easy nor is it done on the first try. It's only finished after multiple developers have tried it out. Then, you'll probably be able to live with it.

  8. Most developers would have used new [] { expression }, which I think is kind of ugly.

Beware the Hype: .NET Core

The article .NET Core, a call to action by Mark Rendle exhorts everyone to "go go go".

I say, "pump the brakes."

RC => Beta => Alpha

Mark says, "The next wave of work must be undertaken by the wider .NET community, both inside and outside Microsoft."

No. The next wave of work must be undertaken by the team building the product. This product is not even Beta yet. They have called the last two releases RC, but they aren't: the API is still changing quite dramatically. For example, the article Announcing .NET Core RC2 and .NET Core SDK Preview 11 lists all sorts of changes and the diff of APIs between RC1 and RC2 is gigantic -- the original article states that "[w]e added over a 1000 new APIs in .NET Core RC2".

What?!?!

That is a huge API-surface change between release candidates. That's why I think these designations are largely incorrect. Maybe they just mean, "hey, if y'all can actually work with this puny footprint, then we'll call it a final release. If not, we'll just add a bunch more stuff until y'all can compile again." Then, yeah, I guess each release is a "candidate".

But then they should just release 1.0 because this whole "RC" business is confusing. What they're really releasing are "alpha" builds. The quality is high, maybe even production-quality, but they're still massive changes vis-a-vis previous builds.

An Example: Project files

That doesn't sound like "RC" to me. As an example, look at the project-file format, project.json.

Mark also noted that there are "no project.json files in the repository" for the OData project that comes from Microsoft. That's not too surprising, considering the team behind .NET Core just backed off of the project.json format considerably, as concisely documented in The Future of project.json in ASP.NET Core by Shawn Wildermuth. The executive summary is that they've decided "to phase out project.json in deference to MSBuild". Anyone who's based any of their projects on the already-available-in-VS-2015 project templates that use that format will have to convert them to whatever the final format is.

Wildermuth also wrote that "Microsoft has decided after the RTM of the ASP.NET Core framework to phase out project.json and use MSBuild for build data. (Emphasis added.)" I was confused (again) but am pretty sure that he's wrong about RTM because, just a couple of days later, MS published an article Announcing ASP.NET Core RC2 -- and I'm pretty sure that RCs come before RTM.

Our Experience

At Encodo, we took a shot at porting the base assembly of Quino to .NET Core. It has only dependencies on framework-provided assemblies in the GAC, so that eliminated any issues with third-party support, but it does provide helper methods for AppDomains and Reflection, which made a port to .NET Core nontrivial.

Here's a few things we learned that made the port take much longer than we expected.

  • Multi-target project.json works with the command-line tools. Create the project file and compile with dotnet.
  • Multi-target project.json do not work in Visual Studio; you have to choose a single target. Otherwise, the same project that just built on the command line barely loads.
  • Also, Visual Studio ignores any #IFDEFs you use for platform-specific code. So, even if you've gotten everything compiling on the command-line, be prepared to do it all over again differently if you actually want it to work in VS2015.
  • If you do have to change code per-platform (e.g. for framework-only), then you have to put that code in its own assembly if you want to use Visual Studio.
  • If you go to all the trouble to change your API surface to accommodate .NET Core, then you might have done the work for nothing: many of the missing APIs that we had to work around in porting Encodo.Core are suddenly back in RC2. That means that if we'd waited, we'd have saved a lot of time and ended up in the same place.
  • There are several versions and RCs available, but only the beta channel was usable for us (e.g. the RC3 versions didn't work at all when we tried them).
  • In the end, we didn't have to make a lot of changes to get Encodo.Core compiling under .NET Core.
  • We learned a lot and know that we won't have too much trouble porting at least some assemblies, but the tools and libraries are still not working together in a helpful way -- and that ends up eating a lot of time and effort.

With so much in flux -- APIs and project format -- we're not ready to invest more time and money in helping MS figure out what the .NET Core target needs. We're going to sit it out until there's an actual RTM. Even at that point, if we make a move, we'll try a small slice of Quino again and see how long it takes. If it's still painful, then we'll wait until the first service pack (as is our usual policy with development tools and libraries).

Conclusion

I understand Mark's argument that "the nature of a package-based ecosystem such as NuGet can mean that Project Z can't be updated until Project Y releases .NET Core packages, and Project Y may be waiting on Project X, and so on". But I just don't, as he says, "trust that what we have now in RC2 is going to remain stable in API terms", so I wouldn't recommend "that OSS project maintainers" do so, either. It's just not ready yet.

If you jump on the .NET Core train now, be prepared to shovel coal. Oh, and you might just have to walk to the next station, too. At noon. Carrying your overseas trunk on your back. Once you get there, though, you might be just in time for the 1.0.1 or 1.0.2 express arriving at the station, where you can get on, you might not even have to buy a new ticket -- and you can get there at the same time as everyone else.



  1. The Mark Renton article states boldly that "Yesterday we finally got our hands on the first Release Candidate of .NET Core [...]" but I don't know what he's talking about. The project just released RC2 and there are even RC3 packages available in the channel already -- but these are totally useless and didn't work at all in our projects.

Quino Retrospective and Roadmap

History

Before taking a look at the roadmap, let's quickly recap how far we've come. An overview of the release schedule shows a steady accretion of features over the years, as driven by customer or project needs.

image

The list below includes more detail on the releases highlighted in the graphic.1

  • 0.1: Proof of concept with metadata, PostgreSql (data and schema-migration) and Winforms UI
  • 1.0: First customer product with PostgreSql, DevExpress Winforms UI and Reporting
  • 1.0.5: MS-SQL driver (parity with PostgreSql driver)
  • 1.5.0: Remoting data driver; require .NET 4.0
  • 1.6.0: Mongo/NoSQL data driver
  • 1.8.0: Rewrite data driver to use sessions
  • 1.8.5: Support improved metadata-generation pattern
  • 1.9.0: Add plugin/overlay support
  • 1.10.0: Require .NET 4.5; add JSON-based remoting protocol; Windows-service support
  • 1.13.0: Rewrite security API
  • v2.0-beta1: Rewrite configuration, logging and schema-migration APIs
  • v2.0-beta2: Add V2 generated-code format
  • 2.0: Finish configuration/IOC rewrite; produce NuGet packages for delivery
  • 2.2: Stabilize Winform; support aliased tables in queries
  • 3.0: Rewrite MetaBuilder API; improve support for plugins

We took 1.5 years to get to v1. The initial major version was to signify the first time that Quino-based code went into external production.2

After that, it took 6.5 years to get to v2. Although we added several large products that use Quino, we were always able to extend rather than significantly change anything in the core. The second major version was to signify sweeping changes made to address technical debt, to modernize certain components and to prepare for changes coming to the .NET platform.

It took just 5 months to get to v3 for two reasons:

  1. Although we were able to make a lot of planned changes in v23, we had to leave some breaking changes for future versions.4
  2. We now strictly adhere to the rule that a breaking change anywhere in the software's API -- and Quino's API surface is large -- leads automatically to a major-version change.5

Roadmap

So that's where we've been. Where are we headed?

As you can see above, Quino is a very mature product that satisfies the needs of a wide array of software on all tiers. What more is there to add?

Quino's design has always been driven by a combination of customer requirements and what we anticipated would be customer requirements.

We're currently working on the following features.

Modeling improvements

This work builds on the API changes made to the MetaBuilder in v3. We're creating a more fluent, modern and extensible API for building metadata. We hope to be able to add these changes incrementally without introducing any breaking changes.6

WPF / VSG

A natural use of the rich metadata in Quino is to generate user interfaces for business entities without have to hand-tool each form. From the POC onward, Quino has included support for generating UIs for .NET Winforms.

Winforms has been replaced on the Windows desktop with WPF and UWP. We've gotten quite far with being able to generate WPF applications from Quino metadata. The screenshots below come from a pre-alpha version of the Sandbox application included in the Quino solution.

imageimageimageimage

You may have noticed the lovely style of the new UI.7 We're using a VSG designed for us by Ergosign, for whom we've done some implementation work in the past.

.NET Core

If you've been following Microsoft's announcements, things are moving quickly in the .NET world. There are whole new platforms available, if you target your software to run on them. We're investigating the next target platforms for Quino. Currently that means getting the core of Quino -- Quino.Meta and its dependencies -- to compile under .NET Core.

imageAs you can see in the screenshot, we've got one of the toughest assemblies to compile -- Encodo.Core. After that, we'll try for running some tests under Linux or OS X. The long-term goal is to be able to run Quino-based application and web servers on non-Windows -- and, most importantly, non-IIS -- platforms.8

These changes will almost certainly cause builds using previous versions to break. Look for any additional platform support in an upcoming major-version release.



  1. There were, of course, more minor and patch releases throughout, but those didn't introduce any major new functionality.

  2. Punchclock, our time-entry and invoicing software -- and Quino "dogfood (When a developer uses their own code for their own daily needs. Being a user as well as a developer creates the user empathy that is the hallmark of good software.)" product -- had been in use internally at Encodo earlier than that.

  3. E.g. splitting the monolithic Encodo and Quino assemblies into dozens of new, smaller and much more focused assemblies. Reorganizing configuration around the IOC and rewriting application startup for more than just desktop applications was another sweeping change.

  4. One of those breaking changes was to the MetaBuilder, which started off as a helper class for assembling application metadata, but became a monolithic and unavoidable dependency, even in v2. In v3, we made the breaking changes to remove this component from its central role and will continue to replace its functionality with components that more targeted, flexible and customizable.

  5. In the years between v1 and v2, we used the minor-version number to indicate when breaking changes could be made. We also didn't try as hard to avoid breaking changes by gracefully deprecating code. The new approach tries very hard to avoid breaking changes but accepts the consequences when it's deemed necessary by the team.

  6. That is, when users upgrade to a version with the newer APIs, they will get obsolete warnings but their existing code will continue to build and run, as before the upgrade. In this way, customers can smoothly upgrade without breaking any builds.

  7. You may also have noticed that the "Sandbox Dialog View" includes a little tag in it for the "XAML Spy", a tool that we use for WPF development. Just so you know the screenshots aren't faked... :-)

  8. As with the WPF interface, we're likely to dogfood all of these technologies with Punchclock, our time-tracking and invoicing system written with Quino. The application server and web components that run on Windows could be migrated to run on one of our many Linux machines instead.

v3.0: Metadata builders and code-generation improvements

The summary below describes major new features, items of note and breaking changes. The full list of issues is also available for those with access to the Encodo issue tracker.

Highlights

Breaking changes

  • IDataSession and IApplication now directly implement the IServiceRequestHandler and helper methods that used to extend IApplication now extend this interface instead, so calls like GetModel() can now be executed against an IApplication or an IDataSession. Many methods have been moved out of the IServiceRequestHandler interface to extension methods declared in the Encodo.IOC namespace. This move will require applications to update the usings. ReSharper will automatically find the correct namespace and apply it for you.
  • Similarly, the extension method ApplicationExtensions.GetInstance() has been replaced with a direct implementation of the IServiceRequestHandler by IApplication.
  • MetaBuilder.Include() has been replaced with Dependencies.Include()
  • When you call the new version of CreateModel(), you can no longer call CreateMainModule() because the main module is set up automatically. Although the call is marked as obsolete, it can only be combined with the older overload of the CreateModel(). Using it with the newer overload will cause a runtime error as the main module is added to the model twice.
  • The various methods to create paths with the MetaBuilder have been replaced by AddPath(). To rewrite a path, use the following style:
Builder.AddPath(
  Elements.Classes.A.FromOne("Id"), 
  Elements.Classes.B.ToMany("FileId"), 
  path => path.SetMetaId(new Guid("...")).SetDeleteRule(MetaPathRule.Cascade),
  idx => idx.SetMetaId(new Guid("..."))
);
C# Handbook Rewrite Coming Soon

Encodo published its first C# Handbook and published it to its web site in 2008. At the time, we also published to several other standard places and got some good, positive feedback. Over the next year, I made some more changes and published new versions. The latest version is 1.5.2 and is available from Encodo's web site. Since then, though I've made a few extra notes and corrected a few errors, but never published an official version again.

This is not because Encodo hasn't improved or modernized its coding guidelines, but because of several issues, listed below.

  • At 72 pages, it's really quite long
  • A more compact, look-up reference would be nice
  • It contains a mix of C#-specific, Encodo-specific and library-specific advice
  • It's maintained in Microsoft Word
  • Code samples are manually formatted
  • New versions are simply new copies in versioned folders (no source control)
  • Collaboration is nearly impossible
  • There is nothing about any .NET version newer than 3.5
  • There is no mention of any other programming language (e.g. TypeScript, JavaScript)
  • A lot of stuff is overly complicated (e.g. var advice) or just plain wrong (e.g. var advice)

To address these issues and to accommodate the new requirements, here's what we're going to do:

  • Convert the entire document from Word to Markdown and put it in a Git repository

    • Collaboration? Pull requests. Branches.
    • Versioning? Standard diffing of commits.
    • Code samples? Automatic highlighting from GitLab (Encodo's internal server) or GitHub (external repository).
  • Separate the chapters into individual files and keep them shorter and more focused on a single topic

  • Separate all of the advice and rules into the following piles:

    • General programming advice and best practices
    • C#-specific
    • Encodo-specific
    • Library-specific (e.g. Quino)

These are the requirements and goals for a new version of the C# handbook.

The immediate next steps are:

  1. Convert current version from Microsoft Word to Markdown (done)
  2. Add everything to a Git repository (done)
  3. Overhaul the manual to remove incorrect and outdated material; address issues above (in progress)
  4. Mirror externally (GitHub or GitLab or both)

I hope to have an initial, modern version ready within the next month or so.

Networking Event 2016.1

imageOn Wednesday, Encodo had its first networking event of the year. Our very own Sebastian Greulach presented Code Review Best Practices. A bunch of our friends and colleagues from the area showed up for a lively discussion that, together with the presentation, lasted over 90 minutes.

We heard from people working with remote teams -- off- and near-shored -- as well as people working locally in both small and large teams and for small to large companies. We discussed various review styles, from formal to informal to nonexistent as well as the differences in managing and reviewing code for projects versus products. Naturally, we also covered tool support and where automation makes sense and where face-to-face human interaction is still better.

The discussion continued over a nice meal prepared on our outdoor grill. We even had a lot more vegetables this time! Thanks to lovely weather, we were able to spend some time outside and Pascal demonstrated his L337 drone-flying skills -- but even he couldn't save it from a rain gutter when a propeller came off mid-flight.

Thanks to everyone who helped make it happen and thanks to everyone who showed up!

API Design: The Road Not Taken

Unwritten code requires no maintenance and introduces no cognitive load.

As I was working on another part of Quino the other day, I noticed that the oft-discussed registration and configuration methods1 were a bit clunkier than I'd have liked. To whit, the methods that I tended to use together for configuration had different return types and didn't allow me to freely mix calls fluently.

The difference between Register and Use

The return type for Register methods is IServiceRegistrationHandler and the return type for Use methods is IApplication (a descendant), The Register* methods come from the IOC interfaces, while the application builds on top of this infrastructure with higher-level Use* configuration methods.

This forces developers to write code in the following way to create and configure an application.

public IApplication CreateApplication()
{
  var result =
    new Application()
    .UseStandard()
    .UseOtherComponent();

  result.
    .RegisterSingle<ICodeHandler, CustomCodeHandler>()
    .Register<ICodePacket, FSharpCodePacket>();

  return result;
}

That doesn't look too bad, though, does it? It doesn't seem like it would cramp anyone's style too much, right? Aren't we being a bit nitpicky here?

That's exactly why Quino 2.0 was released with this API. However, here we are, months later, and I've written a lot more configuration code and it's really starting to chafe that I have to declare a local variable and sort my method invocations.

So I think it's worth addressing. Anything that disturbs me as the writer of the framework -- that gets in my way or makes me write more code than I'd like -- is going to disturb the users of the framework as well.

Whether they're aware of it or not.

Developers are the Users of a Framework

In the best of worlds, users will complain about your crappy API and make you change it. In the world we're in, though, they will cheerfully and unquestioningly copy/paste the hell out of whatever examples of usage they find and cement your crappy API into their products forever.

Do not underestimate how quickly calls to your inconvenient API will proliferate. In my experience, programmers really tend to just add a workaround for whatever annoys them instead of asking you to fix the problem at its root. This is a shame. I'd rather they just complained vociferously that the API is crap rather than using it and making me support it side-by-side with a better version for usually feels like an eternity.

Maybe it's because I very often have control over framework code that I will just not deal with bad patterns or repetitive code. Also I've become very accustomed to having a wall of tests at my beck and call when I bound off on another initially risky but in-the-end rewarding refactoring.

If you're not used to this level of control, then you just deal with awkward APIs or you build a workaround as a band-aid for the symptom rather than going after the root cause.

Better Sooner than Later

So while the code above doesn't trigger warning bells for most, once I'd written it a dozen times, my fingers were already itching to add [Obsolete] on something.

I am well-aware that this is not a simple or cost-free endeavor. However, I happen to know that there aren't that many users of this API yet, so the damage can be controlled.

If I wait, then replacing this API with something better later will take a bunch of versions, obsolete warnings, documentation and re-training until the old API is finally eradicated. It's much better to use your own APIs -- if you can -- before releasing them into the wild.

Another more subtle reason why the API above poses a problem is that it's more difficult to discover, to learn. The difference in return types will feel arbitrary to product developers. Code-completion is less helpful than it could be.

It would be much nicer if we could offer an API that helped users discover it at their own pace instead of making them step back and learn new concepts. Ideally, developers of Quino-based applications shouldn't have to know the subtle difference between the IOC and the application.

A Better Way

Something like the example below would be nice.

return
  new Application()
  .UseStandard()
  .RegisterSingle<ICodeHandler, CustomCodeHandler>()
  .UseOtherComponent()
  .Register<ICodePacket, FSharpCodePacket>();

Right? Not a gigantic change, but if you can imagine how a user would write that code, it's probably a lot easier and more fluid than writing the first example. In the second example, they would just keep asking code-completion for the next configuration method and it would just be there.

Attempt #1: Use a Self-referencing Generic Parameter

In order to do this, I'd already created an issue in our tracker to parameterize the IServiceRegistrationHandler type in order to be able to pass back the proper return type from registration methods.

I'll show below what I mean, but I took a crack at it recently because I'd just watched the very interesting video Fun with Generics by Benjamin Hodgson, which starts off with a technique identical to the one I'd planned to use -- and that I'd already used successfully for the IQueryCondition interface.2

Let's redefine the IServiceRegistrationHandler interface as shown below,

public interface IServiceRegistrationHandler<TSelf>
{
  TSelf Register<TService, TImplementation>()
      where TService : class
      where TImplementation : class, TService;

  // ...
}

Can you see how we pass the type we'd like to return as a generic type parameter? Then the descendants would be defined as,

public interface IApplication : IServiceRegistrationHandler<IApplication>
{
}

In the video, Hodgson notes that the technique has a name in formal notation, "F-bounded quantification" but that a snappier name comes from the C++ world, "curiously recurring template pattern". I've often called it a self-referencing generic parameter, which seems to be a popular search term as well.

This is only the first step, though. The remaining work is to update all usages of the formerly non-parameterized interface IServiceRegistrationHandler. This means that a lot of extension methods like the one below

public static IServiceRegistrationHandler RegisterCoreServices(
  [NotNull] this IServiceRegistrationHandler handler)
{
}

will now look like this:

public static TSelf RegisterCoreServices<TSelf>(
[NotNull] this IServiceRegistrationHandler<TSelf> handler)
  where TSelf : IServiceRegistrationHandler<TSelf>
{
}

This makes defining such methods more complex (again).3 in my attempt at implementing this, Visual Studio indicated 170 errors remaining after I'd already updated a couple of extension methods.

Attempt #2: Simple Extension Methods

Instead of continuing down this path, we might just want to follow the pattern we established in a few other places, by defining both a Register method, which uses the IServiceRegistrationHandler, and a Use method, which uses the IApplication

Here's an example of the corresponding "Use" method:

public static IApplication UseCoreServices(
  [NotNull] this IApplication application)
{
  if (application == null) { throw new ArgumentNullException("application"); }

  application
    .RegisterCoreServices()
    .RegisterSingle(application.GetServices())
    .RegisterSingle(application);

  return application;
}

Though the technique involves a bit more boilerplate, it's easy to write and understand (and reason about) these methods. As mentioned in the initial sentence of this article, the cognitive load is lower than the technique with generic parameters.

The only place where it would be nice to have an IApplication return type is from the Register* methods defined on the IServiceRegistrationHandler itself.

We already decided that self-referential generic constraints would be too messy. Instead, we could define some extension methods that return the correct type. We can't name the method the same as the one that already exists on the interface4, though, so let's prepend the word Use, as shown below:

IApplication UseRegister<TService, TImplementation>(
  [NotNull] this IApplication application)
      where TService : class
      where TImplementation : class, TService;
{
  if (application == null) { throw new ArgumentNullException("application"); }

  application.Register<TService, TImplementation>();

  return application;
}

That's actually pretty consistent with the other configuration methods. Let's take it for a spin and see how it feels. Now that we have an alternative way of registering types fluently without "downgrading" the result type from IApplication to IServiceRegistrationHandler, we can rewrite the example from above as:

return
  new Application()
  .UseStandard()
  .UseRegisterSingle<ICodeHandler, CustomCodeHandler>()
  .UseOtherComponent()
  .UseRegister<ICodePacket, FSharpCodePacket>();

Instead of increasing cognitive load by trying to push the C# type system to places it's not ready to go (yet), we use tiny methods to tweak the API and make it easier for users of our framework to write code correctly.5


Perhaps an example is in order:

interface IA 
{
  IA RegisterSingle<TService, TConcrete>();
}

interface IB : IA { }

static class BExtensions
{
  static IB RegisterSingle<TService, TConcrete>(this IB b) { return b; }

  static IB UseStuff(this IB b) { return b; }
}

Let's try to call the method from BExtensions:

public void Configure(IB b)
{
  b.RegisterSingle<IFoo, Foo>().UseStuff();
}

The call to UseStuff cannot be resolved because the return type of the matched RegisterSingle method is the IA of the interface method not the IB of the extension method. There is a solution, but you're not going to like it (I know I don't).

public void Configure(IB b)
{
  BExtensions.RegisterSingle<IFoo, Foo>(b).UseStuff();
}

You have to specify the extension-method class's name explicitly, which engenders awkward fluent chaining -- you'll have to nest these calls if you have more than one -- but the desired method-resolution was obtained.

But at what cost? The horror...the horror.


  1. See Encodos configuration library for Quino Part 1, Part 2 and Part 3 as well as API Design: Running and Application Part 1 and Part 2 and, finally, Starting up an application, in detail.

  2. The video goes into quite a bit of depth on using generics to extend the type system in the direction of dependent types. Spoiler alert: he doesn't make it because the C# type system can't be abused in this way, but the journey is informative.

  3. As detailed in the links in the first footnote, I'd just gotten rid of this kind of generic constraint in the configuration calls because it was so ugly and offered little benefit.

  4. If you define an extension method for a descendant type that has the same name as a method of an ancestor interface, the method-resolution algorithm for C# will never use it. Why? Because the directly defined method matches the name and all the types and is a "stronger" match than an extension method.

  5. The final example does not run against Quino 2.2, but will work in an upcoming version of Quino, probably 2.3 or 2.4.

v2.2: Winform fixes and Query Improvements

The summary below describes major new features, items of note and breaking changes. The full list of issues is also available for those with access to the Encodo issue tracker.

Highlights

  • Lots of bug fixes and improvements for the Winform UI and German translations with the release of Punchclock on this version. (QNO-5162, QNO-5159, QNO-5158, QNO-5157, QNO-5156, QNO-5140, QNO-5155, QNO-5145, QNO-5111, QNO-5107, QNO-5106, QNO-5104, QNO-5015)
  • DateTimeExtensions.GetDayOfWeek() had a leap-day bug (QNO-5051)
  • Fixed how the hash code for GenericObjects is calculated, which fixes sorting issues in grids, specifically for non-persisted or transient objects (QNO-5137)
  • Improvements to the IAccessControl API for getting groups and users and testing membership (QNO-5133)
  • Add support for query aliases (e.g. for joining the same table multiple times) (QNO-531) This changes the API surface only minimally. Applications can pass an alias when calling the Join method, as shown below,
query.Join(Metadata.Project.Deputy, alias: "deputy")

You can find more examples of aliased queries in the TestAliasedQuery(), TestJoinAliasedTables(), TestJoinChildTwice() defined in the QueryTests testing fixture.

  • Add a standalone IQueryAnalyzer for optimizations and in-memory mini-drivers (QNO-4830)

Breaking changes

  • ISchemaManager has been removed. Instead, you should retrieve the interface you were looking for from the IOC. The possible interfaces you might need are IImportHandler, IMappingBuilder, IPlanBuilder or ISchemaCommandFactory.

  • ISchemaManagerSettings.GetAuthorized() has been moved to ISchemaManagerAuthorizer.

  • The hash-code fix for GenericObjects may have an effect on the way your application sorts objects.The IParticipantManager (base interface of IAccessControl) no longer has a single method called GetGroups(IParticipant). This method was previously used to get the groups to which a user belongs and the child groups of a given group. This confusing double duty for the API led to an incorrect implementation for both usages. Instead, there are now two methods:

    • IEnumerable<IGroup> GetGroups(IUser user): Gets the groups for the given user
    • IEnumerable<IGroup> GetChildGroups(IGroup group): Gets the child groups for the given group

The old method has been removed from the interface because (A) it never worked correctly anyway and (B) it conflicts with the new API.

Voxxed Zürich 2016: Notes

This first-ever Voxxed Zürich was hosted at the cinema in the SihlCity shopping center in Zürich on March 3rd. All presentations were in English. The conference was relatively small -- 333 participants -- and largely vendor-free. The overal technical level of the presentations and participants was quite high. I had a really nice time and enjoyed a lot of the presentations.

There was a nice common thread running through all of the presentations, starting with the Keynote. There's a focus on performance and reliability through immutabiliy, sequences, events, actors, delayed execution (lambdas, which are relatively new to Java), instances in the cloud, etc. It sounds very BUZZWORDY, but instead it came as a very technically polished conference that reminded me of how many good developers there are trying to do the right thing. Looking forward to next year; hopefully Encodo can submit a presentation.

You can take a look at the VoxxedDays Zürich -- Schedule. The talks that I visited are included below, with links to the presentation page, the video on YouTube and my notes and impressions. YMMV.

Keynote: Life beyond the Illusion of the Present

Life beyond the Illusion of the Present -- Jonas Bonér

media

Notes

  • He strongly recommended reading The Network is reliable by Peter Bailis.
  • This talk is about event-driven, CQRS programming.
  • Focus on immutable state, very much like JoeDuffy, etc. transactional accrual of facts.
  • Never delete data, annotate with more facts.
  • The reality at any point can be calculated for a point in time by aggregating facts up to that point. Like the talk I once wrote up some notes about (Runaway Complexity in Big Data, and a Plan to Stop It by Nathan Marz).
  • Everything else is a performance optimization. Database views, tables are all caches on the transaction log. Stop throwing the log away, though.
  • Define smaller atomic units. Not a whole database. Smaller. Consistency boundary. Services?
  • Availability trumps consistency. Use causal consistency through mechanisms other than time stamps. Local partial better than global.
  • He talked about data-flow programming; fingers crossed that we get some language support in C# 7
  • Akka (Akka.NET) is the main product.

Kotlin - Ready for production

Kotlin - Ready for production -- Hadi Hariri

media

  • Used at JetBrains, open-source. 14k+ users. It's not a ground-breaking language. They tried Scala and Scala was the first language they tried to use (Java already being off the table) but they didn't like it, so they invented Kotlin.

  • Interoperable with Java (of course). Usable from all sorts of systems, but intelliJ Idea has first-class support.

  • Much less code, less maintenance. Encapsulates some concepts like "data classes" which do what they're supposed for DTO definitions.

    • Inferred type on declarations. No nulls. Null-safe by design. Opt-in for nulls.
    • Implicit casts as well
    • Interface delegation
    • Lazy delegation
    • Deconstruction
    • Global infix operators; very expressive
    • Also defaults to/focuses on immutability
    • Algebraic data types/ data flow
    • Anglo is statically typed XML views for android
  • JavaScript target exists and is the focus of work. Replacement for TypeScript?

Reactive Apps with Akka and AngularJS

Reactive Apps with Akka and AngularJS -- Heiko Seeberger

media

  • He strongly recommended reading the reactive manifesto
  • Responsive: timely response / non-functional / also under load / scale up/down/out
  • Resilient: fail early
  • Message-driven: async message-passing is a way of getting reactive/responsive. Automatic decoupling leads to better error-handling, no data loss
  • Akka provides support for:
    • Actor-based model (actors are services); watch video from Channel Nine
    • Akka HTTP Server is relatively new
    • Akka is written in Scala
    • There's a Scala DSL for defining the controller (define routes)
    • The Scala compiler is pure crap. Sooooo slooooowww (62 seconds for 12 files)

During his talk, he took us through the following stages of building a scalable, resilient actor-based application with Akka.

  • First he started with static HTML
  • Then he moved on to something connected to AKKA, but not refreshing
  • W3C Server-sent events is unidirectional channel from the server to the client. He next used this to have instant refresh on the client; not available on IE. Probably used by SignalR (or whatever replaced it)? Nothing is typed, though, just plain old JavaScript
  • Then he set up sharding
  • Then persistence (Cassandra, Kafka)

AKKA Distributed Data

  • Deals with keeping replicas consistent without central coordination
  • Conflict-free replicated data types
  • Fully distributed, has pub/sub semantics
  • Uses the Gossip protocol
  • Support various consistency strategies
  • Using AKKA gives you automated scaling support (unlike the SignalR demo Urs and I did over 2 years ago, but that was a chat app as well)

AKKA Cluster Sharding

  • Partitioning of actors/services across clusters
  • Supports various strategies
  • Default strategy is to distribute unbalanced actors to new shards
  • The ShardRegion is another actor that manages communication with sharded actors (entities). This introduces a new level of indirection, which must be honored in the code (?)

AKKA Persistence

  • Event-sourcing: validate commands, journal events, apply the event after persistence.
  • Application is applied to local state only after the journal/persistence has indicated that the command was journaled
  • On recovery, events are replayed
  • Supports snapshotting (caching points in time)
  • Requires a change to the actor/entity to use it. All written in Scala.

Akka looks pretty good. It guarantees the ordering because ACTORS. Any given actor only exists on any shard once. If a shard goes down, the actor is recreated on a different shard, and filled with information from the persistent store to "recreate" the state of that actor.

DDD (Domain-Driven Design) and the actor model. Watch Hewitt, Meijer and Szyperski: The Actor Model (everything you wanted to know, but were afraid to ask).

Code is on GitHub: seeberger/reactive_flows

Lambda core - hardcore

Lambda core - hardcore -- Jarek Ratajski

media

Focus on immutability and no side-effects. Enforced by the lambda calculus. Pretty low-level talk about lambda calculus. Interesting, but not applicable. He admitted as much at the top of the talk.

Links:

expect("poo").length.toBe(1)

expect("poo").length.toBe(1) -- Philip Hofstetter1

media

This was a talk about expectations of the length of a character. The presenter was very passionate about his talk and went into an incredible amount of detail.

  • What is a string? This is the kind of stuff every programmer needs to know.2
  • String is not a collection of bytes. It's a sequence of graphemes. string <> char[]
  • UTF-16 is crap. What about the in-memory representation? Why in God's name did Python 3 use UTF32? Unicode Transformation format.
  • What is the length of a string? ä is how many? Single character (diuresis included) or a with combining diuresis?
  • In-memory representation in Java and C# are UCS-2 (UNICODE 1); stuck in 1996, before Unicode 2.0 came out. This leaks into APIs because of how strings are returned ... string APIs use UTF-16, encoding with surrogate pairs to get to characters outside of the BMP (understood by convention, but not by the APIs that expect UTF-16 ... which has no idea what surrogate pairs are ... and counting algorithms, find, etc. won't work).
  • ECMAScript hasn't really fixed this, either. substr() can break strings charAt() is still available and has no idea about code points. Does this apply to ES6? String-equality doesn't work for the diuresis above.
  • So we're stuck with server-side. Who does it right? Perl. Swift. Python. Ruby. Python went through hell with backwards compatibility but with 3.3 they're doing OK again. Ruby strings are a tuple of encoding and data. All of the others have their string libraries dealing in graphemes. How did Perl always get it right? Perl has three methods for asking questions about length, in graphemes, code points or bytes
  • What about those of us using JavaScript? C#? Java? There are external libraries that we should be using. Not just for DateTime, but for string-handling as well. Even EcmaScript15 still uses code points rather than graphemes, so the count varies depending on how the grapheme is constructed.
  • Security concerns: certificate authorities have to be aware of homographs (e.g. a character that looks like another one, but has a different encoding/byte sequence).
  • He recommended the book Unicode explained by Jukka K. Korpela.

How usability fits in UX - it's no PICNIC

How usability fits in UX - it's no PICNIC -- Myriam Jessier

media

What should a UI be?

  1. Functional
  2. Reliable
  3. Usable
  4. Convenient
  5. Pleasurable

Also nice to have:

  1. Desirable
  2. Delightful
  3. memorable
  4. Learnable
  5. 3 more

Book recommendation: Don't make me think by Steve Krug

  • Avoid mindless and unambiguous clicks. Don't count clicks, count useless shit you need to do.
  • Let the words go. People's attention will wander.
  • UX is going to be somewhat subjective. Don't try to please everyone.
  • OMG She uses hyphens correctly.
  • She discussed the difference between UX, CX, UI.
  • Personas are placeholders for your users. See Personapp to get started working with personas.

Guidelines:

  • Consistent and standardized UI
  • Guide the user (use visual cues, nudging)
  • Make the CallToAction (CTA) interactive objects obvious
  • Give feedback on progress, interaction
  • Never make a user repeat something they already told you. You're software, you should have eidetic memory
  • Always have default values in forms (e.g. show the expected format)
  • Explain how the inputed information will be used (e.g. for marketing purposes)
  • No more "reset" button or mass-delete buttons. Don't make it possible/easy to wipe out all someone's data
  • Have clear and explanatory error or success messages (be encouraging)
  • Include a clear and visual hierarchy and navigation

Guidelines for mobile:

  • Make sure it works on all phones

  • Give incentives for sharing and purpose (engagement rates make marketing happy. CLICK THE BUTTON)

  • Keep usability and conversion in mind (not necessarily money, but you actually want people to be using your app correctly)

  • Usability (can you use your app on the lowest screen-brightness?)

  • ...and more...

  • Make it pretty (some people don't care, e.g. She very clearly said that she's not aesthetically driven, it's not her field; other people do care. A lot).

  • Give all the information a customer needs to purchase

  • Design for quick movement (no lag)

  • Do usability testing through video

  • Leverage expectations. Fit in to the environment. Search is on the left? Behind a button? Do that. Don't make a new way of searching.

  • If you offer a choice, then make them as mutually exclusive as possible. When a company talks to itself (e.g. industry jargon), then users get confused

  • The registration process should be commensurate to the thing that you're registering for

  • Small clickable ads on mobile. Make click targets appropriate.

  • Don't blame negative feedback on "fear of change". It's probably you. If people don't like it, then it might not be user-friendly. The example with Twitter's star vs. heart. It's interesting how we let the world frame our interactions. Why not both? Too complex? Would people really be confused by two buttons? One to "like" and one for "read later"?

Suggested usability testing tools:

  • Crazy Egg is $9 per month for heatmaps.
  • Qualaroo
  • Optimizely (A/B testing)
  • Usabilia
  • Userfeel
  • Trymyui

React - A trip to Russia isn't all it seems

React - A trip to Russia isn't all it seems -- Josh Sephton[^3]

media

This talk was about Web UI frameworks and how his team settled on React.

  • Angular too "all or nothing".
  • Backbone has no data-binding.
  • React looks good. Has its own routing for SPAs. Very component-heavy. Everything's a component. Nothing new here so far.
  • They built their React to replace a Wordpress-based administration form
  • Stateful components are a bad idea
  • React components are like self-contained actors/services
  • They started with Flux, but ended up with Redux. We're using Redux in our samples. I'm eyeballing how to integrate Akka.Net (although I'm not sure if that has anything to do with this.
  • ReactNative: write once, use on any device
  • Kind of superficial and kinda short but I knew all about this in React already

The reactor programming model for composable distributed computing

The reactor programming model for composable distributed computing -- Aleksandar Prokopec[^4]

media

  • Reactive programming, with events as sequences of event objects
  • Events are equivalent to a list/sequence/streams (enumerable in C#)
  • This talk is also about managing concurrency
  • There must be a boundary between outer concurrent events vs. how your application works on them
  • That's why most UI toolkits are single-threaded
  • Asynchronous is the antonym of concurrency (at least in the dictionary)
  • Filter the stream of events to compress them to frames, then render and log, so the events come in, are marshaled through the serializing bottleneck and are then dispatched asynchronously to different tasks
  • Reactor lets clients create their own channels (actors) from which they read events and which they register with a server so that it can publish
  • Akka supports setting up these things, Reactor is another implementation?
  • Dammit I want destructuring of function results (C# 7?)
  • It's very easy to build client/server and broadcast and even ordered synchronization using UIDs (or that pattern mentioned by Jonas in the keynote) The UID needs to be location-specific, though. That's not sufficient either, what you need is client-specific. For this, you need special data structures to store the data in a way that edits are automatically correctly ordered. Events sent for these changes make the events are ordered correctly
  • What is the CRDT? We just implemented an online collaborative editor: composes nicely and provides a very declarative, safe and scalable way of defining software. This is just a function (feeds back into the idea of lambdas here, actually, immutability, encapsulation)
  • Reactors


  1. I am aware of the irony that the emoji symbol for "poo" is not supported on this blogging software. That was basically the point of the presentation -- that encoding support is difficult to get right. There's an issue for it: Add support for UTF8 as the default encoding.

  2. In my near-constant striving to be the worst conversational partner ever, I once gave a similar encoding lesson to my wife on a two-hour walk around a lake when she dared ask why mails sometimes have those "stupid characters" in them.

Finovate 2016: Bank2Things

image

image

At the beginning of the year, we worked on an interesting project that dipped into IOT (Internet of Things). The project was to create use cases for Crealogix's banking APIs in the real world. Concretely, we wanted to show how a customer could use these APIs in their own workflows. The use cases were to provide proof of the promise of flexibility and integrability offered by well-designed APIs.

Watch 7--minute video of the presentation

The Use Cases

Football Club Treasurer

imageThe first use case is for the treasurer of a local football club. The treasurer wants to be notified whenever an annual club fee is transferred from a member. The club currently uses a Google Spreadsheet to track everything, but it's updated manually. It would be really nice if the banking API could connected -- via some scripting "glue" -- to update the spreadsheet directly, without user intervention. The treasurer would just see the most current numbers whenever he opened the spreadsheet.

The spreadsheet is in addition to the up-to-date view of payments in the banking app. The information is also available there, but not necessarily in the form that he or she would like. Linking automatically to the spreadsheet is the added value.

Chore & Goal Tracker

imageimageImagine a family with a young son who wants to buy a drone. He would have to earn it by doing chores. Instead of tracking this manually, the boy's chores would be tabulated automatically, moving money from the parents' account to his own as he did chores. Additionally, a lamp in the boy's room would glow a color indicating how close he was to his goal. The parents wanted to track the boy's progress in a spreadsheet, tracking the transfers as they would have had they not had any APIs.

The idea is to provided added value to the boy, who can record his chores by pressing a button and see his progress by looking at a lamp's color. The parents get to stay in their comfort zone, working with a spreadsheet as usual, but having the data automatically entered in the spreadsheet.

The Plan

It's a bit of a stretch, but it sufficed to ground the relatively abstract concept of banking APIs in an example that non-technical people could follow.

So we needed to pull quite a few things together to implement these scenarios.

  • A lamp that can be controlled via API
  • A button that can trigger an API
  • A spreadsheet accessibly via API
  • An API that can transfer money between accounts
  • "Glue" logic that binds these APIs together

The Lamp

imageimage We looked at two lamps:

Either of these -- just judging from their websites -- would be sufficient to utterly and completely change our lives. The Hue looked like it was going to turn us into musicians, so we went with Lifx, which only threatened to give us horn-rimmed glasses and a beard (and probably skinny jeans and Chuck Taylor knockoffs).

Yeah, we think the marketing for what is, essentially, a light-bulb, is just a touch overblown. Still, you can change the color of the light bulb with a SmartPhone app, or control it via API (which is what we wanted to do).

The Button

The button sounds simple. You'd think that, in 2016, these things would be as ubiquitous as AOL CDs were in the 1990s. You'd be wrong.

imageThere's a KickStarter project called Flic that purports to have buttons that send signals over a wireless connection. They cost about CHF20. Though we ordered some, we never saw any because of manufacturing problems. If you thought the hype and marketing for a light bulb were overblown, then you're sure to enjoy how Flic presents a button.

We quickly moved along a parallel track to get buttons that can be pressed in real life rather than just viewed from several different angles and in several different colors online.

imageAmazon has what they have called "Dash" buttons that customers can press to add predefined orders to their one-click shopping lists. The buttons are bound to certain household products that you tend to purchase cyclically: toilet paper, baby wipes, etc.

They sell them dirt-cheap -- $5 -- but only to Amazon Prime customers -- and only to customers in the U.S. Luckily, we knew someone in the States willing to let us use his Amazon Prime account to deliver them, naturally only to a domestic address, from which they would have to be forwarded to us here in Switzerland.

That we couldn't use them to order toilet paper in the States didn't bother us -- we were planning to hack them anyway.

These buttons showed up after a long journey and we started trapping them in our own mini-network so that we could capture the signal they send and interpret it as a trigger. This was not ground-breaking stuff, but we really wanted the demonstrator to be able to press a physical button on stage to trigger the API that would cascade other APIs and so on.

Of course we could have just hacked the whole thing so that someone presses a button on a screen somewhere -- and we programmed this as a backup plan -- but the physicality of pressing a button was the part of the demonstration that was intended to ground the whole idea for non-technical users.1

The Spreadsheet

imageimageIf you're going to use an API to modify a spreadsheet, then that spreadsheet has to be available online somewhere. The spreadsheet application in Google Docs is a good candidate.

The API allows you to add or modify existing data, but that's pretty much it. When you make changes, they show up immediately, with no ceremony. That, unfortunately, doesn't make for a very nice-looking demo.

Google Docs also offers a Javascript-like scripting language that let's you do more. We wanted to not only insert rows, we wanted charts to automatically update and move down the page to accommodate the new row. All animated, thank you very much.

This took a couple pages of scripting and a good amount of time. It's also no longer a solution that an everyday user is likely to make themselves. And, even though we pushed as hard as we could, we also didn't get everything we wanted. The animation is very jerky (watch the video linked above) but gets the job done.

The Glue

imageSo we've got a bunch of pieces that are all capable of communicating in very similar ways. The final step is to glue everything together with a bit of script. There are several services available online, like IFTTT -- If This Then That -- that allow you to code simple logic to connect signals to actions.

In our system, we had the following signals:

  • Transfer was made to a bank account
  • Button was pressed

and the following actions:

  • Insert data into Google Spreadsheet
  • Set color of lamp

The Crealogix API and UI

imageimageimageSo we're going to betray a tiny secret here. Although the product demonstrated on-stage did actually do what it said, it didn't do it using the Crealogix API to actually transfer money. That's the part that we were actually selling and it's the part we ended up faking/mocking out because the actual transfer is beside the point. Setting up bank accounts is not so easy, and the banks take umbrage at creating them for fake purposes.

Crealogix could have let us use fake testing accounts, but even that would have been more work than it was worth: if we're already faking, why not just fake in the easiest way possible by skipping the API call to Crealogix and only updating the spreadsheet?

Likewise, the entire UI that we included in the product was mocked up to include only the functionality required by the demonstration. You can see an example here -- of the login screen -- but other screens are linked throughout this article. Likewise, the Bank2Things screen shown above and to the left is a mockup.

Wrapup

So what did Encodo actually contribute?

  • We used the Crealogix UX and VSG to mock up all of the app screens that you seen linked in this article. We did all of the animation and logic and styling.
  • We built two Google Spreadsheets and hooked them up to everything else
  • We hooked up the Lifx lamp API into our system
  • We hacked the Amazon Dash buttons to communicate in our own network instead of beaming home to the mothership
  • We built a web site to handle any mocking/faking that needed to be done for the demo and through which the devices communicated
  • We provided a VM (Virtual Machine) on which everything ran (other than the Google Spreadsheets)

As last year -- when we helped Crealogix create the prototype for their BankClip for Finovate 2015 -- we had a lot of fun investigating all of these cutting-edge technologies and putting together a custom solution in time for Finovate 2016.



  1. As it turns out, if you watch the 7--minute video of the presentation, nowhere do you actually see a button. Maybe they could see them from the audience.