Integration test experiments - running IdentityServer and ASP.NET Core WebApi inside a Process

Hello.  I'll get to the main topic of this post shortly (and a link to my github repository with the sample code at the end) but just wanted to give you a bit of background first.  Recently I've been studying security engineering with a view toward strengthening my skills in that area. 

- I attended a two-day workshop on Identity and Access Control by Dominic Baier (one of the creators of IdentityServer4) at NDC London.  This was a real eye-opener, and in turn led me to a few other avenues of research.  It turns out this extra time I've got in lockdown can be put to use...

- I'm reading a book he recommended to me which is considered a seminal work in the field, "Security Engineering" by Ross Anderson.  It's a massive tome so I'm working my way through it slowly but it's clearly essential reading.

- I'm reading up on OAuth 2.0 and Open ID Connect with a view toward eventually being thoroughly conversant with all the concepts, terminology, etc.

- I took a Pluralsight course on Authentication and Authorization in ASP.NET Core.

- Trying to thoroughly understand IdentityServer and the various use cases for it, starting by meticulously going through each one of the Quickstart tutorials.  Which brings me to the topic of this post.

Once I'd gone through a given Quickstart, I wanted to write integration tests against it.  So I started with the first one (the "client credentials Quickstart") as an experiment.  My goals were as follows:

- Write integration tests that simulate the client code from the example.

- Ensure that my results are the same as what you get when you run the client code.

This led me to read up on integration testing in ASP.NET Core.  Microsoft recommends using a WebApplicatonFactory to spin up an instance of your ASP.NET Core web application in the context of the integration test.

Well, I quickly ran into a problem.  The Quickstart relies on both the IdentityServer instance and the Web API running on the local development machine, and listening on two different ports.  It turns out that when the client code tries to call the API's "IdentityController" (line 49 of the example), I started getting this error:

System.InvalidOperationException: IDX20803: Unable to obtain configuration from: 'https://localhost:5001/.well-known/openid-configuration'. ---> System.IO.IOException: IDX20804: Unable to retrieve document from: 'https://localhost:5001/.well-known/openid-configuration'.

Googling led me to a comment on a two-year-old issue on the IdentityServer Github page.

I had IdentityServer set up to run on https port 5001, and the web API set up to run on https port 6001; what I realized from that old post (and verified by observing Fiddler on my machine) was that, when the API is called, it in turn calls the endpoint for IdentityServer to get the discovery document.  That happens, as I understand it, because the Quickstart has the JWT handler options.Authority endpoint specified (line 18 of this file), so the API tries to call IdentityServer to get the discovery document... but in the context of an integration test that uses the WebApplicationFactory, when it's inside the API code, it knows about port 6001 and fakes that it's running in that context; but meanwhile, the WebApplicationFactory that I spun up for IdentityServer (which was meant to be running on https port 5001) isn't actually listening on that port.  

The bottom line of all this head-scratching was the realization that the WebApplicationFactory (and/or TestServer) wasn't really cutting it for me.  I needed two web apps to be running for real, and really listening on two separate ports.  That, to me, would be a true integration test which simulates the real-world condition this setup represents:  two web apps running on two separate endpoints that can be called by a client represented by the integration test code.

So I started googling things like "how to run two TestServer instances at the same time" etc - and found creative solutions to the problem in StackOverflow posts like this one or this blog post.  

But (with all due respect) those seemed a bit fiddly and slightly over-complicated.

Well, cutting to the chase - I started asking myself, how could I just run both web apps for real, but execute them programmatically; and why couldn't I just do a "dotnet run", fire them up in the setup of the integration test, run various tests against those same instances, as if they were the real endpoints listening in the real world, then tear them down at the end?  

So a bunch of googling led to lots of examples of running .Net Core apps inside of a Process, for example this one.  That guy found it worked better if he first did a "dotnet publish" then run it from the output directory to which he had published it.  (BTW at the time of this writing I did this on a Windows 10 machine with Visual Studio 2019).

So in the end I managed to make that happen in what I believe is a reasonably elegant way.  I've got the code to a point where it appears to execute quite reliably - it publishes and runs the two ASP.NET Core apps (IdentityServer and the API, as in the Quickstart example), and they're actually listening on the two different https ports; and they're instantiated only once in the setup, the tests run, then they get torn down at the end.  I did this with some test code helper classes like a "ProcManager" (process manager used to spin up the processes at the beginning and kill them at the end) that I think makes the tests pretty readable.  I use the XUnit's "class fixture" so all the setup and teardown code resides in the fixture class, so the tests are nice and tidy.  There's still much to do, for example:

- see how this might work when running in a CI/CD context e.g. Azure DevOps or TeamCity / Octopus

- make things more configurable - currently I extract the ports programmatically from launchSettings.json

- write more tests using their suggestions at the end of the Quickstart

- apply this idea to the subsequent Quickstarts

The point is I think it's a reasonable way of doing it, running multiple instances of ASP.NET Core web apps, each inside of a Process, in the context of an integration test; and it seems to work; but, what do you think?  I'd be happy to hear any feedback you might have.  Thanks for reading.

Here's the code.