Reshare 5.1 Integration - Fixture Failure Reporting

Topics: ReSharper test runner
Feb 20, 2011 at 4:21 PM

Hi,

We've been having two issues when using xUnit.net (1.6.1) with Resharper 5.1:

* If a fixture fails then R# won't count this as a test failure, so if you leave R# in its default display mode then you think everything is fine. The only reasonable way to see the issue is to hit the "Show Only Failed Tests" option and then expand out to see if anything went wrong.

* We have some tests in abstract base classes, if you click the "Show Only Failed Tests" then these tests are in the tree (though in grey with a state of pending).

The combination of these two issues is annoying and results in people accidentally checking in code even when the tests are failing.

So I was wondering if there is anything we can do about it?

Appologies if I should be asking about this on the Resharper forum, not entirely sure if the issue is with it or xUnit or something else entirely.

Ta,

Colin

Coordinator
Feb 22, 2011 at 4:53 PM

Just had a quick look. 

I can repro the first point - if an exception is thrown either before or after the body of the test, the parent node in the tree gets failed, not the test node itself. I seem to remember this was deliberate, to mimic the built-in nunit test runner. But the nunit test runner marks the child test node as failed now, so either I'm mis-remembering, or JetBrains have changed the behaviour. Either way, as you say, it's not obvious, and the failure did happen while running the child test, so I'll get that changed.

As for the second point, can you expand a bit? What's the exact scenario? Is it a failing test in an abstract base class?

Thanks
Matt 

Feb 23, 2011 at 10:15 AM

Superb on the first one, it'll definitely make a difference to us.

I can give you more info on second one if you want but actually the first one is by far the bigger issue so if you fix it I personally won't care at all about the second one.

One other question, when running the tests through R# we sometimes want to put out some debugging information. It can be useful if you quickly want to diagnose why a test is slow. This doesnt' seem to be working if you use Console.WriteLine in tests for recent versions, is this something that changed?

Coordinator
Feb 23, 2011 at 1:25 PM

Nope, nothing's changed with that, as far as I can tell. xunit captures (and the runner reports) console stdout/stderr, Debug.Write + Trace.Write (+ Trace.TraceInformation, TraceWarning, TraceError).

One thing that did change, and I can't remember when, is that resharper used to show all output when you select the node for a test class. Now it only gives an overview, and to see output, you need to select the test itself.

Mar 1, 2011 at 6:15 PM

Wierd, couldn't get any output to appear, no biggie though.

Was wondering, do you know when the failing fixture causing test run to fail issue will be resolved? Just wondering whether its weeks or away or will need to wait for a major release?

Coordinator
Mar 1, 2011 at 11:54 PM
Hmm. If you get the source and open the tests.vs2010.sln, there are a couple of (manual) tests about capturing output. Might be worth giving that a go, if you fancy it.

And I'm hoping to get the fixture failures thing sorted this week. It's very much a case of as-and-when I get time, unfortunately. And it's in the messiest bit of the codebase (well, it's got the worst tests I think I've ever written...) But all being well, I'll be working on it this week, aiming to release over the weekend.

Matt

On 1 March 2011 18:15, colinjack <notifications@codeplex.com> wrote:

From: colinjack

Wierd, couldn't get any output to appear, no biggie though.

Was wondering, do you know when the failing fixture causing test run to fail issue will be resolved? Just wondering whether its weeks or away or will need to wait for a major release?

Read the full discussion online.

To add a post to this discussion, reply to this email (xunitcontrib@discussions.codeplex.com)

To start a new discussion for this project, email xunitcontrib@discussions.codeplex.com

You are receiving this email because you subscribed to this discussion on CodePlex. You can unsubscribe on CodePlex.com.

Please note: Images and attachments will be removed from emails. Any posts to this discussion will also be available online at CodePlex.com


Mar 2, 2011 at 10:08 AM

Ahh perfect, no rush on it I was just wondering if it would be weeks or months or more. Meantime we're looking at other options (running from batch file or xunit runner) so wasn't sure if we should continue with that is all (as it'd only be a temporary approach at best).

Coordinator
Mar 8, 2011 at 12:01 AM
This discussion has been copied to a work item. Click here to go to the work item and continue the discussion.
Coordinator
Mar 8, 2011 at 12:15 AM

If you don't mind compiling from source, I've just committed a quick fix which means methods are marked as failures if the class level (IUseFixture) setup/teardown fails