Problem with showing complete list of sub tests when running Theory

Topics: ReSharper test runner
Aug 18, 2012 at 9:53 PM

It looks like Resharper is receiving incomplete list of sub tests when running Theory where individual data passed to theory result in same sub test description. Here is example to reproduce the issue:

public static IEnumerable<object[]> GetConstructorParameterNameValidCases {
    get {
        yield return new object[] { new Type[] { typeof(string) }, 0, "firstParam" };
        yield return new object[] { new Type[] { typeof(Func<>) }, 0, "firstParam" };
        yield return new object[] { new Type[] { typeof(string), typeof(string) }, 1, "secondParam" };
    }
}
            
[Theory]
[PropertyData("GetConstructorParameterNameValidCases")]
public void GetConstructorParameterName(Type[] constructorParameterTypes, int parameterOrder, string expectedName) {
    // ...
}

This example should lead to three sub test showed under GetConstructorParameterName but only two are showed because first two cases have the same "description":

GetConstructorParameterName (2 tests)
- GetConstructorParameterName(constructorParameterTypes: System.Type[], parameterOrder: 0, expectedName: "firstParam")
- GetConstructorParameterName(constructorParameterTypes: System.Type[], parameterOrder: 1, expectedName: "secondParam")

I think it is an issue in xUnit.Contrib because the test rewritten to NUnit produces complete list of sub tests

Coordinator
Aug 20, 2012 at 12:20 PM

Ah. This is because the ReSharper runner is using the test name as the key for the node, and you're creating test cases that duplicate test names. The nunit resharper runner doesn't suffer this problem because (I think) nunit itself assigns a unique test name to each test and test case, and resharper uses that as a key.

Have you tested this in the xunit gui? As far as I remember, xunit does nothing special to handle this case, and I don't really think there's much I can do here, either. A simple workaround would be to add a description parameter to the test to clarify which test case is being called, e.g. "string ctor", "func ctor", "two strings", etc.)

Nov 8, 2012 at 5:54 PM

We are experiencing this same issue when we have unit tests that use only array parameters. (For example, all int array parameters display as paramName: System.Int32[].)

The xunit.net test runner (xunit.gui.clr4.exe) does not display individual subtests, but it does run all subtests and marks the parent test as failed if any fails. The critical problem with Resharper is that it marks the parent test as passed even if one subtest failed.

This makes the Resharper unusable for our unit tests because we don't know whether failing tests are not being displayed. Instead, we use the xvsr10 xunit.net test runner for Visual Studio (https://github.com/quetzalcoatl/xvsr10) which doesn't have this problem.

Coordinator
Nov 15, 2012 at 3:34 PM

See work item: 4155. Should be fixed soon.