系列

2018年7月21日 星期六

SpecFlow SpecRun.exe+MSBuild+Report


SpecFlow SpecRun.exe+MSBuild+Report

最近想要使用SpecFlow 使用 SpecRun.exe+MSBuild 去產生報表 並且使用排成器來產生測試報表



一開始要先安裝MSBuild
PS.可以參考我寫的安裝 MSBuild 2017 Tools Intall






  <specFlow>
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config-->
    <!--<unitTestProvider name="MsTest" />-->
    <!--<add name="SpecRun" path="../packages/SpecRun.SpecFlow.2-3-0.1.7.1/lib/net45" />-->
    <!-- For additional details on SpecFlow configuration options see http://go.specflow.org/doc-config-->
    <!-- use unit test provider SpecRun+NUnit or SpecRun+MsTest for being able to execute the tests with SpecRun and another provider -->
    <unitTestProvider name="SpecRun" />
    <plugins>
      <!--<add name="SpecRun" path="../packages/SpecRun.Runner.1.7.2/tools/"/>-->
      <add name="SpecRun" path="../packages/SpecRun.SpecFlow.2-3-0.1.7.2/lib/net45/"/>
    </plugins>
  </specFlow>
接下來開始到web.config or app.config 去設定使用SpecRun執行





<?xml version="1.0" encoding="utf-8"?>
  <Settings projectName="SpecFlowTestProject.Test" projectId="{b2ada03f-013d-4569-a618-f4bfb5df6594}" outputFolder="/Report" />
  <!--<Report disable="false" copyAlsoToBaseFolder="true">
    <Template name="CustomReportTemplate_1.cshtml" outputName="Report1.html"/>
  </Report>-->
  <Execution stopAfterFailures="-1" testThreadCount="1" testSchedulingMode="Sequential" retryFor="None"/>
  <!-- For collecting by a SpecRun server update and enable the following element. For using the
      collected statistics, set testSchedulingMode="Adaptive" attribute on the <Execution> element.
    <Server serverUrl="http://specrunserver:6365" publishResults="true" />
  -->
  <TestAssemblyPaths>
    <TestAssemblyPath>SpecFlowTestProject.Test.dll</TestAssemblyPath>
  </TestAssemblyPaths>
  <DeploymentTransformation>
    <Steps>
      <!-- sample config transform to change the connection string-->
      <!--<ConfigFileTransformation configFile="App.config">
        <Transformation>
          <![CDATA[<?xml version="1.0" encoding="utf-8"?>
                                                  <configuration xmlns:xdt="http://schemas.microsoft.com/XML-Document-Transform">
                <connectionStrings>
                  <add name="MyDatabase" connectionString="Data Source=.;Initial Catalog=MyDatabaseForTesting;Integrated Security=True"
                       xdt:Locator="Match(name)" xdt:Transform="SetAttributes(connectionString)" />
                </connectionStrings>
                                                  </configuration>
                                           ]]>
        </Transformation>
      </ConfigFileTransformation>-->
    </Steps>
  </DeploymentTransformation>
</TestProfile>
然後去設定Default.srprofile



參數說明(參照官方文件說明+SpecRunTestProfile.xsd) 

參數名稱
說明
projectName
測試專案名稱
outputFolder
輸出結果的資料夾
stopAfterFailures
失敗測試的次數,之後停止執行所有測試。將此值設置為0(零)以執行所有測試,而不管故障的數量。如果未指定,則默認為10.
注意:.srprofile NuGet包附帶的默認值為3。
testThreadCount
用於執行測試的測試線程數(默認值= 1)。
雖然沒有最大線程數,但將此值設置得太高會降低性能,因為各個線程會爭用處理能力。建議將Process測試線程隔離(testThreadIsolation設置)的線程數設置為小於核心數的一個。您可以使用一般使用較高的數字隔離AppDomain和SharedAppDomains。
注意:僅增加線程數可能會導致問題,例如,如果線程訪問同一個數據庫。因此,使用{TestThreadId}新文件和數據庫實例名稱中的佔位符將其與.config文件的重定位和轉換相結合是有意義的(請參閱下面)。
注意:每個線程執行鉤子之前和之後(例如BeforeTestRun,AfterTestRun等)。
testSchedulingMode
確定SpecRun執行測試的順序
Sequential::測試始終以相同的順序執行(默認)。執行測試的順序是可預測的並且始終相同,但不能手動影響。
Random:測試順序是隨機的。這可用於檢測可能通過/失敗的閃爍場景,具體取決於執行其他測試區域的順序。
Adaptive:SpecRun使用通過/失敗歷史記錄中可用的統計信息來確定首先運行哪些測試。以前失敗的測試和新測試在成功和穩定的測試之前執行。
注意:如果設置testSchedulingMode為Adaptive如果沒有以前的測試執行結果可用(例如,沒有配置服務器,或服務器無法訪問),則使用該Random選項執行測試。
retryFor
確定應根據其狀態重試哪些測試。可用選項::
None不重試測試:重試
Failing失敗測試(默認)
All:重試所有測試,包括通過的測試
注意:在返回相應狀態之後和執行任何後續測試之前,將立即重試測試。
TestAssemblyPath
所述<TestAssemblyPaths>元件是一種用於容器<TestAssemblyPath>的元件,從所述命令行執行測試時,其定義組件的路徑。<TestAssemblyPath>元素中的路徑相對於基礎文件夾。
確保在自己的<TestAssemblyPath>元素中指定所有程序集文件的正確名稱,否則將不會找到任何測試!

也可以參考官方文件




PS.我在實際進行時發現設定檔跟官方文件有些落差,所以我建議參考官方文件然後到SpecRunTestProfile.xsd裡面去看可以設定資訊的位置

SpecRunTestProfile.xsd文件內容
<?xml version="1.0" encoding="utf-8"?>
<xs:schema id="SpecRunTestProfile"
    elementFormDefault="qualified"

  <xs:element name="TestProfile" type="TestProfile" />

  <xs:complexType name="TestProfile">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element minOccurs="0" maxOccurs="1" name="Settings" type="TestProfileSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="Server" type="ServerSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="Execution" type="ExecutionSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="Environment" type="EnvironmentSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="TestAssemblyPaths" type="TestAssemblyPaths" />
      <xs:element minOccurs="0" maxOccurs="1" name="Report" type="ReportSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="Filter" type="xs:string" />
      <xs:element minOccurs="0" maxOccurs="1" name="Targets" type="Targets" />
      <xs:element minOccurs="0" maxOccurs="1" name="DeploymentTransformation" type="DeploymentTransformation" />
      <xs:element minOccurs="0" maxOccurs="1" name="TestThreads" type="TestThreads" />
      <xs:element minOccurs="0" maxOccurs="1" name="VSTest" type="VSTest" />
    </xs:choice>
  </xs:complexType>

  <xs:complexType name="TestProfileSettings">
    <xs:attribute name="name" type="xs:string" use="optional" />
    <xs:attribute name="projectName" type="xs:string" use="required" />
    <xs:attribute name="licenseKey" type="xs:string" use="optional" />
    <xs:attribute name="baseFolder" type="xs:string" use="optional" />
    <xs:attribute name="outputFolder" type="xs:string" use="optional" />
    <xs:attribute name="reportTemplate" type="xs:string" use="optional" />
    <xs:attribute name="projectId" type="guid" use="optional" />
  </xs:complexType>

  <xs:complexType name="ServerSettings">
    <xs:attribute name="serverUrl" type="xs:string" use="optional" />
    <xs:attribute name="publishResults" type="xs:boolean" use="optional" default="false" />
  </xs:complexType>

  <xs:complexType name="ExecutionSettings">
    <xs:attribute name="stopAfterFailures" type="xs:int" use="optional" default="10" />
    <xs:attribute name="testSchedulingMode" type="TestSchedulingMode" use="optional" default="Sequential" />
    <xs:attribute name="testThreadCount" type="TestThreadCount" use="optional" default="1" />
    <xs:attribute name="retryFor" type="RetryMode" use="optional" default="Failing" />
    <xs:attribute name="retryCount" type="xs:int" use="optional" default="2"  />
    <xs:attribute name="apartmentState" type="ApartmentState" use="optional" default="Unknown" />
    <xs:attribute name="isDryRun" type="xs:boolean" use="optional" default="false" />
  </xs:complexType>

  <xs:complexType name="EnvironmentSettings">
    <xs:attribute name="framework" type="FrameworkVersion" use="optional" default="Default" />
    <xs:attribute name="platform" type="Platform" use="optional" default="Default" />
    <xs:attribute name="testThreadIsolation" type="TestThreadIsolation" use="optional" default="AppDomain" />
    <xs:attribute name="apartmentState" type="ApartmentState" use="optional" default="Unknown" />
  </xs:complexType>

  <xs:complexType name="TestAssemblyPaths">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="unbounded" name="TestAssemblyPath" type="xs:string" />
    </xs:sequence>
  </xs:complexType>

  <!-- Report settings -->
  <xs:complexType name="Template">
    <xs:attribute name="name" type="xs:string" use="required" />
    <xs:attribute name="outputName" type="xs:string" use="optional" />
    <xs:attribute name="outputCulture" type="xs:string" use="optional" />
  </xs:complexType>

  <xs:complexType name="ReportSettings">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="unbounded" name="Template" type="Template" />
    </xs:sequence>

    <xs:attribute name="copyAlsoToBaseFolder" type="xs:boolean" use="optional" default="false" />
    <xs:attribute name="disable" type="xs:boolean" use="optional" default="false" />
    <xs:attribute name="existingFileHandlingStrategy" type="ExistingFileHandlingStrategy" use="optional" default="Default" />
  </xs:complexType>
  
  <!-- test targets -->

  <xs:complexType name="Targets">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="unbounded" name="Target" type="Target" />
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="Target">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element minOccurs="0" maxOccurs="1" name="Filter" type="xs:string" />
      <xs:element minOccurs="0" maxOccurs="1" name="Environment" type="EnvironmentSettings" />
      <xs:element minOccurs="0" maxOccurs="1" name="DeploymentTransformationSteps" type="DeploymentTransformationSteps" />
    </xs:choice>
    <xs:attribute name="name" type="xs:string" use="required" />
  </xs:complexType>

  <!-- test threads -->

  <xs:complexType name="TestThreads">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="unbounded" name="TestThread" type="TestThread" />
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="TestThread">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="1" name="TestAffinity" type="xs:string" />
    </xs:sequence>
    <xs:attribute name="id" type="xs:int" use="required" />
  </xs:complexType>

  <!-- deployment transformations -->

  <xs:complexType name="DeploymentTransformation">
    <xs:sequence>
      <xs:element minOccurs="0" maxOccurs="1" name="GlobalSteps" type="DeploymentTransformationSteps" />
      <xs:element minOccurs="0" maxOccurs="1" name="Steps" type="DeploymentTransformationSteps" />
    </xs:sequence>
  </xs:complexType>
  <xs:complexType name="DeploymentTransformationSteps">
    <xs:choice minOccurs="0" maxOccurs="unbounded">
      <xs:element minOccurs="1" maxOccurs="1" name="CopyFolder" type="CopyFolder" />
      <xs:element minOccurs="1" maxOccurs="1" name="ConfigFileTransformation" type="ConfigFileTransformation" />
      <xs:element minOccurs="1" maxOccurs="1" name="Relocate" type="Relocate" />
      <xs:element minOccurs="1" maxOccurs="1" name="RelocateConfigurationFile" type="RelocateConfigurationFile" />
      <xs:element minOccurs="1" maxOccurs="1" name="Custom" type="CustomStep" />
      <xs:element minOccurs="1" maxOccurs="1" name="IISExpress" type="IISExpressStep" />
      <xs:element minOccurs="1" maxOccurs="1" name="CopyFile" type="CopyFile" />
    </xs:choice>
  </xs:complexType>
  <xs:complexType name="CopyFolder">
    <xs:attribute name="source" type="xs:string" use="required" />
    <xs:attribute name="target" type="xs:string" use="required" />
    <xs:attribute name="deleteFolderOnRestore" type="xs:boolean" use="optional" default="true" />
    <xs:attribute name="deleteFolderOnApply" type="xs:boolean" use="optional" default="true" />
  </xs:complexType>
  <xs:complexType name="Relocate">
    <xs:attribute name="targetFolder" type="xs:string" use="required" />
    <xs:attribute name="deleteFolderOnRestore" type="xs:boolean" use="optional" default="true" />
  </xs:complexType>
  <xs:complexType name="ConfigFileTransformation">
    <xs:sequence>
      <xs:element minOccurs="1" maxOccurs="1" name="Transformation" type="xs:string" />
    </xs:sequence>
    <xs:attribute name="configFile" type="xs:string" use="optional" />
  </xs:complexType>
  <xs:complexType name="RelocateConfigurationFile">
    <xs:attribute name="target" type="xs:string" use="required" />
  </xs:complexType>
  <xs:complexType name="CustomStep">
    <xs:attribute name="type" type="xs:string" use="required" />
    <xs:attribute name="arguments" type="xs:string" use="optional" />
  </xs:complexType>
  <xs:complexType name="IISExpressStep">
    <xs:attribute name="webAppFolder" type="xs:string" use="required" />
    <xs:attribute name="iisExpressPath" type="xs:string" use="optional" />
    <xs:attribute name="port" type="xs:string" use="optional" default="8080" />
    <xs:attribute name="useShellExecute" type="xs:boolean" use="optional" default="true" />
  </xs:complexType>
  <xs:complexType name="CopyFile">
    <xs:attribute name="source" type="xs:string" use="required" />
    <xs:attribute name="target" type="xs:string" use="optional" />
    <xs:attribute name="targetFolder" type="xs:string" use="optional" />
    <xs:attribute name="deleteOnRestore" type="xs:boolean" use="optional" default="true" />
  </xs:complexType>
  
  <!-- VSTest -->

  <xs:complexType name="VSTest">
    <xs:attribute name="testRetryResults" type="TestRetryResultsMode" use="optional" default="Separate" />
    <xs:attribute name="passRateRelative" type="RelativePassRate" use="optional" default="1"/>
    <xs:attribute name="passRateAbsolute" type="xs:int" use="optional"/>
  </xs:complexType>



  <!-- simple types-->


  <xs:simpleType name="RelativePassRate">
    <xs:union memberTypes="floatPassRate percentagePassRate" />
  </xs:simpleType>

  <xs:simpleType name="floatPassRate">
    <xs:restriction base="xs:float">
      <xs:minInclusive value="0"/>
      <xs:maxInclusive value="1"/>
    </xs:restriction>
  </xs:simpleType>


  <xs:simpleType name="percentagePassRate">
    <xs:restriction base="xs:string">
      <xs:pattern value="(100(\.0*)?|[0-9][0-9](\.\d*)?)%" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="TestThreadCount">
    <xs:union memberTypes="testThreadCountName testThreadCountNumber" />
  </xs:simpleType>


  <xs:simpleType name="testThreadCountNumber">
    <xs:restriction base="xs:int">
      <xs:minInclusive value="0" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="testThreadCountName">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Max" />
    </xs:restriction>
  </xs:simpleType>
  
  <xs:simpleType name="TestSchedulingMode">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Sequential" />
      <xs:enumeration value="Random" />
      <xs:enumeration value="Adaptive" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="RetryMode">
    <xs:restriction base="xs:string">
      <xs:enumeration value="None" />
      <xs:enumeration value="Failing" />
      <xs:enumeration value="All" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="guid">
    <xs:restriction base="xs:string">
      <xs:pattern value="{[0-9a-fA-F]{8}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{4}-[0-9a-fA-F]{12}}" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="ApartmentState">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Unknown" />
      <xs:enumeration value="STA" />
      <xs:enumeration value="MTA" />
    </xs:restriction>
  </xs:simpleType>
  
  <xs:simpleType name="FrameworkVersion">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Default" />
      <xs:enumeration value="Net35" />
      <xs:enumeration value="Net40" />
      <xs:enumeration value="Net45" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="Platform">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Default" />
      <xs:enumeration value="x86" />
      <xs:enumeration value="x64" />
    </xs:restriction>
  </xs:simpleType>
  
  <xs:simpleType name="TestThreadIsolation">
    <xs:restriction base="xs:string">
      <xs:enumeration value="SharedAppDomain" />
      <xs:enumeration value="AppDomain" />
      <xs:enumeration value="Process" />
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="ExistingFileHandlingStrategy">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Default"/>
      <xs:enumeration value="Overwrite"/>
      <xs:enumeration value="IncrementFilename"/>
    </xs:restriction>
  </xs:simpleType>

  <xs:simpleType name="TestRetryResultsMode">
    <xs:restriction base="xs:string">
      <xs:enumeration value="Separate" />
      <xs:enumeration value="Unified" />
    </xs:restriction>
  </xs:simpleType>
  
</xs:schema>





為什麼會這樣說呢?因為我一開始要設定報表輸出的位置
官方文件這樣寫


<TestProfile xmlns="http://www.specrun.com/schemas/2011/09/TestProfile"> <Settings name="Multiple Reports" projectName="SpecRun Test Project" /> <Report> <Template name="CustomReportTemplate_1.cshtml" outputName="Report1.html" copyAlsoToBaseFolder="true"/> <Template name="CustomReportTemplate_2.cshtml" outputName="Report2.html" copyAlsoToBaseFolder="true"/> </Report> <TestAssemblyPaths> <TestAssemblyPath>SpecRun.TestProject.dll</TestAssemblyPath> </TestAssemblyPaths> </TestProfile>


然後就不過不過不過不過不過不過
所以我自己去參考SpecRunTestProfile.xsd 然後去設定outputFolder才解決的問題




都設定完成就可以用cmd去執行runtests.cmd~如果你的人品還不錯的話應該就會跑開始跑測試並且產出報告囉~



沒有留言:

張貼留言