Está en la página 1de 9

Automation and Performance Tool Evaluation SoapUI Pro vs Parasoft Soatest

SOA Testing Tool Evaluation


The objective of this exercise was to evaluate automation and load tools for SOA implementations. We went through a similar exercise 9 months ago in which SoapUI Pro was not considered. This revaluation of our automation and load tool is driven by three factors: 1. Recent additions to SoapUI Pro functionality (inluding a load tool) which we felt now made SoapUI Pro competitive due to SoapUIs relatively low price. 2. Our contract with Soatest is subscription based, we thought it best to justify the cost of our previous our tool choice before renewal.

Feature Comparison
# Feature 1 Transport Protocol: Ability to test through different protocols (HTTP,HTTPS, JMS, MQ, RMI) 2 XML Schema Validation: Able to validate the data characteristics for each xml element 3 Request and Response SOAP messages - Able to interpret and monitor data - Ability to check for correctness in the response of the service 4 Response message comparison: - Raw String - XPATH Logical expression 5 Asynchronous Testing 6 Data Driven Testing 7 Ability to add check points 8 WSDL Inspection and Invoking soapUI Pro This tool supports HTTP,REST and HTTPS. MQ and JMS not Supported Parasoft SOAtest This tool supports only HTTP,HTTPS,JMS ,REST, RMI, SMTP, TIBCO, .Net WCF HTTP & TCP and MQ (Sonic and WebSphere)

Automatic schema validation

Automatic schema validation

The SOAP request can be parameterized using properties. Different types of assertions are available for checking the SOAP response.This feature is very easy to use in SoapUI. Response message comparision can be done effectively using basic assertions provided in the tool.

Request formation can be done through easy methods and UI availabe. Data parameterization can be done using various sources like, FileSyste, Excel, DB, etc. XML Assertor and Diff tool helps to do various kind of assertions and regression validation. Support for EXamXML tool is also available for XML Comparison. Supported Supports data driven testing by default

Supported Supports data driven testing through scripting Supports adding check points in test cases Supports adding of check points in test cases to break down timings. to break down timings. Supports WSDL inspection and Invoking Supports WSDL Inspection, WSDL Validations, validations against BP 1.1 3

Feature Comparison
# Feature
Not Supported SOAP Test case & REST test cases only

soapUI Pro

Parasoft SOAtest
Supported

9 Ability to create test cases without WSDL 10 Types of Test cases

SOAP, REST, Browser Testing (UI), EJB, ISO 8583, WebMethods 11 Read Data from various input Using Groovy Scripts it is possible to read Data parameterization is supported sources (DB, XLS, text files) data from an external data source and write in easier steps it to another steps' properties. 12 Supports SOAP 1.1, 1.2 Soap UI supports both SOAP versions 1.1 Supports both SOAP 1.1, 1.2 and 1.2 13 WS-I, WS-Addressing, WSSecurity The support for WS-Security and tests are It supports all managed from the WS-Security tab in the Project View and then applied to their desired targets at corresponding locations It is possibile to create Mock Stubs can be created Implementations, or Web Service Simulations, (called "MockServices") of any Web Service from its WSDL contract Supports Attachment testing Supports attachement testing

14 Web Service Simulation

15 Support for Attachment Testing 16 Support for Regression Testing Regression test suites cannot be created but Supports for regression testing by retesting of the test suites can be performed creating regression test suites
4

Feature Comparison
# 17 18

Feature
WSDL Scoring Assertions or Success Criteria

soapUI Pro
Not Supported Few assertions are available such as: SOAP validation, X-path assertion, SOAP contains etc.

Parasoft SOAtest

19 20

Baseline Regression Testing Automated Reporting

WSDL Tests does this XML Assertor and Diff tool helps to do various kind of assertions and regression validation. Support for EXamXML tool is also available for XML Comparison. Not Possible in SoapUI. Regression can be created For Automated Reporting, SoapUI Automated report is available includes a "Launch Test Runner"

21 22

Reporting Types: Reports Coverage:

23

24

Scenario based Reporting: Able to generate execution results reports based on given scenarios Validate WSDL WS standards: Supported Able to verify WSDL schema to WS standards(Basic Profile 1.0,1.1,1.2)

Reports can be generated in XML, HTML, Text PlainText only Reports Generated are moderate Detailed reports are available as they do not give sufficient information and not easily interpreted by the end users Not Supported Supported

Supported

Feature Comparison
# Feature 25 Ability to test the design time interoperabilty of the service 26 Authentication & Authorization 27 Time Stamp 28 Signatures 29 Encryption & Decryption 30 Secure Socket Layer (SSL) 31 Additional Security Features 32 Installation and setup 33 Good user interface 34 Short learning curve required 35 Minimal manual coding effort required to use 36 Minimal supervision during test execution 37 Easy to create test suites 38 Ability to group (add or remove) test cases in an existing test case suites 39 Easy portability for test suites and test cases for different groups to use 40 Ability to store tests and test suites for reuse soapUI Pro Supported Supported Supported Supported Supported Supported Not Supported Easy Moderate Moderate User needs to know scripting Moderate Easy Easy Supported Supported Supported Supported Supported Supported Supported Supported Supported Moderate Good Good Less scripting required Tutorials and Help guide available. Easy Easy Supported Supported Parasoft SOAtest

41 Ability to update or sync up tests and test suites on Not Supported Supported different client machines 42 Support for integration with Rally Not Supported Supported through API 43 Easy integration with other components such as By Scripting we can integrate the Can be done easily Excel, Notepad, ODBC,Access other components like Excel, File Data Sources & Database 44 Compatible with Operating systems like windows Soap UI is compatible with windows, Yes, separate installers available XP,Linux,Mac,Solaris etc. Linux & Mac 6

Cost Comparison
Tool Automation Load Implementation Total Soap UI Pro $11,250.00 $360,000.00 $32,000.00 $403,250.00 Soatest $50,000.00 $36,000 (unlimited) $0.00 $86,000.00

1. These costs are based upon (50) 1 year subscription licenses for automation and 7500 vusers for load 2. Soap UI Pros Load UI is not an acceptable tool for performance/load. A choice of Soap UI Pro requires an investment in Loadrunner vusers. Where we have an enterprise version of Performance Center , we only have 2500 vusers. The load cost for Soap Ui Pro reflects the purchase of an additional 5000 vusers. 3. The cost to switch to rewrite regression scripts in Soap UI Pro reflects 4 offshore developers for 2 months. 4. Costs do not reflect any training expense or professional Services expense.

Scorecard
Features Functional Testing Regression Testing Integration/End-toEnd Testing Database Testing Ease of Use Ease of Modifying the Scripts Reporting Security/Penetration Testing Performance Testing Active Tool Support In House Tool Expertise Cost Total SOAPUI Pro 4 2 1 2 3 2 3 3 1 3 5 31 Soatest 4 5 5 5 4 4 4 5 4 3 4 2 49

Recommendation
While SOAPUI Pro can be used for smoke testing and basic functional testing for simple
request and responses, there is a quite considerable effort required to create and maintain regression suites for complex requests and responses. The recent addition of features to SoapUI Pro make it a more attractive option, but it still falls well short of the mark. The adoption of the tool by the developers in the scrum will help to push testing forward. Parasoft Soatest for Services testing serves all the pre-requisites and its ease to create and maintain automated regression and load scripts makes it by far the right choice for Enterprise Services

También podría gustarte