Documentos de Académico
Documentos de Profesional
Documentos de Cultura
TF
Half-‐day
Tutorials
8:30
AM
Seretta
Gamba
Freelancer
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-‐-‐ 268-‐-‐ 8770
·∙·∙
904-‐-‐ 278-‐-‐ 0524
-‐
info@techwell.com
-‐
http://www.starwest.techwell.com/
Seretta
Gamba
Seretta
Gamba
has
forty
years’
experience
in
development
and
fifteen
in
test
automation.
After
going
through
all
the
usual
developer
roles,
in
2001
she
was
put
in
charge
of
test
automation
for
her
current
company.
She
developed
a
framework
that
enabled
her
company
to
quickly
get
excellent
results.
After
having
talked
about
the
framework
at
a
couple
of
conferences,
she
met
Dorothy
Graham
and
was
invited
to
write
a
chapter
in
the
book
Experiences
of
Test
Automation.
With
Dorothy
she
has
been
developing
the
Test
Automation
Patterns
Wiki
and
is
now
writing
a
story
book
about
the
patterns.
Technical Test Automation Challenges: Patterns and Solutions
TestAutomationPatterns.org
1
© Seretta Gamba and Dorothy Graham 2018
Tutorial objectives
understand some technical issues in test
automation
be aware of the issues and patterns in the Test
Automation Patterns Wiki
identify patterns that could help solve technical
issues
exercise using personas and the wiki
identify your own issues and patterns, share & discuss
explore automation solutions (more than in wiki)
be able to use the wiki for further solutions (after
the tutorial)
NOTE: system level automation, not unit level
NOTE: test automation patterns, not test patterns 2
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
About you
– show of hands
• job role (tester, test manager, automator, other)
• industry sector
– finance / government / embedded / software products / retail /
mobile apps / others
• automation experience
– none / < 6 months / 6 – 24 months / 2 – 5 yrs / > 5 yrs
• tools using / used (major ones)
– brought a laptop or tablet?
About you
– other people here today
• may have similar problems to you
• may use the same tools or tools you want to know about
• may have ideas to help you
• you may have knowledge that will help them
– introduce yourself to your neighbour / partner
• your name
• why are you here? your current problems / issues /
things you want to find out about
• company you work for
• your experience in (system-level) test automation
• tools you use(d)
4
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Issues (problems)
what problems affect you in test
automation?
what causes the most trouble?
what takes longer than it should?
what is holding you back in your
automation?
what are you struggling with?
what would you most like to improve?
(technical or management issues)
5
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 6
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
10
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 13
testautomationpatterns.org 14
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
15
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
17
18
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
21
22
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
23
BRITTLE SCRIPTS
• small changes to the application cause
large changes in the automation
– high maintenance cost
• Patterns:
– TESTWARE ARCHITECTURE
– ABSTRACTION LEVELS
– MAINTAINABLE TESTWARE
– MANAGEMENT SUPPORT
– GOOD PROGRAMMING PRACTICES
– COMPARISON DESIGN
24
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 25
TESTWARE ARCHITECTURE
Organisation of, and relationship
between, artefacts
scripts, input, test data, test descriptions,
expected results, actual results, log files,
etc.
When you create an automated test:
do you know what already exists that you
can re-use?
do you know where to put the artefacts?
(test materials)
do you know what names to use?
26
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
TESTWARE ARCHITECTURE
Organisation of, and relationship
between, artefacts
scripts, input, test data, test descriptions,
expected results, actual results, log files,
etc.
When you execute an automated test:
do you know where to find all the test
results?
and why they should be stored in a different
way to the test
do you know how to analyse a test failure?
27
LOCALISED REGIMES
28
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
ABSTRACTION LEVELS
• most critical factor for success
– worst: close ties between scripts, tool & tester
• separate testers’ view from technical aspects
– so testers don’t need tool knowledge
• for widespread use of automation
• scripting techniques address this
• separate tests from the tool – modular design
– likely changes confined to one / few module(s)
– re-use of automation functions
– for minimal maintenance and long-lived
automation 30
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
ABSTRACTION LEVELS
Data layer:
databases etc.
DB specialists
ABSTRACTION LEVELS
Testers:
Test cases
32
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 33
DATA-DRIVEN TESTING
• test data extracted from scripts
– placed into separate data files
• control scripts read data from data files
– one script implements several tests by reading
different data files (reduces script maintenance
per test)
• reduced build cost
– faster and easier to automate similar test
procedures
– many test variations using different data
• multiple control scripts required
– one for each type of test (with varying data)
34
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Data-driven example
Control script
For each TESTCASE
Data file: TestCase1 OpenDataFile(TESTCASEn)
ReadDataFile(RECORD)
FILE ADD MOVE DELETE
For each record
countries
Sweden ReadDataFile(RECORD)
Data file: TestCase2 Case (Column(RECORD))
USA
FILE ADD MOVE DELETE FILE:
4,1 OpenFile(INPUTFILE)
Europe
Norway
France ADD:
2 AddItem(ITEM)
Germany
7 MOVE:
1,3
MoveItem(FROM, TO)
2,2
DELETE:
1 DeleteItem(ITEM)
5,3 …..
Next record
Next TESTCASE 35
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
About keywords
• single control script (Interactive Test Environment)
– improvements to this benefit all tests (ROI)
– extracts high-level instructions from scripts
• ‘Test Definition’
– independent of tool scripting language
– a language tailored to testers’ requirements
Unit test: calculate one
• software design
interest payment
• application domain System test: summarise
• business processes interest for one customer
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
40
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
41
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
43
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 44
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Test definition
(for every automated test)
• name of test • measures
• purpose of test – expected run time (of the
automated test)
• materials/artifacts used – EMTE (equivalent
– scripts (shared & unique) manual test effort)
– expected results – others?
– data files
• test selector tags
• results produced – to run tests or groups of
• set-up instructions* tests
• execution instructions* – examples: smoke tests,
short tests, bug fix tests.
• tear-down instructions* long tests, specific
• related tests environment tests
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Exercise
in the Test Automation Patterns Wiki
go to Exercises
open Scenario 3
Work out all three exercises
The following exercises are used on tutorials
and workshops. You are welcome to try these
as well, but solutions are only given in the
tutorial or workshop.
Exercise
which Issue(s) did you find that
Ursula has?
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
51
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 52
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
57
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 58
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
59
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 61
TOOL INDEPENDENCE
• Separate the technical implementation
of the tests from tool-specific elements
– modular scripts
– keep tool-specific parts as small as
possible
• and self-contained to minimise change
– patterns include:
• TEST AUTOMATION FRAMEWORK
• OBJECT MAP
• GOOD PROGRAMMING PRATICES
62
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
63
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 64
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
COMPARISON DESIGN
Test sensitivity
• the more data there is available:
– the easier it is to analyse faults and debug
• the more data that is compared:
– the more sensitive the test
• the more sensitive a test:
– the more likely it is to fail
– (this can be both a good and bad thing)
66
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
test
outcome
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 70
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Known bug
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Agenda
Introduction
Issues and Patterns
Using the wiki
Patterns in depth:
TESTWARE ARCHITECTURE
DATA-DRIVEN vs KEYWORD-DRIVEN
DOCUMENT THE TESTWARE
AUTOMATE WHAT’S NEEDED
INDEPENDENT TEST CASES
TOOL INDEPENDENCE
COMPARISON DESIGN
EXPECTED FAIL STATUS
Conclusion 74
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
75
Discussion
76
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org
Technical Test Automation Challenges: Patterns and Solutions
Summary
• we looked at technical issues
– different ways of getting into the wiki
• issues, patterns, diagnostics
• we looked at technical patterns
• and additional information
• I hope this is / will be useful for you
• please help to improve the wiki by
adding your own comments &
experiences
77
Finally
any more questions?
please email me!
srttgmb@yahoo.com
presented by Dorothy Graham & Seretta Gamba © Dorothy Graham and Seretta Gamba 2018
info@dorothygraham.co.uk / srttgmb@yahoo.com www.DorothyGraham.co.uk
TestAutomationPatterns.org