Portfolio Project: Course Design Document

EDUC 765: Trends and Issues in Instructional Design

By: Jennifer Fett

Submitted 2/26/16

PROJECT PROPOSAL MODULE 2

Content Developoment Group: Creating Effective Digital
Assessments
Sponsoring Organization
Mason House Publishing, Content Development Group: Humanities Studio
Mason House Publishing is dedicated to providing a pathway to learning for students of all ages,
backgrounds, and abilities.

Project Description
In recent years, the development of valid, reliable assessments to accompany K-12 educational
content has become increasingly challenging, due to both changes in the rigor of state and
national assessments and the advances in the digital tools used to administer and score the
assessments. As a leading provider of K-12 educational content, customers expect MHP to meet
their changing needs and offer solutions that allow them to prepare their students for high-stakes
assessments and monitor their progress toward mastery of state and national standards. It is
therefore crucial for all Learning Designers in the Humanities Studio to have an understanding of
the current best practices in developing and delivering effective assessments for their ELA
content. Changes in the organization, customer complaints, and the use and implementation of
new digital tools to create content over the past year or more indicate the need for training. The
goals of the training include:
More consistency in the level of rigor and quality of the assessment content delivered to
customers
Better alignment of assessment content to both customer expectations and state/national
standards
Cost savings both in terms of MHP employee time and the frequency with which we hire
outside vendors for content development

Aim
Improve customer satisfaction by increasing the ability of Humanities Studio Learning Designers
to develop high-quality digital assessments and useful metadata (e.g., rubrics, rationales, text
complexity information, etc.) using the current digital toolset
(MasonCMS>Learnosity>MyMHP).

Target Audience
The target audience for the training is the Learning Designers (including Associate LDs and
Senior LDs) of the Humanities Studio who are either currently assigned to or may be assigned to
any English language arts program for the coming year. These LDs will be identified by the Lead
Learning Designers and Directors of the studio, and will be confirmed by the VP of the studio.
1

Delivery Options
The training will consist of a blended strategy, combining on-site/web-based synchronous
sessions with asynchronous training modules delivered online via CDGA (Content Development
Group Academy). The reason for this particular blend is that unless we block out time on LDs
calendars for the synchronous sessions, many are unlikely to be able to prioritize taking the time
for this training, especially for those who are not currently assigned to an English language arts
project. Furthermore, having training modules that are online and available at any time will be
helpful for those who need a refresher months after the synchronous sessions or for those LDs
hired after the synchronous sessions have taken place.

FRONT-END ANALYSIS: INSTRUCTIONAL NEED MODULE 3

Instructional Need
The Humanities Studio within MHPs Content Development Group has
identified a need to develop and train its Learning Designers in the creation,
development, and delivery of high-quality digital assessment content. The
need has arisen for several reasons: 1) recent reorganizations/restructuring
of the group coupled with the need to move LDs on and off of projects as
priorities shift means that any Learning Designer should be able to plan,
develop, and deliver an ELA assessment product, but the reality is that many
have no familiarity with either the new testing standards or the internal MHP
systems used to create assessment products, 2) there is a lack of
transparency surrounding the process/tools/practices/contacts/resources
used to create an ELA assessment product that has only increased in recent
years because of staff turnover and the implementation of new systems and
roles, 3) MHP has received complaints from customers about the level of
rigor of the assessments in the current basal program (Core Literacy), 4) the
increased importance of the program assessments means that they should
be tied to virtually every other aspect of the basal program, so that even if a
particular Learning Designer is not personally working on developing
assessment content, he or she should have an awareness of how the content
being developed will affect the assessments, and 5) the ever-changing
market for educational publishers makes it crucial for its content developers
(Learning Designers) to be well-informed and competent in producing what
the current market demands, which in this case is digital ELA assessment
products that match the level of rigor of current state and standardized tests
as well as the tests created by various educational consortia (PARCC and
SBAC assessments). While some LDs have experience either with the new
assessment standards, the legacy systems (ExamView, MHOAP, MasonCMS),
or both, there are far too few (3 out of ~60) who have knowledge of both as
well as a working knowledge of the new systems (Learnosity, MyMHP).

FRONT-END ANALYSIS: LEARNER CHARACTERISTICS

MODULE 3
Learner Analysis
Primary Audience
Associate Learning Designers currently assigned or with the
potential to be assigned to an ELA project
Learning Designers currently assigned or with the potential to be
assigned to an ELA project
Senior Learning Designers currently assigned or with the potential
to be assigned to an ELA project
Secondary Audience
Lead Learning Designers in ELA division of the Humanities Studio
Humanities Studio Directors
General Learner Characteristics
Ages 2865
65% female; 35% male
Education: all hold at least BA; many hold graduate degree(s)
Work experience: 3 months to 20+ years
All are conscientious people concerned with helping students and
teachers facilitate the learning process
Entry Characteristics
Prerequisite skill and knowledge: all are familiar with traditional
assessments, most are aware of change in market trends in
assessments, few are familiar with current market requirements and
standards, few are familiar with current MHP assessment platforms
Attitudinal and motivational characteristics: LDs with previous
assessment experience may not feel as though they have much to
learn, many may be resentful of having to take time away from busy
schedules for training with little perceived immediate value (for
those not assigned to assessment projects), some learners may be
motivated to take the training because they feel it will help them
understand their own K-12 kids test preparation needs
Prior experience: some have used ExamView to create assessment
content, some have used MHOAP to create assessment content, few
have used MasonCMS/Learnosity/MyMHP

Contextual Analysis
Orienting Context
LDs want to know how to use the newer systems so that the learning
curve will not be so great if/when they are assigned to an assessment
project.
LDs who are not currently assigned to an assessment project may not
see the utility of the training, those with some experience using legacy
systems may perceive slightly less utility in the overall training, but will
be motivated to learn about the new systems.
LDs not currently assigned to an assessment project will have a low
perception of accountability; those assigned to a current or upcoming
assessment project will have a high perception of accountability.
Potential misconceptions include: the new tests are not that different
from traditional tests; the systems are not that different from one
another, creating an assessment product only requires the use of one
platform or system, and the studio relies on outside vendors and/or the
Assessment Studio to create its assessment content.
Instructional Context (On Site)
Recommended scheduling is for 3 synchronous sessions in the
afternoons (14:30) of March 79, and 2 asynchronous modules to be
completed during the week of March 711
Lots of natural light with ability to lower blinds
Conference rooms are quiet
Temperature tends to be cool; dont have ability to adjust
Can arrange seating to preference (open-ended square, facing
projection screen)
Accommodations: N/A (normal working day)
Conference rooms have overhead projector and screen
Transportation: N/A (normal working day)
Instructional Context (Web Meeting)
Same as above with the exception that temperature, noise, and
seating are not able to be controlled (some employees will be calling in
from home)
Technology Inventory
All LDs have access to Mac computers; account credentials will be
given to all LDs for both MasonCMS and Learnosity
Transfer Context
Upon completion of the training, all LDs will have access to both
MasonCMS and Learnosity.
5

Opportunities for using learned information will vary depending on

each LDs current assignment; Directors and LLDs should plan
accordingly to rotate LDs assigned to assessments for the new
programs.
Upon completion of the training, all LDs will have access to all 4
training modules (including recorded training materials presented
during synchronous sessions), the sample test specification
documentation and resources, user guides for MasonCMS and
Learnosity, as well as contact information for experts in each and an
organizational chart.

INSTRUCTIONAL IMPACT BASED UPON LEARNER

CHARACTERISTICS
Application of Learning Theories
Based on the principles of andragogy, the following will be applied to the
training:
Objectives for the training will be shown at the beginning of the
training itself, and again at the beginning of each module. We will take
time for questions about the objectives so that every participant feels
they have a good understanding of what it is they will be learning.
The training will include an introductory video on the impact of highstakes assessments on students lives. This video will show how not
being prepared for high-stakes assessments can negatively impact
students, teachers, schools, and school districts, leading to lower
grades, decreased funding, and limited access to needed diagnosis and
remediation of academic performance problems. This will provide a
meaningful context for the LDs to see that the needed work is
important to real students and teachers and is not just a requirement
for us to sell product.
During the afternoon of the first session, LLDs will spend roughly 30
minutes explaining to LDs both their understanding of the importance
of the studios ability to create effective digital assessment content as
well as their vision for how the studio will use the new assessment
tools so that every LD, whether currently assigned to an assessment
project or not, understands the importance of mastering the
knowledge and skills required to create effective assessment content
and that he or she will most likely be called to work on an assessment
project at some point in the next development cycle.
An agenda will be provided for each session of the training that clearly
shows time allotted for activities, questions, and breaks. We will keep
breaks to 10 minutes in length so that LDs dont go back to their desks
and start working during breaks. Healthy snacks and drinks will be
provided at the halfway point on each day of the training.
Aside from the introductory presentations and discussion, the video
clips, and scheduled Q & A sessions each day, the training will be
largely self-directed. LDs will be broken up into small groups to
complete the synchronous modules activities. After the training
sessions have concluded, each LD will have the rest of the week to
complete the two asynchronous training modules to be submitted first
to the LLD for review and then to the training facilitators for evaluation
and feedback.

Application of Motivational Theories

In as much as possible, the ARCS theory of motivation (Pappas, 2015) will be
applied throughout the training. The following are some broadly based
applications:
Attention: the video mentioned in the previous section (see second
bullet under heading Application of Learning Theories) will be
important in gaining the trainees attention and providing a real-world
context to make the training more meaningful to the trainees. The
presentation by LLDs will also help to underscore the importance of the
training.
Relevance: in order to create a sense of continuity, participants will be
asked to complete a survey prior to beginning the training in which
they recall any prior projects for which they worked on creating
assessments as well as any other relevant item-writing experience. The
survey results will be tallied and displayed for the group during the
training, and one of our introductory discussions will be about trainees
prior experiences creating assessments, either at MHP or another
employer.
Confidence: feedback and encouragement will be provided to LDs
throughout the synchronous training sessions, especially for those who
have expressed aversion to using new technological systems
(identified before the training by LLDs and by survey results); in the
first required asynchronous online training module, there will be a
series of sample assessment items in which the trainee will have to
determine the number and nature of problems with each item. This
interactivity will involve instant feedback (in the form of a green check
mark or red x and a brief bit of voiceover explaining the rationale
behind the correct/incorrect answer(s).
Satisfaction: The aim of the introductory video and discussion sessions
is to provide trainees with true motivation for completing the training
and continuing to develop their skills in crafting and developing highquality assessment products for students. If the motivation is there,
then satisfaction at a job well done should be strong. The level of
satisfaction should be even higher for those LDs who are either
currently assigned to or will be assigned to assessment projects in the
next quarter or two because they will see the immediate application of
the things theyve learned during the training.

Impact of a Diverse Audience on Instruction

Though there is a relatively limited amount of cultural diversity among the
LDs who will be the trainees for the initial group of sessions, it is possible
that the modules posted online to CDGA will be used to train outside vendors
as well where the likelihood of having a much more culturally diverse
audience is greater. Because of that, the following considerations will be
included in the training:
8

Versions of the in-person/web-based synchronous training sessions will

be made into videos that will be posted alongside of the rest of the
self-paced training modules. In these videos, versions of the small
group activities will be demonstrated so that students can watch and
listen as a group of people works through each step of the tasks.
A glossary of all terms will be posted with the CDGA materials. The
glossary will include links to examples where applicable (e.g., the entry
for rubric will include links to example rubrics used by PARCC and
SBAC).
For internal MHP staff, a Humanities Studio Assessments intranet
group will be created so that LDs can ask questions of their colleagues,
share project documentation and lessons learned, and have an active
dialogue about the processes involved in creating and delivering
assessment content.

We are fortunate to have such clearly articulated standards (Common Core

State Standards for ELA/Literacy) for which LDs and vendors will be writing
content. However, there is still room for interpretation in those standards. If
the budget allows, we may also create something similar to our Close
Content app, in which the text is read aloud and highlighted on screen, after
which the people reading have a conversation about what theyve just read.
We could create a version of this in which we record conversations for each
of the Common Core State Standards for ELA/Literacy and show an example
item aligned to each standard to cut down on the room for interpretation of
those standards.

TASK/GOAL/PERFORMANCE ANALYSIS MODULE 5

Goal Analysis
Step 1 - Write down the goals.
Original goals:
Learning Designers are able to plan, develop, and produce high-quality
next generation assessment products using MHPs content
management system (MasonCMS) and the agreed upon third-party
platform (Learnosity) for ultimate delivery in our go-forward platform
(MyMHP).
Step 2 - Write down everything a learner would have to say or do for
you to agree that the learner has achieved the goal. This is not a list
of what you will need to do as the instructional designer or teacher.
explain the key differences between traditional assessments and next
generation assessments
demonstrate a belief in the importance of producing high-quality next
generation assessment products by successfully completing all of the
training modules within the allotted time
gather, evaluate, and synthesize test specifications from a state or
consortium (SBAC or PARCC) into a blueprint for the construction of
assessment units or modules
using the completed blueprints, prepare an estimate of content
development costs for finance
ensure that the final costs of development do not exceed the original
budget estimate provided to finance (unless previously approved by
LLD)
identify opportunities for content repurposing and/or come up with a
strategy to obtain content in the most cost effective manner in an
effort to reduce the costs of the project
collaborate with the key stakeholders in each of the following groups:
Development Solutions (MasonCMS/Learnosity), Product Ownership
(MyMHP), Text Permissions & Licensing, Audio Visual Licensing, Image
Services, Finance, and Project Management
produce a complete set of content development guidelines made
available to all CDG and vendor partners via Box
collaborate with CDG and vendor partners to create high-quality
content that meets the specifications in the content development
guidelines
work effectively with internal and external vendor partners to meet the
specified deadline
sign off on all content in MasonCMS and provide item codes for export
into Learnosity to Development Solutions

10

collaborate with Development Solutions/MyMHP product owners to

create activities that provide a user experience that is compatible with
MyMHP user interface
review, revise, and replace content as needed in Learnosity
include all relevant acknowledgment information in Learnosity
meet market requirements set forth at the beginning of the project
include relevant metadata to be used in reporting
include answer rationales for all EBSR and TEI items
include item-specific rubrics for short constructed response items
include generic multi-domain rubrics for performance tasks and essays
instruct QA teams in performing quality checks in MyMHP
provide QA teams with necessary content inputs
monitor JIRA for bugs related to the completed product in MyMHP and
respond in a timely manner

Step 3 - Sort the items Listed in step 2.

Learning Designer Competencies
gather, evaluate, and synthesize test specifications from a state or
consortium into a blueprint for the construction of assessment units or
modules
using the completed blueprints, prepare an estimate of content
development costs (including permissions costs) for finance
provide a complete set of content development guidelines to all CDG
and vendor partners
collaborate with CDG and vendor partners to create high-quality
content that meets the specifications in the content development
guidelines
revise (as needed) and finalize content and metadata in MasonCMS
and move to Complete (READ ONLY) step
review all content and verify all metadata (including
acknowledgments) once in Learnosity
Project Management
maintain timely communication with the following groups:
Development Solutions (MasonCMS/Learnosity), Product Ownership
(MyMHP), Text Permissions & Licensing, Audio Visual Licensing, Image
Services, Finance, and Project Management
communicate any budget changes to finance in a timely manner and
receive LLD approval for any difference greater than ($10,000)
gather and enter all relevant acknowledgment information from TPL
and AVL into MasonCMS

11

provide test maps with item codes for export into Learnosity to
Development Solutions
provide QA teams with necessary content inputs and instructions for
performing quality checks in MyMHP
respond to assigned JIRA bugs in a timely manner

Product Quality
provides an authentic next generation assessment experience
meets the market requirements set forth by Product Marketing and the
Vice President of the Humanities Studio
includes all relevant metadata for use in scoring, reporting, and
progress monitoring
Step 4 - Write a complete sentence to describe each of the items on
your final list.
Learning Designer Competencies
The LD has successfully gathered, evaluated, and synthesized test
specifications from a state or consortium into a blueprint for the
construction of assessment units or modules.
Using the completed blueprints, the LD has prepared an estimate of
content development costs (including permissions costs) for finance.
The LD provided a complete set of content development guidelines to
all CDG and vendor partners.
In collaboration with CDG and vendor partners, the LD has created
high-quality content that meets the specifications in the content
development guidelines.
The LD has reviewed/revised/finalized all content and metadata in
MasonCMS by moving it to the Complete (READ ONLY) step.
The LD has verified that all content and metadata has been imported
correctly into Learnosity.
Project Management
The LD has maintained timely communication with the following
groups: Development Solutions (MasonCMS/Learnosity), Product
Ownership (MyMHP), Text Permissions & Licensing, Audio Visual
Licensing, Image Services, Finance, and Project Management.
The LD has communicated any budget changes to finance in a timely
manner and has received LLD approval for any difference greater than
($10,000).
The LD has gathered from AVL and TPL and entered all on-screen
acknowledgment information.
The LD has provided test maps with item codes for export into
Learnosity to Development Solutions.

12

The LD has provided content inputs and instructions to QA teams

performing quality checks.
The LD has responded to assigned JIRA bugs in a timely manner.

Product Quality
The final assessment product provides an authentic next generation
assessment experience.
The final assessment product meets the market requirements set forth
by Product Marketing and the Vice President of the Humanities Studio.
The final assessment product includes all relevant metadata for use in
scoring, reporting, and progress monitoring.

13

INSTRUCTIONAL OBJECTIVES MODULE 5

Project (Instructional) Goal
Learning Designers in the Humanities Studio will plan, develop, and deliver
high quality next generation assessment products that meet market
requirements and provide a cohesive user experience in MyMHP. The
Learning Designers will make use of best practices to stay within project
timelines and budget, to maintain open lines of communication with the
relevant cross-functional teams, to provide all necessary content input and
output data, and to maintain customer satisfaction by responding to ongoing
issues as reported in JIRA.

Terminal Objectives and Enabling Objectives

Explain how creation of assessment practice and preparation products

for next-generation assessment is aligned with company values and
mission (understand cognitive)

Demonstrate understanding of rigor and content requirements of nextgeneration assessments by successfully completing two item-writing
modules through CDGA (apply cognitive)

Conduct research into relevant state or consortia test development

specifications, integrating the requirements into blueprints for each
grade level (create cognitive)

Write test development specification document(s) for vendor/internal

content development partner (create cognitive)

Formulate high-level plan for sourcing of passages/media, including

repurposing of legacy content where possible (evaluate cognitive)

Create budget estimate for Finance based on information from TPL,

AVL, and VRM (if applicable) (create cognitive)

Collaborate with cross-functional teams to facilitate workflow steps

from MasonCMS>Learnosity>MyMHP (apply cognitive and affective)
o Collaborate with Project Management, Development Solutions,
and TPS to agree upon workflow steps and create realistic
milestone schedule (apply cognitive and affective)
o Collaborate with Development Solutions group to outline and
construct the assessment project structure in MasonCMS (apply
cognitive)

14

o Create assignments per grade level in MasonCMS for internal or

external content development vendors (apply cognitive)
o Explain to vendor partners the process of manuscript >
MasonCMS review steps so that they understand each step
required of them (understand cognitive)
o Manage content quality in MasonCMS by reviewing and providing
clear and timely feedback per passage/item set to vendor
(evaluate cognitive and affective)
o Collaborate with Learnosity Product Owner to determine and
request any needed enhancements for the project (analyze
cognitive and affective)
o Collaborate with Development Solutions to create activity
templates in Learnosity that reflect agreed-upon features and
user experience (apply cognitive)
o Perform content quality checks in Learnosity so that the final
product is 95% error free (analyze cognitive)
o Liase with MyMHP team to plan for integration of assessment
activities from Learnosity into MyMHP assessment dashboard
(apply cognitive and affective)
o Work to ensure quality of end-user experience of MyMHP
assessment content by creating bugs in JIRA for any existing
issues that fall within the domain of Learning Designerure
(evaluate cognitive)
o Work with cross-functional team to ensure quality of end-user
experience of MyMHP assessment content by responding to all
assigned bugs in JIRA (analyze cognitive)

15

REFERENCES
Clark, D.R. (2004). Blooms Taxonomy of Learning Domains: The Cognitive Domain. Retrieved
from http://www.nwlink.com/~donclark/hrd/bloom.html
Morrison, G.R., Ross, S.M., Kalman, H.K. (2012) . . . Designing Effective Instruction (7th ed.).
Retrieved from https://reader.texidium.com/dist/#/book/205
Pappas, C., (2015, May 20). The Quintessential Of The ARCS Model of Motivation. Retrieved from
http://elearningindustry.com/instructional-design-models-and-theories

Smith, M. K. (1996; 1999, 2010) Andragogy, the encyclopaedia of informal education.

Retrieved from http://infed.org/mobi/andragogy-what-is-it-and-does-it-help-thinkingabout-adult-learning/

16