Introduction

At the direction of the PNBOA Board of Directors, the Ad-Hoc Committee on Evaluations has spent approximately the last 60 days reviewing the current evaluations program and system, in an effort to find ways to improve and maximize the member experience. During the process, the committee, has met 4 times as an entire group, chartered 3 sub-groups to address very specific issues within the system, and conducted an online survey to the membership. The information below represents the committees findings, analysis and recommendations for improvements. It should be noted that the committee strongly believes that this is merely the 1st phase of a multi-phase approach. Given the time considerations, it is unrealistic and potentially harmful to make too many changes without thoroughly understanding the implications. Therefore, you will notice the committee has recommended this team be extended for a 2-year period to continue to work and refine the process going forward.

PNBOA Ad-Hoc Committee on Evaluations Board Recommendations August 18, 2010

A BIG thank you from the committee to all those who participated in the survey, provided the committee with recommendations through the BLOG and by other means. A special thanks to Joe Thompson, Scott Fetterly, Ira Dunbar, any many others who provided input into structure and process. We appreciate your dedication and passion to the process.
Committee Members Lezley Smith Bob Ames Matt Morales Matt Wong Jason Capps Dave Upthegrove Charlie Gorman John Lindsey Andy Reiswig- Chair Preliminary Findings The online survey conducted during July was a useful indicator of sentiment within the membership. Albeit, only a fraction of the entire membership participated, there were many interesting nuggets that we were able to take away from it, and formulate recommendations from. Here are is a brief summary:

Total responses were 104 Varsity: 56%, Registered: 38%, Apprentice: 6% 1-10 years of service in the association accounted for 66% of the respondents 70% indicated they were very familiar with the evaluation process 52% believe the system works but needs improvement 57% indicated desire for some form of peer evaluation 58% believe there should be separate observation forms for 2 and 3 person mechanics 64% believe that the most important purpose of the evaluations system is to improve as an official

Fact Finding & Committee Direction The committee was charged with reviewing and understanding the current system, procedures that work well and procedures that dont work well. After completion of this step, the committee was broken down into 3 sub-groups to address certain specific areas, they include: Reviewing changes to the physical form Scoring system approach Observer training These 3 areas were selected because the committee believes these areas are largely able to be revised and changes implemented in time for the upcoming 10-11 season. Preliminary Recommendations The recommendations have been separated into 2 categories: General & Specific recommendations. A 3rd category identified as Future Considerations will speak to other areas where the committee believes there is opportunity, but not either a priority or that there is confidence it can be implemented prior to the upcoming season. General Recommendations: 1. Recommend implementing a revised Evaluations program over a 2 -year period. The committee recommends that it remain in place during this time to continue to evaluate and oversee. 2. Recommend that the program goals and objectives be specifically outlined to membership to include that the program is largely about professional development and improvement. 3. Recommend developing and implementing a pilot peer evaluation program. The pilot program would only be used to collect data during the upcoming season, and would not be used for the purposes of calculating rankings. Upon completion of the pilot program, the committee would review and determine its effectiveness long term. 4. Recommend developing and implementing a revised training program for observers and varsity officials to be conducted by persons to be identified prior to the season. The committee recommends the following elements be incorporated: a. Re-define what minimum requirements observers must demonstrate in order to be an observer b. Formulate a contract/PSA for observers to sign c. Implement an In-season observer feedback process d. Conduct an observer search e. Implement a post season survey for member feedback on observers

f. Ensuring consistency in training between paid observers and varsity officials 5. Recommend a revised scoring system approach to include scoring range brackets (see details at the end of this presentation). The scoring range bracket allows the observer to check a range of scores based on their observation without giving a specific score to the official. Along with comments, this approach will allow the observer to maintain subjectivity in the score, but yet provide the official with a range that they should expect to be scored for that particular game. 6. Recommend that the Board of Directors research and evaluate the hiring of an independent Evaluations Administrator, who would be responsible for overseeing and implementing the observation program. This position would have direct oversight of hiring observers, coordinating with assignors on scheduling, managing training and tabulating results. 7. Recommend that the Evaluations Manuals be re-written and condensed into a single simpler manual. 8. Recommend leaving existing officials classifications as is for now. However, the committee recommends eliminating the Registered- No Evaluations category. Specific Recommendations: 1. Recommend revising and implement new observation forms to include a two person observation for and a three person observation form. The committee strongly believes that there are substantial differences in how the games should be observed, and that tailoring separate forms will improve the ability of the observer to provide proper feedback. 2. Recommend retaining the First 5 observations that count for varsity officials. The committee discussed switching to a Top 5 format, however it was determined that retaining the current approach with the ability to drop the lowest score was appropriate. 3. Recommend revising the JV observations that count program by changing the format as follows; JV officials must have a minimum of 8 observations. For ranking purposes, the Top 8 shall count. However, if officials have substantially more than 8, only the bottom 25% of those observations will dropped. 4. Recommend first time varsity officials with protected status, shall not be required to observe. 5. Recommend apprentice officials shall be observed using the proper observation form, and shall be scored, however the scores will not be counted. 6. Recommend eliminating the opt out program by which officials can choose not to be observed. Future Considerations: (Note: these are not necessarily recommendations, but potential topics of discussion as this process continues to develop.) 1. Consider revisions to the officials classifications, by implementing a 2-tiered varsity list V1, V2, and remove the numbered rankings. V1 officials would be eligible for playoffs, V2 officials would not be. 2. Consider providing paid observation program for registered officials. 3. Consider re-instating a mentor/big brother program, with incentives for those varsity officials who participate.

4. Consider a fee for use approach to the paid observation program. Allowing officials to pay a fee to participate in the program, which could help offset administrative costs. Scoring Bracket Approach: The following demonstrates the scoring bracket approach being proposed in which the observer would check a box to a particular range of scores. This would be on the bottom of the observation form provided to the official. Overall Evaluation Scoring Range EXAMPLE ONLY Score: 98-100 Professional: This official demonstrates all of the characteristics of a notable official, but also demonstrates leadership and control even in distress. This official elevates the performance of the crew and the integrity of the game. Score: 95-97 Notable: This official demonstrates great judgment, mechanics, and communication skills along with leadership qualities. This official can be trusted to work a difficult game at the highest level. Score: 91-94 Believable: This official demonstrates proficiency with judgment, mechanics and communication, and has some leadership qualities. This official is capable of handling most situations, while being a good partner for the crew. Score: 87-90 OK: This official has a general understanding of rules, mechanics and judgment, but there is opportunity for improvement. This official is comfortable without being in a leadership position, and may not improve the performance of the crew. Score: 80-86 Average: This official does just enough to get by. While they possess adequate mechanics, rules knowledge and judgment, they do not elevate the performance of the crew. Summary The committee believes strongly that substantial opportunity exists in providing a more consistent, fair, and transparent evaluations process. We believe that by enacting the recommendations above, and considering future enhancements, this will ensure that the PNBOA continues to evolve and adapt to the needs of our members, while maintaining our status as the premier officials organization in the Pacific Northwest. Respectfully submitted August 18, 2010 PNBOA Evaluations Ad-Hoc Committee