Introduction Review of Related Literature Methodology

TECS: Test Essay Checking Software

Marlon Fernando, Efren Ver Sia, Cheryleigh Anne Verano and Prospero C. Naval, Jr.
Computer Vision & Machine Intelligence Group Department of Computer Science College of Engineering University of the Philippines-Diliman

July 8, 2013

1 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Outline
1

Introduction Problems Statement of the Hypothesis Objectives Review of Related Literature Existing Systems Concept Indexing Methodology Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis
2 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

3 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic.

4 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic. Evaluating essays can consume a huge amount of time and eort.

5 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic. Evaluating essays can consume a huge amount of time and eort. Teachers are challenged to be consistent on grading essays to ensure no bias and same standards are applied in all essays.

6 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic. Evaluating essays can consume a huge amount of time and eort. Teachers are challenged to be consistent on grading essays to ensure no bias and same standards are applied in all essays. Scores of human raters can be aected by certain factors like fatigue, mood, and distractions.

7 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Problems

Essay-type examination is a useful evaluation tool to fully capture a students knowledge on a certain topic. Evaluating essays can consume a huge amount of time and eort. Teachers are challenged to be consistent on grading essays to ensure no bias and same standards are applied in all essays. Scores of human raters can be aected by certain factors like fatigue, mood, and distractions. Large-scale assessments are usually limited to multiple-choice question type.

8 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Statement of the Hypothesis

TECS will be able to produce essay score and feedback that is almost as good as human evaluation results.

9 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

10 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue

11 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of eort and time consumption for grading multiple essays

12 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of eort and time consumption for grading multiple essays Able to give an instant feedback for an essay

13 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of eort and time consumption for grading multiple essays Able to give an instant feedback for an essay Provide a fast and memory-ecient algorithm for essay grading

14 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of eort and time consumption for grading multiple essays Able to give an instant feedback for an essay Provide a fast and memory-ecient algorithm for essay grading Introduce a breakthrough in education

15 / 49

Introduction Review of Related Literature Methodology

Problems Statement of the Hypothesis Objectives

Objectives

Provide a fast and more accurate essay scoring by eliminating human rater fatigue and inter-rater reliability issue Highly reduce cost of eort and time consumption for grading multiple essays Able to give an instant feedback for an essay Provide a fast and memory-ecient algorithm for essay grading Introduce a breakthrough in education

16 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Outline
1

Introduction Problems Statement of the Hypothesis Objectives Review of Related Literature Existing Systems Concept Indexing Methodology Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis
17 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

18 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

19 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing

20 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing
Starts with tokenization of words in an essay and the generation of Term-by-Document Matrices

21 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing
Starts with tokenization of words in an essay and the generation of Term-by-Document Matrices Instead of SVD, it uses Concept Decomposition (CD).

22 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing
Starts with tokenization of words in an essay and the generation of Term-by-Document Matrices Instead of SVD, it uses Concept Decomposition (CD). K-means Clustering VS. Fuzzy C-means Clustering

23 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing
Starts with tokenization of words in an essay and the generation of Term-by-Document Matrices Instead of SVD, it uses Concept Decomposition (CD). K-means Clustering VS. Fuzzy C-means Clustering In experiments that were conducted, both versions of CI outperform LSI in Exact Agreement Accuracy and Pearsons Product-Moment Correlation Coecient

24 / 49

Introduction Review of Related Literature Methodology

Concept Indexing

Concept Indexing
Starts with tokenization of words in an essay and the generation of Term-by-Document Matrices Instead of SVD, it uses Concept Decomposition (CD). K-means Clustering VS. Fuzzy C-means Clustering In experiments that were conducted, both versions of CI outperform LSI in Exact Agreement Accuracy and Pearsons Product-Moment Correlation Coecient In particular, CI with Fuzzy C-means achieves better results than CI with K-means in essay content evaluation, and CI with Fuzzy C-means is less sensitive to the preprocessing techniques such as stemming and stopwords removal and eventually, to noise.
25 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Outline
1

Introduction Problems Statement of the Hypothesis Objectives Review of Related Literature Existing Systems Concept Indexing Methodology Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis
26 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Procedures and Algorithms

27 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Procedures and Algorithms

Spelling Grammar Checker: GNU Aspell/LanguageTool

28 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Procedures and Algorithms

Spelling Grammar Checker: GNU Aspell/LanguageTool Content Analysis: Concept Indexing

29 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Procedures and Algorithms

Spelling Grammar Checker: GNU Aspell/LanguageTool Content Analysis: Concept Indexing Agile software development

30 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Equipment Specications

31 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Equipment Specications

a web hosting service subscription (approx. P200.00 per month) or a dedicated server with 2-4 GB RAM (approx. P6,000.00 per month) that runs Apache, Linux and PostgreSQL

32 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Equipment Specications

a web hosting service subscription (approx. P200.00 per month) or a dedicated server with 2-4 GB RAM (approx. P6,000.00 per month) that runs Apache, Linux and PostgreSQL Language: PHP(backend) and HTML/JS/CSS (frontend)

33 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Equipment Specications

a web hosting service subscription (approx. P200.00 per month) or a dedicated server with 2-4 GB RAM (approx. P6,000.00 per month) that runs Apache, Linux and PostgreSQL Language: PHP(backend) and HTML/JS/CSS (frontend) Framewok: Yii

34 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entity Relationship Diagram

35 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities

36 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
USERS:
1 2

Attributes: rst name, last name, middle name, and username Roles: 0 for admin, 1 for school administrator, 2 teacher, and 3 for the students The slug eld is composed of characters that will be sent to his/her mail for verication of accounts, and veried eld is to determine if the users already veried his/her account.

37 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
USERS:
1 2

Attributes: rst name, last name, middle name, and username Roles: 0 for admin, 1 for school administrator, 2 teacher, and 3 for the students The slug eld is composed of characters that will be sent to his/her mail for verication of accounts, and veried eld is to determine if the users already veried his/her account.

BASIC INFO: consists of users mobile number and his/her address: the street, municipality and province.

38 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
USERS:
1 2

Attributes: rst name, last name, middle name, and username Roles: 0 for admin, 1 for school administrator, 2 teacher, and 3 for the students The slug eld is composed of characters that will be sent to his/her mail for verication of accounts, and veried eld is to determine if the users already veried his/her account.

BASIC INFO: consists of users mobile number and his/her address: the street, municipality and province. CLASS INFO:
1

class name and class key separated from the class list of a teacher The user id eld is for the id of the teacher who created the class
39 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities

40 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
CLASS:
1 2

This table is for the students who will enroll for the class. It stores the user id of the students, and the class id of the class that was made by the teacher.

41 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
CLASS:
1 2

This table is for the students who will enroll for the class. It stores the user id of the students, and the class id of the class that was made by the teacher.

EXAMS INFO: This is where the essays of the students will be saved, and the date when the student passed his/her essays.

42 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
CLASS:
1 2

This table is for the students who will enroll for the class. It stores the user id of the students, and the class id of the class that was made by the teacher.

EXAMS INFO: This is where the essays of the students will be saved, and the date when the student passed his/her essays. RESULT:
1

It consists of the nal score given to the essay that has been passed by the students. It also has the feedback of the teacher if he/she wants to put comments in the essay of a particular student. And the exam id is from exam info where the body of essay resides.

43 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Entities
CLASS:
1 2

This table is for the students who will enroll for the class. It stores the user id of the students, and the class id of the class that was made by the teacher.

EXAMS INFO: This is where the essays of the students will be saved, and the date when the student passed his/her essays. RESULT:
1

It consists of the nal score given to the essay that has been passed by the students. It also has the feedback of the teacher if he/she wants to put comments in the essay of a particular student. And the exam id is from exam info where the body of essay resides.

EXAMS: This stores the user id of the student who passed an essay, an exam id where the body of essay resides, and the class id where the student is enrolled.
44 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Data Collection and Analysis

45 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Data Collection and Analysis

The proponents will ask at least four teachers in elementary or in high school for their evaluation on at least 50 student essays each.

46 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Data Collection and Analysis

The proponents will ask at least four teachers in elementary or in high school for their evaluation on at least 50 student essays each. We shall have two sets of essays: one will consist of technical or objective essay and the other set will consist of creative or subjective essays.

47 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Data Collection and Analysis

The proponents will ask at least four teachers in elementary or in high school for their evaluation on at least 50 student essays each. We shall have two sets of essays: one will consist of technical or objective essay and the other set will consist of creative or subjective essays. The teachers evaluation results will then be compared to the machines and will be subjected to Pearsons r for the analysis of their correlation.

48 / 49

Introduction Review of Related Literature Methodology

Procedures and Algorithms Equipment Specications Entity Relationship Diagram Data Collection and Analysis

Data Collection and Analysis

The proponents will ask at least four teachers in elementary or in high school for their evaluation on at least 50 student essays each. We shall have two sets of essays: one will consist of technical or objective essay and the other set will consist of creative or subjective essays. The teachers evaluation results will then be compared to the machines and will be subjected to Pearsons r for the analysis of their correlation. For additional training and testing data set, the proponents will also use essays from Kaggle.

49 / 49