1.Introduction
1.1Purpose
The purpose of this project is to implement new reviewer-mapping strategies into the already existing PG system.
1.2Scope
The scope of the assignment is in three files which work together in the PG system. They are:
GenerateReviewers.java.
Within this class is an existing function called createReviewersTable() which is where the bulk of the work will be done.
GenUserTable.java
A class used by GenerateReviewers.java
GenUser.java
A class used by GenerateReviewers.java
We will be modifying the existing code to implement new reviewer mapping strategies for the PG system.
1.3Background
In a nutshell, PG is a peer evaluation tool. All background information on PG can be found at
1.4Documentation
The documents that we will be editing to implement new reviewer-mapping strategies are the files covered in section 1.2 above. Theseare the classes that deal with generating the reviewer mappings.
1.5Standards
We are not aware of any standards that exist for this project other than the format of the code. We will follow the existing format of the code and update any documentation in the code to reflect our changes, as well as adding any new documentation that is required.
1.6Other documents
We will need to conduct research on graph-theoretical solutions to problems.
1.7Algorithms
The algorithms we are current usig are as follows:
- Really Random
Get all users
Create an array with the users
Offset=1 (End of array –1)
Move current user’s ID to the end of the array (offset++)
While num_of_reviewers_chosen < n
Randomly select a number between 0 and size–offset
Add chosen user to review list
Move id to end of array
Offset++
Num_of_reviewers_chosen++
The algorithm above ensures that the user isn’t chosen to review him/herself adnthat no reviewer is chosen twice. By moving the Ids to the end of the array we eliminate the need to check to see if a user has already been assigned, thereby making the algorithm run quicker.
- User Selected
Open current reviewer file
Whle not EOF
Is current user available and not myself?
Yes -> Add to selection menu
User selcts people to review and submits
Update reviewer file
Here, we simply open the output file that holds all of the user mappings and add all of the people available to be reviewed to a menu for the user to choose. This way we don’t have to worry about choosing an invalid reviewer.
- On The Fly
Open current reviewer file
Whle not EOF
Is current user available and not myself?
Yes -> Add to reviewers list
Update reviewer file
As you can see, this is very similar to user selected, except that the system chooses the first available reviewers that aren’t the current user.
2.System Architecture
Proposed Changes
Figure: 1
Proposed Changes
Figure: 2
Proposed Changes
Figure: 3
2.1 System Requirements
As can be seen from the above diagrams, our code will need to interface smoothly with the existing PG system. The only real requirement we have is that the system must use whatever mapping strategy is chosen, and that mapping strategy must produce the desired results. The mapping strategies are covered in more detail in section 2.2.2.
2.2 System Requirements
2.2.1Introduction
This section lists the summary of requirement for the project only. It does not discuss overall product requirements.
The goal of this project is to implement new reviewer-mapping strategies. The new strategies that will be implemented are:
- Really Random
- User Selected
- On-the-fly
Each of these is covered in more detail in section 2.2.2.
2.2.2Summary of Product requirements:
Several new reviewer-mapping strategies need to be implemented in PG. The mapping strategies we plan to implementare as follow:
- Really random reviewer mappings
Currently, PG can map reviewers "randomly," which means that any student is equally likely to be assigned to review any other author. However, if the classlist stays the same over the course of several assignments, PG will produce the same reviewer mapping each time. There should be a way to get different mappings for each assignment, perhaps by using ahash of the assignment name as a random-number seed.
- Self-selected reviewers
In this scheme, a student seeking to review an author is allowed to examine all of the submissions and choose a certain number of them to review (e.g., each student reviews n=3 other students). Only n students would be allowed to review any given student. The idea behind this is that it is easier to get qualified reviewers for a submission if students are allowed to choose which to review.
- On-the-fly reviewer mappings
This scheme is like self-selected reviewers, except that PG, not the student, chooses which assignments the student will review. It is already implemented for reviews of reviews, so it should be possible to "borrow" the code. This insures that no student will be assigned
to review someone who has not submitted the homework
In the last two schemes, it is necessary to assure that the algorithm does not "paint itself into a corner" by getting into a situation where a student must be assigned to review him/herself. We will pursue a graph-theoretic solution if possible.
These strategies will be transparent to the user in that they will not choose which one is being followed. The administrator will choose this in this version by editing a variable. In later versions, there will be an interface for the administrator to choose the current mapping strategy.
2.3Summary of Testing Requirements
This section summarizes all testing requirements.
- Test1:
"Really random" reviewer mappings test requirement:
Check that reviewer assignment changes with each assignment. Currently, reviewer assignment is dependent on the class list. If the classlist does not change, the reviewer mapping does not change either. Check that it varies withassignment name and not justclasslist.
- Test2:
Self-selected reviewers test requirement:
In this scheme a student is allowed to review n=3 other students only. Check that a student can not review more than the specified number. Check that a student is allowed to choose from a list of submissions and chooses a certain number of them to review. Make sure that we do not get into a situation where a student is assigned to review him/herself. Verify that an assignment is reviewed by only a preset number of reviewers.
- Test3:
On-the-fly reviewers mappings test requirement:
Test requirements for this case are similar to requirements of "Self-selected" reviewers mapping case. One major difference is that PG selects assignments the student will review. To test this case make sure that no student will be assigned to review someone who has not submitted the homework. Also ensure that we do not get into a situation where a student must be assigned to review him/herself.
2.3.1Requirements - Testing Cross Reference Matrix
- "Really random" reviewer mappings -Ref. Test1
- Self-selected reviewers -Ref. Test2
- On-the-fly reviewer mappings -Ref. Test3
2.4Interface requirements
Our work is part of an already existing product. It interfaces with other modules in the system. Some of the modules are: Reviewer database and Assignment database. Our module refers to the reviewer database to pick up a reviewer for an assignment. It also needs to access the assignment database to pick an assignment to be assigned to reviewer.
2.5 Quality Requirements
We are making changes to already existing system. We plan to add new code and make changes to some existing code. It is necessary that all quality goals are kept in mind. Quality requirements for the proposed PG system changes are as follows:
- New functionality must not adversely affect product performance in terms of response time
- Navigation within the system must not deviate from existing standards
- New product must be as reliable as the old one
- All changes must be properly documented
- Programming standards must be adhered to
- Product change standards/practices must be followed
2.6Portability Requirements
PG is a web-based system. We have to make sure that it does not become dependent on one platform. It should have the look and feel of the platform on which it is run. Since the code is in Java, it should port nicely. Also, since we will be doing little or no GUI work, all of our code should run on any platform that will run Java.
2.7Performance Requirements
Response time of PG system is reasonably good. In our project we are adding some new code and modifying some existing code. Our goal is to ensure that new/modified code does not impact the current performance level of PG.
2.8Assumptions
Right now, we don't have a very clear picture of the architecture of the PG system. Interfacing with other modules may not be as per our proposal. It is possible that instead of keeping all the student records in a local database PG accesses some other central database files. Also it is not very clear how the PG system chooses assignment lists and assignment names, so we may have to redefine some of our assumptions later.
3. Details of Requirements
3.1Requirement 1: Really random reviewer mappings
The functional requirement for the first reviewer-mapping strategy is as follows: given a list of students (classList) and a list of authors to be reviewed (authorList) or a list of submissions, (where authorListclassList), create a random mapping from students to authors (reviewerMapping) such that even if classList == authorList, (1) a different reviewerMapping is produced for each assignment, and (2) a student/author will not be assigned to review their own work.
3.1.1 Inputs
The inputs to the functional requirement are as follows:
- classList
Description: the list of students in the class
Type of input: An array of student ID’s
Range of input values: all valid student ID’s
Source of input: class list or reviewer database
Major components of the input:
- authorList
Description: the list of students who have submitted work to be reviewed
Type of input: An array of author ID’s or submissions
Range of input values: all valid author ID’s or submissions
Source of input: Probably taken from class list or submission database
Major components of the input:
- assignmentName
Description: the name of the assignment that is being reviewed
Type of input: a string of alphanumeric characters
Range of input values: any alphanumeric string
Source of input: Probably from course instructor or assignment database
Major components of the input:
3.1.2 Processing
3.1.1
A reviewerMapping shall be derived for the inputs. The assignment name shall be hashed and used as a random-number seed to produce a “really” random mapping. In addition, no single student in the classList will be mapped to his/her own ID in the authorList or their submission when the final reviewerMapping is derived.
3.1.23.1.3 Outputs
The outputs of the functional requirement are as follows:
- reviewerMapping
Description: the final mapping of students to authors (or submission by authors)
Type of output: An array of student ID’s and authorID’s(or submissions)
Format of output: An array of student ID’s and authorID’s (or submissions
Components of the input:Range of output values: all possible combinations of studentIDs and authorIDs or submissions except sudentID == authorID
3.1.4Special Considerations
3.1.3None at this time. We will try to produce code that is as short and quick as possible.
3.23.2 Requirement 2: Self-selected reviewers
The functional requirement for the second reviewer-mapping strategy is as follows: given a student (reviewer) from the classList and a list of authors to be reviewed (authorList) or a list of submissions, the student should be allowed to select a number of authors (or submissions) from the authorList to review (studentReviews). The number of reviews (numStudentReviews) should not be greater than a preset maximum(maxStudentReviews). An author or submission can only be selected for review by the student if its current number of reveiwers (numReviewers) is less than a preset maximum number (maxReviewers), and if author student.
3.2.13.2.1 Inputs
The inputs to the functional requirement are as follows:
- student
Description: a single student from the classList
Type of input: a student ID from the classList
Range of input values: any valid student ID
Source of input: class list or reviewer database
Major components of the input:
- authorList
Description: the list of students who have submitted work to be reviewed
Type of input: An array of author IDs or submissions
Range of input values: all valid author IDs or submissions
Source of input: Probably taken from class list or submission database
Major components of the input:
3.2.23.2.2 Processing
The student shall be allowed to select a number of authors/submissions from the authorListor submission list to review, up to the maximum number allowed. This subset of submissions (studentReviews) will be assigned to the student by PG unless one of the submissions is the student’s own (i.e. student == author) or if the submission/author already has the maximum number of reviewers (maxReviewers).
3.2.33.2.3 Outputs
The outputs of the functional requirement are as follows:
- studentReviews
Description: the final mapping between the student and items in the authorList or submission list
Type of output: a list of author Idsor submissions assigned to the student for review
Format of output: valid author Idsor submissions
Components of the input:
Range of output values: a subset of authorIDs or submissions up to maxStudentReviews
3.2.43.2.4 Special Considerations
None at this time. We will try to produce code that is as short and quick as possible.
3.2.5
3.33.3 Requirement 3: On-the-fly reviewer mappings
The functional requirement for the third reviewer-mapping strategy is as follows: given a student (reviewer) from the classList and a list of authors to be reviewed (authorList), the PG system will select a number of authors from the authorList for the student to review (studentReviews). The number of reviews (numStudentReviews) should not be greater than a preset maximum(maxStudentReviews). An author or submission can only be assigned to the student for review if its current number of reveiwers (numReviewers) is less than a preset maximum number (maxReviewers), and if author student.
3.3.13.3.1Inputs
The inputs to the functional requirement are as follows:
- student
Description: a single student from the classList
Type of input: a student ID from the classList
Range of input values: any valid student ID
Source of input: class list or reviewer database
Major components of the input:
- authorList
Description: the list of students who have submitted work to be reviewed
Type of input: An array of author Idsor submissions
Range of input values: all valid author Idsor submissions
Source of input: Probably taken from class list or submission database
Major components of the input:
3.3.23.3.2 Processing
The PG system will select a number of authors/submissions (up to the maximum number allowed) from the authorListor submission list. This subset of submissions (studentReviews) will be assigned to the student for review unless one of the submissions is the student’s own (i.e. student == author) or if the submission/author already has the predefined maximum number of reviewers (maxReviewers).
3.3.33.3.3 Outputs
The outputs of the functional requirement are as follows:
- studentReviews
Description: the final mapping between the student and authorList or submissions
Type of output: a list of author Idsor submissions assigned to the student
Format of output: valid author Idsor submissions
Components of the input:
Range of output values: a subset of authorIDs or submissions up to maxStudentReviews
3.3.43.3.4 Special Considerations
None at this time. We will try to produce code that is as short and quick as possible.
4 Development Environment Requirements
4.1 Hardware requirements
Our part of the project does not add any special hardware requirements to the already existing project requirements.
4.2 Software requirements
Our code will be written in Java 1.2. For the piece of code we are writing, any compiler should work fine, but we will be using NCSU’s existing Java compiler for final testing.
4.3 Deviations
We do not anticipate any deviations from the current development platform.
4.4 Target Environment Requirements
The target platform is the same as the development platform.
4.5 Testing Environment Requirements
The test platform is the same as the target environment.
5 Post Development Requirements
5.1 Training Requirements
The users of the system will need to be trained on how to choose their own reviews if that is the option that has been set. The other two possible options will not require any special training for the users.
The admin will need to be trained on how to select one of the implementedmapping schemes.
Training for both will be minimal since all of the aspects of what we are adding will be simple.
5.2 Technology Transfer Requirements
Currently the customer for this project is also the admin. This design document should provide the customer with all of the information s/he needs. Any other training that would be required is covered in section 5.1.
5.3 Maintenance Requirements
We are not aware of any maintenance requirements at this time. Since we will be thoroughly testing the part of the code we write and will verify that it can only produce valid mappings, little or no maintenance to our code should be needed. However, in the event someone would like to expand on our code, we will be providing documentation inside the code that describes the functionality of each piece of code.