Criteria for Evaluating Portal Framework Software

Criteria for evaluating portal framework software

(based on San Diego State University's Rubric for Rating Commercial Portal Vendors and Paul Browning's adaption of same)

S1= score for Oracle Portal

S2 = score for u-portal

S3 = score for Luminis

S4 = score for SAP EP

Insufficient
(0 points) / Adequate
(1 point) / Excellent
(2 points) / S1 / S2 / S3 / S4 / Comments
1.1
Look and feel (of front-end)
Aesthetics / Static background with few or no graphic elements. No ability for variation in layout or typography. / There are a few graphic elements and there is limited ability for variation in type size, colour, and layout. / UOL has full control of look and feel and changes can be made quickly. Appealing graphic elements are included appropriately. Differences in type size and/or university colours and logos are used well. / 2 / 2 / OP: uses wizard for PL/SQL, not for Java
uP: no wizard; XSLT or CSS
1.2
Look and feel (of front-end)
Ease of use / Counter-intuitive interface, requiring greater than two hours of user training. / Somewhat intuitive interface, requiring two hours or less of user training. / Intuitive interface, requiring little or no user training. / 1 / 2 / OP: needs training, navigation difficult
uP: layout easy to understand
2.1
Security
Authentication / No authentication. / Requires multiple log-ins in order to access different databases. / Single sign-on for multiple functions from one central database (assuming appropriate infrastructure exists). / 1.5 / 1.5 / Both retain and pass on credentials
2.2
Security
Authentication method / No authentication or authentication only against a local database. / One authentication method that can use an existing database. / Multiple authentication methods that can use existing databases (for example, can authenticate using an existing LDAP server or using Active Directory via Kerberos). / 2 / 2
2.3
Security
Encryption / No access via secure connection. / Secure connection can be specified but only in non-standard way. / Secure connection can be specified using standard method. / 2 / 2 / Both use SSL
2.4
Security
Access / Information access is all or nothing. / User is allowed to access certain information based on their user type. / User is allowed to access and change certain information based on who they are and their user type. / 2 / 2
2.5
Security
Roles / Information access is all or nothing. / There is a limit to the number of roles a user can have in the system. / There is no limit to the number of roles a user can have in the system. / 2 / 2
2.6
Security
Hosting / Server(s) under control of vendor at location undetermined by UOL. / 1 points
Server(s) located locally. / Server(s) located locally or at a location approved by UOL. / 2 / 2
3.1
Personalisation
Information push / User receives targeted information relevant to their constituency but no dynamic data. / User receives information relevant to the individual, but limited dynamically updated data available. / User receives specific information relevant to the individual and available in real-time. For instance, student-specific course-schedule, enrolment details, and degree checklist. / 2 / 2
3.2
Personalisation
Information pull (portal editor) / No editing tool to customise the portal environment. / Editing tool for customising tabs, panel buttons colours, and fonts.
Personalised view of all the information relevant to user-specific needs and preferences. / Editing tool for full customisation as well as the ability to subscribe to new channels (discussion boards, chat etc). User has ability to add/edit/remove information from a list of internal and external resources that the university approves. / 2 / 2
3.3
Personalisation
Link to exisiting VLE / No link. / Partial access to web-enabled course information/classes. / Full interoperability with VLE.
4.1
Interaction
Email / No email. / Portal only accommodates proprietary email system. / Supports multiple email standards and protocols.
4.2
Interaction
Chat & message boards / No chat or message board functionality. / Only chat or only message board. / Supports real time chat and message boards.
4.3
Interaction
Electronic balloting and polling / No electronic balloting and polling functionality. / Criteria for balloting and polling are only partially supported. / Electronic balloting and polling are fully supported.
4.4
Interaction
Multimedia / No streaming audio or video. / Options for streaming audion and video are limited. / Support for plug- ins that allow for streaming audio and/or video.
5.1
Productivity tools
Search engine / No search engine. / Limited search engine for UOL intranet and/or internet only. / Natural language search engine for both intranet and internet (eg internal Ask Jeeves).
5.2
Productivity tools
Calendar / No calendar. / Shared calendar(s) available. / Personalised calendar is available.
Allows others (with user approval) to populate their calendar.
Synchronisation with Palm OS is available.
5.3
Productivity tools
Meeting scheduler / Does not support a campus-wide meeting scheduler / Limited campus-wide meeting scheduler available. / Campus-wide meeting scheduler for specific users with ability to select and reserve specific rooms and equipment.
5.4
Productivity tools
To-do-list / No to-do-list. / To-do-list available but with limited features. / To-do-list available, but with many features, eg items can be placed in categories and ranked in priority order.
5.5
Productivity tools
Address book / No address book. / Address book is limited and proprietary. / Address book can interface with other contact lists and databases.
6.1
E-commerce
Advertising control / Banner advertising on every page of portal or not able to control per user group. / Banner advertising is optional. / Banner advertising is optional, controllable and can be targeted to specific user groups based on their role within the university.
6.2
E-commerce
Advertising revenue / No advertising revenue possible. / Advertising revenue is limited and shared with the vendor. / Advertising revenue goes directly to the university
6.3
E-commerce
Transactions / Cannot be integrated with campus systems offering web-based transactions. / Can be integrated with campus systems offering web-based transactions. / Can be integrated with campus systems offering web-based transactions with option of one "shopping cart" enabling the user to credit the appropriate entity.
7.1
Workflow
Forms routing / No forms routing. / Forms routing available - paper documents can be replaced by web-based forms. / Forms routing available - paper documents can be replaced by web-based forms. In addition, forms tracking software built in.
8.1
Support
Integration / No fit with existing relational databases, for example Oracle RDBMS / Some fit with existing relational databases, for example Oracle RDBMS / Great fit with existing relational databases, for example Oracle RDBMS / 2 / 2 / Both can use ODBC and JDBC
8.2
Support
Implmentation / Vendor relies solely on UOL staff. / Vendor provides implementation training to UOL staff and consultants for a fee. / Vendor provides implementation training to UOL staff and on-site consultants for free. / 2 / 1 / OP: training more readily available
8.3
Support
Scalability / Not currently in use at a large-scale reference site. / Currently being implemented at a large-scale reference site. / Existing large-scale reference site demonstrating that application server scales on Solaris. / 2? / 2? / Reference site may not be Solaris
8.4
Support
Reliability / No reference site. / Some evidence from reference sites that service is reliable. / Clear evidence from reference sites that production service is reliable. / 1 / 1 / Need to contact reference sites
8.5
Support
Load balancing / No vendor-provided load-balancing. / Vendor provides some assistance with enabling load-balancing. / Vendor provides load-balancing as part of solution (or via alliance with commercial partner). / 2 / 1
8.6
Support
Maintenance / No plan for ongoing support or maintenance. / Weak plan for ongoing support or maintenance. / Strong plan for ongoing support or maintenance. / 2 / 1
8.7
Support
Vendor support / Vendor requires toll call during business hours. No email or Web-based help. / Vendor provides email and Web-based help. No phone or fax help (24/7). / Vendor provides email, Web-based and toll-free help (24/7) for free. / 1.5 / 1
8.8
Support
Dependencies / Requires installation of several packages (other than product) that are not already in use locally. / Requires installation of one or two packages (other than product) that is not already in use locally. / Does not require any software that is not already in use locally. / 1 / 2 / OP: needs mod_jserv
8.9
Support
Long-term viability - vendor experience / Vendor is in pilot phase and has no experience or references. / Vendor has experience in higher education portal development but has limited references. / Vendor has significant higher education portal development experience and can provide numerous references. / 1 / 1.5
8.10
Support
Long-term viability - commitment to product / Vendor company is small with limited funding to develop product. / Vendor company is investing some funding in product. / Vendor company has ample financial backing that is committed to developing the product. / 1 / 1
9.1
Standards
API (Application Programmer Interface) / Portal API cannot pass information to other applications, or is not available to campus. / Portal API can pass some information to other applications and is available on a limited basis. / Portal API can pass information to other applications, seamlessly integrating multiple sources of information and campus can write their own interface, for example passing user preferences between applications / 2 / 2
9.2
Standards
LDAP (Lightwieght Directory Access Protocol) / Portal is not LDAP compliant (it will not allow a user to query a database via the internet). / Portal is LDAP compliant, but only allows limited online querying. / Portal is LDAP compliant and allows user to actively manage and customise a personal database.
9.3
Standards
Accessibility - output in different formats / No tools for outputting in different formats / Vendor provides tools for a limited number of standard formats / Vendor provides tools giving full flexibility of output format / 1 / 2
9.4
Standards
Accessibility - administrative tools / No tools for ensuring that site meets WAI accessibility guidelines. / Some help provided to ensure that site meets WAI accessibility guidelines. / Good set of tools provided to ensure that site meets WAI accessibility guidelines. / 0 / 0
10.1
Administration
UOL staffing / Seven or more full-time UOL staff required for managing and maintaining system software. / Between four and six full time UOL staff required for managing and maintaining system software. / Three or fewer full time UOL staff required for managing and maintaining system software. / 2 / 2 / Both: 1 person to maintain
10.2
Administration
Vendor staffing / Vendor relies solely on UOL staff. / Vendor provides training to UOL staff and consultants for a fee. / Vendor provides training to UOL staff and data migration/integration consultants for free. / 1 / 1
10.3
Administration
Fit with existing staff skill set - management of software / No existing staff currently have skills required for management of system software. / Only one or two members of staff currently have skills required for management of system software. / Three or more members of staff currently have skills required for management of system software. / 1 / 1 / OP: Oracle, Portal interface, Apache
uP: Oracle, Apache, Tomcat
10.4
Administration
Fit with existing skill set - development / No existing staff currently have skills required for developing applications using the software. / Only one or two members of staff currently have skills required for developing applications using the software. / Three or more members of staff currently have skills required for developing applications using the software. / 2 / 1 / OP: PL/SQL, Java
uP: Java, XML/XSLT
10.5
Administration
User definition / System will NOT allow UOL administrator to define custom user types. / System will allow UOL administrator to define limited user types. / System will allow UOL administrator to define custom user types. / 2 / 2
10.6
Administration
Information channels / System will NOT allow UOL administrator to define custom information channels. / System will allow UOL administrator to define limited information channels. / System will allow UOL administrator to define custom information channels. No limit to number of information channels. / 2 / 2
10.7 Administration
Time to market / System will take greater than:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout / System will take:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout / System will take less than:
- 8 weeks to define
- 12 weeks to design
- 8 weeks to prototype
- 16 weeks to rollout
10.8
Administration
Hardware resources / Hardware requirements do not coincide with university standards. / Hardware requirements loosely coincide with university standards. / Hardware requirements coincide with university standards. / 2 / 2
10.9
Administration
Pricing - software / Annual licence fee and Service Level Agreement costs not based on fixed price schedule or exceeds budget. / Annual licence fee and Service Level Agreement costs based on fixed price schedule. / Annual licence fee and Service Level Agreement costs based on fixed price schedule and are under budget. / 2 / 2
10.10
Administration
Pricing - production service hardware / Large increase in hardware required for production service / Modest increase in hardware required for production service / Will be able to run production service on current hardware / 0? / 1?
10.11 Administration
Pricing - development hardware / Large increase in hardware required to develop service / Modest increase in hardware required to develop service / Will be able to develop service on current hardware / 1? / 2?
10.12
Administration
Online help, documentation and training / No help putting portal applications into production. / Plan for integration, but little useful documentation and training. / Clear integration, with plenty of supporting documentation and face-to-face train-the-trainer training.
Several online help features, for example tutorials, job aids and FAQ?s. / 1.5 / 1 / OP: lots of documentation!
11.1
General
Essential channels / No indication of off-the-shelf channels for HE. / Some off-the-shelf channels for HE available or being developed. / Many off-the-shelf channels for HE available or being developed. / 0 / 1 / OP: can use in-line frames
uP: Aleph and Blackboard
11.2
General
Exit strategy / Channels are non-standard and would require a lot of work to transfer to another framework. / Channels are written using standards but would require some work to transfer them to another framework. / Channels are written using standards and can be very easily transferred to another framework using those standards. / 0.5 / 1 / OP: less work needed if only Java used (thought to be unlikely)

Total scores

Weighting / WS1 / WS2 / WS3 / WS4
1.1 Look and feel (of front-end)
Aesthetics
1.2Look and feel (of front-end)
Ease of use
2.1 Security
Authentication
2.2 Security
Authentication method
2.3 Security
Encryption
2.4 Security
Access
2.5 Security
Roles
2.6Security
Hosting
3.1 Personalisation
Information push
3.2 Personalisation
Information pull (portal editor)
3.3 Personalisation
Link to exisiting VLE
4.1 Interaction
Email
4.2 Interaction
Chat & message boards
4.3 Interaction
Electronic balloting and polling
4.4 Interaction
Multimedia
5.1 Productivity tools
Search engine
5.2 Productivity tools
Calendar
5.3 Productivity tools
Meeting scheduler
5.4 Productivity tools
To-do-list
5.5 Productivity tools
Address book
6.1 E-commerce
Advertising control
6.2 E-commerce
Advertising revenue
6.3 E-commerce
Transactions
7.1 Workflow
Forms routing
8.1 Support
Integration
8.2 Support
Implementation
8.3 Support
Scalability
8.4 Support
Reliability
8.5 Support
Load balancing
8.6 Support
Maintenance
8.7 Support
Vendor support
8.8 Support
Dependencies
8.9 Support
Long-term viability - vendor experience
8.10 Support
Long-term viability - commitment to product
9.1 Standards
API (Application Programmer Interface)
9.2 Standards
LDAP (Lightwieght Directory Access Protocol)
9.3 Standards
Accessibility - output in different formats
9.4 Standards
Accessibility - administrative tools
10.1 Administration
UOL staffing
10.2 Administration
Vendor staffing
10.3 Administration
Fit with existing staff skill set - management of software
10.4 Administration
Fit with existing skill set - development
10.5 Administration
User definition
10.6 Administration
Information channels
10.7 Administration
Time to market
10.8 Administration
Hardware resources
10.9 Administration
Pricing - software
10.10 Administration
Pricing - production service hardware
10.11 Administration
Pricing - development hardware
10.12 Administration
Online help, documentation and training
11.1 General
Essential channels
11.2 General
Exit strategy