Website: taloe.up.pt
eAssessment Web Tool: e-assessment.innovate4future.eu

TALOE (Time to Assess Learning Outcomes in E-learning) intends to promote the internal consistency of online courses by using the ALOA model (Aligning Learning

1. Project Objectives
The main goal of TALOE was to develop a web-based platform to help teachers and trainers decide on the e-assessment methods to use in their online courses. The main idea is that a teacher will describe the learning outcomes of their course or module and the TALOE platform will analyse them and provide a proposal for assessment methods that is consistent with the intended learning outcomes. During the project implementation, in the phases of testing and of improvement, it was acknowledged that web-tool could be used in other educational contexts and modes. It is in fact a web-tool that can be used globally.

The objectives of the TALOE project were:

  1. Research and select innovative e-assessment practices that take can be used as case studies and examples;
  2. Develop a web-based tool that is easy to use by the stakeholders;
  3. Test the implementation of the tool with real case studies and outside the partnership for validation and improvement;
  4. Distribute and disseminate the TALOE web-tool among the communities of stakeholders.

The project development was based on a pedagogical framework sustained by the ALOA (Aligning Learning Outcomes with Assessment – Rita Falcão, PhD Thesis, U. Porto and U. Austin) model. This framework was created for the use of e-assessment and for linking specific e-assessment tasks with the learning outcomes statements of the courses or modules. The method was based on existing research on assessment and findings were extended for different areas of application like e-learning.

It was planned since the beginning that the TALOE web-tool would be available in an open mode to different stakeholders. These stake holders included Higher and Continuing Education institutions as well as vocational training organizations. It is also available for stakeholders associated with processes of recognition and accreditation. The main target groups of TALOE project were teachers and trainers from all levels of education. The TALOE web-tool was created to help teachers and trainers to define e-assessment strategies for their courses or modules. It is curious that in one of the dissemination sessions a director of quality in teaching suggested that learners should use the web-tool to verify the quality of the assessment methods of the courses they would attend. In fact it is another possible objective that was not included in the partnership plans.

Other targets groups include:

  • Researchers of learning outcomes (LO) and assessment.
  • Recognition and accreditation staff, to verify the validity of assessment methods for VNIL (Validation of Non-formal and Informal Learning) or to provide evidence for accreditation.
  • Programme developers, to define assessment strategies recommendations for new programmes.
  • Decision makers and quality evaluators, to define valid assessment strategies for their institutions.
  • Networks and initiatives operating in the fields of LO, assessment, vocational training, higher education, continuing education, recognition and accreditation.

To reach these objectives the TALOE partnership worked closely with groups of teachers and trainers that have actively contributed to the outcomes of the project. The partnership invested in dissemination of the project activities and has reached stakeholders through online communication, participation in conferences, seminars, workshops, direct contacts and different types of publications.

2. Project Approach
TALOE project partnership developed and implemented a web-based platform for assessment of learning outcomes. The major goal was to help teachers and trainers of Europe decide on the assessment methods that can be used to match their courses or modules intended learning outcomes. The tasks of the TALOE project aimed at developing and implementing a web-tool that could be global in terms of scope and easy to apply in the alignment of learning outcomes with assessment methods. TALOE platform is ready to be used by other stakeholders in the educational system, to verify the validity and appropriateness of assessment methods used in different learning contexts and in processes of recognition and accreditation of the competences defined in the learning outcomes. The project was structured around three main phases: research, development, and implementation/testing. The work packages were defined, assigned to partners and planned taking into account the three phases of the project. The activities of TALOE project were distributed in seven work packages that are intrinsically related with the tasks arising from the project goals.

During the first year of the project, TALOE focused on research about e-assessment methods, on existing case studies and on the development of the web-tool. Concerning research, TALOE was focused on a theoretical model for the alignment of assessment and learning outcomes named as ALOA model. However, TALOE has pushed the model forward to incorporate a component of assessment methods considered adequate and common. The project used a diversity of case-studies to analyse and to collect e-assessment practices that could be used as examples of implementation of assessment methods online. In terms of development of the web-tool, TALOE used the ALOA model to define the structure of the web-tool in terms of functionality and of interface with users. The web-tool was developed collaboratively, involving a team of developers from the partnership and the author of the ALOA model.

During the first part of the second year the existing deliverables were reviewed as a function of the evolution of the testing of the web-tool. Following the testing of the web-tool by the partnership web-tool was adapted as a result of the findings. It is relevant to consider the increase of the accuracy of the tool, the interface with users, the cultural diversity, the language issues and the scope of users. After that adaptation and improvement the web-tool was tested on a wider scale using the networking and the partnerships established. Some organizations and experts were directly contacted and invited to cooperate in this testing. The final phase during the last semester of 2015 was dedicated to reach a larger audience in terms of dissemination and of exploitation of results.

The second year was indeed focused on the implementation with pilot case studies and with interaction with stakeholders. These stakeholders were invited to participate during the presentations, the seminars, the workshops, the webinars and the online discussions. The reaction from participants in these presentations and in ensuing debates was always positive and useful. The reactions and participations not only allowed a validation and an improvement of the web-tool but were also an encouragement to continue the pursuit of the project objectives. In the end the partnership is convinced that the web-tool is useful for teachers and trainers of all levels if education.

The research and development components had the collaboration and inputs of several partners of the consortium. There was also a great effort in dissemination of the results by the partnership. This effort is justified by the number of publications, presentations, workshops in educational conferences and in other events. The partnership is convinced that an adequate dissemination strategy and exploration plans helped the promotion of the web-tool among a community of users even after the end of the project. The members of TALOE partnership have produced papers and other materials that are available for researchers at the project website.  The website of the project and social network activities have operates online with a large number of participations and hits. The involvement of all partners was achieved with an effective and regular communication strategy among the partnership. The numerous virtual meetings and the online collaboration platform (Moodle learning management system) were important to achieve this interaction among partners.

3. Project Outcomes & Results
The TALOE project proposal and partnership identified the following specific objectives:

  • Research and select innovative e-assessment practices that take advantage of the use of technology;
  • Develop a web-based tool that is easy to use by the stakeholders;
  • To test the implementation of the tool with real case studies;
  • To distribute and disseminate the TALOE tool among the communities of stakeholders.

Globally and the end of the project it can be stated that the specific objectives were accomplished:

  • The researched and collected assessment practices are valid and interesting.
  • The web-tool is online and ready to use.
  • The web-tool was extensively tested by the end-users, in different stages, and the results of the evaluation process contributed to improve the web-tool.
  • The partners made an impressive effort in the dissemination of the results of the projects, nationally and internationally.

In the first year of project, TALOE partnership focused on the first two objectives related with the web-tool that represents the major outcome:

  • Research and select innovative e-assessment practices that take advantage of the use of technology;
  • Develop a web-based tool that is easy to use by stakeholders.

Besides these two content oriented deliverables activities other tasks were performed in terms of project management, of quality control of the project, of dissemination of project and of some initial exploitation actions.

In the second year the TALOE partnership concentrated in the testing and tuning of the web-tool using case studies and contributions from other stakeholders:

  • Implement the web-tool features and functions;
  • Test and improve the web-tool for public use and wider audience.

The project outcomes also included a wide dissemination and exploration outcomes due to the interested involvement of the partnership and of other stakeholders.

4. Research and selection based outcomes
TALOE has produced two deliverables, WP3.01 and WP3.02, concerning the first research objective.

WP3.01 “E-Assessment practices” consisted in the collection of 18 cases of assessment of online courses from different institutions around Europe, mostly from the project partners. These case studies were analyzed by the partnership in terms of classification under two perspectives: these were classified in terms of learning outcomes definition and in terms of the assessment modes used to verify these learning outcomes. The case studies classifications were also revised by the partnership during a second round. The goals of this work were to obtain examples of case studies that can be used as a showcase of current practice and also as testing material during the second year of the project. Since these case studies are available to the partnership these can be scrutinized during the tuning of the web-tool. WP3.02 “ALOA model with integration of E-Assessment practices” was developed on a second stage of the research component where the consortium developed an extension of the ALOA model to include the e-assessment practices described in the case-studies. This outcome generated the list of e-assessment methods to be considered in the web-tool. The relationship of these methods with the different types of learning outcomes was defined based on the revised Bloom Taxonomy and on the alignment theory of Anderson et al.

Development based outcomes
WP4.01 “Web-based e-assessment platform” addressed the second main objective of the project. The partnership has produced a first version of the web-tool that is ready to test the first functionalities of the intended platform. Due to the complexity of the ALOA model it was decided to simplify the tool procedures during a first phase of testing it. The first phase addresses only the simplest forms of knowledge. The partnership tested the web-tool until the next project meeting in February 2015. The development and consequent testing was done in phases of complexity of the definition of the procedures relating learning outcomes and assessment methods. This was an extra effort to achieve consistency of the web-tool performance and simplicity of procedures by the potential users. The second deliverable WP4.02 “User guides of the e-assessment platform” was developed for partners to test the web-tool until the public phase of development was initiated.

Web-tool outcomes
WP5.01 and WP5.02 “Global Report Implementation” is related with case studies implementation. Within this activity the web-tool was tested and evaluated and especially its usability and the quality of the results on selected case studies and by users outside the partnership active members. In order to verify if the produced TALOE web-tool had succeeded in achieving the set goals the testing phase was a necessary and important step in the life cycle of application development. It was planned that the cases chosen were diverse and representative of various learning contexts, including higher education, vocational training, online modules etc. In the end, testing was done on many more case studies than planned and with four invited stakeholders by each project partner. This increase of case studies enabled a better feedback and quality of testing results. The present version of the TALOE web-tool does not discriminate between knowledge types, as suggested by the theoretical model. After initial testing of the set up matrix it has been confirmed that the matrix is working properly. The best (most appropriate) assessment methods are selected on the base of the absolute matches between input (learning outcome) and the six assessment methods considered. First phase testing was done with eighteen case studies and performed by the project partners to see if the suggested assessment methods were closely related to the defined learning outcomes. Second phase of testing was done with invited stakeholders. There were thirty one participants (stakeholders) that evaluated the web-tool. The analysis of received feedback showed a positive evaluation regarding the usability of the tool and the quality of the output/results, i.e. the e-assessment method suggestions. Invited users found this tool easy to use and useful.  It also increases the accuracy of assessment methods used by the tool and the alignment between learning outcomes, assessment techniques and teaching methods. Final evaluation on the satisfaction with the web tool was done with random users. . Received feedback confirms a positive attitude towards the tool, describing it as a valuable tool that can be used to improve learning outcomes and align them with specific assessment methods. Participants in the survey found the tool easy to use and expressed an intention to use it in future as well.