Friday, March 20, 2009

Back to Basics #3 - Selecting Development Tools

OK. Another in my back to basics series but one I think is needed. If we are to believe that more and more companies will jump into eLearning to reduce costs (another arguement I will try and deal with later) then having a rigorous process behind how we select development tools is very important. Do we use our own development tools nowadays? If so this will help.

This is a bit long but I find publishing on the blog will allow you to cut and paste as required. As always, feel free to do so.

Development Tools Selection Process

Outlined below is a process for evaluating eLearning authoring/development systems. This same process can be used for a variety of development tools, although the detailed criteria will vary depending on the type of development tool.

Stage 1: Determine Needs

Determine requirements that the development tool must meet:

What is the nature of the training/performance solution that the tool would be use to create?

  • Information transfer
  • Tutorial
  • Simulation
  • Skill building
  • Conceptual
  • Cognitive

What are some of the characteristics being considered?

  • High level of interactivity
  • Engaging
  • Ease of use
  • Easily accessible
  • Available on demand
  • Accommodate large learning audience
  • Ability to modularize to meet diverse learning needs
  • Deliverable to geographically dispersed learning audience
  • Ease of interface - keyboard and mouse interaction
  • Embedded computer-managed instruction
  • Flexible time offered
  • Ability to connect to multiple database sources
  • Ability to deliver on firm standard hardware platform
  • Can be packaged for portable use (vs. requiring participants to take on a special computer in the office)

Stage 2: Conduct Industry Survey

Survey the marketplace to identify candidates. Don't be shy!

Stage 3: Collect Systems Information

Gather information on:

  • Delivery platform: What is the delivery platform? Is it multi-platform (in the event of sales)? What delivery media are supported (CD-ROM, networks, etc.) Which operating systems does it run under? Is the delivery platform compatible with our existing technology?
  • Development platform: What development platform is required? Is there any special equipment needed? What are the requirements vs. what is already installed in-house? Are there separate development and delivery environments and, if so, what does this entail (e.g., after development, does the program have to be ported over to a different delivery system?)
  • Development interfaces: What interface is desired/required? Systems (icons/menus) give the developer more friendly access to development functionality, reduces the learning curve and increases productivity, but at the cost of flexibility and advanced capabilities. A language-based system allows for more complex courseware, but requires substantial training time and has a steep learning curve. Hybrid systems offer an iconic or menu interface along with a language that underlies the system. This dual functionality allows new developers to start developing courses fairly quickly as well as allow for increasing instructional complexity.
  • LMS coniderations: What record keeping, tracking/reporting features of participants’ progress does the tool have that is compliant with the LMS?
  • Cost and licensing: What are the costs? Are there any runtime license fees? Royalty fees for external sales?
  • Training and support: What training and support are available and what is the cost (including any travel to the training)? What consulting services/freelance expertise is available?
  • Product stability: How long has the vendor been in this industry? How new is the product? How stable is the product (will it/the vendor be around next years? Three years from now?) How widely is it used in the industry? It would be a bad situation to implement something nationwide, then have the vendor go out of business.

Stage 4: Perform Initial Selection of Several (about 5) Systems for Further Analysis

Based on the criteria above, compare development tools and select several systems that best fit the majority of the requirements.

Stage 5: Down Select to 2-3 Systems for Detailed Analysis

Contact vendors directly to arrange demonstrations of their products and receive an evaluation copy of their software. Assess the systems ease of use, features, etc. Have the vendor demonstrate the system so that the system’s full capabilities can be assessed. Based on demonstrations, data collected, and initial experience, rank systems based on training requirements criteria.

Determine two or three final candidate systems. Selecting more than three increases the evaluation time and makes the final decision more difficult.

Stage 6: Perform Detailed Analysis/Benchmarking on each final candidate

  • Talk to typical users of the program: What do they like? Dislike? What are the limitations? What is the learning curve? Development cycle?
  • Conduct benchmark testing to determine each system’s capabilities. Develop a “storyboard” for a sample interaction/design of the system. Use the vendor/someone experienced in the tool, to develop this interaction vs. attempting to develop it yourself to eliminate training time and ease of learning (which should be considered separately outside of these tests).
  • Evaluate the following criteria in the benchmarking:- Development or revision time (including time set aside for planning)- Number of authoring screens used in development and revision- System response time between actions or screens for each phase (be sure to test executable on the same machine to eliminate computer configuration variables)- Quality of display in delivery
  • Evaluate systems using this criteria by developing sample interaction specification documents in the content of the program to be developed. These documents should contain screen displays and directions for response analysis, branching and feedback. Give these documents to each vendor to develop this interaction in a specific period of time (such as two days).
  • Monitor their process, measuring each of the criteria during the specified time frame and then judge the quality of the display during delivery.
  • Text abilities:
    When developing a piece of elearning, the range of facilities available for text is important, since this is likely to be the primary means of communicating with the student. The range of options available for text is likely to give you a good indication of the overall flexibility of the system.
  • Graphics
    Graphics used in programs can either be generated by the application itself, or imported from files.
  • Sound
    Since future applications may utilise sound, the abilities of the authoring environment to control sound is important. With interactive-audio drawn from a CD, the ability of the system to interface with the necessary software required to control the CD player will be fundamental. Sound can also be supplied from the hard drive.
  • Software friendliness
    Where the program is to be developed/updated partially in-house, the ease of use of the software development environment is important. This reduces the need to subcontract the development of elements of the program to others outside of your organisation. The more complex the software is to use, the more technical expertise will be needed to develop and maintain the program.
  • Design assistance
    When developing the applications, the more tools there are to assist the author, the quicker it will be to develop systems and modify them.
  • Learner control
    The software environment must be capable of allowing the development of systems which allow the inclusion of sophisticated learner control.
  • Question handling
    The ease of programming interactions is also a pointer to the overall abilities of systems. The range and variety of these are important, as is the ability to modify, add or remove interactions from within a program.
  • Branching
    Most applications developed require branching. The route taken by the application should be capable of being decided in a number of ways.
  • Other user resources
    Ability and ease in adding other resources to an application, such as context-sensitive help screens, dictionaries or glossaries, a built-in calculator, student note-taking resources, etc.
  • Printing options
    Printing is not only required for reporting the student progress, but also for documentation and retention purposes.
  • Documentation and tutorials
    Consider how well put together the paper-based documentation and other support for the product is.
  • Hardware and software
    The software should be capable of running on a range of PwC hardware.

Stage 7: Review Detailed Criteria & Recommend Authoring System

Detailed Criteria Program

Finally there are all the little detailed, but important things you need to consider:

  • Text abilities
    Variety of fonts, sizes & colors
    Variety of styles, bold, italic, underline, highlighted
    Variety of formats, centered, justified, double spaced
    Ability to import text from other sources
    Automatic re-formatting of text on the screen
    Search & replace ability
  • Graphics development capabilities
    (circles, lines, boxes, etc)
    Transition effects (wipes, fades, dissolves, etc)
    Color graphics abilities/range of colors
    Animation abilities
    Generated by the application
    Ability to import from other programs
    Ability to import graphics in several formats
  • Sound
    Use computer-generated sounds (tones, beeps, etc)
    Use sound cards
    Utilize sound stored on other media (digital files on hard-disk, CD-ROM)
  • Software friendliness
    Ability to structure layout of learning sequence
    Ease of modification of the structure of the program
    Ability to change screen layouts quickly
    Ease of programming
    Availability of developers
    Ability to link to other applications
    Ability to call software written in another language
  • Design assistance
    Range of pre-written structures available
    Ability to reuse code within other programs
    Integral flowcharting of the application
    System checks for incomplete branches
    Ability to track testing of the system to help debugging
    Ability to run sections of program, to aid debugging
  • Learner control
    Pacing & sequence control by learner or program
    Able to provide gradual learner control
    Learner can go forward or backwards within a lesson
    Programmable learner control keys
  • Question handling
    Ease of creating student interactions
    Ability to modify, add or remove interactions
    Variety of question types
    Variety of answer matching, key word, exact, spelling errors
    Able to use questions with several correct answers
    Able to generate specific feedback for each student answer
    Random generation of questions or feedback
  • Branching
    Unlimited branching for menu choices
    Able to branch based on correct/incorrect responses
    Able to branch based on elapsed time
    Able to branch based on number of tries
  • Other user resources
    Easy access to built-in, context-sensitive help screens
    Easy access to available dictionaries or glossaries
    Access to built-in calculator
    Access to student note-taking resources
  • LMS considerations
    Student management/CMIManagement easily turned on or off for specific program
    Able to record small or large amounts of data for each student
    Able to store elapsed time or level of achievement for each student
    Course restart options available to LMS
    Able to generate reports for each student or each lessonOutput available as an CSV file
  • Printing options
    Able to print: program text or graphics, branching information, program documentation,
    all or part of program
  • Documentation & tutorials
    Well-indexed reference manual included
    Tutorial included
    Telephone support available
    External training available
  • Hardware & software
    Memory (RAM) requirements for delivery
    Memory requirements for authoring
    Required disk space
    Processor requirements
    Ability to run on organisational supported hardware
    Need for other software (such as Windows)
    Need for other hardware (graphics card)
    Vendor stability/Support available from software distributors & system developers for continued viability/support

I know this is a bit detailed. Maybe this is why it is called back to basics. How much of this do you take for granted?

No comments: