Friday, May 01, 2015

Parsing KMworld 2014 -- Day 1

KMworld has graciously posted all of the presentations from its 2014 conference. Let's take a look at the presentations from the first day, November 5 2014:

Pugh presents Knowledge is Business. She is apparently pushing her book: Smarter Innovation -- using interactive processes to drive better business results.
  • There's some stuff about the transition from a knowledge-enabled strategy to a knowledge-driven strategy.
  • Knowledge products include frameworks, research and development, integrated services, education, consulting. Knowledge can be instilled (in the product), distilled (knowledge for sale), embodied (people, community).
  • She cites some stuff from Ark Group
  • Uses of knowledge include: bridging, social and operational integration, capabilities validation, market and industry exploration, commercialization
  • Organizational-level capacities: shared language/metadata, consensus innovation process, sense-making/benchmarking, swaps, sabbaticals, mindfulness
  • Individual-level: mastery, self-awareness, boundary tracking, attracted to foreign, comfort with ambiguity, convening skills
  • Team/community level: integrity/courtesy/inclusion/translation, "desire-to-know" vs. "need-to-know"
  • Overall -- interesting but maybe a bit high level.

  • So we've got silos… no surprise
  • Difference between complicated and complex problems
  • So we need systems thinking. Okay, very Senge.
  • "Improving a part can destroy a system" i.e., you can't put a jet engine in a car
  • KM includes interactions and interdependency
  • Good stories…
  • Theory of Constraints.
  • KDSD = knowledge discovery sharing and distribution
  • Model-based repositories and visualization
  • Either very wise or very dated

McCabe (Unisys) provides "Knowledge transfer across generations":
  • She has a business method patent or two, one apparently with a guy named Goodall so it can't be all bad!
  • Some decent stats about aging in the work force
  • Some more good stats about "employee turnover as % of salary":
    • Clerical 50-80%
    • Skilled hourly 75-100%
    • Professional: 75-125%
    • Technical: 100-150%
    • Engineers: 200-300%
    • Supervisors: 100-150%
    • Mid managers: 125-200%
  • Employee k lifecycle: recruit, onboard, develop expertise, transfer k, exit
  • Great case study for workforce renewal training

  • I like Richard's stuff so let's see what we get.
  • Identified learning objectives. Nice.
  • Emailing is big… but poorly done.
  • Decent stats on usage of information services
  • Interesting. I don't think that most people are here yet but it provides some decent guidance.

Nelson of Mayo clinic on "Pushing the envelope : from CMS to KCMS":
  • This could be cool, particularly given my recent interest in the KCS stuff. And the guy has "Ontologist" in his title!
  • It looks to be a big case study
  • We have a "content value chain" that goes from "unstructured -- human consumption" up to "smart content -- human consumed, machine organized and interpreted"
  • Used semantics for targeted ads and dynamics links
  • It all seems to be driven by Sitecore and TopBraid (from TopQuadrant)

Most of the presentations are available as PPTs:

Peterson and Phair of MITRE provide "Using tangible interfaces for predictive knowledge delivery":
  • So… capture whiteboards to SharePoint
  • Add a keypad to the whiteboard so that authors can introduce their credentials (i.e., metadata for discoverability)
  • Write commands or specific instructions in a specific part of the whiteboard
  • Future: sensing presence (i.e., an employee enters a building a needs an office space), sensing conversation.

Minicucci of IBM with "Enterprise search for Knowledge Management":
  • Combine structured and unstructured
  • Clustering with text analytics, entity extraction, sentiment analysis,
  • Next: "Cognitive exploration"
  • Hmmm… there's a difference between search and expert analysis.  The expert has to understand question, produce answers and evidence, analyze, assess confidence, and deliver

Brenkovich of Accenture "Enterprise search for KM: improving content access for consultants across Accenture."
  • Ooo! A case study!
  • Only 37% of consultants could find what they needd
  • Phases:
    1. Content accessibility. Focus on best content, noise reduction, awareness raising. Content mgmt scorecard quarterly. 30% increase in content access (37% to 48%)
    2. Establish content ready state and commit to communities. Integrated social learning; established CM accountabilities; more content quality; calendar with monthly content tasks; focus on communities with an "Enablement Toolkit" or cookbook for community; 58% increase in content access (48% to 76%)
    3. Maintain content steady state and community uplift. Content management CoE; professional communities; community engagement; community governance; leadership accountability; showcase curated content; improve mobile collaboration.
  • Content scorecard:
    • Provide the best content. Improved harvesting; simplified contribution; content rating campaigns; reminders to rate downloaded material
    • Reduce the noise. Quarterly review of content older than 3 years; improved archival processes; clean up of duplicates
    • Improve access. Monthly contribution review for tagging; search best practices (best bets, type aheads, relevancy tuning); CM enablement for content creators; clean up of incomplete or incorrect tagging
    • Raise awareness. "Find content" communications in global newsletter; inclusion of content and collaboration in onboarding and other training
  • Results:
    • Provide best content: 50% of content rated (up from 13%); engagement profile completions up 80%; proposal completions up 180%; deliverable completions up 320%
    • Reduce noise: 73% reduction in content older than 3 years; eliminated >1000 duplicate titles
    • Improve access: fixed tagging on 7000 contributions; 58% reduction in bad searches
    • Raise awareness: 42% increase in use of our collaboration tools
  • They build content dashboards to track progress
  • CoE. Knowledge Specialist responsibilities. Content harvesting, content audit, contribution review, content archiving, content requests; repackaging content; search optimization; support for practice enablement; managing community sites; developing offering essentials and key documents; driving CoP communication; training content support
  • They talk about the 3 types of communities (a la the ARMY): professional communities, communities of practice, communities of interest.
  • Measuring community success: dashboard, survey, program scorecard

Cheryl McKinnon "Go beyond traditional ECM : new approaches to managing enterprise content"
  • It's always nice to see something from Cheryl, formerly of Open Text and Nuxeo. She even came to visit me once when she was with Nuxeo. Let's see what she has to say.
  • Four disruptors -- transform the customer experience; embrace the mobile mind shift; turn big data into business insights; become a digital disruptor
  • Drivers for ECM: search, content sharing, compliance, archive consolidation, cost-effective automation, paper reduction
  • Analytics seem to be a trend. Data on usage monitoring/system auditing, user interviews, bugs/requests, PoC, log files, etc.

Varney "Evolving a premier digital workplace at KCTCS"
  • Kentucky community colleges?
  • PSFT, Sitecore, Blackboard, OnBase, Inteliworks, AdPro
  • Yeah, yeah… And they use ADKAR from Prosci (which Brian F. mentioned just last week) for change management
  • Awareness; Desire; Knowledge; Ability; Reinforcement

Schneier "When knowledge walks out connecting, sharing and capturing alumni knowledge for development"
  • I had seen this one before
  • World Bank Group
  • Alumni want to help, primarily around providing informal expert help, expert interviews, etc.
  • Introduced a skill finder

Horn "Talent development: a knowledge sharing lens"
  • MWH
  • Real process for new hires:
    • Onboarding. Brand, core values, purpose, people, professional culture
    • Induction. Clarify roles and expectations; compliance training. Two weeks.
    • Orientation. Basic training. 90 days.
  • Integrate knowledge transfer into the plan
  • Use events: annual conference, community calls, pair and share
  • Recognition and incentives. It's on the PA; awards.
  • Hero spotlight
  • Community success factors:
    • Community measurement
    • Motivation, recognition, rewards
    • Supporting technology
    • Standard collaboration processes
    • Development of trusted relationships
    • Leadership, governance, and sponsorship
    • Clear business case and objectives
    • Adequate resources and support
    • Member engagement and defined roles
    • Clear deliverables and activities
  • Knowledge transfer: mentor relationships; knowledge forums; work assignments; k-transfer plan
  • Identify risk of knowledge loss: single points of failure; retirement
  • Identify key knowledge.

Wolf "Knowledge capture and transfer at Kraft Foods"

  • They used a knowledge modeling process called MASK
  • MASK: Method for Analyzing Structured Knowledge
  • Developed by Louis Ermine at the French Commission de l'energie atomique in 1993
  • It needs a KM Lead; a sponsor; an expert; a recipient who owns the knowledge books; and a facilitator
  • The MASK process
    • Needs assessment. Lead and sponsor. Criteria: rarity, strategic fit, difficult in acquiring, difficulty in applying
    • Scoping. Led by MASK facilitator. Input from expert and recipient. Validated by sponsor. Provides a base for elicitation.
    • Eliciting and modeling. Led by MASK facilitator. 1 expert per 4 hour interview. Output captured on paper and audio records. Models generated by facilitator.
    • Validation. Completed by Expert with facilitator and lead support. Detailed editing of tech content.
    • Sharing. Recipient introduces the book to relevant groups. Expert attends and supports.
  • Body of Knowledge = basic phenomena + concepts + processes + history + know how + evolution
  • Activity model+phenomenon model+sample concept model+task model + history model + lineage model 

Wednesday, April 29, 2015

Best Training Guide -- US Marine Corps Systems Approach to Training (SAT) Manual

We have a winner! This document is perhaps the best guide to training. It has pretty much everything.

Let's go back to some of the wonderful stuff from the US military.

US Marine Corps Systems Approach to Training (SAT) Manual. June 2004.

I really like this thing. It has some great detail. It conforms to an ADDIE model has has details on each section. I particularly like the breakdown of inputs, outputs, etc.


Job task data
Job analysis
Task list

Task analysis
Individual training standards (ITS)

Instructional setting
Define student population
Target Population Description (TDP)

Conduct learning analysis
Learning Objectives

Define evaluation
Test Items

Select media and method
Delivery System

Organize Instruction
Sequenced Terminal Learning Objectives (TLO)
Learning Objectives
Organize course
Course Schedule

Develop Instruction
Master Lesson Files (MLF)

Delivery System
Develop media

Test Items
Validate Instruction
Revised instructional materials

Develop supporting course materials
Course descriptive data

Program of instruction
Instructional materials
Prepare for instruction
Delivery of instruction

Implement instruction
Course data
Course data
Conduct formative evaluation
Revisions to instruction

Conduct summative evaluation
Data on instructional effectiveness

Manage data
Course Content Review Board (CCRB)

  • Analyze
    • Job analysis
    • Task analysis
    • Selection of tasks for instruciton
  • Design
    • Write a Target Population Description (TPD)
    • Conduct Learning Analysis to develop learning objectives
    • Select Delivery System
    • Sequence Learning objectives
  • Develop
    • Develop course schedule
    • Develop instruction
    • Develop media
    • Validate instruction
    • Develop Course Descriptive Data (CDD) and Program of Instruction (POI)
  • Implement
    • Prepare for instruction
    • Implement instruction
  • Evaluate
    • Plan and conduct
    • Analyze and interpret
    • Document and report

A task must be:
  • Logical and necessary unit of work
  • Be observable and measurable or produce an observable and measurable result
  • Have one action verb and one object
  • Be a specific act done for its own sake
  • Be independent of other actions
  • Have a specific beginning and ending
  • Occur over a short period of time

Task lists must be verified and refined. And then you have to identify tasks for instruction based on:

  • Percent of jobholders performing the task
  • Percentage to time spend performing the task
  • Criticality of the task to the job
  • Frequency of task performance
  • Probability of inadequate performance
  • Task learning difficulty
  • Time between job entry and task performance (task delay tolerance)
  • Resource constraints for delivery
There might be particular standards for task delivery.

Target Knowledge Description: who is coming and what knowledge, skills, and attitudes (KSAs) do they need to have?

Learning objectives should:
  1. Tell students what they will be able to perform (behavior)
  2. Describes conditions under which the performance will occur (condition)
  3. Tells how well someone will perform (standard -- completeness, accuracy, time,)
  4. Establishes the basis for measurement of the performance
  5. Provides a focus for the instructor and the student

Tremendous detail on how to actually test these things, particularly multiple choice (pg. 63).

Figure 2-17 is the Methods Selection Matrix that shows the appropriateness for different training methods for different types of learners and environments.

You need a lesson plan. It gives the instructor a script. It establishes continuity and standardization of instruction. Historical document. A lesson plan should include:
  1. Title page
  2. Introduction
  3. Body (transitions, instructor notes, cues)
    1. Establish main ideas
    2. Sequence main ideas
    3. Insert content
    4. Provide examples
    5. Determine placement of methods
      1. Lecture
      2. Demonstration
      3. Practical application
    6. Insert methods
  4. Summary

Videos should be no more than 5-8 minutes.

An instructor preparation guide should include:
  • Lesson title and lesson designator
  • Total lesson time
  • References
  • Location of test
  • Personnel required
  • Facilities
  • Review course materials
  • Materials/equipment
  • Exercise setup and planning

A Master Lesson File (MLF) serves as a repository for all materials. It must include at a minimum:
  1. Learning Analysis Worksheet
  2. Learning Objective Worksheet
  3. Concept Card (snapshot of class -- objectives, method, media, instructor/learner ratios, references)
  4. Risk Assessment Worksheet
  5. Instructor Preparation Guide
  6. Lesson Plan
  7. Student Outline
  8. Supplemental Student Materials
  9. Media

The delivery and evaluation stuff is really good.

Course history folders should include:
  • Enrollment rosters
  • Student data sheet information
  • Test results
  • After Instruction Reports
  • End of Course Critique Summary
  • Graduation Roster

Course content should be reassessed by a review board.

Chapter 6 is all about adult learners.

What is this? Constructivist Learning Environments (CLEs)? They are apparently intended for "ill-structured problems". It is best accomplished with "situated learning" to promote "authentic learning." Problem-based learning is also an effective approach. There's also "Cognitive apprenticeship" (i.e., apprentice, intern, etc.). Best practices for cognitive apprenticeships include:
  • Mentoring/coaching relationship
  • Modeling and explaining of expert knowledge via think aloud procedures and worked examples
  • Encourage collaboration
  • Social interaction to enculture students into authentic practices

Chapter 7 is about developing trainers and then we get a whole lot of worksheets. Awesome. Included sheets include:
  • Learning analysis worksheet
  • Learning objective worksheet
  • Concept card
  • Lesson plan
  • Master Lesson File checklists

Great references. In short, it's the best.

Developing training material -- a round up

Okay. I've made the trip to the business library… which isn't actually where I thought it was. The new library is beautiful but lacks the quaintness of the old library. That said, let's get into this project.

Training. Specifically, I have been tasked to develop some training material for something we do internally and that I have been doing for a long time. I'm developing some training that should be delivered electronically in either a seminar format or self-serve from the LMS.

I've ripped a bunch of books from HF5549.5.T7. Let's see what we find.

  1. Hardingham. 1996. Designing training.

The intro looks good. It lays out a thought model and reviews choices for different types of training. Venue choices. Costs. You have to consult to understand training objectives. These things should be SMART. Manage politics, particularly blockers and contributors.

There are five key concepts in training design:
  1. Credibility
  2. Commitment
  3. Risk
  4. Attention
  5. Maneuverability

Fundamental design principles:
  1. Establish credibility
  2. Elicit Commitment
  3. Attract attention
  4. Allow maneuverability

Signpost. Lots. Vary pace and rhythm. Chunk content. Map the participants world. Give participants choices. Surface objections. Balance theory and practice. Design for closure (entertainment, presentation of certificates, closing speeches, closing feedback, presentation of gifts).

Process interventions are a form of training that could involve small group work, facilitation, direct process manipulation.

There might be differences in groups. Give participants choices and design in layers.

Classic challenges: training the unwilling; money; training for the easily bored; training for people who won't work together.

Overall, it's a little book but I like it. Perhaps we could use it as a basis for building workshops i.e., training on training.

  1. Stolovitch and Keeps. 2011. Telling ain't training.

This book was my reference anchor since it was published by the ASTD. Let's see what it has to say.

It starts off with a decent overview of how humans learn. There are different types of learning: declarative and procedural (e.g., riding a bike). Learning requires ability, prior knowledge, motivation (value, confidence, and mood).

We've got some more principles of adult learning:
  • Readiness. Adult learners must participate in and contribute to the learning process.
  • Experience. Ault learners must be able to immediately apply what they are learning.
  • Autonomy. They must see the benefits of what they are doing.
  • Action. Content and activities should be integrated.

Participatory learning: scenarios, role plays, cases, brainstorming, practice activities.

The learner must be the center of attention. Trainer success comes from the success of the attendees. Andragogy = adult learning. Pedagogy = kid learning. Answer: What's in it for me? Never assume they don't have experience. Let them contribute. Focus on immediate application. Training must be applied on the job. Train others as you would be trained.

There's a five step model for structuring training. See:


There's also some good planning sheets.



Learners need to be engaged.  They have different meta-cognitive skills: planning, selecting, connecting, tuning, monitoring. There are also the deficient factors: ability, prior knowledge, and motivation.


We have a few different cognitive strategies: clustering, spatial (hierarchies, flow-charts, matrix), advance organizers, image-rich comparisons (analogies, metaphors, comparisons), repetition, memory aids (mnemonics: acronyms, acrostics, rhymes, key words).

Four types of training: receptive (telling), directive (do it this way), guided discovery (cases), exploratory learning.

Gg - Perhaps guided discovery is the best way to start F2F training.



Potential activities:
  1. Better me. How can you do better than me at a procedural task?
  2. Concentration. Match cards, etc.
  3. Confrontation. Groups of three. Two in confrontation, one observer. Timed rounds. Participants cycle through five strategy choices and observe outcomes. Six scenarios, each lasting 5 minutes. Debrief.
  4. Critical list. After instruction, members have to produce the most important points. Iterate to find most important points.
  5. Crypto cluster. Decipher a puzzle.
  6. Domino Effect. 20-30 cards with a term on one side and a definition on the other. Participants must make a chain.
  7. Exam cram. Divide into study groups. Give a test to each team. They have 20 minutes to cram. The leader can actually test or not.
  8. Facts-in-Five. Give a card with a 5x5 matrix. Columns are categories; rows have letters of the alphabet. Fill the matrix with aligning keywords.
  9. Great debate. Take some debatable issues. Divide into teams. They have 10 minutes to prep pro or con for a 2 minute debate.
  10. Hit or myth. Participants get a list of 10 statements that are either true or false. They have to decide.
  11. Jeopardy. Given answers; have to determine questions. Could be timed, scored, or involve increased difficulty.
  12. Jigsaw. Class is divided into teams that are given documentation about one part of a topic. The teams prepare a presentation about their specific part including format and how to make it interesting. Chain the presentations into a lecture. End with a quiz and a debrief.
  13. Lecture team quiz. Break into teams. Lecture. Teams must prepare a quiz question and answers. Exchange.
  14. Letter game. Break into teams. Each team gets an envelope with a problem written on it. Each team gets one fewer cards than envelopes. They write their solution to the problem on a card, their identifier on the back, insert into the envelope, and pass to the next group who does the same thing. Teams than rank order the solutions and points are assigned.
  15. Listening teams. Each team has to listed for key parts of information and repeat.
  16. Mismatch. Describe 4-8 scenarios involving a question and an answer. The question and answer are mismatched.
  17. Ours versus theirs. Create a lists of 10 to 15 statements. Beside each statement create three checkboxes associated with competitive providers or products. Ask participants for feedback.
  18. Police interrogation. There is a topic and quiz. There is 30 minutes. The participants must force the content from the participants.
  19. Press conference. Teams of reporters. They must draw out information from an expert. The conference has four or five themes. Teams create questions related to the themes on colour-coded cards. Teams then take on a theme. They have 15 minutes to review questions and then 7 minutes to grill the expert.
  20. Quiz game. After a day of instruction, participants are given a slip of paper to identify a single point worth remembering. This is the answer. The participant also comes up with a question to elicit the answer.
  21. Slap jack. Instructor creates cards with a term on each card. Participants are grouped into teams of 3-5. The instructor calls out knowledge domain, each participant throws a card, participant has to correctly identify the domain and touch/slap the card.
  22. Techno challenge. Alternate between roles of customer and sales consultant. Rounds are 99 seconds. Clients ask questions; consultants respond to as many as possible. Good review.
  23. Terminology tussle. Everyone gets a BINGO card with terms. Definitions are read out related to the terms.
  24. They say, we say. Teams of 3 or 4. Half as competitor sales consultants; half as home team. A round = 15 seconds for one side to introduce a feature + 30 seconds for the other side to rebut.
  25. True grid. Blank grid. Participants can apply axis titles and fill it in or the axis titles can be preassigned.

Tests and verification are different things.

There are advantages to each approach:


Four steps to training evaluation: 1. participant reaction; 2. immediate learning at the end of the session; 3. on-the-job application post-training; 4. results -- did something change organizationally?

RLO = reusable learning object.

Online learning best practices:
  1. Right content
  2. Strong alignment with business goals
  3. True interactivity
  4. Valued experience i.e., fun
  5. Few distractions
  6. Useful on the job
  7. Powerful feedback
  8. Valid assessments
  9. Good human factors
  10. Integrated follow-up

Important decisions:
  1. Who is the course for and why?
  2. What are the content and context?
  3. How deeply should it go?
  4. What type of learning is needed?
  5. How much time do you have?
  6. How will the training be delivered?
  7. What are the metrics?

"Blended Learning" involves information (repositories, job aids, procedures), instruction (training, online, external), and collaboration (social media, CoP).

See: smilkstein 1993. acquiring knowledge and using it. ERIC 382 238.

That was a good one. My brain is getting full and we're almost two hours into this effort. What's next?




  1. Kosalka, et al. 2013. Instructional designer competencies.

It is a volume of the IBSTPI (International Board of Standards for Training, Performance and Instruction). So that's interesting.

So this is about competencies for instructional design… cool. If I was coming up with job descriptions. This thing is very much an auditing standard rather than a how-to guide. Regardless, perhaps I should draw a distinction between "training" and "instructional design".

  1. Phillips. 2010. Measuring and evaluating training.

Another ASTD title. It is a collection of essays that basically follow Evaluation Planning, Data Collection, Data Analysis, Measurement and Evaluation. Some interesting looking stuff in here but it's probably more academic/mature than my current information needs mandate.

  1. Russ-Eft, et al. 2008. Evaluator competencies.

As long as we're on the evaluator side of things, let's take a look at this book. Koszalka is an author and the forward is by the past president of the American Evaluation Association. Yet another association to deal with!

And… it's not what I'm looking for. Next.

  1. Lawson. 2009. The trainer's handbook.

The title looks promising.  Let's go.

So we need to an needs assessment: identify problem or need; determine needs-assessment design (interviews, surveys, etc.); collect data; analyze data; provide feedback. The feedback report should have these sections: executive overview, description of process, summary of findings, preliminary conclusions, recommendations, potential barriers.

We get some more background on androgogy and adult learning styles.

Preventing cognitive overload: minimize lecture and use key learning points, checklists, graphs, models, etc.; have participants do the work; create chunks of content and distribute it incrementally; design workbooks for participants to follow-along; create job aids.

We've got some in-depth instructional style surveys which are pretty cool. Basically, you have sellers, professors, entertainers, and coaches.

We have to write some learning objectives. There needs to be an instructional plan, including:
  1. Part 1
    1. Title
    2. Course description
    3. Learning outcomes
    4. Length
    5. Format/methodology
    6. Audience
    7. Participant preparation
    8. Instructional materials -- document list; equipment list; media list
    9. Reference list
    10. Facility check-off list (i.e., how do we put the room together with materials, etc.)
  2. Part 2
    1. Time frames
    2. Content outline -- ideas, concepts, principles, skills
    3. Training aids and materials
    4. Trainer's notes

Other considerations include instructional methods and developing materials. A trainer's guide should include: TOC, introductory background material, presentation guide with facilitation tips, instructional plan, master copies of materials, list of materials, and resources.

Co-operative training is good. Videos can be good. Lectures are okay but some things are better: question and answer; interesting visual aids; stories and anecdotes; case problems; examples and analogies. Other techniques include group inquiry, guided discussion, active knowledge sharing, peer lessons, jigsaw, learning tournament, etc.

Role plays -- scripted, coaching, spontaneous, or rotating trio.
Case studies -- story form, with characters, and realistic dialog, and appropriate specific details, make the story easy to follow, provide discussion questions
Inbasket activities
Instructional games

The book then gives us some details about room design. Nice.

We have to be aware of openings or "set induction". We could use icebreakers or openers.
  • Human scavenger hunt. "Find someone in the room who…"
  • The party… I don't think I would use this.
  • Instant assessment. Prepare cards with different colors and letters. You basically get them to hold up cards when you ask questions such as: "My main motivation for attending this session is…"
  • What do you want to know?
  • What's in it for them?

Interactive delivery involves things like videos, role plays, simulations, storytelling (parables, fables).

Clothing: dress a bit above your audience. Don't wear anything distracting.

Make good slides. Use flip charts appropriately.

Theatrical tools might be valuable. For example, a cow bell to get people's attention. Giveaways are helpful. As are games.

Finally, we have some more guidelines on evaluation of training.

  1. Talbot. 2011. Training in organizations : a cost-benefit analysis.

The TOC looks good but my brain is fairly shot so I'm skipping it. I can always come back to the library!

I still have the question about best practices in instructional design. This is the type of question that has likely been answered by some sort of government body at some point or by the aforementioned IBSTPI.

First off:

  1. The 2012 IBSTPI Instructional Design Competencies (

Basically, they need to have professional foundations, planning and analysis skills, design and development skills, evaluation and implementation skills, and management skills. There is a breakdown of performance statements. Moving on...

  1. There's an "instruction design criteria checklist" from Michael Fors of Microsoft ( The list is actually a great synopsis of everything I just researched. It will go to Evernote.

  1. There's something called the Canadian Association of Instruction Designers. It, too, has a competency model (

  1. The IEEE offers some resources, including a Reference Guide for Instruction Design ( It offers some great links to a variety of different resources. The "Strategies for teaching at a distance" from Idaho State University looks interesting. And, there's yet another association: International Association for Continuing Education and Training. This site is worth a return visit. The whole model is based on the Dick and Carey model, see:

  1. Let's look at IACET. It offers accreditation for training centers, research, resources and references. It also has some sort of standard. The standard measures a provider's program development across a 10 different categories ( And it complies with ANSI recommendations for standards development. The standard is compatible with the "ADDIE" model, which is a framework of generic processes for instructional designers and training developers. ADDIE stands for Analysis, Design, Development, Implementation, Evaluation. It is an "Instruction Systems Design" (ISD) model such as the Dick and Carey model or the Kemp ISD model. Apparently the ADDIE model was initially create for military inter-service training... so there might be some .mil information on it. Part of IACET efforts, however, are to create a standard based on a canonical list of resources ( The most recent reference is Morrison et al. 2006. Designing effective instruction (5th ed.). Wiley & Sons.

  1. For what it's worth, Wikipedia has some interesting content on the history of instructional design ( It provides some other acronyms of interest for learning design: PALO, IMS LEARNING DESIGN, LDL,SLD 2.0, TELOS, RELOAD, etc.

  1. Let's go back to our list of federal information. There's another checklist available from the USGS which I should be able to parse (

  1. The Department of Labor has published a collection of best practices for Web-based training ( It uses ADDIE and the Kirkpatrick model of evaluation. It provides a pretty cool taxonomy of course breakdowns, etc.


It also talks about training flow:


We get some good tactical guidelines for interactivity:
  • engage the learner. vivid and lively language, fascinating facts; speak directly to the learner; active voice; second person; imperative voice
  • address WIIFM. explain relevance; explain long and short term benefits; consequences of not learning the information
  • provide interaction every 3 to 5 screens. But don't introduce unnecessary breaks
  • groups content into segments with questions, reviews, and summaries
  • ask as many questions as possible
  • ask questions at the on-the-job level
  • use rhetorical questions to get learners to think ahead
  • consider presentations where learners can learn by exploring

There are different types of web learning.
  • Category 1. sequential telling with little user control.
  • Category 2. Includes simple simulations (e.g., turn dials, rotate switches, could include basic images, videos, and audio).
  • Category 3. High simulation. Learners might have to alternate between multiple screens. Uses more peripherals. Good for operation and maintenance procedures.
  • Category 4. Emersion. Generally supports certification or qualification requirements. Lots of interactivity and branching but no AI.

You need to be s.508 compliant.

Section 1.2 provides course structure which is typically: course introduction, module, lesson, topic.

A module is a group of lessons. The module introduction should have two screens: overview of the module and description of what the learner will learn; second screen has module objectives. There should also be a module menu, lessons, and a module summary.

Lessons correspond to Terminal Learning Objectives and include various topics, each associated with a Enabling Learning Objective. The maximum time for lesson completion (seat time) is 45 minutes.

Lessons should have the introduction screens, content and practice screens, lesson summary screens, and a lesson knowledge check.

At the end of the course, there should be a "comprehensive knowledge check" (a test). Provide remediation for questions that learners answer incorrectly.

Scenarios are good. Use case studies, stories, etc. Ask: "What would you do in this situation?"

You generally need a Detailed Content Outline or DCO.

the overall Course Design Plan (CDP) should include:
  • Introduction. project overview, target audience, purpose
  • Course structure. description, course outline, course flow chart, estimated number of screens, contact time
  • Course design strategy. overall approach to course, scenarios, and plan for supplemental material
  • Course practice and assessment strategies
  • Design strategy for each module and lesson. Terminal Learning Objectives, Enabling Learning Objectives, descriptions, instructional strategies, assessment strategies, content outline

The document contains guidelines for actually building out the storyboard.

It provides some best practices for screen design:
  • Graphics left; text right
  • Top down, left-to-right presentation
  • Fewest steps necessary to deliver the right information
  • No timed effects
  • Once concept, procedure, or item of instruction per screen
  • Consistent use of color
  • Stay compliant with s508 of ADA
  • Don't violate copyright

recommendations for language:
  • Active voice, 2nd person, conversational tone
  • Simple, concise, consistent language
  • Avoid hyphenation (except for compounds)
  • Avoid jargon and slang
  • Maintain parallel construction and noun-pronoun agreement

For audio:
  • use narration when the visual channel is overloaded, there is a need for immediate learner response, or if the message is simple
  • Narration should complement text. It should be somewhat different from onscreen text.

For audio development/script development:
  • an audio script for every synched element
  • Avoid long pauses in visuals to support narration
  • Make clear concept transitions
  • The storyboard should conain the script
  • Keep language simple, active, and direct
  • Format text appropriately if the acronym is to be spelled out (e.g., D-O-L)

For visual elements:
  • They should relate to the content
  • Provide recurring information in consistent locations
  • Maintain a constant perspective
  • Get consent for logos
  • Avoid clutter
  • Re-use graphics for basic concepts
  • Use these rations: 10% complex (VR, 3D); 30% simple animations; 60% photo /illustrations

For graphics:
  • Don't use too many colours at once
  • Have sufficient contrast between background and text

For videos:
  • Use short clips to reinforce learning objective
  • Use appropriate techniques (talking head, show and tell, interview, panel, simulation, dramatization)
  • User video if the content requires motion

For animation:
  • Only use special effects rarely for emphasis
  • Should initiate only on learner action

We have the four level Kirkpatrick evaluation model. DOL aims for level 2 or 3.

Level 2 can be assessed via practice questions, knowledge checks, and pre-post testing.

Level 3 testing is conducted 6 weeks to 6 months after training.

Overall, good document.

  1. Now we have something from the United States Distance Learning Association (another one!?!) called _An instructional media selection guide for distance learning_ ( The USDLA has conferences and certifications. It seems to focus more on providers, online universities, etc. 

The document notes that distance learning must include -- at a minimum -- physical distance between student and teacher; content providing organization; curriculum; measurement of learning. Apparently there have been over 70 different learning styles identified but low validity and reliability casts some questions. It also cites UCLA's Cooperative Institutional Research Program work on generational differences. Apparently, they are slow and gradual. There are no significant differences from one generation to the next (the study did, however, note a pattern of increased entitlement, decreasing literacy, and decreasing factual knowledge).


Instructional issues to be considered include:
  • Identification of knowledge and skills gaps
  • Assessment and measurement tools
  • Level of interaction
  • Instructional strategies
  • Complexity of content
  • Rate of content change
  • Level and domain (cognitive, affective, psychomotor) of learning objectives
  • Delivery issues to consider include audience size and distribution, cost (in-house vs outsource; availability of infrastructure; delivery hardware endpoints -- teleconferencing, satellite, WAN/LAN, TV/monitor, portability)




  1. The eLearning Guild.

Lots of good weeds-level content about various topics, including case studies.