Politely stated, many people think there is room for improvement in the FAA’s knowledge test questions and associated guidance documents (e.g., Airplane Flying Handbook). The FAA agrees. That’s why the agency decided to seek expert advice on shaping the content, process, methodology, and priorities for these critical materials to enhance GA safety. To that end, in September 2011 the FAA assembled a group comprised of more than a dozen people whose names and/or organizations you would recognize as “the” experts in the GA education, training, and test preparation fields. Officially chartered as an “ARC,” or Aviation Rulemaking Committee, that can recommend both regulatory and non-regulatory changes, the group met for the first time in early October 2011. The Airman Testing Standards and Training ARC submitted its report and recommendations – essentially, a blueprint for building better knowledge tests – to the agency in April 2012. The FAA has made this document available on its website (www.faa.gov/aircraft/draft_docs/arc/), but here’s a summary of its key proposals.

Content – Airman Certification Standard

Though not listed first (it is actually Recommendation 3), the core concept is the ARC’s recommendation for the FAA to transition to a single “Airman Certification Standards” (ACS) document for each certificate and rating. Under this approach, the aeronautical knowledge topics enumerated in 14 CFR part 61 will be clearly aligned with, and integrated into, the appropriate Areas of Operation in the existing Practical Test Standards (PTS). The ARC also recommended adding task-tailored sections on risk management to each Area of Operation.

This approach has several benefits.

• First, it will help the FAA identify and eliminate “irrelevant” questions, because it will provide the framework needed to clearly align and map specific areas of aeronautical knowledge and risk management to the stick-and-rudder skills needed for safe operation in today’s National Airspace System (NAS).
• Second, it will offer one-stop-shopping guidance to students, instructors, DPEs, and training providers on what the FAA expects in terms of knowledge, skills, and attitudes.
• Third, the new ACS, once developed, will help the FAA align and streamline guidance material and develop targeted test questions.

Process – Stakeholder Body

The ARC’s first recommendation offers the means to bring the ACS to life with a view to real-world training, testing, and operation. Specifically, the ARC recommended that the FAA establish a stakeholder body consisting of subject matter experts from industry and relevant FAA policy offices to develop and review the ACS, undertake the review and development of handbook content, and assist with the developing and “boarding” (review) of knowledge test questions. In a related point (Recommendation 6), the ARC also recommended that the FAA revise its internal processes to ensure that all relevant FAA policy offices participate meaningfully in the development and review of airman testing standards and training materials.

Methodology – Question Development and Review

Given that the ARC arose from concerns about the quality and relevance of FAA knowledge test questions, its members devoted considerable time to discussing options and ideas for improvement. Although development of the ACS (see above) offers the principal means to a stable and long-term resolution to existing concerns, two of the ARC’s recommendations (4 and 5) address members’ views on test question development – that is, what kinds of questions the FAA should seek to develop. Though recognizing that some knowledge areas require rote memorization, the ARC urged the FAA to incorporate scenario-based questions wherever appropriate and, in all cases, to ensure that questions are: pertinent to safe operations, necessary for sound airmanship, and relevant to the way pilots operate in the real world.

As you may have seen in press reports on the ARC’s work, ARC members also recommended that the FAA return the knowledge test item data bank to the public domain. Though the FAA concurs with the spirit and intent of this recommendation, which is to use a test development process open enough to benefit from expert input and review, the ARC report also recognizes the FAA’s obligation to safeguard both the real and perceived integrity of its testing process. The FAA is exploring a range of options for meeting both objectives.

Priorities – What’s Up First?

Recognizing that the FAA must set priorities, the ARC recommended that the stakeholder body focus initially on developing the ACS and associated guidance and testing materials for three “foundational” certificates and ratings: private pilot certificate, flight instructor certificate, and instrument rating.

Stay tuned as the FAA and stakeholders work to advance from blueprint to building.  (FAA Safety Briefing – SepOct 2012)