Defense Acquisition Research Journal Thinking Critically about Defense Acquisition. The Defense Acquisition Review Journal (ARJ) is a scholarly peer-reviewed journal published by the Defense Acquisition University (DAU). All submissions receive a blind review to ensure impartial evaluation. Articles represent the views of the authors and do not necessarily reflect the opinion of the DAU or the Department of Defense. January 2020 Vol. 27 No. 1 | Issue 91 Journal DOI: 10.22594/dauARJ.issn.2156-8391 Elect ISSN: 2156-8405
DEFENSE ACQUISITION RESEARCH JOURNAL A Publication of the Defense Acquisition University
THINKING CRITICALLY about DEFENSE ACQUISITION
| ISSUE 91
January 2020 Vol. 27 No. 1
Critical Thinking for the Federal Auditor Gabrielle G. McClure-Nelson
Complexity in an Unexpected Place: Quantities in Selected Acquisition Reports Gregory A. Davis and David M. Tate
Risk-Based ROI, Capital Budgeting, and Portfolio Optimization in the Department of Defense Johnathan Mun
ARTICLE LIST ARJ EXTRA
The Defense Acquisition Professional Reading List Perspectives on Defense Systems Analysis: TheWhat, theWhy, and theWho, but Mostly the How of Broad Defense Systems Analysis Written by William P. Delaney, with Robert G. Atkins, Alan D. Bernard, Don M. Boroson, David J. Ebel, Aryeh Feder, Jack G. Fleischman, Michael P. Shatz, Robert Stein, and Stephen D. Weiner Reviewed by Kevin Garrison
DEFENSE ACQUISITION RESEARCH JOURNAL A Publication of the Defense Acquisition University
Ms. Ellen Lord Under Secretary of Defense for Acquisition and Sustainment Mr. James P. Woolsey President, Defense AcquisitionUniversity Mr. Joseph Johnson Chief of Staf, Defense AcquisitionUniversity Mr. Leo Filipowicz Director, DAUOperations Support Group
Editorial Board Dr. Larrie D. Ferreiro Chairman and Executive Editor RADM James Greene, USN (Ret.) Naval Postgraduate School Dr. Joseph L. Ilk Defense AcquisitionUniversity Mr. William Lucyshyn University ofMaryland Dr. Thomas A. Mazzuchi The GeorgeWashingtonUniversity
Mr. Richard Altieri Dwight D. Eisenhower School for National Security andResource Strategy Dr. Michelle Bailey Catholic University of America Dr. Don Birchler Center for Naval Analyses Corporation Mr. Kevin Buck TheMITRECorporation Mr. John Cannaday Defense AcquisitionUniversity Dr. John M. Colombi Air Force Institute of Technology Dr. William T. Eliason Dwight D. Eisenhower School for National Security andResource Strategy Dr. Steve Fasko Defense AcquisitionUniversity Dr. J. Ronald Fox Harvard Business School Mr. David Gallop Defense AcquisitionUniversity
Dr. Christopher G. Pernin RANDCorporation Dr. Mary C. Redshaw Dwight D. Eisenhower School for National Security andResource Strategy Dr. Yvette Rodriguez Defense AcquisitionUniversity Dr. Richard Shipe Dwight D. Eisenhower School for National Security andResource Strategy Dr. Keith Snider Naval Postgraduate School Dr. John Snoderly Defense AcquisitionUniversity Ms. Dana Stewart Defense AcquisitionUniversity Dr. David M. Tate Institute for Defense Analyses Dr. Trevor Taylor Royal United Services Institute (UK) Mr. Jerry Vandewiele Defense AcquisitionUniversity
Mr. John McCormack CranfeldUniversity (UK) Dr. John G. McGinn
GeorgeMasonUniversity Dr. James Moreland Ofce of Under Secretary of Defense for
Acquisition and Sustainment Dr. Robert F. Mortlock Naval Postgraduate School Dr. Troy J. Mueller TheMITRECorporation
ISSN 2156-8391 (print) ISSN 2156-8405 (online) DOI: https://doi.org/10.22594/dau.012020-91.27.01
The Defense Acquisition Research Journal , formerly the Defense Acquisition Review Journal , is published quarterly by the Defense Acquisition University (DAU) Press and is an ofcial publication of the Department of Defense. Postage is paid at the U.S. Postal facility, Fort Belvoir, VA, and at additional U.S. Postal facilities. Postmaster, send address changes to: Editor, Defense Acquisition Research Journal , DAU Press, 9820 Belvoir Road, Suite 3, Fort Belvoir, VA 22060-5565. The journal-level DOI is: https://doi.org/10.22594/ dauARJ.issn.2156-8391. Some photos appearing in this publication may be digitally enhanced.
Articles represent the views of the authors and do not necessarily refect the opinion of DAU or the Department of Defense.
DEFENSE ACQUISITION RESEARCH JOURNAL A Publication of the Defense Acquisition University
Managing Editor, Chief of Visual Arts & Press Norene L. Johnson Assistant Editor Emily Beliles Art Director Michael Bubar-Krukowski ProductionManager
Frances Battle Graphic Designer, Digital Publications Nina Austin Technical Editor Collie J. Johnson Associate Editor Michael Shoemaker Copy Editor,
CirculationManager Debbie Gonzalez Multimedia Assistant Noelia Gamboa Editing, Design, and Layout Chickasaw Nation Industries The C3 Group
2 28
Critical Thinking for the Federal Auditor Gabrielle G. McClure-Nelson
Constituents agree that critical thinking is an important competency for the federal auditor. Herein, the author researches howthe federal auditor perceives the importance of critical thinking and what practical means exist to develop this important skill.
Complexity in an Unexpected Place: Quantities in Selected Acquisition Reports Gregory A. Davis and David M. Tate Quantity reporting in the SelectedAcquisitionReport (SAR) is the focus of this article. The authorsmake the case that SARs aremuch like custommanufactured parts in that each one is unique, but good processes could still make themmore uniformand useful.
60 Risk-Based ROI, Capital Budgeting, and Portfolio Optimization in the Department of Defense Johnathan Mun
This research illustrates and recommends approaches of modeling methodology and development of military value metrics, and how to combine them into a defensible, reusable, extensible, and practical approach within portfolios of programs. The author showcases how the methodologies can be applied to develop a comprehensive and analytically robust case study that senior leadership at the DoDmight utilize to make optimal decisions.
CONTENTS | Featured Research vii From the Chairman and Executive Editor
x
Research Agenda 2020
108 Professional Reading List Perspectives on Defense Systems Analysis: The What, the Why, and the Who, but Mostly the How of Broad Defense Systems Analysis Written by William P. Delaney, with Robert G. Atkins, Alan D. Bernard, Don M. Boroson, David J. Ebel, Aryeh Feder, Jack G. Fleischman, Michael P. Shatz, Robert Stein, and Stephen D. Weiner and reviewed by Kevin Garrison 112 Current Research Resources in Defense Acquisition A selection of new research curated by the DAU Research Center and the Knowledge Repository 118 2020 Edward Hirsch Acquisition and Writing Competition 120 Defense ARJ Guidelines for Contributors The Defense Acquisition Research Journal (ARJ) is a scholarly peer-reviewed journal published by the Defense Acquisition University. All submissions receive a blind review to ensure impartial
evaluation. 130 Call for Authors
We are currently soliciting articles and subject matter experts for the 2020 Defense Acquisition Research Journal (ARJ) print year. Please see our guidelines for contributors for submission deadlines 134 Recognition of Reviewers 2019 We would like to express our appreciation to all of the subject matter experts who volunteered to participate in the Defense Acquisition Research Journal peer review process.
FROM THE CHAIRMAN AND EXECUTIVE EDITOR Dr. Larrie D. Ferreiro
The t heme of t h i s ed it ion of t he Defense Acquisition Research Journal is “Thinking Critically about Defense Acquisition.” In order to open the aper ture for cr itica l thinking , the Defense AcquisitionResearch Journal has updated its guidelines for contributors to now include submissions for case histories based on defense acquisition programs or eforts. Case histories difer from case studies in that case histories (like research papers) draw specific
conclusions based on analysis as opposed to case studies, which are primarily intended for classroom and pedagogical use, and generally terminate with a jumping-off point for the student or class to come to decisions. We invite potential authors to consider submitting case history manuscripts. Cases from all acquisition career felds and/or phases of the acquisition life cycle will be considered. They may be decision-based, descriptive, or explanatory in nature. Cases must be sufficiently focused and complete (i.e., not open-ended like classroom case studies) with relevant analysis and conclusions. All cases must be factual and authentic. [Please note that we do not accept fctional cases.] The frst research article in this issue, “Critical Thinking for the Federal Auditor" by Gabrielle G. McClure-Nelson, identifies to what extent critical thinking skills are considered an important competency for federal auditors, given the often tightly constrained and rules-focused nature of auditing government contractors.
ix
A Publication of the Defense Acquisition University
January 2020
The second article, authored by Gregory A. Davis and David M. Tate and titled, “Complexity in an Unexpected Place: Quantities in Selected Acquisition Reports," notes that the definition of unit quantities in acquisition programs is not consistent (for example, the units produced at the end of a long production run are substantially diferent from the early ones). The authors ofer explanations as to why this is the case, and possible methods for improving the reporting requirement. The third article, "Risk- based ROI, Capital Budgeting, and Portfolio Optimization in the Department of Defense" by JohnathanMun, describes an analytical modeling process to help the U.S. Department of Defense (DoD) senior leadership with making decisions about risk-based capital budgeting and optimizing acquisition and program portfolios. This issue’s Current Research Resources in Defense Acquisition focuses on the use of Other Transaction Authority (OTA) in government contracting. It contains descriptions of several key resources, along with links to the DAUKnowledge Repository sites. The featured reading in this issue’s Defense Acquisition Professional Reading List is Perspectives on Defense Systems Analysis: The What, the Why, and the Who, but Mostly the How of BroadDefense Systems Analysis byWilliamP. Delaney, withRobert G. Atkins, Alan D. Bernard, Don M. Boroson, David J. Ebel, Aryeh Feder, Jack G. Fleischman, Michael P. Shatz, Robert Stein, and Stephen D. Weiner, reviewed by Kevin Garrison. Dr. Michael J. Pryce has departed the Defense ARJ Editorial Board. We thank him for his service and wish him well. We welcome Mr. John McCormack to the Editorial Board. Please note at the end of this journal the re-issued Call for Papers for the 2020 DAU Alumni Association Edward Hirsch Acquisition and Writing Competition. Due date is March 15, 2020.
Dr. Larrie D. Ferreiro Chairman and Executive Editor Defense ARJ
x
https://www.dau.edu
From the Art Director Michael Bubar-Krukowski
As we start a new decade, so does the Defense Acquisition Research Journal (ARJ)— and with some big changes. Like most publications, the Defense ARJ tries to keep up with the latest design trends and push the standards for design in the research journal world. We pride ourselves on this at the Defense ARJ and are always trying to keep the journal fresh and accessible to everyone. One of the big changes you will notice is the updated logo. Like most great brands, the Defense ARJ logo should evolve over the years while still keeping aspects that make it recognizable. The Defense ARJ has been long overdue for an update. Our new logo still pays homage to the previous version that everyone knows, but the updated version has been cleaned up to be more on trend. The signature quill exhibits the most drastic change, becoming cleaner while also creating more opportunities to add color. Also the chosen typeface is a slight update from the previous version that adds a cleaner feel, allowing the quill to shine and creating a timeless look. Other changes have been made to the design of the journal that should hopefully revitalize the aesthetic and keep the Defense ARJ at the top of its class. You will notice a redesigned table of contents, new ads, and more! For several years now the Defense ARJ has been winning awards for design. Going forward, we strive to keep up the same standards of excellence while also becoming more competitive in the world of design and publication.
A Publication of the Defense Acquisition University
https://www.dau.edu
DAU CENTER FOR DEFENSE ACQUISITION RESEARCH AGENDA 2020
This ResearchAgenda is intended tomake researchers aware of the topics that are, or should be, of particular concern to the broader defense acquisition community within the federal government, academia, and defense industrial sectors. The center compiles the agenda annually, using inputs from subject matter experts across those sectors. Topics are periodically vetted and updated by the DAU Center’s Research Advisory Board to ensure they address current areas of strategic interest. The purpose of conducting research in these areas is to provide solid, empirically based fndings to create a broad body of knowledge that can informthe development of policies, procedures, and processes in defense acquisition, and to help shape the thought leadership for the acquisition community. Most of these research topics were selected to support the DoD’s Better Buying Power Initiative (see http:// bbp.dau.edu). Some questions may cross topics and thus appear in multiple research areas. Potential researchers are encouraged to contact the DAU Director of Research (research@dau.edu) to suggest additional research questions and topics. They are also encouraged to contact the listed Points of Contact (POC), who may be able to provide general guidance as to current areas of interest, potential sources of information, etc.
xii
January 2020
Competition POCs • John Cannaday, DAU: john.cannaday@dau.edu • Salvatore Cianci, DAU: salvatore.cianci@dau.edu • Frank Kenlon (global market outreach), DAU: frank. kenlon@dau.edu Measuring the Efects of Competition • Whatmeans are there (or can be developed) tomeasure the efect on defense acquisition costs of maintaining the defense industrial base in various sectors? • What means are there (or can be developed) to measure the effect of utilizing defense industrial infrastructure for commercial manufacture, and in particular, in growth industries? In other words, can we measure the efect of using defense manufacturing to expand the buyer base? • What means are there (or can be developed) to determine the degree of openness that exists in competitive awards? • What are the different effects of the two best value source selection processes (trade-of vs. lowest price technically acceptable) on programcost, schedule, and performance? Strategic Competition • Is there evidence that competition between system portfolios is an effective means of controlling price and costs? • Does lack of competition automatically mean higher prices? For example, is there evidence that sole source can result in lower overall administrative costs at both the government and industry levels, to the effect of lowering total costs? • What are the long-term histor ica l trends for competition guidance and practice in defense acquisition policies and practices?
xiii
A Publication of the Defense Acquisition University
https://www.dau.edu
• To what ex tent a re contracts being awa rded noncompetitively by congressional mandate for policy interest reasons? What is the efect on contract price and performance? • What means are there (or can be developed) to determine the degree to which competitive program costs are negatively afected by laws and regulations such as the Berry Amendment, Buy AmericanAct, etc.? • The DoD should have enormous buying power and the ability to infuence supplier prices. Is this the case? Examine the potential change in cost performance due to greater centralization of buying organizations or strategies. Efects of Industrial Base • What are the efects on program cost, schedule, and performance of having more or fewer competitors? What measures are there to determine these efects? • Whatmeans are there (or can be developed) tomeasure the breadth and depth of the industrial base in various sectors that go beyond simple head-count of providers? • Has change in the defense industrial base resulted in actual change in output? How is that measured? Competitive Contracting • Commercial industry often cultivates long-term, exclusive (noncompetitive) supply chain relationships. Does this model have any application to defense acquisition? Under what conditions/circumstances? • What is the effect on program cost, schedule, and performance of awards based on varying levels of competition: (a) “Efective” competition (two or more offers); (b) “Ineffective” competition (only one offer received in response to competitive solicitation); (c) split awards versus winner take all; and (d) sole source.
xiv
January 2020
Improve DoD Outreach for Technology and Products from Global Markets • How have militaries in the past benefted from global technology development? • How/why have mi l ita r ies missed the la rgest technological advances? • What are the key areas that require theDoD’s focus and attention in the coming years to maintain or enhance the technological advantage of its weapon systems and equipment? • What types of eforts should theDoDconsider pursuing to increase the breadth and depth of technology push eforts in DoD acquisition programs? • How effectively are the DoD’s global science and technology investments transitioned into DoD acquisition programs? • Are the DoD’s applied research and development (i.e., acquisition program) investments efectively pursuing and using sources of global technology to afordably meet current and future DoD acquisition program requirements? If not, what steps could the DoD take to improve its performance in these two areas? • What are the strengths and weaknesses of the DoD’s global defense technology investment approach as compared to the approaches used by other nations? • What are the strengths and weaknesses of the DoD’s global defense technology investment approach as compared to the approaches used by the private sector—both domestic and foreign entities (companies, universities, private-public partnerships, think tanks, etc.)? • Howdoes theDoDcurrently assess the relative benefts and risks associated with global versus U.S. sourcing of key technologies used inDoD acquisition programs? Howcould theDoD improve its policies and procedures in this area to enhance the benefts of global technology sourcing while minimizing potential risks?
xv
A Publication of the Defense Acquisition University
https://www.dau.edu
• How could current DoD/U.S. Technology Security and Foreign Disclosure (TSFD) decision-making policies and processes be improved to help the DoD better balance the benefts and risks associated with potential global sourcing of key technologies used in current and future DoD acquisition programs? • How do DoD primes and key subcontractors currently assess the relative benefts and risks associated with global versus U.S. sourcing of key technologies used in DoD acquisition programs? How could they improve their contractor policies and procedures in this area to enhance the benefts of global technology sourcing while minimizing potential risks? • How could current U.S. Export Control System decision-making policies and processes be improved to help the DoD better balance the benefits and risks associated with potential global sourcing of key technologies used in current and future DoD acquisition programs? Comparative Studies • Compare the industrial policies of military acquisition in different nations and the policy impacts on acquisition outcomes. • Compare the cost and contract performance of highly regulated public utilities with nonregulated “natural monopolies” (e.g., military satellites, warship building). • Compare contracting/competition practices between the DoD and complex, custom-built commercial products (e.g., ofshore oil platforms). • Compare programcost performance in variousmarket sectors: highly competitive (multiple oferors), limited (two or three oferors), monopoly. • Compare the cost and contract performance ofmilitary acquisition programs in nations having single “purple” acquisition organizations with those having Service- level acquisition agencies.
xvi
- Norene Johnson, Emily Beliles, and Michael Bubar Krukowski Defense Acquisition University Press Fort Belvoir, VA
WINNER
We’re on the Web at: http://www.dau.edu/library/arj
ISSUE 91 JANUARY 2020 VOL. 27 NO. 1
1
CRITICAL THINKING AUDITOR FEDERAL for the
Gabrielle G. McClure-Nelson
In the overly constrained space of the federal audit environment, to what extent can critical thinking skills be applied in a profession characterized by arduous public trust expectations, controlling auditing standards, prescriptive federal acquisition policies, frequently changing guidance, continual peer oversight, and the slow implementation of audit findings? Promoting the increased use of private sector auditors may suggest that federal auditors perceive competencies differently. However, a recent survey administered to 645 auditors of a federal audit agency region indicated that the majority of the core competencies identified by the American Institute of Certified Public Accountants are perceived as relevant in auditing government contractors. However, of concern, the data were mixed in support of critical thinking as an important competency. Given employer preference for skills in this area, the author attempts to identify applications to increase auditor critical thinking skills and to offer suggestions for increasing the relevance of the federal audit.
- DOI: https://doi.org/10.22594/dau.19 - 830.27.01 Keywords: Generally Accepted Government Auditing Standards (GAGAS),
Yellow Book, American Institute of Certifed Public Accountants (AICPA), Core Competencies, Section 809 Panel
Image designed by Michael Krukowski
Critical Thinking for the Federal Auditor
https://www.dau.edu
The ability to succeed in an over-constrained space was recently identified as an important leadership attribute by Microsoft Chief Executive Officer SatyaNadella (Jones, 2019). But, in the overly constrained space of the federal audit environment, what does success look like for the federal auditor who aspires to exercise leadership skills? To what extent can critical thinking skills necessary for leadership be applied in a profession characterized by arduous public trust expectations, controlling auditing standards, prescriptive federal acquisition policies, frequently changing guidance, continual peer oversight, and the slow implementation of audit findings?
Importance of the Federal Auditor in Acquisition
Federal auditors play a vital role in the acquisition process. In 2018, federal spendingwas subject to evaluationby about 11,000auditors primarily employed by the Departments of Defense (DoD) and Health and Human Services (DHHS) (Office of Personnel Management, 2018). The Section 809 Panel, established by Congress in the Fiscal Year 2016
National Defense Authorization Act (National Defense Authorization Act [NDAA], 2016), describes defense auditors as “essential components of the Department of
Defense’s system of contracting internal controls” (Section 809 Panel, 2018a, p. 54).
Impediments to Auditor Critical Thinking Several factors can inhibit the federal
auditor’s exercise of critical thinking and leadership skills, among them: untimely response to audit findings, auditing standards oversight, changing mission guidance, and threats to job stability.
4
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
Untimely Response to Audit Findings Inaction to audit findings can erode federal auditor morale and compromise auditor commitment to success. Annually, theDHHS publishes the top un implemented recommendations from its Office of Inspector General audits and evaluations (DHHS, 2018). The Department of Defense Inspector General (DoDIG) distributed a similar 2018 compendium describing about $2 billion of potential monetary benefits from open recommendations (DoDIG, 2018a). Unsustained audit fndings on contractor business systems, federal cost accounting standards, andmillions of dollars of audit exceptions reported by other DoD agencies were disclosed by the DoDIG for the years 2016 through 2019 (DoDIG, 2016, 2017a, 2018b, 2019). In 2018, federal spending was subject to evaluation by about 11,000 auditors primarily employed by theDepartments of Defense (DoD) and Health and Human Services (DHHS). Auditing Standards Oversight During this same period, federal auditors were cited for insufcient adherence to auditing standards. In 2017, the DoDIG reported the Army failed its 2017 quality control system review (DoDIG, 2017b); and in that same year, the U.S. Government Accountability Office (GAO) identifed untimely audits as a reason for delinquent contract close-outs at the Departments of Defense, State, Transportation, and the National Aeronautics and Space Administration (GAO, 2017b). Changing Mission Guidance Rapidly changing mission guidance can trigger auditor fatigue. The customer identity confusion at the Defense Contract Audit Agency (DCAA) is a signifcant example. The Section 809 Panel (2019, p. 25) reports that about 10 years ago, in response to critical audit independence findings, DCAA identifed the “taxpayer” customer in itsmission statement. TheGAO promptly challenged this action by noting DCAA’s primary role is to advise contracting ofcers. The Section 809 Panel, in turn, bested GAO’s challenge by recommending DCAA not only advise contracting ofcers, but provide “education and training”—a recommendation that appears eligible for unintended independence abuse (Section 809 Panel, 2018a, pp. 64–65, 67).
5
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
Contradictory guidance not only promotes auditor weariness, but wariness, regarding management trustworthiness. In spite of its best intentions to recognize contracting ofcer needs, the Section809Panel recommendations appear redundant in that DCAA’s current mission statement (available on its external website) already acknowledges both the acquisition team (of which the contracting officer is part) and the taxpayer (DCAA, n.d., Mission ). Likewise, the DoDIG describes its audit function as including actionable recommendations, that is, actions that improve DoD programs and operations (DoDIG, n.d.). Finally, providing advice to the contracting officer is already compatible with government auditing standards that require the auditor to assist oversight ofcials by “making recommendations for corrective action” (GAO, 2018c, para. 7.50, p. 139).
Threats to Job Stability Lastly, the federal auditor deals with the continual threat of job encroachment by private sector auditors. Both the National Defense Industrial Association (Thomas, 2017) and the Section 809 Panel recently recommended the hire of “independent professional auditors” (Section 809 Panel, 2019, p. 25). However, the promotion of commercial auditors
6
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
as a remedy for federal audit failings appears an easy suggestion that may warrant further deliberation in light of the millions of dollars recently paid for audit failure by the most prestigious accounting frms: a $335 million Price Waterhouse professional negligence settlement in 2019 (Johnson & Schroeder, 2019), a $12 million Ernst & Young failed-audits settlement in 2016 (Securities &Exchange Commission, 2016), and a recent consideration by General Electric to fire KPMG after a 109-year relationship due to signifcant undisclosed liabilities and other accounting issues (Gryta & Lublin, 2018, pp. B.3, 3). In fact, some government acquisition ofcials have already expressed concern that public accounting frmsmay lack “sufcient understanding” of federal contractor business systems (GAO, 2019a, p. 29). Commercial auditors may not be prepared for the complexity of subject matter with which the federal auditor deals. In March 2019, a GAO review identified more than $3.4 billion in subcontract costs incurred over a 10-year period that had not been audited due, in part, to complex ownership relationships among contractors and subcontractors (GAO, 2019b). Likewise, in November 2018, in connection with its audit of the Internal Revenue Service (IRS) 2017–2018 fnancial statements, the GAO noted that the complex statistical process the IRS uses to estimate the amounts of taxes receivable contributed to material weakness in internal control over unpaid assessments (GAO, 2018b). Private sector auditors may be less familiar with these complexities than their federal counterparts. Auditor Competencies Research Promoting the use of nonfederal auditors may suggest that federal auditors perceive the importance of competencies differently than their private sector colleagues. However, recent research conducted by the author fnds federal auditors agree with the professional core skills identified by the American Institute of Certified Public Accountants (McClure-Nelson, 2013). The American Institute of Certifed Public Accountants (AICPA) publishes a listing of core competencies needed to enter the accounting profession—a listing that hasn’t signifcantly changed since the AICPA 2011 “Horizons 2025” report, or AICPA 1999 competency listing (AICPA, 2018a). For the federal auditor, many of whom are certified public accountants (CPAs), the AICPA competencies are largely compatible with the proficiencies identifed by the GAO generally accepted government auditing standards (GAGAS) in its Yellow Book. (See Table 1 for a crosswalk of the majority of these competencies.)
7
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
TABLE 1. PARTIAL CROSSWALK OF AICPA COMPETENCIES TO GAO YELLOW BOOK AUDITOR PROFICIENCIES Partial Crosswalk of AICPA Competencies to GAO Government Auditing Standards
AICPA 2018 Precertification Core Competency Framework
GAO 2018 Yellow Book
Pillar
Competency
¶ Competency
Accounting Risk assessment, analysis 3.116 An auditor’s consideration of the risk level of each engagement …
The subject matter of an attestation engagement may take many forms, including the following: e. systems and processes …
Accounting System and process management
1.19
Accounting Reporting
1.06 GAGAS contains requirements and guidance dealing with …. reporting.
5.30 Consultation uses appropriate research resources ….
Accounting Research
Subject matter that directly enhances auditors’ professional expertise to conduct engagements may include … h. information technology
Accounting Technology and tools
4.24
Those charged with governance refers to the individuals responsible for overseeing the strategic direction of the entity …. The audit organization’s policies and procedures may address consistency in the quality of engagement performance. This is often accomplished through …. industry- specific or subject matter-specific guidance materials. Examples of internal control audit objectives include determining whether b. resources are used in compliance with laws, regulations …
Business
Strategic perspective
1.04
Global and industry perspectives
Business
5.26
Business
Resource management
1.24
… Matters addressed may include the following: j. … applicable legal and regulatory requirements.
Governance perspective - Legal and regulatory
Business
5.26
8
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
Auditors often conduct GAGAS engagements under a contract with a party other than the officials of the audited entity or pursuant to a third party request. Subject matter that directly enhances auditors’ professional expertise to conduct engagements may include … b. general ethics and independence …
Business
Customer perspective
7.04
Professional
Ethical conduct
4.24
Examples of prospective analysis objectives include providing conclusions … b. program or policy alternatives …
Professional
Decision-making
1.26
Objectively identify ….alternative courses of action
Appropriate teamwork and training help less experienced members of the engagement team to clearly understand the objectives of the assigned work. Subject matter that directly enhances auditors’ professional expertise to conduct engagements may include … f. leadership … Subject matter that directly enhances auditors’ professional expertise to conduct engagements may include … d. communicating clearly and effectively, both orally and in writing … Subject matter that directly enhances auditors’ professional expertise to conduct engagements may include … e. managing time and resources …
Professional
Collaboration
5.38
Professional
Leadership
4.24
Professional Communication
4.24
Professional
Project management
4.24
Since the opinions of federal auditors have been inadequately solicited regarding competencies required for success, a recent studywas undertaken by the author to assess the relative importance of theAICPAcore competency framework to the federal auditor of government contractors (McClure- Nelson, 2013). The study was based on the 1999 AICPA competency framework that categorized skills as either functional, broad business, or personal (presently referred to by the AICPA as accounting, business, and professional categories). A survey was administered to 645 auditors of a federal audit agency region requesting opinions of the importance of AICPA-defined competencies.
9
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
Research Questions The main research question of the study was whether the categorized AICPA core competencies adequately describe the skills and attributes required in a federal auditing environment. (SeeTable 2 for a complete listing of the skills included in these categories.) Questions andhypotheses included: • Research Question 1: To what extent are the AICPA core competencies relevant in auditing federal contractors? Auditors were expected to find that the majority of AICPA core competencies were relevant and that risk analysis would be identifed as an important competency. • Research Question 2: To what extent are some AICPA core competencies more important than others to auditors of federal contractors? Auditors were expected to rank the functional accounting skills as well as the personal skills as more important than the business skills. Of the business skills, auditors were expected to identify strategic critical thinking as the most important of these competencies. • Research Question 3: To what extent are opinions diferent regarding the relevance and ranking of the AICPA core competencies given increased job experience of the auditor of federal contractors? The more experienced federal auditors were expected to demonstrate greater appreciation for the business competencies than their junior counterparts. • ResearchQuestion4: Towhat extent are other competencies, not identifed by the AICPA, important to the work of auditors of federal contractors? Senior auditors were expected to identify maintaining independence as an additional required competency.
10
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
TABLE 2. LISTING OF CORE COMPETENCY CATEGORIES AND COMPETENCIES CATEGORIES AND COMPETENCIES Functional Personal Broad Business Decision Modeling
Communication Interaction Leadership Professional Demeanor Project Management Problem Solving
International Global Industry Sector Legal Regulatory
Measurement Risk Analysis
Research Reporting
Marketing Client Focus Resource Management Strategic Critical Thinking
Leverage Technology—applicable to all categories
Protocols to Protect Human Subjects Protection of Human Subjects (2019) protocols were based on the requirements set forth in 21 C.F.R. Pts. 50, 56 and 45 C.F.R. Pt. 46. The researchwas reviewed and approved by theWilmingtonUniversityHuman Subject Review Committee. Procedures for obtaining informed consent included notifying participants that participation would contribute to academic research regarding the best preparation for a career in federal auditing, communicating that the study did not involve payments or incentives, and fnally, that participation was voluntary and anonymous. Limitations of the Research The study is limited in that not all federal auditors audit federal contractors, nor do all federal auditors follow the same auditing procedures for all types of audits. For instance, some federal auditors audit other government components. The research, however, may prove applicable to nongovernment auditors who audit federal contractors. Research Methodology The population was a geographic region of a sizeable federal audit agency that audits the cost representations of government contractors. The single geographic region was representative of other agency regions in that all auditors receive the same training at a common educational site, follow the same audit policy prescribed by agency headquarters, adhere to the same agency-prescribed audit standards, and use the same agency audit programs. In addition, auditors transfer among the agency regions. Two groups were included in the population; the frst consisted of junior auditors at General Service (GS) grade levels GS-7, 9, and 11; and the second consisted of senior-level auditors, at GS-12 and 13 grades. At the time of the
11
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
study, the group of senior-level auditors totaled about 380 and the group of junior auditors totaled about 265 (for a total of 645) in the single region. A total of 263 usable responses was received (109 junior auditors and 154 senior auditors)—a 41% response rate (263/645). Managers above the GS-13 level were excluded from the population given their small number relative to the other GS levels.
Research Instrument Likert Scale survey questions were developed from the AICPA core competencies described on theAICPAEducational CompetencyAssessment website. The AICPA grouped core competencies into three broad categories as shown in Table 2. Likert questions were coded to the AICPA core competencies in order to draw meaningful conclusions from analysis of the data. Since the survey was newly developed for purposes of this study, three federal CPA auditors were requested tomatch the Likert questions to the AICPA categories and competencies identifed by the researcher. Fleiss’ Kappa coefcients were then computed to measure inter-rater agreement. Acceptable Cronbach alpha statistics were obtained that measured internal consistency and reliability of the questions developed for each competency and for competencies within the three categories. SurveyMonkey was used to administer the survey November 1–16, 2012. Survey questions are shown in Table 3.
12
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
TABLE 3. RESEARCH SURVEY QUESTIONS Question: How important is it for the federal auditor in your Agency to…?
No. AICPA Core Competencies
Analyze changes in the fnancial risks of the contractor’s industry/sector
1
Understand why controls cannot completely eliminate the risk of fraud
2
Demonstrate objectivity and integrity consistent with the standards of auditing
3
Establish working relationships with audit requestors
4
Interact and cooperate productively and maturely with others
5
Use technology-assisted tools to assess and control risk and document work Communicate information and concepts with conciseness and clarity when writing and speaking Interpret research fndings from a variety of viewpoints
6
7
8
Communicate the contractor’s planning process, strategy, and goals
9
Identify pros and cons of alternative methods of measurement
10
Consider how human resource management afects a contractor
11
Develop innovative or creative solutions to problems
12
Report fndings in accordance with auditing standards
13
14
Inspire and motivate team members
Analyze the impact of changes in contracting laws and regulations
15
Use mathematical or scientifc models to evaluate decision alternatives
16
Prioritize and delegate various aspects of a project in order to allocate resources Identify global threats and opportunities impacting contractors
17
18
13
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
Data Collection Procedures Data collected fromrespondentswere coded and entered into a computer data fle for analysis usingMinitab, IBMSPSS (ver. 20.0) statistical software and Microsoft Excel. Demographic data related to the number and nature of the two groups of junior and senior auditors were determined using frequencies and percentages in order to develop a profle of the respondents. Descriptive statisticswere computed regarding frequencies and percentages of responses. Multiple regression determined whether competency rating was a function of GS level or a function of GS levels plus gender or age. In order to determinewhether diferences existed between the two groups of junior and senior auditors, an independent t -test was computed to compare the means between the two groups for each of the competencies. Multiple regression determined which independent variables (demographic factors) were statistically signifcant in infuencing the outcome of the dependent variable (responses to Likert questions). Results of the Study • Question 1. The data supported that federal auditors found the majority of the AICPA core competencies relevant in auditing government contractors and that the competency of risk analysis is important. (See Table 4 for the listing of competencies ranked by importance.) • Question 2. The data supported that auditors found the functional accounting skills aswell as the personal skills, more important than the business competencies. However, the data only partially supported the hypothesis that auditors would identify strategic critical thinking as the most important business competency. • Question 3. Increasing age correlated with more importance assigned to three of the six business competencies (industry/ sector, strategic critical thinking and international/global), providing limited support to the hypothesis that senior auditors would appreciate business competencies more than the junior auditors. • Question 4. Finally, the data did not support the hypothesis that senior auditors would identifymaintaining independence as an additional required competency.
14
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
TABLE 4. RESULTS - COMPETENCIES RANKED BY IMPORTANCE SURVEY RESULTS NO. Competencies Ranked By Importance SURVEY RESULTS NO.
Competencies Ranked By Importance
1
Reporting
10
Leadership
2
Communication
11
Research
3
Professional Demeanor
12
Problem Solving
4
Interaction
13
Decision Modeling
5
Marketing
14
Strategic Critical Thinking
15
Measurement
6
Leverage Technology
16
Industry Sector
7
Risk Analysis
8
Legal Regulatory
17
Resource Management
9
Project Management
18
International Global
Demographic Findings The results of multiple regression indicated gender correlated with the response to many of the Likert questions. Therefore, in order to determine if women tended to give a higher rating in general, t -tests were conducted between men and women for their responses to all competency questions and for their responses to categories of competencies questions. The results indicated that women tended, in general, to rate higher than men. However, in order to determine whether differences existed in the importance assigned to the categories of competencies between men and women, t -tests were conducted between two diferent categories of competencies, at a time, for women and for men. The results indicated that even though women tend to give a higher rating, no diference existed in how they would assign importance to the three categories of competencies when compared with men. The Importance of Critical Thinking Of concern, the data were mixed in support of critical thinking as an important competency. Instead of identifying strategic critical thinking as the most important business competency, auditors ranked it third, behind the marketing and legal/regulatory competencies. Respondents also gave relatively low rankings to competencies similar to strategic
15
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Critical Thinking for the Federal Auditor
https://www.dau.edu
critical thinking, such as decisionmodeling and problem solving. However, when asked to rank the threemost important competencies, they identifed strategic critical thinking as the third out of 18 competencies. The mixed response is of concern given the importance of strategic critical thinking to thework of the federal auditor. Numerous researchers have found strategic critical thinking to be important for the profession of accounting/ auditing (Bolt-Lee & Foster, 2003; Daigle, Hayes, &Hughes, 2007; Gupta & Marshall, 2010; Jim, Damtew, Banatte, & Mapp, 2009; Kaciuba & Siegel, 2009; Thomas, 2000). Auditors require the critical thinking skills that allow for analyzing business risk (McKnight &Wright, 2011). The federal auditor’s intent on meeting the rigidity of auditing standards or preoccupation with the knowledge content required for passing the CPA exam may account for the low standing of critical thinking as a required competency. University accounting programs continue to stress content memorization required for passing the CPA exam instead of emphasizing the critical thinking skills required of auditors by employers (Gupta & Marshall, 2010). The implications of this finding are significant with regard to what an already overburdened accounting curriculumcanbe expected to deliver. The accounting course of study is currently expected to provide training in the emerging areas of forensic accounting, the international fnancial reporting standards, and enhanced internal controls resulting fromSarbanes-Oxley legislation, but still deliver the knowledge content demanded by the CPA exam. The inclusion of other important competencies such as ethics, identifcation of fraud, information literacy, communication capabilities, and strategic critical thinking challenges accounting faculty to fnd the time to introduce these topics without displacing other key topics in accounting courses (Young &Warren, 2011). Faculty are additionally subject to what Vance and Stephens (2010, p. 6) refer to as the increasing pressures within colleges to “acquiesce to the needs” of the current generationwhose sufciency in a well-developedwork ethic is questioned by the authors. In fact, these authors specifcally note the absence of competencies such as behavioral drive and self-motivation. While the responsibility of educating accounting students includes preparation for professional work and professional identity (Wilkerson, 2010), the research indicates graduate education or employer training programsmay be better able to address the development of strategic critical thinking skills.
16
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
January 2020
Other Findings Other noteworthy fndings resulted fromthe author’s research. Auditors identifed communication andmeasurement as skills needing improvement. Concerning communication, one auditor noted, “efective writing skills are essential, we must document everything we do. Our audit reports are our product and the communication given must be clear and able to stand the test of time.” With respect to measurement, the importance assigned to this competency may have fowed from the well-recognized auditing term "criteria" in the AICPA defnition; i.e., auditors are trained to evaluate contractor performance against criteria —most often, the Federal Acquisition Regulation (FAR). The study also indicated that, contrary to prevailing literature, auditors of federal contractors want better training in conventional accounting content such as general ledger accounting and, especially, cost accounting. For instance, one respondent noted “the ability to understand how the contractor's accounting systems work is very important. Having previous experience as an accountant (general ledger, accounts payable, payroll) has been extremely helpful in understanding how diferent systems work.” accounting, the international fnancial reporting standards, and enhanced internal controls resulting from Sarbanes-Oxley legislation, but still deliver the knowledge content demanded by the CPA exam. Knowledge of cost accounting is important in auditing cost representations of federal contractors, especially with regard to rates computed to recover indirect cost. A respondent noted, “I remember taking cost (or managerial) accounting; but not to the extent that is needed” for federal auditing. Other respondents identifed a need for “knowledge of pools and bases and the application of indirect cost to direct cost” and a need for training in “the development of a predetermined and actual indirect rate, including the allocation process for those rates.” Many of the comments referred to the need for specialized undergraduate training in the FAR criteria. For instance, one auditor noted, “they don't stress the auditing environment in the undergraduate curriculum. I had 1 The accounting course of study is currently expected to provide training in the emerging areas of forensic
17
Defense ARJ, January 2020, Vol. 27No. 1 : 2-26
Page 1 Page 2 Page 3 Page 4 Page 5 Page 6 Page 7 Page 8 Page 9 Page 10 Page 11 Page 12 Page 13 Page 14 Page 15 Page 16 Page 17 Page 18 Page 19 Page 20 Page 21 Page 22 Page 23 Page 24 Page 25 Page 26 Page 27 Page 28 Page 29 Page 30 Page 31 Page 32 Page 33 Page 34 Page 35 Page 36 Page 37 Page 38 Page 39 Page 40 Page 41 Page 42 Page 43 Page 44 Page 45 Page 46 Page 47 Page 48 Page 49 Page 50 Page 51 Page 52 Page 53 Page 54 Page 55 Page 56 Page 57 Page 58 Page 59 Page 60 Page 61 Page 62 Page 63 Page 64 Page 65 Page 66 Page 67 Page 68 Page 69 Page 70 Page 71 Page 72 Page 73 Page 74 Page 75 Page 76 Page 77 Page 78 Page 79 Page 80 Page 81 Page 82 Page 83 Page 84 Page 85 Page 86 Page 87 Page 88 Page 89 Page 90 Page 91 Page 92 Page 93 Page 94 Page 95 Page 96 Page 97 Page 98 Page 99 Page 100 Page 101 Page 102 Page 103 Page 104 Page 105 Page 106 Page 107 Page 108 Page 109 Page 110 Page 111 Page 112 Page 113 Page 114 Page 115 Page 116 Page 117 Page 118 Page 119 Page 120 Page 121 Page 122 Page 123 Page 124 Page 125 Page 126 Page 127 Page 128 Page 129 Page 130 Page 131 Page 132 Page 133 Page 134 Page 135 Page 136 Page 137 Page 138 Page 139 Page 140 Page 141 Page 142 Page 143 Page 144 Page 145 Page 146 Page 147 Page 148 Page 149 Page 150 Page 151 Page 152 Page 153 Page 154 Page 155 Page 156 Page 157 Page 158 Page 159 Page 160Made with FlippingBook Learn more on our blog