How do third-party evaluations of judgments affect decision quality on ill-structured problems? An interactive experiment

Garza, Brent Anthony

Permalink

https://hdl.handle.net/2142/97703 Copy

Description

Title

How do third-party evaluations of judgments affect decision quality on ill-structured problems? An interactive experiment

Author(s)

Garza, Brent Anthony

Issue Date

2017-04-19

Director of Research (if dissertation) or Advisor (if thesis)

Hobson, Jessen L.

Doctoral Committee Chair(s)

Hobson, Jessen L.

Committee Member(s)

• Peecher, Mark E.
• Thompson, Anne M
• Douglas, Jeffrey A.

Department of Study

Accountancy

Discipline

Accountancy

Degree Granting Institution

University of Illinois at Urbana-Champaign

Degree Name

Ph.D.

Degree Level

Dissertation

Keyword(s)

• Third-party evaluations
• Inspections
• Reviews
• Judgment
• Public Company Accounting Oversight Board (PCAOB)

Abstract

Using an interactive laboratory experiment, I start to investigate how third-party evaluations of judgments on ill-structured problems affect the decision quality of those being evaluated (“evaluatees”). First, an evaluatee completes an ill-structured estimation task. I deem the amount and type of costly procedures performed by the evaluatee to make that estimate their “judgment.” An evaluator then evaluates this judgment to determine whether the evaluatee should perform additional costly work to improve the estimate. Finally, the evaluatee may perform any additional work and adjust their estimate. I manipulate the evaluators’ level of focus on the quantity of work performed by evaluatees (more vs. less) and when evaluators document their assessment of evaluatees’ judgment (before vs. after conversing with evaluatees). I find that evaluators focused more on the quantity of work who document their initial assessment before conversing with evaluatees request greater amounts of the procedure they (i.e., evaluators) rate as more diagnostic but do not request greater amounts of the procedure they rate as less diagnostic. Evaluators’ behavior is consistent with theories of confirmation bias. Second, I find that evaluatees perform the additional work requested and, over time, learn to perform some of it before the evaluation. Third, this additional work generally improves evaluatees’ decision quality. I then examine the tradeoffs between performing procedures of differently-rated diagnosticities on decision quality. While results suggest that better decision quality occurs when evaluatees use a combination of the procedures, evaluators seem to over-request (and, thus, evaluatees over-perform) procedures that evaluators rate as more diagnostic. Therefore, while third-party judgment evaluations can improve decision quality on ill-structured problems, they may suboptimally do so. The theory and results of this study have implications for both accounting and non-accounting areas where judgments on ill-structured problems are evaluated by third parties.

Graduation Semester

2017-05

Type of Resource

text

Permalink

http://hdl.handle.net/2142/97703

Copyright and License Information

Copyright 2017 Brent Garza

Owning Collections