Cross-Examining Child Custody Evaluations

Caveat:  Dr. Childress is not an attorney, he is a psychologist.  His commentary on this blog is as a psychologist, not an attorney.  For legal advice consult an attorney and follow the advice of the attorney.

---

In my professional consultation work with attorneys, I am frequently asked to help them prepare their lines of cross-examination for child custody evaluators who have reached problematic conclusions relative to the pathology of attachment-based “parental alienation” (AB-PA).

There are so many problems inherent to the practice of child custody evaluations that it’s hard to know where to begin.

The multitude of problems all stem from how the practice of child custody evaluations began.  The practice of child custody evaluations developed gradually and piecemeal over the course of time in response to the needs of the Court for guidance in family matters.  But this gradual and piecemeal origin means that the practice of child custody evaluation is not well thought out.

With the increase in the divorce rate occurring in the 1970s and 80s, professional psychology was increasingly asked to provide input to the Court regarding child custody arrangements in high-conflict families.  The initial response of professional psychology was to provide haphazard and idiosyncratic assessments and opinions of varying quality based on variable methodology.

In an effort to improve the practice of offering child custody recommendations to the court, various “standards” were developed by professional organizations, such as the American Psychological Association in 1991 (Specialty Guidelines for Forensic Psychology) and the Association of Family and Conciliation Courts in 2006 (AFCC: Model Standards of Practice for Child Custody Evaluations) which sought to provide standardized methodology for the conduct of child custody evaluations.

However, these “standards” exclusively address the role-relationships of the evaluator to the client and court, and the methodology for collecting the data – they do NOT address how the data is to be interpreted, and it’s the interpretation of the data that is central to forming the conclusions and recommendations provided by the evaluation. The interpretation of the data collected in child custody evaluations is left entirely (100%) to the discretion of the particular evaluator, as is the decision regarding what information from professional psychology is to be applied – or not applied – to the interpretation of the data.

I want to be entirely clear on this, child custody evaluations consider themselves exempt from the standards governing clinical psychology, and so they believe themselves exempt from systematically applying all of the established constructs or principles drawn from professional psychology to the interpretation of the data, including the DSM diagnostic system for identifying pathology (see AFCC Standard 4.6c), or they apply these principles and constructs in such haphazard, random, and idiosyncratic ways as to be essentially worthless in providing a standardized professional interpretation of the data based on established psychological principles and constructs.

Let me just add something about Standard 4.6c of the AFCC standards; while “diagnostic labels” may have a “prejudicial” impact against the person who is diagnosed with a pathology, such as a narcissistic or borderline personality parent, intentionally withholding relevant diagnostic information from the Court in order to influence the Court’s decision would seemingly have an equally “prejudicial” impact in the opposite direction (against the normal-range parent seeking custody) and would be equally – if not more – problematic in my opinion.  I believe professional psychology has a fundamental obligation to provide the Court with ALL of the relevant mental health information, including relevant DSM diagnostic information, and allow the Court to determine the relative weight given to this information.  In my opinion, it is not within the purview of professional psychology to preempt the decisional authority of the Court by unilaterally making  a preemptive decision about the potential prejudicial or probative value of mental health information and to then intentionally withhold relevant DSM diagnostic information from the Court in order to influence – in any direction – the decision of the Court. We should simply be reporting on the relevant mental health information, not making preemptive decisions that would bias – in any way – the reporting of the relevant mental health information.

Instead of being circumscribed within the confines of established psychological constructs and principles, child custody evaluators are free to just make up their conclusions and recommendations based on their personal beliefs and personal biases, which are highly vulnerable to “counter-transference” issues (influence from the evaluator’s own childhood and family-of-origin experiences that affect the evaluator’s interpretation of the custody evaluation data). 

When it comes to the interpretation of the data, they just make it up based on their own idiosyncratic beliefs and biases.

At times, child custody evaluators use “psychology-sounding” words to make it seem like they are applying established constructs and principles, and these “psychology-sounding” words fool legal professionals and the public who don’t know the literature and research in professional psychology. But I’m a psychologist – I know the literature and research in psychology – and these child custody evaluators are talking complete nonsense when they use these “psychology-sounding” words… 100% nonsense.  They are making it up, completely and totally making this stuff up.

Child custody evaluators are not applying any existing models or principles of professional psychology to the interpretation of the data, or they are doing so in odd and idiosyncratic ways that essentially fit with their preconceived biases.  The “Standards” for Forensic Psychology produced by the APA and AFCC are essentially putting lipstick on a pig – it’s still a pig.  And not a very pretty pig at that. 

Read them. The standards are entirely about how to collect data and about how the professional relationship is to be established with the client and court.  There is nothing about how the data should be interpreted – specifically interpreted.  There are statements about “best interests of the child” and applying the knowledge from child development, and divorce, and other related areas of professional psychology, but in any specific case, the application – or non-application – of this professional knowledge base is left entirely – 100% – to the idiosyncratic and highly variable discretion of the individual evaluator.

The very foundation for the practice of child custody evaluation is inherently flawed.  So let me begin with this foundational line of questioning which is available to attorneys:

Line 1:  There is no established scientific foundation for the practice of child custody assessments.

The core premise – the foundational premise – upon which child custody evaluations are based is deeply and inherently flawed, resulting in a deeply and inherently flawed application of this premise.  The scientific foundation for the practice of child custody evaluations is non-existent, and as a result, the practice of child custody evaluation is built on a fragile house-of-cards.

While it looks like an elegant construction, if we breathe on this house-of-cards it will collapse.

The Emperor Has No Clothes

Hans Christian Anderson wrote a short story about an emperor who wanted a wonderful new set of clothing.  The emperor hired two (charlatan) tailors who said that they could create a “magical” suit of clothing that would be invisible to anyone who was incompetent and unfit for their position.  Anyone who was incompetent and stupid would be unable to see the “magical” clothing.

As these charlatan tailors began their work on sewing their supposedly “magical” clothing, the emperor and all of his courtiers were afraid to say that they couldn’t see the clothing because that would mean that they were unfit for their positions, that they were stupid and incompetent, so everyone pretended to see the clothes.  They all praised the magnificent beauty and quality of the “magical” clothing, and since everyone else seemed to see the clothing each person thought that they alone could not see the “magical” clothing.

When the clothes were finally finished, the tailors were paid and they quickly left the kingdom (making a lot of money for their supposedly “magical” clothing).  The emperor decided to proudly display his magnificent new clothing in a parade before his people.  Again, all of the people along the parade route also pretended to see the clothing because no one wanted to admit that they couldn’t see the emperor’s new clothes, because that would mean that they were incompetent and stupid since everyone else obviously saw the clothing.  But as the emperor paraded past his people, a young child turned to his mother and said, “Look mommy, the emperor has no clothes.”  With that, the collusion of silence was broken and everyone began to admit that they too saw no clothing, and they began to laugh at the emperor who stood naked before his people, duped by the charlatan tailors.

With regard to child custody evaluations, I am that little kid on the parade route; “Look mommy, the emperor has no clothes.”

The practice of child custody evaluations violates every standard of professional practice regarding assessment.  But no one in professional psychology is saying anything.  It’s a collusion of silence.

So I’ll say it, “the emperor has no clothes” – and family law attorneys need to begin saying it too.

There is no scientifically established validity to the conclusions or recommendations of child custody evaluations.  None.  Zero.

Don’t believe me, or don’t agree?  There is a Comment section to the blog. I invite anyone to provide a single citation in the research literature that establishes the inter-rater reliability or validity (construct validity; content validity; predictive validity; convergent validity, discriminant validity) for the conclusions and recommendations of child custody evaluations.  Not for the data collection procedures, but for the conclusions and recommendations of child custody evaluations, which are made based on the interpretation of the data.

<crickets>

Child custody evaluations do not apply in any systematic or rational way the established constructs and principles of professional psychology, and the conclusions and recommendations offered by child custody evaluators are simply made up based on the individual idiosyncratic biases of the individual custody evaluator.

They simply make it up.

Child custody evaluations are little more than voodoo assessment.  Rattle some beads, recite some magical incantations, and read the entrails of a goat. Seriously.

• Rattle some beads: The procedures used for data collection
• Recite some magical incantations: Write a report with psychology-sounding words
• Read the entrails of a goat: Make recommendations about the “best interests of the child”

Here’s what two leading figures in Forensic psychology, Stahl and Simon, say about the definition of “best interests of the child” (and notice who published this work, The Family Law Section of the American Bar Association):

“A critical subject facing those working in the field of family law, whether they’re legal professionals or psychological professionals, is the concept of the best interests of the children. Even recognized experts in this concept differ with regard to what it means, how it should be determined, and what factors should be considered in determining what is in the best interest of a child. Thus, this ubiquitous term escapes consensus and remains fundamentally vague.” (Stahl & Simon, 2013, p. 10-11)

“It is defined differently from state to state; and even in Arizona, where there are nine statutory factors associated with the best interest of the child, the meaning behind many of the factors is obscure.  Additionally, when psychologists refer to the best interests of children, they are referring to a hierarchical set of factors that may have different meanings to different children with different families and that may be understood differently by psychologists with different backgrounds and different training.” (Stahl & Simon, 2013, p. 11)

Stahl, P.M. and Simon, R.A. (2013). Forensic Psychology Consultation in Child Custody Litigation: A Handbook for Work Product Review, Case Preparation, and Expert Testimony, Chicago, IL: Section of Family Law of the American Bar Association

“may have different meanings to different children with different families and that may be understood differently by psychologists with different backgrounds and different training.”

This is related to a property of the assessment called “inter-rater reliability” – the stability of the conclusions and recommendations derived from the data across different evaluators.

They just make it up.  Seriously.  They – just – make – it – up.

Voodoo assessment.  Rattle some beads (data collection procedures), recite some magical incantations (write a report with psychology-sounding words), and read the entrails of a goat (make recommendations from the “spirit world” of professional psychology regarding the supposed “best interests” of the child).

I am making a strong allegation.  This blog has a Comment section.  I invite any advocate for the practice of child custody evaluations to provide a citation for a single study – just one – that has demonstrated the inter-rater reliability and validity (construct validity, content validity, predictive validity, convergent validity, divergent validity) for the conclusions and recommendations produced by child custody evaluations.

<crickets>

There is NO scientific foundation for the practice of child custody evaluations.

Inter-Rater Reliablity

The construct of “reliability” is the professional term for the stability of an assessment’s results.  If I give you an IQ test this week and your score is 100 (normal-range), then your IQ score should be roughly the same when I retest you one week later. 

If, one week later, your IQ score drops to 70 (cognitively deficient), then my assessment procedure for establishing your IQ is NOT reliable.  My assessment procedure does not yield stable results from one assessment administration to another.

In the IQ example given above, the type of reliability is called “test-retest reliability,”  There are four methods in the professional practice of assessment that are used to establish an assessment procedure’s reliability:

• Test-retest reliability
• Inter-rater reliability
• Split-half (internal consistency) reliability
• Alternate forms reliability

This is all standard – and basic – professional psychology assessment stuff.  I teach it every semester to graduate students in my Psychometrics of Assessment class.  Basic stuff.

By definition, an assessment procedure cannot be valid if it is not reliable. If your IQ changes from normal-range to developmentally delayed because of the inherent instability of my assessment procedure, my assessment procedure is NOT a valid assessment of your intelligence.

Validity is the professional psychology term that refers to the actual truth or accuracy of the findings from an assessment procedure.

A fundamental axiom within the field of professional assessment is that:  An assessment procedure CANNOT BE VALID if it is not reliable.  In the example above, my assessment procedure for your IQ cannot be valid (true and accurate) if the findings from the assessment procedure fluctuate radically from one test administration to the next (in one assessment you’re deemed to be of normal-range intelligence but in the next assessment a week later you’re deemed to be cognitively delayed).

In the case of child custody evaluations the appropriate form of reliability would be inter-rater reliability, which means that two raters (two evaluators) would reach the same conclusions and recommendations based on the same data set.  Inter-rater reliability.

No research has ever been conducted to establish the inter-rater reliability for the conclusions and recommendations reached by child custody evaluations.  None.

And note, this is in the context of Stahl and Simon’s (2013) analysis regarding the fundamental construct being assessed in child custody evaluations, the “best interests” of the child:

“Even recognized experts in this concept differ with regard to what it means, how it should be determined, and what factors should be considered in determining what is in the best interest of a child.” (Stahl & Simon, 2013, p. 10)

The inter-rater reliability for the conclusions and recommendations of child custody evaluations has NEVER been scientifically established.

So if two different custody evaluators can reach two radically different sets of conclusions and recommendations from the same data – based on their differing interpretations regarding the meaning of the data – then the conclusions and recommendations from child custody evaluations are not reliable.

And if the conclusions and recommendations from child custody evaluations are not reliable (are not stable from one custody evaluator to the next; inter-rater reliability), then the conclusions and recommendations from child custody evaluations CANNOT, by definition, be valid.

Let that sink in for a moment.  If the conclusions and recommendations are not reliable (stable) from one evaluator to the next (inter-rater reliability), then the conclusions and recommendations cannot, by definition, be valid.

There is no scientific evidence establishing the inter-rater reliability for the conclusions and recommendations from child custody evaluations.

Foundational Line of Questioning

This is the basis for the first line of questions; to question the custody evaluation in such a way so as to:

1.)  Expose the absence of any data in the scientific literature that establishes the “inter-rater” reliability for the conclusions and recommendations of child custody evaluations.

1A.) Walk the evaluator through definitions for the constructs of “reliability” and “validity” – defining the four types of reliability, with a focus on inter-rater reliability.

• Reliability: The stability of the assessment findings across situations (raters)
• Validity: The truth or accuracy of the findings

1B) Ask for citations from the scientific literature that establish the “inter-rater reliability” and “validity” (to be discussed later) for the conclusions and recommendations of child custody evaluations – there are none.

1C.) Lead the evaluator into a key question, “Can the findings from an assessment procedure be valid if they are not reliable? – A: No.

2.)  Expose the meaning of this: “Can two different child custody evaluators reach markedly different conclusions and recommendations based on the same data set?” A: Yes.

2A.) If the child custody evaluator tries to assert that other psychologists would agree with his or her interpretation of the data, ask how the evaluator knows this if there has never been any research done on the inter-rater reliability for the conclusions and recommendations of child custody evaluations (that they all collect data in the same way does not establish that they would all interpret the data in the same way).

2B)  Use the phrase “conclusions and recommendations” when referring to reliability and validity because you’ll want to avoid issues surrounding data collection procedures. The data collection procedures are highly standardized – they are likely to be reliable-stable in the data they produce – it’s the interpretation of the data to reach conclusions and recommendations that is at issue.  What is being set up by this line of questions is a second line of question regarding what constructs and principles from professional psychology did the child custody evaluator apply to the interpretation of the data – this second line will be a key line of questioning.

3.)  Expose that the conclusions and recommendations of the child custody evaluator are simply the opinions of one psychologist and may not represent the opinions of other psychologists who would review the data;

3A.) Q: “So there’s no way of knowing whether your opinions represent the opinions of other psychologists, or whether a vast majority of other psychologists would disagree with your conclusions and recommendations, there’s simply no way of knowing that is there?”

4.)  Expose that the conclusions and recommendations of the child custody evaluator are simply the opinions of one psychologist and may be biased by the personal attitudes and beliefs of the evaluator.

4A.) This sets up a second line of questions regarding what established constructs from professional psychology were applied in interpreting the data (not collecting the data; interpreting the data) as a means to limit the potential introduction of bias.

4B.) The custody evaluator may try to answer the bias question by pointing out the standardized collection of data.  That’s irrelevant.  The issue is NOT the collection of data, it’s the interpretation of data.  The cross-examination seeks to expose that NO established principles of professional psychology (specifically, principles from family systems theory; triangulation, cross-generational coalition, inverted hierarchy, emotional cutoff – and attachment theory) were applied to interpreting the data.  This will be addressed in the second line of questioning.

The second part of this scientific foundation line of questioning is about the construct of “validity.”  It’s a short line of questions, but on the same topic area of undercutting the scientific foundation for the practice of child custody evaluation.  The emperor has no clothes.

I’ll cover issues of “validity” (construct validity; content validity; predictive validity; concurrent validity; discriminant validity) in a future blog post, but essentially the issue is that no studies have ever been conducted to establish the “validity” for the conclusions and recommendations of child custody evaluations.

There is no established scientific validity to the conclusions and recommendations of child custody evaluations.  None.  Zero.  Nothing.  The emperor has no clothes.

A Monkey Throwing Darts

Child custody evaluations are voodoo assessment. Rattle some beads (data collection procedures), recite some “magical” incantations (write a report with “psychology-sounding” words), and read the entrails of a goat (offer conclusions and recommendations from the “spirit world” of professional psychology).

But as for the scientifically established validity (truth and accuracy) for the conclusions and recommendations of child custody evaluations… we might as well have a monkey throwing darts at a dartboard.

Seriously, neither the monkey throwing darts nor child custody evaluations have any data whatsoever regarding the reliability and validity of these respective “assessment” procedures.  And at least the monkey throwing darts approach would be much cheaper, and probably more entertaining.

Q: “Dr. So-n-So, if we had a monkey throwing darts at a dartboard regarding various custody time-share options, where in the scientific literature is there evidence that the conclusions and recommendations reached by child custody evaluations are more reliable and any more valid than the outcome of a monkey throwing darts at a dartboard?”

A: There is no evidence from the scientific literature that the conclusions and recommendations from child custody evaluations are any more reliable or valid than the conclusions and recommendations reached by a monkey throwing darts at a dartboard.

Again, for all the advocates who are in favor of the “assessment” practice of child custody evaluations, there is a Comment section on this blog.  Feel free to prove me wrong.  Cite for me a single research study demonstrating the inter-rater reliability and construct validity, or content validity, or predictive validity, or convergent validity, or discriminant validity for the conclusions and recommendation provided by child custody evaluations.

<crickets>

Craig Childress, Psy.D.
Clinical Psychologist, PSY 18857