|
|
Home
| Databases
| WorldLII
| Search
| Feedback
Precedent (Australian Lawyers Alliance) |
FORENSIC LINGUISTICS AND PSEUDOSCIENCE
HOW TO RECOGNISE THE DIFFERENCE
By Alex Bowen and Adjunct Professor Diana Eades
Forensic linguistic analysis, based on the scientific study of language, can add value to language evidence through admissible expert opinion. But how can non-experts distinguish genuine, valid forensic linguistic analysis from weaker kinds of analysis, including pseudoscience? In the spirit of peer review, this article aims to help investigators and lawyers to identify reliable forensic linguistic analysis. The promotion by a television documentary of an expert report on the authorship of anonymous incriminating documents provides a case study. We hope to provide useful information about authorship analysis and linguistic analysis, not just to critique a single case.
In 2020, a two-part episode of the ABC documentary series Australian Story[1] dealt with a complainant’s experience of being stalked, and prominently featured ‘linguistic analysis’ by a former ‘FBI profiler’ which purportedly established that anonymous documents were written by the alleged stalker. The first part of the documentary detailed the complainant’s relationship with the defendant and his increasingly concerning behaviour. Among the most threatening and harmful actions were the posting of offensive anonymous fliers and letters in various locations. Still, the complainant had little doubt about the identity of her stalker and repeatedly appealed to police for help, but found protection orders ineffective.
The second part of the documentary followed the complainant’s strenuous efforts to have police act on the escalating stalking. Police uncovered evidence linking the defendant to the anonymous offensive documents but were not sure if it was enough for a conviction. The documentary then introduced the concept of linguistic analysis; the process of obtaining an expert report; and ultimately the suspect’s guilty plea, which, it implied, was likely facilitated by the report.
The case (referred to in this article as ‘the stalking case’) has attracted media attention for legitimate reasons, and we recognise that stalking is serious and can be difficult to prosecute. However, this story unfortunately promotes a kind of language analysis that in our opinion is scientifically invalid.[2] We have no reason to doubt that this defendant was guilty (noting that he pleaded guilty in court), but unreliable forensic analysis can lead to miscarriages of justice, or wasted time and resources, or both.
Linguistics is the scientific study of language in general and of specific languages, based on the comprehensive and systematic analysis of empirical language evidence. The term ‘forensic linguistics’ refers to the use of expert linguistic evidence in courts and tribunals, and it can also refer more broadly to the linguistic study of language in legal contexts.
The linguistic analysis report in the stalking case was not tested in court (which could suggest it had no impact), and if it had been, of course, the defence may well have opposed it. But we are concerned that investigators, journalists and others appear to have been so persuaded by it that it was featured prominently on a respected media platform. The Australian Story documentary suggests that obtaining the purported forensic linguistic report was a turning point in prosecuting the case. We acknowledge that the complainant struggled for several years to have the case prosecuted and obtaining a seemingly definitive report would have been a great relief. However, taking a systemic perspective, the documentary shows how easy it is to promote misleading and misconcieved ideas about language analysis, and highlights the need for linguists to communicate more broadly about forensic linguistics. We hope that forensic linguistic analysis can be used more reliably in future cases where appropriate.
The defendant in the stalking case pleaded guilty to some charges (after denying involvement during police interviews), which might suggest that the language analysis was a valid way of independently establishing that he wrote the documents. This is a notable and common fallacy: that because a conclusion is true, evidence or reasoning used to support that conclusion is reliable, and therefore good for other cases. This is no basis for evaluating forensic analysis for use in court. Forensic science generally is at risk of circular thinking, where convictions are taken to validate forensic analysis, and forensic conclusions are taken to validate convictions. Correspondingly, we emphasise that by pointing out problems with purported expert analysis in the stalking case, we take no position about the strength of other evidence or the court outcome in that case.
The stalking case perhaps exemplifies the often-proclaimed ‘CSI effect’,[3] which may cause inflated expectations about forensic science, as well as dismissive responses to real, cautious forensic science, and increased acceptance of other forms of analysis which may amount to pseudoscience. A classic indicator of pseudoscience is over-confident conclusions, which may match unrealistic expectations created by fictional science.
Meanwhile, a broader reckoning in real forensic methods is underway. Some forensic methods, notably DNA analysis, are generally very reliable. But others have been found to be unreliable, or carried out with inadequate safeguards against cognitive bias.[4]
It is not always easy to distinguish pseudoscience from science, and the dividing line between them can be complex.[5] Genuine forensic linguistics can do both more and less than non-linguists might expect, because scientific findings about language sometimes contrast with general educated understandings about language. This is unlike some other forensic sciences (for example, DNA analysis) where non-experts have little background knowledge.
Limited awareness about linguistic science is a problem for linguists to address, and clearer guidance is required to help lawyers to work out when expert evidence about language can be useful, valid and admissible. The stalking case provides a useful example.
In this case, alleged stalking behaviour generated language evidence: anonymous letters, fliers and posters referring offensively to the complainant and others. Numerous fliers targeted the complainant with sexual slurs, while some blamed the complainant for the suicide of her friend. There was only ever one suspected author. Because police were not sure that the physical evidence linked the suspect sufficiently to the documents, expert assistance was sought to examine the language evidence.
The Australian Story documentary suggested that James R Fitzgerald was identified as a possible expert witness based on a fictionalised portrayal of his work in the series ‘Manhunt: Unabomber’. While it is possible for good science to be fictionalised, the fact that a potential expert witness was portrayed in a television series provides no indication that the purported expert uses a valid method. It is essential for lawyers to check the credentials of expert witnesses carefully. In the stalking case, we have been unable to find, through internet searches and a review of Fitzgerald’s website,[6] any description by Fitzgerald of how he uses a method which has been peer reviewed or validated.
Peer-reviewed critique has observed that, in previous cases, ‘Fitzgerald [used] a form of Forensic Stylistics that on its surface may look like science, but, on closer inspection, turns out not to be scientific at all.’[7] (This is not a criticism of all forms of Forensic Stylistics,[8] and we make no suggestion that Fitzgerald or the report in this case are intentionally misleading). Fitzgerald’s expert report claimed that comparison of language in the anonymous letters and fliers with documents known to be written by the defendant, such as love letters, established that the defendant wrote both sets of documents. We have read the full report, but the examples discussed in this article are drawn from the televised documentary.
Identifying the author from a sample of language depends on an assumption (which has not been shown to be true) that different people use language in recognisably different ways. Intuitively, we believe we can recognise the writing style of people we know well. This may suggest that when an investigation involves close attention to language, and we notice similarities between two text samples, we are tuning in to an individual’s language and can potentially make an identification.
Sometimes an investigator or witness has a hunch about who wrote a text, and then requests expert assistance. In the Unabomber case in the United States, the language similarity of written evidence was first noticed by the Unabomber’s family.[9] In the stalking case, the detective said about the posters and previous love letters: ‘I could see similarities there myself’.[10] Having a suspicion confirmed by apparently scientific analysis could be persuasive and reassuring but does not necessarily make that analysis valid expert evidence, even if the original suspicion is correct. Hunches can, of course, be wrong, and can be affected by cognitive bias.[11]
The detective, on receiving the report, looked first at its conclusion, expressing relief at its confident finding. But this report’s unrealistically strong conclusion is the clearest warning that it is, in our opinion, unsound:
‘the writing style [of the two sets of documents] is CONSISTENT to the degree of Exceptionally Distinctive...
language commonalities ... when taken in totality, firmly establishes the linkage between them and the fact that one author, [the defendant], is responsible for writing [the anonymous documents].’[12]
In our view it should be apparent to a critical reader that the tone of this conclusion does not reflect the need for careful comment about complex evidence. This can be seen in the combination of descriptions such as ‘exceptionally distinctive’ and ‘firmly establishes’, and the presentation of its conclusion as ‘fact’. Further, naming the defendant not only as the ‘one author’ but also as ‘responsible’ risks confusing textual analysis with attribution of legal or moral responsibility.
Compare this with a measured expert conclusion from an English case:[13]
‘Linguistic features identified in Mr Hodgson’s and the suspect texts are compatible with their having been produced by the same person.’[14]
This conclusion reflects cautious evaluation of evidence and leaves the court to decide who the author was,[15] rather than asserting that as a ‘fact’ established by the analyst.
Linguistic research demonstrates that language is systematic, and that speakers, largely unconsciously, use language creatively but with certain regularities. Forensic linguistic authorship analysis depends on observations that different people use language with statistically significant patterns of difference.
It is not established that individuals can consistently be distinguished by their language use, and especially not from small samples of language.[16] Language samples are not like fingerprints. However, forensic linguistic authorship analysis has been demonstrated and evaluated in a range of situations, including in emails, suicide notes, text messages, chat logs, novels and student plagiarism. Statistical methods and ways of framing conclusions are debated in forensic linguistics.[17] Linguistic authorship analysis can be described as a search for ‘consistent patterns of language use’, followed by analysis to ‘determine how distinctive any such patterns are’.[18]
The analysis in the stalking case highlighted similarities between the two text samples without demonstrating how distinctive they were. In fact, the wording of the conclusion – ‘consistent to the degree of exceptionally distinctive’[19] – seems to conflate the two concepts, wrongly implying that a high level of consistency can make a feature distinctive.
Two texts could have an extremely consistent feature; for example, always spelling the word ‘and’ in the same way. But this does not make that feature distinctive, because while an ampersand (&) is sometimes used, many writers consistently use ‘and’. However, spellings like ‘meany’ (for ‘many’) and ‘gowing’ (for ‘going’) may be so demonstrably rare among hundreds of thousands of authors that their repeated use in two texts is material evidence of a single author.[20]
This highlights the need to compare features observed in the samples in light of appropriate hypotheses about who could have written them. Perhaps the comparison group of possible authors is everyone in Melbourne. How will a linguistic analyst measure the probability that any other person could have authored the documents? In theory, this could be done by comparing each identified language feature with a corpus containing large amounts of comparable naturally occurring language, to measure how often it occurs. This requires a way of controlling for topic, genre and form, which influence language choices – and can require the use of complex statistics.[21]
Sometimes it is possible for investigators to identify a small number of possible authors prior to linguistic analysis.[22] This depends on factors specific to the individual case, and should be carefully discussed with the analyst before forensic analysis begins – taking care to avoid cognitive bias.
There was only one suspect in the stalking case, and no indication that any other language sample or database was used for comparison with other possible authors. Without such a comparison, the most any linguist could say is that the suspect’s language is not inconsistent with the anonymous writing; and that such a conclusion is not a positive identification.
In our opinion, the language features presented do not support the strongly expressed conclusion in the stalking case. The analysis appears to build on intuitive responses to written documents focusing on presentation, formatting, and ‘mistakes’ or non-standard writing.
One feature relied on is the mixing up of apostrophes in possessives and plurals (such as ‘mens’ vs ‘men’s’). However, errors of punctuation like this are frequently observed and many authors may produce them in unedited writing.[23] Ensuring they are distinctive for authorship analysis in a particular case would require careful justification.
Another feature related to ‘first line indenting frequency’ and single-sentence paragraphs, which were said to constitute a pattern of inconsistencies, or ‘non-pattern pattern’. To support authorship identification, such a pattern would have to be described, quantified and measured, with strong evidence to suggest it was distinctive to one writer.
When reviewing an expert report, it is useful to ask what kind of evidence it identifies and how the features used in the analysis reflect application of the specialised field of knowledge. The language similarities identified by Fitzgerald (formatting, spelling and punctuation) are features that a non-linguist might notice. The report may lend apparent authority to intuitions about the language evidence, without doing anything the trier of fact could not do.
An expert in forensic linguistics should identify analytically relevant features based on specialised understanding of language evidence and ways of clearly describing and categorising language features, and use a valid method when estimating or measuring distinctiveness.
Finally, Fitzgerald refers to ‘the highly personal and thematic topic of the [documents] as they relate to [the complainant]’.[24] The personal information in the fliers is relevant evidence which might persuasively exclude some authors. However, knowledge demonstrated by the author of the letters and fliers is not a matter of linguistic expertise, as it does not relate to patterns of language use. Its inclusion in an expert report gives undue weight to something that would be readily apparent to the trier of fact.
A critical lawyer may have concerns about the opinion in the stalking case, beginning with its overly definitive conclusion. We have discussed further weaknesses which are made apparent by knowledge of linguistics. Accordingly, in our view there are three basic arguments that the opinion is inadmissible:
1. The opinion is not relevant because it provides no better analysis of language than a lay person could do.[25]
2. However, it appears authoritative, which risks encouraging deference to the purported expert.[26]
3. It is not an opinion based on appropriate specialised knowledge about language;[27] in fact, it is inconsistent with relevant knowledge from linguistics, a discipline it claims to represent.
Forensic authorship analysis, and forensic linguistics more broadly, can add value to language evidence if done according to appropriate standards. The growing body of research in forensic linguistics includes specialised international peer-reviewed journals, and many publications in law and linguistics journals. The University of Melbourne has recently launched a Research Hub for Language in Forensic Evidence.[28]
Risks of mishandling complex language evidence are significant for lawyers on both sides of a case. Linguists identify situations where, without expert involvement, language evidence is not used to its full potential.[29] On the other hand, when inappropriate language analysis is presented it may be ruled inadmissible, after having wasted time and resources and distracted from other evidence.
There are no straightforward guidelines to resolve these problems. While there are increasing efforts to codify general principles of forensic science, every forensic field is different, and not all admissible expert evidence is, or can be, scientific. However, reference to general standards, many of which are matters also considered by courts,[30] is one way to explore the reliability of a potential expert opinion. For example, the following questions should be asked:
• Does the expert have relevant qualifications?
• Is the expert, taking into account the way they were briefed, acting independently and impartially?
• Is the analysis based on an identified field of knowledge and consistent with its core methods and findings? Is that knowledge and are those methods transparently identified or explained in the report? Have they been peer-reviewed?
• How is the identified method linked to the evidence?
• Are the conclusions cautious and measured, or hyped and expressed in absolute language? Are there acknowledgements of limitations of the analysis?
Finally, while we welcome linguists from overseas contributing appropriate expertise, there are linguists in Australia who are willing to advise, formally or informally, about evidentiary questions, including numerous language issues beyond the scope of this article – some of which are also relevant to civil litigation.
The authors acknowledge the input and feedback from Helen Fraser (Professor of Linguistics, University of Melbourne and Director, Research Hub for Language in Forensic Evidence) at several stages, and from Ed Finegan (Emeritus Professor of Linguistics and Law, University of Southern California Gould School of Law) and Tim Grant (Professor of Forensic Linguistics, Aston University and Director of Aston Institute for Forensic Linguistics) on a previous version of the article. The authors are solely responsible for the opinions, statements and any errors in the article.
Alex Bowen has practised as a solicitor in both criminal and commercial law. His research on language in police interviews has resulted in peer-reviewed publications in international journals. He is currently a PhD student in the School of Languages and Linguistics, University of Melbourne. EMAIL alex.bowen@gmail.com.
Diana Eades PhD, FAHA, is Adjunct Professor in Linguistics at the University of New England. In addition to over 35 years’ experience in researching language in the law, she has presented expert linguistic evidence in courts and tribunals in three states and the Northern Territory. EMAIL diana.eades@une.edu.au.
[1] ABC, ‘To catch a stalker (Parts 1 and 2)’, Australian Story (Episodes 26 and 27, 2020)
<https://iview.abc.net.au/video/NC2002Q026S00>.
[2] We acknowledge that adherence to a scientific method is not the only way to generate useful knowledge or analysis. In particular, we do not mean to imply that knowledge which aims to be scientific in a Western sense is superior to Aboriginal and Torres Strait Islander knowledge about language or other topics. However, the purported linguistic analysis in this case is described as ‘forensic linguistics’ and there is no apparent basis for relying on the analysis except if it is reasonably scientific.
[3] J Wise, ‘Providing the CSI treatment: Criminal justice practitioners and the CSI effect’, Current Issues in Criminal Justice, Vol. 21(3), 2010, 383–99.
[4] See for example SM Kassin, IE Dror and J Kukucka, ‘The forensic confirmation bias: Problems, perspectives, and proposed solutions’, Journal of Applied Research in Memory and Cognition, Vol. 2(1), 2013, 42; LM Solan, ‘The forensic linguist: The expert linguist meets the adversarial system’ in M Coulthard, A May and R Sousa-Silva (eds), The Routledge Handbook of Forensic Linguistics, 2nd ed, Routledge, London, 2020, 349–63.
[5] LM Jupe and V Denault, ‘Science or pseudoscience? A distinction that matters for police officers, lawyers and judges’, Psychiatry, Psychology and Law, Vol. 26(5), 2019, 753–65.
[6] Note that this was correct at the time of writing. <https://www.jamesrfitzgerald.com>.
[7] BS Howald, ‘Authorship attribution under the rules of evidence: Empirical approaches in a layperson’s legal system’, International Journal of Speech, Language & the Law, Vol. 15(2), 2008, 219-47 at 239.
[8] Further, see T Grant, ‘TXT 4N6: Method, consistency and distinctiveness of SMS text messages’, Journal of Law and Policy, Vol. 1(2), 2013, 467–94.
[9] T Grant, The Idea of Progress in Forensic Authorship Analysis, Cambridge University Press, 2022, 16.
[10] ABC, above note 1.
[11] See Solan, above note 4.
[12] See ABC, above note 1.
[13] R v Hodgson [2009] EWCA Crim 742.
[14] T Grant, ‘Text messaging forensics: Txt 4n6: Idiolect-free authorship analysis?’ in M Coulthard, A May and R Sousa-Silva (eds), The Routledge Handbook of Forensic Linguistics, 2nd ed, Routledge, London, 2021, 558–75 at 567.
[15] Ibid.
[16] Ibid.
[17] M Coulthard, ‘On admissible linguistic evidence’, Journal of Law and Policy, Vol. 21(2), 2013, 441–66.
[18] Grant, above note 14, 559.
[19] See ABC, above note 1.
[20] Partial example from G Heydon, Researching Forensic Linguistics: Approaches and Applications, Routledge, London, 2019, ch 1.
[21] S Ishihara, ‘Strength of forensic text comparison evidence from stylometric features: A multivariate likelihood ratio-based analysis’, International Journal of Speech, Language & the Law, Vol. 24(1), 2017, 67–98.
[22] See Grant, above note 8.
[23] See for example L Squires, ‘Whos punctuating what? Sociolinguistic variation in instant messaging’ in AM Jaffe (ed), Orthography as Social Action: Scripts, Spelling, Identity and Power, De Gruyter Mouton, 2012, 289.
[24] See ABC, above note 1.
[25] Smith v R [2001] HCA 50; (2001) 206 CLR 650.
[26] This can be grounds for exclusion under ss135 or 137 of the Evidence Act 2008 (Vic).
[27] Evidence Act 2008 (Vic), s79.
[28] See <arts.unimelb.edu.au/language-forensics>; H Fraser, ‘Introducing the research hub for language in forensic evidence’, Judicial Officers’ Bulletin, Vol. 32(11), 2020, 117.
[29] See for example Heydon, above note 20. Also <arts.unimelb.edu.au/language-forensics>.
[30] A useful overview is found in what some courts require expert witnesses to state and disclose in a report. See for example Supreme Court of Victoria, Practice Note SC CR 3: Expert Evidence in Criminal Trials, [6]; also [4].
AustLII:
Copyright Policy
|
Disclaimers
|
Privacy Policy
|
Feedback
URL: http://www.austlii.edu.au/au/journals/PrecedentAULA/2022/54.html