Uncle Dale Posted August 26, 2010 Share Posted August 26, 2010 During the last several months I've frequently been asked to comment upon the content and findings published by Matt Jockers' team of Stanford computer analysts in their 2008 Literary and Linguistic Computing paper, entitled: "Reassessing Authorship of the Book of Mormon"Not being a professional statistician, I've resisted saying much about this publication, but have long hosted a web-page devoted to the subject, here:http://solomonspalding.com/Lib/Jock2008.htmI am informed that my reproduction of the Jockers paper must be drastically reduced in content, in order to comply with copyright restrictions. In the next week I'll delete much of my on-line file.If anybody wishes to retain a copy of this html version of the full Jockers paper, I suggest that they download the file in the next few days, before I have to remove it from the web.I've also been asked to reproduce G. Bruce Schaalje's recent rebuttal of the Jockers paper; but since I have no permission to create such a web-edition, I've confined my references to his material to a few short comments, here:http://solomonspalding.com/Lib/Jock2008.htm#commentsIf anybody notices obvious errors or missing important information, please let me know, and I'll make the necessary changes.Thank you,Uncle Dale Link to comment
Glenn101 Posted August 26, 2010 Share Posted August 26, 2010 Dale, I think, from my layman's perspective, you have summarized the Jockers studies pretty well, and included the criticisms that have been brought to bear. In your summary of the Schaalje paper, which I think will be published in the near future if I understand Bruce correctly, you summarized the pca plots but did not include a summary of the enhancements that Bruce made to the Jockers and the results that Bruce obtained when performing tests on the Federalist papers.Thanks,Glenn Link to comment
Uncle Dale Posted August 26, 2010 Author Share Posted August 26, 2010 Dale, I think, from my layman's perspective, you have summarized the Jockers studies pretty well, and included the criticisms that have been brought to bear. In your summary of the Schaalje paper, which I think will be published in the near future if I understand Bruce correctly, you summarized the pca plots but did not include a summary of the enhancements that Bruce made to the Jockers and the results that Bruce obtained when performing tests on the Federalist papers.Thanks,GlennIf you'd like to type up a couple of paragraphs on that subject, I'd be happy to add them into the comments section.Archive.org had a copy of Bruce's paper on-line, but it seems to have disappeared -- so I don't currently have access to the text.UD Link to comment
Glenn101 Posted August 26, 2010 Share Posted August 26, 2010 If you'd like to type up a couple of paragraphs on that subject, I'd be happy to add them into the comments section.Archive.org had a copy of Bruce's paper on-line, but it seems to have disappeared -- so I don't currently have access to the text.UDDale, here is the data. I am laying it out pretty much as Bruce did with no commentary.Author Candidates were Joseph Smith, Sidney Rigdon (1831-1846), Sidney Rigdon (1863-1873), Solomon Spalding, Oliver Cowdery, and Parley P. Pratt using the NSC methods from the Jockers study. The tests were run on the fifty-one papers known to be authored by Alexander Hamilton without Hamilton included as a possible author.Results:Sidney Rigdon - 28 of 51 with posterior probabilities ranging as high as 0.9999Parley P. Pratt - 12Olivery Cowdery - 11Joseph Smith Jr. - 0Solomon Spalding - 0Bruce added a "latent author" variable to test for the possibility that none of the candidate authors were actually the true author of the text. The math is beyond me but to quote from Bruce's paper "We propose a latent author with a distribution of literary features just barelyconsistent with the new text."Using the latent author method and the same candidate author set, two of the papers were attributed to Rigdon and the rest to the latent author.A third experiment was run, this time using Hamilton as an author candidate, using twenty-five of the 51 papers as a training text for Hamilton and using the remaining twenty-six texts for the actual tests.Hamilton was identified as the author of all of those texts.Please note that this information was taken from a draft preprint of the paper. Bruce has polished it up because he said that "there has been some movement" on the publication of the paper, so we would expect some differences in the final product. Bruce did not indicate that there were any changes to the conclusions and data.From what I can understand of the graphs that Bruce has produced on his paper, the two texts that Rigdon scored on against the latent author were included in the twenty-six final texts using Hamilton as an additional candidate.That is the only conjecture that I will make.Thanks,Glenn Link to comment
Uncle Dale Posted August 26, 2010 Author Share Posted August 26, 2010 Dale, here is the data. I am laying it out pretty much as Bruce did with no commentary.Author Candidates were Joseph Smith, Sidney Rigdon (1831-1846), Sidney Rigdon (1863-1873), Solomon Spalding, Oliver Cowdery, and Parley P. Pratt using the NSC methods from the Jockers study. The tests were run on the fifty-one papers known to be authored by Alexander Hamilton without Hamilton included as a possible author.Results:Sidney Rigdon - 28 of 51 with posterior probabilities ranging as high as 0.9999Parley P. Pratt - 12Olivery Cowdery - 11Joseph Smith Jr. - 0Solomon Spalding - 0Bruce added a "latent author" variable to test for the possibility that none of the candidate authors were actually the true author of the text. The math is beyond me but to quote from Bruce's paper "We propose a latent author with a distribution of literary features just barelyconsistent with the new text."Using the latent author method and the same candidate author set, two of the papers were attributed to Rigdon and the rest to the latent author.A third experiment was run, this time using Hamilton as an author candidate, using twenty-five of the 51 papers as a training text for Hamilton and using the remaining twenty-six texts for the actual tests.Hamilton was identified as the author of all of those texts.Please note that this information was taken from a draft preprint of the paper. Bruce has polished it up because he said that "there has been some movement" on the publication of the paper, so we would expect some differences in the final product. Bruce did not indicate that there were any changes to the conclusions and data.From what I can understand of the graphs that Bruce has produced on his paper, the two texts that Rigdon scored on against the latent author were included in the twenty-six final texts using Hamilton as an additional candidate.That is the only conjecture that I will make.Thanks,GlennLooks good, Glenn -- I've added those comments here:http://solomonspalding.com/Lib/Jock2008.htm#comments5I've also begun to shorten the web-page contents in order to comply with copyright restrictions; so several pages have already been removed.UD Link to comment
Glenn101 Posted August 27, 2010 Share Posted August 27, 2010 Looks good, Glenn -- I've added those comments here:http://solomonspalding.com/Lib/Jock2008.htm#comments5I've also begun to shorten the web-page contents in order to comply with copyright restrictions; so several pages have already been removed.UDI think that is as far as we can go right now until Bruce's paper is published and the experts weigh in on it. That should be interesting and hopefully there will be some informed consensus.Glenn Link to comment
why me Posted August 27, 2010 Share Posted August 27, 2010 Using the latent author method and the same candidate author set, two of the papers were attributed to Rigdon and the rest to the latent author.A third experiment was run, this time using Hamilton as an author candidate, using twenty-five of the 51 papers as a training text for Hamilton and using the remaining twenty-six texts for the actual tests.Hamilton was identified as the author of all of those texts.Thanks,GlennSo this means that Hamiliton could have written the book of mormon? Was he still alive at tbat date? He could have done it if he were still alive. By why would he include JS in the plot? JS was just a little beneath him in intellect. Link to comment
Glenn101 Posted August 27, 2010 Share Posted August 27, 2010 So this means that Hamiliton could have written the book of mormon? Was he still alive at tbat date? He could have done it if he were still alive. By why would he include JS in the plot? JS was just a little beneath him in intellect. I assume that you are writing a bit tongue in cheek here?The texts had nothing to do with the Book of Mormon. They were all Federalist Papers known to have been written by Hamilton. This was a test of the Jockers NSC methods applied to the Federalist papers.The Federalist papers have been used by different people to test different authorship attribution techniques because the provenance of most of the papers are known, i.e. authored by either James madison or Alexander Hamilton.Glenn Link to comment
Uncle Dale Posted August 27, 2010 Author Share Posted August 27, 2010 ...The Federalist papers have been used by different people to test different authorship attribution techniques because the provenance of most of the papers are known, i.e. authored by either James Madison or Alexander Hamilton.GlennBut there is an extended purpose in Bruce's citing this example, and that is because Jockers himself has tested the NSC classification methodology against Federalist papers authorship and that study has been received by statisticians and historical scholars as valid.The leap in application Jockers has attempted, is to bring NSC methods over to a set of texts which has disputed authorship, and the set of known authors is also disputed. He justifies his experimental application of these methods in the case of Book of Mormon chapter attribution, based upon at least three points -- #1: The method accurately attributes Isaiah and Malachi to the BoM's Isaiah and Malachi (albeit it with a few false positives) -- #2: The method accurately excludes known non-BoM authors (Barlow and Longfellow) from the attributions (albeit it with a few false positives) -- #3: The method produces authorship attributions for Spalding and Rigdon, which in some cases are so strong as to be statistically impossible, if credited to coincidence. To these three points might be added one more: #4: The NSC results generally correspond to independent "Delta" methods also applied to the Book of Mormon chapters.Whether or not Jockers' transfer of a computerized methodology recognized as applicable to the Federalist Papers, over to study of the Book of Mormon, will be supported by independent, disinterested experts, remains to be seen.But we might also consider what would happen if Joseph Smith were added to the mix -- and his 1830 BoM "Preface" were added to the computerized testing. Jockers has updated his study recently, by adding Smith's word-print to the testing, but he did not add in Smith's 1830 "Preface." If that text was indeed composed by Smith, then he would be among the author-candidates for the BoM as a whole, and should be "testable."An accurate, valid application of NSC classification methods to the BoM, using word-prints for Smith, Isaiah and Malachi, should produce results showing that Smith wrote the 1830 "Preface;" that Isaiah and Malachi wrote the BoM chapters independently attributable to them --- and that none of the three authors wrote any of the remaining BoM chapters. That is, if the Mormons are right, in saying that Smith only wrote the "Preface."But there is a possible "fly in the ointment" with my latter suggestion.Perhaps NSC classification testing is only appropriate when ALL of the author-candidates are among the list tested for authorship by that method. If this is the case, then simply applying the word-prints of Smith, Isaiah and Malachi to the BoM would be useless ----- unless we also tested the text for the word-prints of Lehi, Nephi, Mormon, Moroni, Ether, Helaman, etc.I'd be curious to hear what the LDS statisticians have to say about this.What if the actual author-candidate list for the Federalist Papers is one day shown to include Thomas Jefferson? What if we found out, for certain, that Jefferson composed two of the papers? Would such a discovery render past NSC testing invalid -- because one of the actual authors had not been included in the study?Or, so long as SOME of the actual writers' word-prints are included in the computerized testing, is the NSC method robust and flexible enough to turn out "none of the above" percentage results for texts NOT composed by those SOME?Before Jockers can be fully relied upon, these are the sorts of issues that the experts will have to resolve. And I doubt very seriously that the controversy will be conclusively decided in the pages of BYU Studies or in Maxwell Institute publications. The issue will be resolved by an eventual scholarly consensus in the peer-reviewed, non-sectarian professional literature; or it will not be resolved at all.UD Link to comment
Glenn101 Posted August 27, 2010 Share Posted August 27, 2010 But there is an extended purpose in Bruce's citing this example, and that is because Jockers himself has tested the NSC classification methodology against Federalist papers authorship and that study has been received by statisticians and historical scholars as valid.The leap in application Jockers has attempted, is to bring NSC methods over to a set of texts which has disputed authorship, and the set of known authors is also disputed. He justifies his experimental application of these methods in the case of Book of Mormon chapter attribution, based upon at least three points -- #1: The method accurately attributes Isaiah and Malachi to the BoM's Isaiah and Malachi (albeit it with a few false positives) -- #2: The method accurately excludes known non-BoM authors (Barlow and Longfellow) from the attributions (albeit it with a few false positives) -- #3: The method produces authorship attributions for Spalding and Rigdon, which in some cases are so strong as to be statistically impossible, if credited to coincidence. To these three points might be added one more: #4: The NSC results generally correspond to independent "Delta" methods also applied to the Book of Mormon chapters.I do not know what methodology that Jockers et al used in their Federalist papers experiments. Did they use the same candidate set that they did for the Book of Mormon? Did they also use a candidate set that excluded the known authors but included the Book of Mormon candidate set?I do not understand the "statistically impossible" attributions assertion when all of the attribution results are relative to the the other included candidates.When Bruce excluded Hamilton as an author candidate for the 51 papers he is known to have authored, the authorship attribution results were similar to some of the results that the Jockers study obtained. The exception was for Spalding. He had zero texts attributed to him.Rigdon had the most attributed to him, 28, "with posterior probabilities as high as .9999.However, when Hamilton was added to the mix, none of the Federalist papers were attributed to Rigdon, or anyone else but Hamilton, which would seem to indicate that it is important to have the real author included in any candidate set.That is why Bruce made a few enhancements to the Jockers NSC methodology to ensure the results will not be skewed to the point of being unreliable if the actual author of a text is not included. There will probably be some enhancements to his "latent author" algorithm, but it shows some promise.I am going to quote a few sentences from Bruce's draft detailing some of the problems that Bruce saw with the Jockers study. And remember, this is from the draft, not the final edition that will be published."While NSC and related methodologies have much to offer to stylometric analysis, the recent use of NSC in a study of Book of Mormon authorship (Jockers et al. 2008) is problematic. The training data set was assumed to contain writings of the true author, goodness-of-fit of the procedure was not checked, posterior authorship probabilities of the test texts did not reflect text size, and multiplicity was ignored when interpreting the results. These issues would not usually be a problem in genomics investigations using NSC, but they are important issues in most stylometric problems."Whether or not Jockers' transfer of a computerized methodology recognized as applicable to the Federalist Papers, over to study of the Book of Mormon, will be supported by independent, disinterested experts, remains to be seen.I hope that it will not be another year before Bruce's paper is published. Then we can let the professionals duke it out and see the lay of the land after the dust settles.Glenn Link to comment
Uncle Dale Posted August 27, 2010 Author Share Posted August 27, 2010 ...I hope that it will not be another year before Bruce's paper is published. Then we can let the professionals duke it out and see the lay of the land after the dust settles.GlennI don't have any special information on this subject, other than what everybody else has heard. I do think, however, that not EVERY SINGLE writer of a controversial body of literature need be known in advance, before that text can be tested for probable authorship. I think that if Smith did indeed write the "Preface" to the 1830 BoM (and no other part) then there is probably some way of demonstrating that -- even if we do not add Moroni, Nephi, etc., to the list of author-candidates.As for the future -- some other statisticians are already citing the 2008 Jockers paper as providing some degree of reliable method/conclusions.Until we see some more publications, that's about all we have to go on in the professional literature right now.UD Link to comment
why me Posted August 28, 2010 Share Posted August 28, 2010 If anybody notices obvious errors or missing important information, please let me know, and I'll make the necessary changes.Thank you,Uncle DaleWell, I don't know about that Uncle Dale. But what I do know is that your former mormon turned catholic friend contacted you for some information and he posted it on a different board. Some catholics are now celebrating that the book of mormon is a fraud and the study proves it. Do you think that they are partying just a little too early? Interested to know. Link to comment
Uncle Dale Posted August 28, 2010 Author Share Posted August 28, 2010 Well, I don't know about that Uncle Dale. But what I do know is that your former mormon turned catholic friend contacted you for some information and he posted it on a different board. Some catholics are now celebrating that the book of mormon is a fraud and the study proves it. Do you think that they are partying just a little too early? Interested to know.Actually, the lady you refer to comes from several generations of Catholics, some of whom were Christians before the time of George Washington (or so I'm told). She has spent some time near Nauvoo, and knows something about Mormons, since one of her ancestors was in the anti-Mormon Hancock Co. Militia (if I recall correctly).I've never known a Mormon who decided to follow after the Bishop of Rome -- though I'm sure there have been such apostates, somewhere and somewhen.I see the Jockers' report as something like a fellow reaching his hand out the window, and telling people that it's raining outside.Another person comes along and disagrees, saying that the moisture thus felt came from some other source and it isn't raining at all.Yet another person comes along and professes that sticking a hand out a window to check for rain is not a reliable method, etc. etc.Perhaps it is raining -- perhaps it is not raining. If enough people actually take the trouble to go outside and check for themselves, then perhaps we can reach a consensus on the controversy.In Jockers' case, some additional capable people need to check out his methodology and his results. Until that happens, the best we can say is that he remains unchallenged in the professional forum where he published his report. And, in fact, is being cited as a reliable source by other scientists in the field.For those readers who are convinced by Jockers' arguments, the lack of any professional rebuttal, after many months, speaks for itself.Uncle Dale. Link to comment
why me Posted August 28, 2010 Share Posted August 28, 2010 For those readers who are convinced by Jockers' arguments, the lack of any professional rebuttal, after many months, speaks for itself.Uncle Dale.I don't think that most people care one way or another...at least not those who don't have a stack in the issue. Of course, we have Dan telling us that a rebuttal is on its way. I would think that the lds will be the only ones offering a rebuttal. And I am sure that it will be a good one. Not that it will make a difference. Link to comment
Uncle Dale Posted August 28, 2010 Author Share Posted August 28, 2010 ...we have Dan telling us that a rebuttal is on its way....Dan Vogel?I would think that he would wait until Jockers publishes his updated Bom authorship report, including Joseph Smith's word-print.Or, perhaps you mean Dan Vogel's supporter, Chris Smith? His rebuttal is on-line hereOr, perhaps you mean Dan Peterson is going to publish something by a Mormon author in a Maxwell Institute journal?That will be practically meaningless and will receive no professional attention outside of the LDS cloister. I can write a paper on the true origin of L. Ron Hubbard's stuff, but if the only rebuttal it ever receives comes from Scientology itself, the rebuttal will be practically meaningless.I can write a paper on the true origin of Rev. Sun Moon's sermons, but if the only rebuttal it ever receives comes from the Unification Church itself, the rebuttal will be practically meaningless. Even an article in the "Washington Times" would be practically meaningless.If LDS statisticians intend on joining a professional dialogue about Jockers' work, they'll have to do so in the non-sectarian professional literature, and go through proper peer-review, without Maxwell Institute editors holding their hands.UD Link to comment
Glenn101 Posted August 29, 2010 Share Posted August 29, 2010 Uncle Dale, on 28 August 2010 - 02:32 PM, said:For those readers who are convinced by Jockers' arguments, the lackof any professional rebuttal, after many months, speaks for itself.Uncle DaleIf LDS statisticians intend on joining a professional dialogue about Jockers' work, they'll have to do so in the non-sectarian professional literature, and go through proper peer-review, without Maxwell Institute editors holding their hands.UDDale, I am sure you are aware that Bruce Schaalje has presented a paper for publication critiquing the Jockers study. It was presented to the Literary and Linguistic Computing Magazine in April of 2009. It has yet to be accepted for publication, although Bruce did say a couple of weeks ago that there is now some movement on it. I have not been informed as to why the delay nor when we can expect to see the actual publication. Just to note, Bruce has authored and coauthored other publications which have been had favorable peer reviews, so it is not as if he is a tyro in statistics or publishing.In other words, there has been a professional rebuttal. Those who are convinced by the Jockers study arguments have not even attempted to engage the points that Bruce made. Some of the math is beyond most of us, but the fact that when Bruce ran the unmodified NSC algorithms on the Federalist papers known to be authored by Hamilton without including Hamilton as an author candidate, Rigdon came up as the probable author for 28 of those 51 papers with some very high posterior probabilities. However, when Hamilton was included as an author candidate, he was correctly chosen every time. Those facts speak for themselves.Glenn Link to comment
Uncle Dale Posted August 29, 2010 Author Share Posted August 29, 2010 ...there has been a professional rebuttal...And, when it finally reaches the literature, it will be the beginning of a discussion -- perhaps even a conversation and a dialogue.Nobody other than Mormons will read a response in the Maxwell Institute publications. Sorry about that, but it's true.If the response is meant primarily for faith promotion, then I suppose it's OK. If it is meant as a means to engage other scholars in a scientific inquiry, then there will be a problem.Where I see the real problem arising for Mormons, is if Jockers' new use of NSC classification for texts with contested authorship lists is picked up and applied by other scientists in other authorship studies.Today there is only Jockers to deal with. In a year or two we may see half a dozen statisticians using Jockers' methods for purposes entirely unrelated to the LDS "restoration scriptures."A rebuttal paper in BYU Studies (which is probably more widely read than Maxwell Institute articles), may generate a bit of interest, so long as its contents are focused upon LDS-related topics. But who on earth will read a Provo journal, when it comes to second-generation topics relating to Jockers' methods, but NOT to Mormon topics.If the LDS scholars are going to nip this problem in the bud, the place to do that is out in the general public, far away from BYU and from Utah.I wonder if anybody has given that any thought yet?UD Link to comment
Glenn101 Posted August 29, 2010 Share Posted August 29, 2010 If the LDS scholars are going to nip this problem in the bud, the place to do that is out in the general public, far away from BYU and from Utah.I wonder if anybody has given that any thought yet?UDI'm sort of scratching my head here. I thought that I pointed out in the post you just snipped from, that the problem is being addressed, and will be in the same venue that the Jockers Study was published.As I mentioned before, once that happens and the dust settles, we will see better how things lie.Glenn Link to comment
Uncle Dale Posted August 29, 2010 Author Share Posted August 29, 2010 I'm sort of scratching my head here. I thought that I pointed out in the post you just snipped from, that the problem is being addressed, and will be in the same venue that the Jockers Study was published.As I mentioned before, once that happens and the dust settles, we will see better how things lie.GlennYou can check with Bruce to find out for certain, but I think that the "movement" in his paper's eventual publication has not come from the reviewers at the computing journal. Last I heard, he was also submitting it for publication in Provo, and had submitted a more arcane mathematical paper to a stats journal, which is not the pre-pub we've read.As you say -- we shall see.UD Link to comment
why me Posted August 29, 2010 Share Posted August 29, 2010 Dan Vogel?.UDDaniel Peterson. I keep hearing from him on the board that a rebuttal is own its way. Link to comment
BCSpace Posted August 29, 2010 Share Posted August 29, 2010 Readers of these comments should keep in mind that the 2008 Jockers paper identifies Solomon Spalding, Sidney Rigdon, Oliver Cowdery and Parley P. Pratt as contributors to the Book of Mormon, only in terms of relative probability. If more author-candidates were appended to Jockers' list, the new additions would likely score at least a few "hits," and thus reduce the probability percentages already obtained for the original author-candidates. If the "true writers" of the Book of Mormon have been somehow left off of Jockers' list, then of course they would score no probability percentages at all. It is also possible that no "true writers" of the book were included among Jockers' selection of 19th century writers, and that no matter what probability percentages were assigned in the 2008 paper, all those authorship figures are meaningless in the real world. So...back to square one for modern authorship proponents. Especially since we're dealing with a translation, why would it be suprising to find a "true" modern author anyway? That would not speak to false claims on the part of JS & Co. Link to comment
why me Posted August 29, 2010 Share Posted August 29, 2010 I just hope that the rebuttal comes soon. Of course, it will not matter for many critics who wish the lds church to fall. But this study is being paraded around on critic sites as an end all end all for the lds faith. The rebuttal should come sooner than later. Link to comment
Benjamin McGuire Posted August 29, 2010 Share Posted August 29, 2010 I just hope that the rebuttal comes soon. Of course, it will not matter for many critics who wish the lds church to fall. But this study is being paraded around on critic sites as an end all end all for the lds faith. The rebuttal should come sooner than later.The wait is worth it. Once it's out there, this will all have seemed a very temporary issue.Ben M. Link to comment
Glenn101 Posted August 29, 2010 Share Posted August 29, 2010 why me, on 29 August 2010 - 09:58 AM, said:I just hope that the rebuttal comes soon. Of course, it will not matter for many critics who wish the lds church to fall. But this study is being paraded around on critic sites as an end all end all for the lds faith. The rebuttal should come sooner than later.The wait is worth it. Once it's out there, this will all have seemed a very temporary issue.Ben M.Is the rebuttal the one that Bruce Schaalje presented for publication. The one that the Literary and Linguistic Computing periodical has been dragging its feet on? Or is there another in addition?Thanks,Glenn Link to comment
why me Posted August 29, 2010 Share Posted August 29, 2010 The wait is worth it. Once it's out there, this will all have seemed a very temporary issue.Ben M.I am glad to hear it. But I am afraid the damage has been done. With the Internet the critics will keep dragging out this study to prove their point. And of course, there will be no rebuttal included when they pull it out. If it is a good rebuttal, I think that FAIR should put it in their wiki and in their regular website and Dale should include it in his website. Link to comment
Recommended Posts
Archived
This topic is now archived and is closed to further replies.