Expert Wary Of NJPS Findings

The long-awaited National Jewish Population Study 2000-01 is flawed to the degree that many of its key findings should not be considered reliable, and ought to be reworked and re-analyzed before being used by communal policy planners.
That is the upshot of a new report from an independent expert brought in to evaluate the landmark survey.
Mark Schulman, a social science research expert and chair of the NJPS 2000-01 Review Committee, was commissioned by the survey’s sponsors, the United Jewish Communities, to produce a definitive analysis of "what went wrong" with the study, which cost a record $6 million to produce.
"Study design issues and administrative problems" make this NJPS "particularly prone" to error, Schulman writes in his 24-page report, which includes multiple warnings to continue to vet the findings.
After initial results of NJPS were released last October, senior UJC executives got word that pieces of the data had been lost and that there were other serious questions about the reliability of its conclusions.
At the 11th hour, UJC president and chief executive officer Stephen Hoffman shelved the study pending a full inquiry. The committee Hoffman appointed brought in Schulman to investigate.
Schulman, in polite but clear wording, says the survey’s sample "shows signs of a skew" and "strongly recommends follow-up research to gauge the extent to which the Jewish sample may skew toward Jews who are more religiously identified and who reside in completely Jewish households."
Aspects of the study’s design also "may have produced an estimate of the Jewish population that is slightly lower" than it should be.
Schulman credits UJC for airing issues related to the survey’s methodology and for its openness in establishing the evaluation process.
One big problem, he notes, is the matter of the "lost data," which has plagued the study since parts of it were made public last year. Screening information from two-thirds of the households initially called, in which at least one adult was Jewish, was lost by the research firm, Roper Audits & Surveys Worldwide.
The study’s population estimates are based on information from the remaining one-third of responding households with at least one Jew, and the researchers weighted the findings to compensate for the missing data. The way in which it was weighted is reason for caution, Schulman writes.
Furthermore, the population estimates do not factor in the potential responses of those people who refused to respond to the religion items, Schulman writes. UJC researchers decided there were no Jews among the missing cases, "the most conservative assumption possible."
"While such an estimate should be reported, it is unlikely to be the best estimate," he writes.
Though NJPS officials have been claiming a 28 percent response rate, in fact it is significantly lower in the key population: for Jews and people of Jewish background it is below 20 percent, according to Schulman’s report. And "response rate issues may have skewed the final sample," he writes.
Even the 28 percent response rate "is considered at the low end for public policy and population studies," he writes, "and does raise at least a ‘yellow flag’ of caution in interpreting the results."
Schulman acknowledges that response rates on sensitive topics, like religion and personal finance, often experience lower response rates than surveys on more general topics. And refusal rates in the New York metropolitan area, where NJPS over-sampled, tend to be higher than refusal rates elsewhere.
Based on analysis by Len Saxe, co-director of the North American Jewish Data Bank, the archive which holds the NJPS, the result may be that the Jewish sample undercounted those in interfaith marriages.
In an interview with The Jewish Week, Saxe said the true response rate was 16 percent and that "a response rate below 20 percent raises serious questions about the kinds of inferences that can be made. You can’t be confident" of the results.
Schulman, a principal in the Manhattan-based research firm Schulman, Ronca & Bucuvalas Inc., is immediate past president of the industry’s professional group, the American Association for Public Opinion Research. He returned this week from meetings in Europe and did not return phone messages seeking comment.
Some aspects of the NJPS remain credible, Schulman says in his report. The most valuable information will come from "the analysis of relationships between variables," or subgroups, "within this data set," Schulman writes.
"Analysis of these relationships will provide valuable insights into the relationships between the varying backgrounds of Jews, their beliefs, religious practice and the role of religion in family life."
But even that must be viewed cautiously, Saxe said in an interview.
"The data set needs to be re-looked at, possibly re-weighted," he said. "We draw conclusions from the data, particularly conclusions about the percentage of people who do X, and we need to be very careful in assuming that that’s a representative group of engaged American Jews.
"The Schulman report identifies a number of potentially serious problems which make using the survey difficult," Saxe said. "I hope that researchers will look at the data set and, following the cautions of Schulman and the committee, do re-analysis to draw out appropriate conclusions. With this report, people can use the data set much better."
In addition to criticisms about the study’s findings, the motives of the UJC were called into question this week. J.J. Goldberg, editor of the Forward newspaper, asserted in an opinion piece in The New York Times that UJC knowingly undercounted the Jewish population.
"Whether out of ideology, ego, incompetence or, as I suspect, a combination of all three," he wrote, "the respected charity invented a crisis."
The UJC’s Hoffman called Goldberg’s charge a "slander" and said that "nothing could be further from the truth."