NJPS Credibility Questioned

Leading demographers are raising serious doubts about the credibility of the long-awaited $6 million National Jewish Population Study 2000, now that its sponsor has shelved the findings pending an investigation into the loss of some research data.
The integrity of the study — data from which could shape communal policy for years to come — is potentially undermined simply by being pulled from public view on the eve of its release, these demographers say. And that is true, they say, though only a small amount of data was lost, too little to seriously impact the ultimate numbers.
“The legitimacy of the study and the personnel in charge, as well as the judgment of the leadership of the United Jewish Communities, is now being questioned,” said Calvin Goldscheider, a sociology professor at Brown University.
“I am not optimistic about the future political/policy value of this expensive study,” said Goldscheider, a critic of the methodology used in the 1990 NJPS.
Much-anticipated findings about synagogue affiliation and the intermarriage rate were slated for release at this week’s UJC General Assembly in Philadelphia, the annual meeting bringing together thousands of Jewish organization staff and communal policy makers.
When UJC president Stephen Hoffman announced last week that all NJPS 2000 information and related programming was being pulled from the GA schedule because of his questions about the ramifications of the lost data, most demographers were surprised.
“Any delay of this sort can be damaging because it increases the likelihood that people out there won’t have confidence in the findings,” said Frank Mott, co-chair of the NJPS national technical advisory committee.
Hoffman, for his part, said he no longer trusts the leaders of the technical advisory committee.
The decision to delay the findings “finally pointed out what has been looming in the background” of the NJPS 2000, said Egon Mayer, a member of the advisory committee — “that this is not an independent study, but one with an organizational agenda” driven by internal political concerns.
“In social science we make mistakes all the time,” Mayer said, “but we publish them and take responsibility when we are right and when we are wrong. Now you no longer know why information was allowed to come out or not come out.”
After three years of work and nearly twice the initially budgeted outlay, Hoffman last week became aware that a small percentage of the calling records assembled by the polling firm hired to conduct the research had been lost some time ago.
Demographers who worked on the study say the missing data was not significant and their loss should not have impeded the study’s release. At most, they say, the American Jewish population estimate may be 5.3 million rather than 5.2 million. That figure was announced last month when the first round of NJPS 2000 data were released.
In addition, experts both inside and outside of the NJPS project are raising questions about why Hoffman until now has evidently been unaware of the process and results of his organization’s largest domestic project.
The results of some 3,500 initial screening calls, of a total of 175,000, were lost by the firm hired to conduct the NJPS 2000 research, Roper Audits & Surveys Worldwide. The vanished records were from the first round of calls to some 2 percent of randomly dialed phone numbers. These records relate to which calls were answered by someone Jewish or living with a Jew and which were not. Lost was screening data, not information gathered from actual interviews, say those involved.
The records were likely misplaced or erased in the spring of 2001, when UJC’s research funds ran out and the fieldwork was put on hold until more money was raised a few months later, those involved said.
First indications that data might be missing came in May, according to David Marker, associate director of Westat, the federal government’s largest data collection contractor and a member of the NJPS technical advisory committee. In September, Marker said he spoke with the Roper employee weighting the sample to reflect the American Jewish population.
“He clarified how he did it and I was told that the missing information was not anything we would be concerned about,” Marker said.
In September more information was requested from Roper by members of the advisory committee, he said. But it wasn’t until Nov. 6 that the extent of the situation became clear to committee members, Marker said.
Only last week, on Nov. 12, did UJC executive Lorraine Blass, the NJPS project manager, inform Hoffman, according to Marker.
By the end of the next day, over strong objections from advisory committee leaders, Hoffman put the NJPS 2000 findings on hold.
“He jumped too quickly,” said Mott of the advisory committee, a demographer based at Ohio State University. While a small portion of data being lost isn’t a good thing, “it’s not a fatal flaw,” Mott said.
Hoffman made clear his loss of confidence in Mott and his committee co-chair, University of Delaware researcher Vivian Klaff.
“Leaders of [the technical advisory committee] have made a habit of reducing things to insignificance,” Hoffman told The Jewish Week. “I no longer believe that everything is insignificant.”
“What’s missing could be imputed,” countered Ira Sheskin, an advisory committee member based at the University of Miami. Sheskin, however, agreed with Hoffman. Stopping the release until all questions are answered was “the right call,” he said.
By Monday, Hoffman had appointed McGill University President Bernard Shapiro to lead a task force to investigate what happened. It will begin its work in early December, said Hoffman. He has not given the task force a deadline and declined to speculate how long it would take.
“Everything will be double-checked now,” said Sheskin. He and others said the process would likely take months.
Hoffman said the task force would seek outside experts to evaluate the situation, but that current technical advisory committee members also were free to participate.
“I hope that as quickly as possible we’ll identify the questions that have come up around the data, determine what is statistically insignificant, what needs to be looked at further and what aspects of the study needs more data collection,” said Hoffman. “This will be done in the full light of day by knowledgeable people who are not interested in covering up mistakes.”
Initial NJPS 2000 findings released last month were made public without acknowledging the error in data collection. Hoffman said at this point he is not considering disciplinary action against members of UJC’s staff, including Jim Schwartz, the director of research, or Blass.
Nor have sanctions against Roper been discussed.
“My focus is not on discipline and on who did what when,” Hoffman said. “My focus is on does the study hold together and hold up to questions that have been raised? My belief is currently no, because I stopped it.”
Roper spokesperson Malkie Bernheim said the firm “is cooperating vigorously with the UJC independent review of the recent survey data and expects that the integrity of the study will be validated as soon as possible.”
Hoffman said he has additional questions, beyond those relating to the lost data, involving Roper and the advisory committee leadership, but he gave no specifics. If the missing data “was the only question I had, I probably wouldn’t have stopped” the study’s release, he said.
The advisory committee’s Mayer, who heads the sociology department at Brooklyn College, said: “I’m not sure that Hoffman appreciated how central this project was in the life of UJC. If somebody wanted to be briefed, I’m sure he could have been.
“I also don’t know how much he appreciated the extent to which you can’t take a research project and play it like a yo-yo, letting information in and then out,” he said.
But one major NJPS funder came to Hoffman’s defense.
“In fairness, Hoffman has been in crisis overload, what with Israel and Argentina” needing so much fund-raising attention from UJC, said the funder, who spoke on the basis of anonymity. “He doesn’t have a team around him” that can handle these things, he said.
Details of the NJPS 2000 findings are eagerly awaited by those involved in planning Jewish communal policy.
The most highly anticipated figure is the intermarriage rate. When the 1990 NJPS reported that just over half — 52 percent — of Jews marrying within the preceding five years had wed outside the faith, it shocked the Jewish community into a new era.
A source close to the NJPS said documents indicate that the new intermarriage rate is 55 percent.
That figure was denied by Jerusalem-based demographer Steven Cohen, a longtime critic of the approach used in the 1990 study who was hired by UJC to consult on the NJPS.
“I was calculating” the intermarriage rate, he said in an e-mail interview. “I can’t say more, but that is absolutely wrong.”
Hoffman could neither confirm nor deny the 55 percent figure.
“I haven’t seen a hint of a number, not a whisper,” he said. “There were a few things I was informed about, but nothing comprehensive. If there was a big intermarriage number I think I would have known, but who knows, maybe not. That’s why I have to stop this now.”
Still, many demographers expect the NJPS intermarriage rate to come in close to 52 percent this time, too. That would mean the rate has remained essentially flat over the past 17 years, taking into account the 1990 study reflected marriages within the preceding five years.
If intermarriage rates remain essentially unchanged, does that mean the incalculable investment of the last decade on “inreach” and “outreach” has failed?
Brandeis University sociologist Sylvia Barack Fishman says no.
While “there is a drifting away on the part of a certain segment of the community, the other development is a much more intensive involvement ” among other Jews, she said. “If people would also pay attention to the great success of revitalization efforts, the picture would look very different.”
As to the work ahead for the new UJC task force, Hoffman was asked if he hoped to have a preliminary report by the end of December.
“All I want to do by the end of the calendar year,” he said, “is survive it.” n
With reporting by editor and publisher Gary Rosenblatt and staff writer Stewart Ain.