Critics Question Value Of Population Study

On the eve of the release of the long-awaited and delayed National Jewish Population Study 2000, sources close to the review of the data worry that the study is too flawed to be effectively salvaged.

And they warn that making the NJPS public could prove harmful, setting off a range of communal analysis, planning and spending based on inaccurate information.
A chief concern of the sources is that the response rate was too low to reflect the numbers and attitudes of American Jewry.

The study, whose fieldwork began three years ago and cost $6 million — twice the original estimate — is “an unmitigated disaster,” said one source familiar with the study. This source asserted that the NJPS “was poorly designed, implemented and evaluated.”
The study’s release was delayed last fall after it was learned that data had been lost, among other problems.
Officials of the United Jewish Communities, the umbrella group for the federation system in North America that oversaw and funded the study, declined to comment on the record other than to say the findings will be released to its leadership Sept. 9. But privately they insisted the study is solid and highly informative, that researchers used the highest standards and that the full data set will be made public within days after the release.

Calvin Goldscheider, a sociology professor at Brown University who has read some of the methodological reports UJC has done on the study, was skeptical. He said the NJPS “had problems” and “will not be very useful” in providing the kind of “descriptive information that most people will be interested in” — namely, statistics on the growth or decline of the U.S. Jewish population, as well as of the various religious denominations and the intermarriage rate.
“The data collected can’t answer those questions accurately,” he said, because of problems in the way questions were formulated and asked, as well as data that was lost and “is not correctible.”

On the other hand, Goldscheider said the study contains “rich data” on more “analytical issues,” like whether the amount of education Jews have affects who they marry, or what it means to people to be Jewish.

He suggested UJC learn more from social scientists about “how to ask the right questions.”
Several experts echoed this theme of focusing less on what one called “the numbers game” — the size of the U.S. Jewish population, etc. — and more on examining closely “those Jews we know and how they look at their identity.”
The experts said this was in part because communal officials put too much stock in “definitive statistics,” and in part because, given the problems associated with this NJPS, even when the study is vetted and made public, its findings should be approached with a degree of caution.

“This study should have an asterisk next to it,” one expert said. “The only question is how big that asterisk should be.”

Even before its release, the NJPS has undergone a tortuous process that in recent months has taken on a Humpty Dumpty aura, with teams of demographers, consultants and UJC leaders working feverishly to repair the damaged data.

Touted by UJC as “the most detailed study of American Jewry ever conducted,” the NJPS originally was scheduled to be rolled out as the centerpiece of the annual UJC General Assembly in November, and a preview of the study was made public in October. It found that the Jewish community had grown older (median age 41, up from 37 a decade ago) and smaller (about 5.2 million, a 5 percent drop).

But the study was shelved after it was learned that some research data was lost. A UJC task force was appointed to review and report on what went wrong, and release the revised study. That process has been delayed several times this year, but UJC announced this week that the findings will be provided to its executive committee at a meeting in Chicago early next month.

Mark Schulman, president of the American Association for Public Opinion Research and a highly respected leader in the field, is preparing an independent report for the UJC task force due to be submitted to officials within days. Insiders say that in addition to making transparent the data and methodology used, the report will be strongly critical of how the NJPS was done, though couched in delicate language.

Schulman, who was unavailable for comment, was said to be working on the nuances of what all agree is very complex data and analysis.

The source most familiar with the review, who requested anonymity, said a chief concern is that the response rate was so low — 28 percent, according to researchers — that the study’s findings would not be representative of the American Jewish community it seeks to profile and examine. (Government surveys require a 70 percent response rate, and 50 percent is the minimum used in many national opinion polls, experts say.)

Goldscheider said that “if the response rate was around 30 percent, you could throw the descriptive parts of the study out the window. More importantly,” he added, “you can challenge the policies based on the study and seriously question again the value of a considerable expenditure with very low returns for the community.”

Sociologists, demographers and others in the field worry that a low response rate calls into question the reliability of the respondents, suggesting that a disproportionate number in this case were “elderly, lonely or just liked to talk on the phone,” one expert said.

UJC officials maintained they used a highly rigorous standard to be most accurate, even knowing it would produce a lower response rate. They said the full data set and methodology would be posted in early September on the Web site of the North American Jewish Data Bank, recently relocated to Brandeis University, at www.jewishdatabank.org. They noted that no other Jewish population study has made such detailed information available.
NJPS planners had hoped to survey 5,000 Jews around the country, but ran into difficulty in finding Jews and getting them to agree to be interviewed. They ended up with about 4,000. One apparent problem was in relying on random-digit dialing, which proved costly and ineffective. Critics say the procedure tends to skew toward older people who are more likely to be home and willing to participate in a lengthy phone interview.
Respondents were asked 250 questions and the average length of an interview was about 40 minutes, far longer than many people are willing to be interviewed.

(One reason so many questions were asked is that UJC sought additional funding from private foundations to subsidize the cost of the study, and some foundations agreed in return for including questions they wanted answered, pertinent to their philanthropic activities.)

Eventually, the researchers directed more of the random-digit dialing calls to areas with larger Jewish populations, and the results were then weighted to account for less-populated areas.

Just how much weight to give to respondents is a delicate and debatable issue since there is no U.S. census information on Jews to compare with, as is done with other data in national surveys.

The researchers ended up offering people $25 as an incentive to agree to respond to the lengthy list of questions.

Another major concern is that the NJPS findings will not be comparable to the last national study, released in 1990, since different techniques were used.

The most memorable finding of the previous NJPS was the rate of intermarriage, which was said to have reached 52 percent. That number continues to be debated, but it alarmed the community to the point that it launched a number of outreach and “Jewish continuity” programs in response.
Communal officials are eager to learn what the current intermarriage rate is and assess whether they are on the right track in combating assimilation. The NJPS data will also be used to determine future planning, particularly in terms of young families and children, noted sociologist Egon Mayer, “but everything hinges on the underlying validity” of the methodology.

Mayer is one of about two dozen experts in the field who make up the NJPS national technical advisory committee. He said this week that the committee has been bypassed in recent years by UJC and has not been apprised of the work of the UJC’s national review body. “We’ve seen nothing,” he said.
The advisory committee has been invited to take part in a Sept. 4 conference call, apparently to fill in the members on the review body’s progress, Mayer said.

Mayer published his American Jewish Identity Survey in 2001, concluding that there are 5.5 million Jews in the U.S. and that nearly 10 million people live in households with a Jew or someone who considers himself Jewish. He said he used the same methodology as the 1990 NJPS so that direct comparisons could be made.