The vision to be ‘Viksit Bharat’ by 2047 can become reality only when we closely monitor progress towards our development targets and can prove at the end that we successfully met them. This rests on the institutional capacity to measure defined indicators and present them in an unbiased manner. Official statistics, considered an integral part of a country’s development infrastructure, perform this role.
The United Nations has recognized that “official statistics provide an indispensable element in the information system of a democratic society, serving the government, the economy and the public with data about the economic, demographic, social and environmental situation.” However, this element functions within the domain of the government, making its framework and architecture heavily reliant on the priorities and perceptions of the government of the day. Can India’s statistical system surmount these challenges?
The foundation for official data collection was laid immediately after Independence when the country opted for a planned economic path. Before that, the decennial censuses, commercial trade statistics and other administrative reports were the primary source of data. The wholehearted support of the government and the data that the planning process needed ensured that India adopted several innovative statistical processes. In fact, few may remember that one of the first mainframe computers imported in the country in the mid-1960s was installed in the ministry of statistics!
The technical support from the Indian Statistical Institute saw the successful implementation of statistical sampling at a scale untested till then. This was widely recognized globally and formed the cornerstone of the national statistical edifice. It was complemented with censuses covering agriculture, livestock, manufacturing, enterprises, and more. Statutory and administrative reporting like the labour market information through employment exchanges also strengthened the database.
However, by the turn of the century, overreliance on conventional methods for data collection and processing, and the failure to produce timely data led to a steady decline of the statistical system. The need for timely data beyond the needs of the planning process was felt with non-governmental users, too.
Added to this was the global scrutiny of economic data after the East Asian crisis. The statistical system also failed to adapt to new challenges like the spread of the Internet and information technology (IT), and the growing complexity and size of the economy and society. In short, a failure to modernize.
One notable effort to identify and suggest corrective steps the government made was to appoint a National Statistical Commission under Dr R. Rangarajan. However, the recommendations of this commission could not address all the maladies, as the digitalization and data explosion changed the supply-and-demand environment in new ways not foreseen by the commission.
Today’s official statistical system functions in an environment significantly different from the past, making many parts of the established processes obsolete.
Official data now has many stakeholders beyond the government, including business, academia, general public, media and international agencies. Even adhering to the sustainable development goals requires monitoring of datasets covering an array of topics.
This has enlarged the scope of official data and broadened the dimensions to include not just quantitative information of economic units, but also spatial and qualitative aspects.
Unfortunately, we now have a situation where Indian development data is under a cloud leading to growing discontent among users of Indian official statistics. Instead of a concerted effort to reorient the system, there is growing disregard and slow corrosion of key elements of the system like the population census, national sample surveys, revision of base years of economic indicators, etc.
Rather than generating independent objective data based on established procedures, there is greater reliance on data generated through portals in the form of ‘dashboards’ with completely opaque metadata. This has also meant a centralization of data gathering, disregarding the responsibility of state governments to track subjects under their remit, further degrading the statistical capacity of states. There are growing concerns about the privacy issues in the data assembled through such processes.
The recent labour ministry criticism of the India Employment Report 2024 brought out by the International Labour Organization and the Institute for Human Development using data from official surveys is a case in point. Such instances can discourage independent analysis.
We have also witnessed government agencies widely publicizing global indices when they are favourable even if these have very weak data support.
On the flip side, results of the government’s own surveys are criticized when they are in conflict with official claims. The government’s sensitivity to data is a new dimension negating the advantages of having multiple sources of data that can actually help cross-validate official data.
The lack of interest in strengthening official data by central agencies within the federal framework impacts the states in more serious ways. For example, the gross state domestic product (GSDP) is a key metric used in the devolution of central funds to states. The current methodology has very little room for use of state-specific data in GSDP estimation.
These developments are in the backdrop of growing digitalization of business and social interactions and the growing interest in data by the media and researchers.
The high standards of data journalism and the incisive data analytics used by researchers are in sharp contrast to the official indifference to data. These independent agents have the capacity to work on underlying data and bring out insightful findings rather than reporting official conclusions.
Lack of auditing of statistical processes is another area that impacts accountability. We have had national censuses and surveys that have cost huge resources but did not produce any results due to reasons casually ascribed as quality issues, without any professional justifications.
The path to becoming a developed nation by 2047 needs milestones in the form of socioeconomic indicators meeting global standards based on transparent and verifiable methodologies open to scrutiny and criticism. These steps are essential to restore public trust in data, both nationally and globally. This requires a series of new initiatives in statistical capacity building and institutional reforms. These include revisiting institutional structures for managing official data leading to more autonomy for the statistical system to restore the primacy of data. This can be along the lines of reforms in the UK statistics in recent times.
India’s size and diversity mean it has to develop its own templates for data collection with proportionate increase in resources. The uneven capacity of states for data gathering and dissemination is an area of concern which, in the short run, needs the central agencies to be more proactive and produce national and sub-national data.
The integration of the many parts of the statistical system also calls for better metadata management techniques, without which data silos will have very little interconnectivity. Non-standardization of data elements has always been a problem for data users. Despite efforts in establishing e-governance standards in administrative data, at the ground level the statistical system is yet to derive its benefits. Same is the story of nearly two decades of efforts towards integration and access to spatial data where there are also issues of security involved.
The challenges faced by India’s statistical system will have few parallels anywhere and even fewer models to emulate. The population size and the informality of economic activities will make measurement issues complex here unlike in developed countries. But transparent and autonomous institutions alone can defend the system.
The first set of reforms to enhance credibility is to build an independent institution in charge of national statistics with powers to break down the silos. Such an institution will help identify operations that have outlived their initial objectives like economic census, agricultural census, etc., and focus on production of key macroeconomic and social indicators.
Taking into account the new dimensions of data and the growing economy, human resources should be enhanced with more stress on the multidisciplinary nature of data and its handling. Experience also shows that initiatives like the open data portal of the National Informatics Centre, data and analytics platform of NITI Aayog, etc., have to be part of the statistical infrastructure and not just IT initiatives.
Though most statistical databases are freely given to researchers and are available in data centres of research institutions, data gathered by official agencies for various schemes are not in the public domain.
Making these available with safeguards for privacy and their objective analysis can also contribute to understanding the developmental milestones as we progress towards 2047.
P.C. Mohanan is a former acting chairman of the National Statistical Commission.
