Archived - Evaluation of the Economic and Fiscal Policy Branch

Archived information

Archived information is provided for reference, research or recordkeeping purposes. It is not subject to the Government of Canada Web Standards and has not been altered or updated since it was archived. Please contact us to request a format other than those available.

Prepared by
Internal Audit and Evaluation
Department of Finance Canada

Approved by the Deputy Minister of Finance on the recommendation of the Audit and Evaluation Committee on May 26, 2015

Table of Contents

Executive Summary

Executive Summary

This report presents the results of the evaluation of the Economic and Fiscal Policy Branch (EFPB) of the Department of Finance Canada, which was conducted between April and October 2014. The evaluation covered approximately $7.2 million in annual direct program spending and examined the relevance and performance of EFPB’s activities and operations over fiscal years 2009–10 to 2013–14.

The evaluation relied on three lines of evidence: document review and analysis, key informant interviews and a case study. The following are the key findings and conclusions of the evaluation.

Relevance

EFPB’s activities and objectives are clearly aligned with the roles, responsibilities and priorities of the Government of Canada and the Department of Finance Canada. EFPB analyzes the economic and fiscal situation of the country and advises the Minister on the government’s fiscal framework. It also provides the Minister with analytical support on a wide range of economic and financial issues related to the government’s macroeconomic and structural policies and plays the lead role in the federal budget process. EFPB conducts research and analysis that is highly relevant to the policy decisions facing the government.

Performance

EFPB is a high-performing branch—effective, efficient and economical—having solid leadership, highly skilled and experienced staff, and effective processes to support its activities.

The quality of its policy research, analysis and advice is consistently high across its three divisions and is noted for its timeliness, relevance and soundness of evidence. EFPB has effective working relationships and collaborates with its main partners and stakeholders, who have a high level of confidence and trust in EFPB.

Despite operating with reduced resources over the past five years, EFPB has successfully met a high level of demand, maintained high-quality and timely outputs, and continued to make improvements to its processes, including its coordination of the annual Budget. However, continuing to operate with sustained resource pressures poses some long-term capacity risks. In recent years, EFPB has been more limited in its ability to invest in important long-term planning activities, including anticipatory analytic work, such as model development and in-depth analysis of emerging and medium- to long-term policy issues, and development of staff and knowledge transfer through directed exposure of staff to different tasks to develop new technical and managerial capabilities. These are risks for the long-term sustainability of the current high level of productivity and the ability to supply relevant research and analysis, as well as for succession planning.

It may be useful to explore opportunities to further increase consultations, communications and coordination related to the development of economic policy analysis and advice. These opportunities could include promoting more consultations with external stakeholders, including other policy branches, other economic institutions (for example, think tanks), and the Canadian business sector.

Over time, EFPB’s senior management has taken steps to better integrate the role of the Economic Studies and Policy Analysis Division (ESPAD) within the branch and the Department, to promote more policy-relevant research and analysis. Although this has been an important positive development, the roles and responsibilities of ESPAD and of EFPB’s current analysis division, the Economic Analysis and Forecasting Division (EAFD), for analysis of some economic policy issues have become less distinct. In this context, it is important that EFPB’s senior management continue its efforts to ensure that the roles of these two divisions are clear and that work is well coordinated in order to effectively integrate their respective capabilities and perspectives.

Notwithstanding the numerous strengths and achievements of EFPB, the following recommendations are made in the spirit of continual improvement.

Recommendation 1: Ensure adequate attention to longer-term planning activities, including capacity development and knowledge transfer.

Recommendation 2: Take steps to improve internal communication and coordination on policy analysis and advice to more effectively integrate the current and medium- to long-term economic analysis roles and perspectives.

Recommendation 3: Consider taking steps to ensure that adequate consultations with other relevant stakeholders are taking place on important analytical issues, within the constraints of sensitivity of information and tight timelines.

1.0 Background

1.1 Introduction

This report presents the results of the Evaluation of the Economic and Fiscal Policy Branch (EFPB) of the Department of Finance Canada, which was conducted by Internal Audit and Evaluation (IAE) between April and October 2014. The evaluation covered approximately $7.2 million in annual direct program spending by the Department. The evaluation was listed in the 2014–19 Departmental Evaluation Plan, which was approved by the Deputy Minister of Finance on May 26, 2014.

1.2 Profile of the Economic and Fiscal Policy Branch

EFPB analyzes the economic and fiscal situation of the country and advises the Minister on the government’s fiscal framework, including revenues and expenditures. It also provides the Minister with analytical support on a wide range of economic and financial issues related to the government’s macroeconomic and structural policies and plays the lead role in the federal budget process. EFPB prepares the Department’s publicly distributed monthly fiscal reports, fiscal reference tables, and long-term economic and fiscal projections. It also closely monitors economic developments in Canada’s major trading partners and continuously tracks fiscal developments at all levels of government.

EFPB has three divisions. The Fiscal Policy Division(FPD) is responsible for the analysis and forecasting of the government’s revenues and expenses and plays a major role in the preparation of the Budget and the Update of Economic and Fiscal Projections. The Economic Studies and Policy Analysis Division (ESPAD) has a mandate to conduct policy-focused research to supply strategic advice to the Minister on economic and fiscal issues that are of medium- and long-term significance. The Economic Analysis and Forecasting Division(EAFD) tracks, analyzes and forecasts Canada’s economic performance, including analyzing risks surrounding economic forecasts.

Collectively, the activities of EFPB support effective management of the fiscal framework, and fiscal planning that is transparent and supports long-term fiscal stability; and the departmental Strategic Outcome of a strong economy and sound public finances for Canadians.

In 2013–14 EFPB had a staff of 68 full-time equivalents (FTEs) and operating costs of about $7.2 million.

Appendix B depicts a logic model for the evaluation of EFPB and provides further details on key activities, outputs, partners and stakeholders, and expected outcomes.

2.0 Evaluation Objectives and Scope

The objective of the evaluation was to assess the relevance and performance (effectiveness, efficiency and economy) of the key activities and operations of EFPB. The scope of the evaluation covered the last five years of operations, 2009–10 to 2013–14, with a focus on more recent operations.

The evaluation risk for EFPB was assessed as medium, given recent audits and an evaluation of important branch functions. At the same time, a branch-level evaluation had never been conducted. The evaluation relied on recent studies, including the Evaluation of the Economic Studies and Policy Analysis Division (IAE, 2012); the Audit of the Federal Budget Process (IAE, 2012); the Audit of Long-Term Fiscal Sustainability (Office of the Auditor General of Canada [OAG], 2012), and audits of Canada’s Economic Action Plan (OAG, 2010 and 2011). Data collection efforts were calibrated to build on existing studies and to focus on areas most likely to add value.

3.0 Methodology

3.1 Evaluation Approach

Evaluation of the policy function—the provision of policy research, analysis and advice—has become an integral part of the evaluations conducted by IAE. IAE has developed an approach that primarily examines the quality and usefulness of the organization’s policy-focused outputs and the appropriateness or adequacy of its processes and resources for producing and monitoring these outputs. The approach aims to assess whether key stakeholders (senior departmental and government officials) have timely access to high-quality policy-focused research, analysis and advice, using criteria developed by Schacter (2006) for the assessment of high-quality policy advice (Table 3, subsection 4.2.1). This approach is based on the premise that if policy work is relevant, useful and of high quality, then it is reasonable to suggest that it contributes to the achievement of departmental outcomes.

The evaluation approach also incorporates elements of organizational assessment inspired by the Balanced Scorecard methodology by assessing performance from the following four perspectives:

  • Client or stakeholder perspective, through the quality of EFPB’s policy analysis and advice and the effectiveness of collaboration with its key partners and stakeholders;
  • Internal processes and employee perspectives, via EFPB’s organizational capacity, structures, processes, and human and other resources in place; and
  • Financial perspective in terms of efficiency and economy over time in the usage of financial resources.

The overall methodology is based on the premise that for an organization to achieve its goals cost-effectively the needs of clients and stakeholders should be well understood and employees should have the required skills and resources to implement appropriate processes to meet client and stakeholder needs. Furthermore, all activities should be undertaken in a manner that reflects effective stewardship of the organization’s financial resources.

3.2 Evaluation Issues

The evaluation issues, the evaluation questions and the corresponding lines of evidence are presented in Table 1.

Table 1
Evaluation Issues and Lines of Evidence
"X" indicates that this line of evidence was used
Evaluation Issues Evaluation Questions Data Collection Methods

Document Review Interviews Case Study
Relevance (R)
Demonstrated need and alignment with government priorities, roles and responsibilities R1: To what extent are the objectives and activities of EFPB still relevant and in line with the priorities, roles and responsibilities of the Department and the federal government? X X  
Performance (P)
Design and implementation P1: To what extent do the organizational structure, work environment and internal processes support the effective execution of EFPB’s mandate and achievement of its objectives? Have these factors changed over time? X X  
Human and other resources P2: To what extent do human resources (HR) practices of EFPB effectively support the recruitment, development and retention of highly skilled staff? Does EFPB have appropriate tools and other resources to achieve its objectives? X X  
Achievement of expected outcomes P3: To what extent has EFPB achieved its objectives and expected outcomes, in particular with respect to providing high-quality policy analysis and advice? (Schacter’s criteria) X X X
Efficiency and economy P4: What steps or measures has EFPB taken during the past five years to improve its efficiency and economy? How have the trends in EFPB’s operational resources ($ and FTEs) compared with the trend of its outputs and activities? Are there ways to undertake key activities and deliver key outputs more economically and efficiently? What has been the impact of the Deficit Reduction Action Plan on its operations? X X  

3.3 Data Collection Methods

IAE used the following data collection methods:

3.3.1 Document review and analysis

Documents related to EFPB activities, outputs and objectives that were reviewed included previous audits, evaluations and reviews; EFPB planning documents, including divisional work plans and organizational charts; branch policy-related outputs such as annual Budgets and Updates of Economic and Fiscal Projections; historic financial and FTE data; results of the 2011 Public Service Employee Survey; and corporate planning and performance reporting documents such as Integrated Business Plans and Corporate Risk Profiles. The purpose of the document review was to understand the context in which EFPB operates and conducts its activities, to inform the development of data collection tools, and to support the assessment of elements of EFPB’s organizational performance.

3.3.2 Key informant interviews

Key informant interviews were a main source of information to assess EFPB performance. A total of 36 interviews (including for the case study) were conducted in roughly equal numbers of internal and external interviewees. Table 2 shows the number of interviews conducted by participant type.

Nineteen comprehensive internal interviews were conducted with EFPB management and staff, focusing on quality of policy analysis and advice, organizational capacity, and economy and efficiency of operations. A purposive sample was selected to include management of all divisions and a cross-section of staff from each division at various levels. Interviewees were also selected according to their years of experience in EFPB (minimum of two years).

Seventeen targeted external interviews were conducted with key EFPB stakeholders, including departmental senior management, executives in other branches, other government departments, and other non-government organizations. External interviews focused on effectiveness of collaboration and quality of policy analysis.

3.3.3 Case study of horizontal policy coordination

This case study examined in greater depth the structures and networks that EFPB has established to effectively deal with horizontal policy issues. The objective of the case study was to draw lessons and identify best practices to contribute to more effective horizontal policy coordination. Two EFPB coordination activities were examined: the coordination of the Organisation for Economic Co-operation and Development’s (OECD) Economic Survey of Canada, and the coordination of research within the Department of Finance Canada. The case study relied on a review of internal documentation related to these coordination processes and on interviews with EFPB staff and external stakeholders involved in these coordination processes and activities.

Table 2
Evaluation Interviews by Participant Type
Participant Type Interviews Horizontal Policy Coordination Case Study
Branch management 11 3
Branch staff 8 2
Departmental stakeholders 10 2
External stakeholders 7 2
Total 36 9

3.4 Limitations

There are inherent limitations to evaluating the policy advice function, which the approach and methodology described in subsection 3.1 sought to overcome. The following are additional limitations:

  • A trade-off between breadth and depth in assessing the relevance and performance of the EFPB as a whole. Use of a case study allowed for more in-depth analysis of EFPB activities where evaluation information could add value.
  • Because limited quantitative data were available, the evaluation relied heavily on qualitative data and analysis to address some evaluation issues. This limitation was mitigated by triangulating data collected from various methods and sources and using purposive sampling to target relevant key informants.

4.0 Evaluation Findings

The following section presents the main findings on relevance and performance (effectiveness, efficiency and economy) in accordance with the evaluation issues. Data from multiple lines of evidence were triangulated, whenever possible, to confirm the findings presented in this section and to formulate the conclusions and recommendations in section 5.0.

4.1 Relevance

This section examines the relevance of EFPB activities and their alignment with the priorities, roles and responsibilities of the federal government.

Key findings

EFPB’s activities and objectives are clearly aligned with the roles, responsibilities and priorities of the Government of Canada and the Department of Finance Canada. EFPB conducts research and analysis that is highly relevant to the policy decisions facing the government.

The Department of Finance Canada is the federal government’s central agency that has the mandate to provide advice on all aspects of Canada’s economic and financial affairs. EFPB’s role within the Department can be described as “at the centre of the centre” in supporting the implementation of government policy objectives in coordination with other central agencies, the Treasury Board of Canada Secretariat and the Privy Council Office. EFPB plays a lead role in core departmental functions such as the setting of fiscal policy and the development of the government’s fiscal framework, budget preparation and planning, preparation of economic outlooks and forecasts, and analysis of the economic impacts of major policy initiatives and options. Within the government, EFPB is a focal point for macroeconomic analysis, including the medium- to long-term challenges facing the economy. EFPB’s activities are clearly aligned with the roles, responsibilities and priorities of the federal government.

EFPB activities have been extremely relevant to current government priorities of promoting employment, economic growth, prosperity and an increased standard of living for Canadians, alongside managing a return to balanced budgets by 2015. EFPB has made key contributions to defining government policy initiatives, including the Economic Action Plan since 2009. EFPB activities have supported the government’s efforts to support jobs and growth and its commitment to fiscal responsibility, and contributed to the departmental Strategic Outcome of a strong economy and sound public finances for Canadians.

Departmental senior management has noted that the policy relevance of EFPB’s research and analytic work has improved over time, an important positive direction for EFPB and the Department as a whole. Research and analysis is well linked to the relevant policy decisions facing the Department, the government and the country.

4.2 Performance

This section examines EFPB’s performance in achieving its expected outcomes in support of the departmental Strategic Outcome of a strong economy and sound public finances for Canadians. It assesses effectiveness, efficiency and economy of EFPB’s activities and operations, drawing on the Balanced Scorecard methodology. This includes assessing the quality of EFPB’s policy analysis and advice and the effectiveness of collaboration with key partners and stakeholders (client or stakeholder perspective); its organizational capacity, including internal resources, structures and process (internal processes and employee perspectives), and the extent to which EFPB operates efficiently and economically (financial perspective).

Key findings

EFPB is a high-performing branch—effective, efficient and economical—having solid leadership, highly skilled and experienced staff, and effective processes to support its key activities.

The quality of EFPB’s policy research analysis and advice is consistently high and is noted for its timeliness, relevance and soundness of evidence. Opportunities might exist, within the constraints faced by EFPB, to continue to improve consultations, communications and coordination related to the development of policy analysis and advice.

Meeting a high level of demand in a context of continued resource pressures poses some long-term capacity risks for EFPB. These risks relate to ensuring adequate attention to capacity development and to knowledge transfer initiatives.

4.2.1 Effectiveness

Quality of policy analysis and advice

A principal focus of the evaluation was to assess the quality of EFPB’s policy analysis and advice, in accordance with the ten criteria proposed by Schacter and presented in Table 3.

The evaluation found that the quality of EFPB’s policy research, analysis and advice was consistently high across its three divisions. There was a high level of consensus among internal and external informants interviewed on the overall high quality of its analysis and advice, as well as on its strengths and the main areas for improvement.

Table 3. Schacter’s Criteria for High-Quality Policy Advice[1]

1
Is it timely?
2
Is it based on adequate consultations?
3
Does it articulate a clear purpose?
4
Is it logical, differentiating facts from assumptions and linking conclusions to recommendations?
5
Is it based on sound evidence?
6
Is it balanced, and does it include multiple viewpoints?
7
Does it present viable options for action?
8
Is it relevant to the actual situation, and does it anticipate related developments?
9
Is it well organized and well presented to the reader?
10
Is it pragmatic, bearing in mind implementation issues?

EFPB’s policy analysis and advice was found to be strong on virtually all of the ten criteria. Particular strengths identified were analysis and advice that are timely and relevant and that are based on sound evidence. These characteristics were judged as reflecting steadfast leadership and management having set high standards for timeliness and analytic quality.

External stakeholders who regularly interact with EFPB were impressed with its ability to consistently provide relevant analysis and advice, quickly and with clarity, even on short notice. They appreciated EFPB’s strong technical capacity and sound policy judgment, as well as the professionalism of its staff. It was further noted that EFPB provides usable information and analysis with clear explanations that facilitate its use in policy advice on a range of issues, and advice that can be adapted to the circumstances. Overall, there was a high level of trust and confidence in EFPB among its main stakeholders and departmental senior management.

Central agency stakeholders interviewed expressed an interest in having EFPB’s analysis and advice on important policy issues for the government sooner, to increase their lead time for analysis and use of these inputs. At the same time stakeholders acknowledged the constraints faced by EFPB in sharing information—including secrecy and sensitivity of information, and time frames that EFPB itself must respect—but which are well communicated to them by EFPB. Given these constraints, stakeholders appreciated efforts to share draft documents earlier, where feasible. Overall, collaboration with other central agencies was found to be effective.

Internal interviewees noted a trade-off between achieving timeliness and the depth of analysis, including ensuring the soundness of evidence and the ability to consult adequately. It was acknowledged that EFPB is often required to provide analysis and advice within short time frames, and that this requirement poses risks, particularly for complex issues that could benefit from more time to develop high-quality advice.

With regards to balance, EFPB was generally considered to do a good job of providing unbiased analysis and advice, based on facts (data-driven) and sound evidence.

EFPB was found to have formal, well-established and effective processes in place for consultations and coordination related to its principal budget-related activities. These activities include coordinating the annual development of the federal Budget with relevant stakeholders within the Department and regular consultations with private sector forecasters on the macroeconomic forecast and risks to the economy to develop its fiscal projections and updates. Through its central role in the annual budget process, EFPB has established a broad network of contacts to support regular interactions and effective consultations, particularly with policy branches that produce a high volume of budget notes, including the Economic Development and Corporate Finance Branch, the Federal-Provincial Relations and Social Policy Branch, and Tax Policy Branch.

Ensuring adequate consultations with other relevant stakeholders on important economic policy issues for the Department and the Government of Canada was identified as the principal area on which to focus efforts for continual improvement. This includes reaching out to, and consulting with, other departments, economic institutions (for example, think tanks) and the Canadian business sector, to take these other viewpoints into account. It was recognized by internal and external stakeholders that EFPB faces constraints in sharing information owing to tight timelines and sensitivity of information (budget secrecy). However, EFPB could benefit from more outreach and consultations within these constraints—for example, in the early stages of policy analysis and development. Internal interviews indicated some of the other barriers to effective outreach and consultations include limited formal processes for consulting outside the Department and difficulties in identifying relevant external contacts. The former was linked in part to limited ability to present externally analytic work that is relevant to current policy decisions, given that it is not advisable to present work that may flag potential future policy changes. But more fundamentally, the importance and expectations for consultations (including what information can be shared externally) may not always be clear at the working level. In practice the level of consultations depends to some extent on the initiative of the individuals involved. Establishing clear processes and expectations, including direction on when and how to consult externally, within existing constraints could help promote more and better consultations in developing analysis. At the same time, it was noted by EFPB senior management that for risk-management purposes, constraints related to budget secrecy and sensitivity of information may limit the Department’s ability to allow staff, especially more junior staff, to consult directly with external stakeholders.

According to branch senior management, since the onset of the financial crisis in 2008, EFPB has increasingly been tasked to lead and contribute to analysis on policy issues that intersect with the mandates of other policy branches and other federal departments. It was recognized that EFPB was working more collaboratively with other policy branches and other departments in recent years, in such areas as the monitoring of risks to the global economy and international economic coordination, housing markets, labour markets, manufacturing and infrastructure. Departmental senior management noted that EFPB has improved its ability to conduct applied research and analysis in concert with other policy branches. Given the increasingly multidisciplinary nature of policy work, it is important that EFPB continue to foster these opportunities.

The case study on horizontal policy coordination, discussed below, further explores some lessons and practices to continue to increase consultations and coordination of analysis and advice on horizontal policy issues.

Achievement of expected outcomes

The evaluation evidence indicates that EFPB contributes effectively to the achievement of the departmental Strategic Outcome of a strong economy and sound public finances for Canadians. EFPB effectively provides the Minister of Finance and senior government officials with timely access to relevant information and evidence-based analysis and research on matters that impact the government’s economic and fiscal agenda. Effective coordination of the annual federal budget process is an example of how EFPB contributes to effective management of the fiscal framework, and fiscal planning that is transparent and supports long-term fiscal sustainability. The case study below further examines some of EFPB’s contributions to expected outcomes for the Department.

Horizontal policy coordination

Through a case study on horizontal policy coordination, the evaluation examined the structures and networks EFPB had in place to effectively deal with horizontal policy issues. The case study looked at two specific policy-related coordination activities led by EFPB: the Government of Canada’s participation in the Organisation for Economic Co-operation and Development’s (OECD’s) Economic Survey of Canada, and the coordination of research within the Department. Details of the case study findings are presented in Appendix C.

ESPAD has lead responsibility for coordinating the Government of Canada’s participation in the biennial OECD Economic Survey of Canada. ESPAD’s coordination of this process was found to be efficient and effective. ESPAD completed the necessary tasks on time, despite tight deadlines, and provided effective support to the OECD, ensuring that selected topics were well developed. Expectations were well communicated to Canadian stakeholders. ESPAD was found to have the appropriate set of skills and knowledge to effectively analyze, filter and prioritize feedback to the OECD. The coordination of this exercise could be further standardized by establishing more formal guidelines on how to contact other departments, and by systematically updating the list of steps involved in the process and a list of people contacted in previous years. In addition, providing more context to stakeholders on the survey could generate better input from stakeholders.

Regarding research coordination, ESPAD has circulated its annual research work plan since 2012 to obtain input from senior management across the Department. ESPAD has also been proactive in reaching out to other branches and economic analysis groups in other federal departments through various channels, including email, workshops and in-person meetings; however, limited input has been received. The reasons for this are not entirely clear; stakeholders may be satisfied with the work plan and have limited input to provide, or it may represent an opportunity to improve awareness of possible synergies between ESPAD’s research and work in the policy branches and other government research groups. Greater use and more frequent in-person meetings with other directors and directors general seeking more structured input may yield better external input to ESPAD’s research plan.

Based on the findings of the case study and the information collected through general interviews on the effectiveness of EFPB’s collaboration with its main partners and stakeholders, a summary of lessons learned and effective practices is provided below. These practices aim to support EFPB in continuing to increase consultations and coordination of policy work with other policy branches and other government departments.

Practices to Promote Consultations, Horizontal Coordination and Information Sharing

Consultations

  • Establish clear expectations and provide guidance and protocols for targeted consultations with relevant external stakeholders in developing policy analysis and advice (within the constraints of budget secrecy and sensitivity of information).
  • Adapt the presentation of analytic work to enable sharing with external stakeholders and consulting on non-sensitive aspects of policy analysis and advice.
  • Provide sufficient context (within the constraints of timeliness and sensitivity of information) to external stakeholders when seeking input to obtain more relevant input.

Horizontal Coordination and Information Sharing

  • Seek out and engage early with relevant internal and external stakeholders in the development of policy analysis and advice to avoid potential duplication and to better integrate different perspectives.
  • Bring together the main internal stakeholders at the planning stage to clarify objectives and responsibilities for developing policy analysis and advice.
  • Promote regular communications and information sharing, internally across divisions and with important stakeholders in other policy branches.
  • Conduct in-person meetings for selected consultation and coordination activities.
  • Share draft documents whenever possible, clearly communicating important limitations, to promote better information sharing.

Organizational capacity

Another important element of the evaluation approach to assess EFPB’s performance relates to the appropriateness of its organizational structure and internal processes and to the level of its financial and human resources.

Organizational structure and processes

EFPB’s organizational structure was generally found to be appropriate, with clear and distinct roles and responsibilities among its divisions and sections. EFPB’s organizational structure and mandates have evolved appropriately over the period covered by the evaluation to meet changing demands. These changes include consolidating groups conducting research and analysis on medium- to long-term policy issues within ESPAD and greater integration of ESPAD within the branch and the Department through more demand-driven work, to promote more policy-relevant research and analysis.

Although the mandates and responsibilities for each of the three divisions and their sections were generally found to be clear, internal interviews indicated that in practice roles and responsibilities are fluid, particularly among the current and medium- to long-term economic analysis roles. The mandates of ESPAD and EAFD are less “black and white” compared with the past. Although the broad mandates and responsibilities of each division for routine activities are clear, ESPAD and EAFD may on occasion work on the same policy issues. The evaluation did not find evidence of duplication— in fact, having the two groups contribute to policy analysis and advice from different perspectives was viewed as positive. However, similar mandates also make responsibilities for analytic work on some files less clear. Based on the information collected through internal interviews, more top-down coordination and regular communication and information sharing between the divisions (at all levels) could promote more effective integration of policy perspectives and the capabilities of the current and medium- to long-term economic analysis roles. Indeed, there is evidence that branch senior managers are well aware of the issue and have taken steps to promote more effective coordination of mandates and work. Efforts to improve information sharing that were identified by EFPB in response to the 2011 Public Service Employee Survey (PSES) data remain relevant in this context. Circulating a weekly list of memos to the Deputy Minister’s Office was perceived as a good practice to increase awareness of the work of others, as was considering the scope for cross-divisional collaboration when initiating new projects. Also, analysts from different divisions could be attending the same meetings with the goal of consolidating viewpoints, presenting integrated analysis and conveying a single coherent message. These practices are equally applicable for collaboration across policy branches.

EFPB was found to have effective practices for internal review and quality control. It undertakes regular analysis of its fiscal projections against actual outcomes to understand and learn from past performance, and it completes annual budget post-mortems to make practical improvements to the budget coordination process on an ongoing basis. Review of policy analysis through the internal hierarchy is the main process used to ensure the quality of policy research, analysis and advice. This process was found to be effective in obtaining relevant and useful feedback and perspectives at various stages of development. Obtaining early input from relevant internal stakeholders and establishing clear objectives for analytic work could help reduce the number of iterations required to finalize work. It was also noted that limited external review of policy analysis and advice poses some risks for quality, particularly on more complex policy issues. As discussed previously, ongoing efforts to improve external consultations and outreach, taking into account time and external information-sharing constraints, would help mitigate such risks.

Resources

Most EFPB staff members interviewed indicated that they had adequate access to the tools and data they needed to perform their jobs. IT constraints in recent years were cited as the most significant barrier. However, it was noted that these issues are gradually being addressed in collaboration with Shared Services Canada and as part of the relocation of the Department of Finance Canada to a new building.

EFPB has been successful in recruiting skilled staff and retaining experienced staff. According to internal interviewees, EFPB offers relevant and interesting work for economists, including certain applied economic and fiscal policy work that is uniquely available within the branch. Skilled staff is effectively recruited through the Department’s university recruitment program. Interviews and a review of branch organizational charts over the past five years indicate that EFPB has been successful in integrating new staff and developing and retaining key expertise. The majority of EFPB’s management (executive) staff has worked in the branch for over five years, and about half of the Economics and Social Science Services (EC) staff has been with EFPB for at least five years. These overall findings are consistent with the 2011 PSES data, which indicated that 90 per cent of EFPB staff believed that people are hired who can do the job. In addition, compared with the rest of the Department, more EFPB staff members believed that they had opportunities for promotion within the Department, and fewer intended to leave their jobs within the next two years.

Ensuring appropriate training and development of staff was identified as an ongoing challenge for EFPB, based on both 2011 PSES data and on internal interviews. Although adequate funds are available, identifying meaningful training opportunities is difficult for both staff and supervisors. On-the-job training was widely viewed as the most important means of developing technical and management skills of staff. Mentorship of new employees and development of analysts were identified as areas where EFPB could do better. This finding was linked in large part with EFPB having to do more with less in recent years.

Internal interviews indicated that since the onset of the financial crisis in 2008, EFPB has had increased demands for analysis and advice. For example, there have been significant demands for economic analysis and advice related to fiscal planning and management of the fiscal framework associated with the Economic Action Plan, which brought about significant changes to the fiscal context and the return to balanced budgets. At the same time, EFPB has operated under sustained resource pressures over the last five years, as illustrated in subsection 4.2.2, “Efficiency and Economy” (Figures 1 and 2). These pressures pose some long-term capacity risks for EFPB. Internally, EFPB has been viewed as more limited in its ability to invest in planning activities.

Internal interviews indicated that developing the capacity of staff through exposure to different tasks is a challenge in an environment where high-quality work must be delivered within short timelines. To achieve timeliness and high quality, new files are more likely to be assigned to the most experienced or competent staff member. This reliance on the more experienced staff rather than exposing someone else to a new file limits knowledge transfer among staff and increases risks related to succession planning.

Several branch managers and staff also noted that there has been less time for anticipatory analytic work, including model development (re-evaluating and re-estimating key models and tools), ongoing monitoring of, and anticipatory thinking on, emerging economic issues, and in-depth analysis of medium- to long-term economic policy issues. These types of activities were viewed as important for building and maintaining analytic capacity and expert knowledge, and hence long-term productivity and the ability to supply relevant research and analysis. It was noted by some internal interviewees that meeting increased demands with fewer resources has led to a more “generalist” capacity within EFPB. This development was viewed as positive and important from the perspective of staff being more flexible to fulfill different roles as needed. However, it was also noted that less-defined roles limit the depth of expertise and knowledge behind analysis and advice. In summary, there are concerns about sustainability and the long-term impacts on quality of continuing to operate with stretched resources.

Based on the information collected through the evaluation, a summary of practices to promote long-term capacity development and knowledge transfer is provided below, for EFPB to continue and consider augmenting where possible.

Practices to Promote Capacity Development and Knowledge Transfer

Capacity Development

  • Establish goals and targets for developing organizational and staff capacity, and include knowledge transfer in work plans, performance agreements and learning plans.
  • Proactively manage training and conference participation to align the development of staff with organizational needs.
  • Assign work with the additional objective of training and building capacity of staff, when time constraints permit.
  • Encourage on-the-job training opportunities through mentoring, job shadowing, coaching, and peer-to-peer and team training.
  • Allocate time and resources in work plans to the ongoing monitoring and analysis of medium- to long-term economic issues, and the review and development of tools and models. Monitor the outputs and the achievement of work plan objectives.

Knowledge Transfer

  • Identify, as early as possible, staff members who intend to leave their jobs, to promote a smooth transition.
  • Expose staff members to different tasks, databases and models, to diversify their experience and increase their capacity to replace colleagues.
  • Assign responsibilities for the management of key documents and files, to avoid losing important knowledge when staff members leave.
  • Ensure that knowledge gained through participation in external training and conferences is widely shared and accessible within EFPB.
  • Encourage the attendance of working-level staff members at senior management meetings where issues relevant to their work are discussed, to promote a better understanding of context and senior management expectations.
  • Encourage staff to work with other divisions and branches on projects to promote skills development and the sharing of best practices, as well as intra- and inter-branch collaboration.

4.2.2 Efficiency and economy

This section examines the efficiency and economy of the operations of EFPB. These issues were addressed using interviews, as well as the analysis of personnel and non-personnel expenditures and FTE information obtained from the Corporate Services Branch of the Department. As noted previously, according to senior management and staff, the demands on resources continued to increase over the period covered by this evaluation. Nevertheless, EFPB managed to produce high-quality and timely policy analysis and advice, despite often being pushed to the limits of its capacity in responding to various demands. This finding is supported by the 2011 PSES data, which indicated that only 29 per cent of EFPB staff always completed or almost always completed the assigned work during regular working hours. At the same time, almost 90 per cent of EFPB staff was willing to put in the extra effort to get the job done.

Figure 1. EFPB Financial Data and Total FTEs, 2007–08 to 2013–14 Figure 1. EFPB Financial Data and Total FTEs, 2007–08 to 2013–14 - For details, please see following paragraph.

Figures 1 and 2 depict EFPB financial and FTE resources over the past seven fiscal years. Because total expenses and actual FTEs were unusually high in 2009–10, the average expenses and FTE use over the previous two-year period, 2007–08 to 2008–09, were used as the baseline for comparison. As depicted in Figures 1 and 2, FTE use decreased by 7.8 per cent between 2007–08 and 2013–14[2] (EAFD, 21 per cent; ESPAD, 7 per cent; and Fiscal Policy Division (FPD), 2 per cent), whereas total expenditures remained relatively unchanged (although non-personnel expenditures decreased by 58 per cent). Actual expenditures by EFPB averaged $226,000 per year less than budgeted during the period. Examples of measures taken by EFPB to reduce its operational costs included a reduction in travel for face-to-face meetings and increased use of teleconferencing. Travel costs averaged $159,000 from 2007–08 to 2009–10, compared with $57,000 from 2010–11 to 2013–14.

Figure 2. FTEs Per EFPB Division, 2007–08 to 2013–14 Figure 2. FTEs Per EFPB Division, 2007–08 to 2013–14

Management and staff have implemented a series of measures aimed at enhancing the efficiency and economy of branch operations. For example, EFPB obtains more free data (for example, Statistics Canada CANSIM [Canadian Socio-Economic Information Management System] data) and shares the cost of licences among its divisions and with other branches and organizations.

EFPB senior management noted that the Department may want to examine whether the cost of maintaining databases is appropriately shared across branches. Further opportunities may exist to improve information management and to make better use of existing tools—for example, by being able to search past briefing notes or to generate tables and reports using commands in the Excel database.

The budget process post-mortems and the feedback provided by budget coordinators allowed EFPB to make significant productivity gains related to budget coordination by implementing a series of new initiatives. For example, these processes led to the implementation of the Budget boot camp, which allowed new budget coordinators and other branches to become familiar with the budget process and learn about new tools, templates and processes. EFPB centralized its administrative resources in November 2012 and reduced the number of administrative FTEs. At the time of data collection for the evaluation, views were mixed about the operational efficiencies gained from the centralization. Through buy-in and commitment from management and administrative staff, centralization was expected to deliver efficiencies—for example, through specialization of tasks and better workload distribution. Subsequent discussion with EFPB’s Assistant Deputy Minister indicates that these efficiencies have been achieved.

5.0 Conclusions and Recommendations

EFPB is a highly relevant, effective, efficient and economical branch.

The quality of its policy and analysis and advice is consistently high and is noted for its timeliness, relevance and soundness of evidence. The branch has effective working relationships and collaborates with its main partners and stakeholders, who have a high level of confidence and trust in the branch.

EFPB has solid leadership, highly skilled and experienced staff, and effective and efficient processes to support its key activities. Despite operating with reduced resources over the past five years, EFPB has successfully met a high level of demand, maintained high-quality and timely outputs, and improved its internal processes, including coordination of the annual federal budget process.

Key risks relate to EFPB’s ability to sustain its capacity to deliver the highest quality policy analysis and advice, in the context of continued resource pressures. EFPB has been more limited in its ability to invest in longer-term planning activities, including anticipatory analytic work, the development of staff, and knowledge transfer. These limitations increase the risks to long-term productivity and succession planning. Some opportunities also exist to further improve consultations, communications and coordination related to the development of policy analysis and advice.

Appendix D summarizes the analysis of EFPB’s main strengths, limitations, opportunities and risks, based on the evidence collected though this evaluation.

While recognizing the numerous strengths and achievements of EFPB, the following recommendations are made in the spirit of continual improvement:

Recommendation 1: Ensure adequate attention to longer-term planning activities, including capacity development and knowledge transfer.

Recommendation 2: Take steps to improve internal communication and coordination on policy analysis and advice to more effectively integrate the current and medium- to long-term economic analysis roles and perspectives.

Recommendation 3: Consider taking steps to ensure that adequate consultations with other relevant stakeholders are taking place on important analytical issues, within the constraints of sensitivity of information and tight timelines.

Appendix A: Management Response and Action Plan

Management Response and Action Plan
Recommendations Management Response Planned Action Lead Target Date
Recommendation 1:

Ensure adequate attention to longer-term planning activities, including capacity development and knowledge transfer.
Management agrees with this recommendation. Directors will ensure that divisions’ work plans include the development of new analytical tools and a database and that more than one person knows how to use the new and existing tools. Directors, with support of the General Director (GD) and the Assistant Deputy Minister (ADM) September 30, 2015
Recommendation 2:

Take steps to improve internal communication and coordination on policy analysis and advice to more effectively integrate the current and medium- to long-term economic analysis roles and perspectives.
Management agrees with this recommendation. Management has already taken steps to ensure better cross-divisional communication and integration of work between Economic Studies and Policy Analysis Division (ESPAD) and Economic Analysis and Forecasting Division (EAFD). Directors’ Performance Management Agreements indicate that they must regularly meet and consult to leverage the relative strengths of Divisions and to avoid duplication. Meetings take place every two weeks. Directors of ESPAD and EAFD Completed (Ongoing)
Recommendation 3:

Consider taking steps to ensure that adequate consultations with other relevant stakeholders are taking place on important analytical issues, within the constraints of sensitivity of information and tight timelines.
Management agrees with this recommendation but notes that there are risks to be weighed when considering greater external consultations (e.g., secret and sensitive information and tight timelines). Branch management committee will weigh the risks associated with greater external consultations, and if judged manageable, will develop a protocol and guidelines for ensuring adequate external consultations, in consultation with staff and the Deputy Minister. Branch management committee (ADM, GD, Directors and Senior Chiefs) December 31, 2015

Appendix B: Economic and Fiscal Policy Branch Logic Model

Appendix B: Economic and Fiscal Policy Branch Logic Model

[Appendix B: Economic and Fiscal Policy Branch Logic Model - text version]

Appendix C: Case Study on Horizontal Policy Coordination

This case study assessed the extent to which the Economic and Fiscal Policy Branch (EFPB) has in place the necessary structures and networks to effectively deal with horizontal policy issues. Two branch coordination activities were selected for in-depth review: the coordination of the Organisation for Economic Co-operation and Development’s (OECD’s) Economic Survey of Canada, and the coordination of research within the Department of Finance Canada.

OECD Economic Survey of Canada

The Economic Studies and Policy Division (ESPAD) has lead responsibility for coordinating the Government of Canada’s participation in the biennial OECD Economic Survey of Canada. The coordination process involves participating in discussions on potential topics, identifying Canadian stakeholders to provide input into the study, preparing for various meetings between the OECD Canadian delegation and Canadian stakeholders,[3] distributing the survey questionnaire to the Canadian stakeholders, collecting and consolidating comments made by Canadian stakeholders on the OECD’s findings, and ensuring federal government stakeholders and senior officials are well informed of the process.

ESPAD’s coordination of this process was found to be efficient and effective. ESPAD completed the necessary tasks on time, despite tight deadlines associated with the different steps of the process, successfully supported the OECD in identifying relevant subject-matter expertise within the federal government, and ensured that selected topics were well developed. Expectations were well communicated to Canadian stakeholders, including the communication of deadlines and the need to use track changes to share comments and to incorporate a rationale for each change suggested to the report. The topics addressed in the economic studies (microeconomic and macroeconomic issues such as the labour market, housing market and fiscal sustainability) require a well-diversified set of skills and knowledge to effectively analyze, filter and prioritize feedback to the OECD. ESPAD was found to have the appropriate skills set and knowledge to effectively perform this important coordination function. The OECD appreciated ESPAD’s professionalism and efficiency in consolidating and delivering input from Canadian stakeholders in a suitable time frame and format.

The coordination of this exercise could be further standardized by establishing guidelines on how to contact other departments, keeping a list of steps involved in the process, and maintaining a list of contacts. These guidelines would reduce the time needed to contact representatives from other departments and would facilitate the transfer of responsibility for this coordinating exercise within ESPAD. In addition, providing more context to stakeholders about the survey (for example, the names of the other stakeholders providing input on the same issues) could generate better input from stakeholders.

Research Coordination

The 2012 ESPAD evaluation recommended that ESPAD should establish a systematic process for annual sharing and gathering input from senior management across the Department into ESPAD’s annual research work plan. The objective was to further promote policy-relevant research that meets departmental needs. ESPAD has circulated the annual research work plan and has been proactive in reaching out to other branches and departments since 2012. However, limited input has been received thus far. The reasons for this are not entirely clear; stakeholders may simply be satisfied with the work plan and have limited input to provide, or it could indicate an opportunity to improve awareness of possible synergies between ESPAD’s research and work in the policy branches and other government research groups.

To the extent that branch and Departmental senior management wish to actively pursue more research and analysis projects in concert with the needs of other policy branches, there may be some opportunities to enhance the annual research plan consultation process. Organizing in-person meetings with other directors to seek more structured input may also yield better external input to ESPAD’s research plan. The evaluation also noted that there is no formal mechanism for disseminating research findings widely within the Department (or externally). However, evidence suggests that the research findings are effectively communicated to known relevant stakeholders and are reflected in policy advice to the Minister.

Appendix D: Analysis of Strengths, Limitations, Opportunities and Risks

The following table provides a summary of the relative strengths and limitations of the Economic and Fiscal Policy Branch, potential opportunities to further improve its performance, and important risks to its ability to continue to provide the highest-quality analysis and advice.

Strenghts
  • Solid and steadfast leadership that has established high standards for quality.
  • Ability to attract talent (offers unique and relevant work for economists).
  • Highly regarded by external stakeholders for its expertise on economic and fiscal policy issues and for the quality of analysis and advice.
  • Formal and well-institutionalized processes for consultations and coordination on core budget-related activities.
Limitations
  • Constraints on information sharing, consultations and external review of analysis and advice owing to budget secrecy and short timelines.

 

 

 

 

Opportunities
  • Institutionalizing information sharing and collaboration on horizontal economic policy issues outside the budget coordination process, both internally (Economic Studies and Policy Division and Economic and Analysis and Forecasting Division—Recommendation 2) and with key departmental partners.
  • Establishing norms and practices to promote more effective external consultations. (Recommendation 3)
Risks
  • High level of demand and limited resources pose long-term capacity risks (Recommendation 1):
    • Maintaining analytic capacity and development of staff; and
    • Knowledge transfer and succession planning, including reliance on key individuals.
  • Limitations on external review of analysis and advice.

 

1 Mark Schacter, “ ‘The Worth of a Garden’: Performance Measurement and Policy Advice in the Public Service – A Discussion Paper.” (Commissioned by the Treasury Board of Canada Secretariat, 2006), p. 7.

2 A group of six FTEs moved from FPD to ESPAD toward the end of 2013–14, but this did not impact FTE data for that fiscal year.

3 Canadian stakeholders include the Bank of Canada, other central agencies and various federal government line departments and agencies, depending on the topics covered.