use of a project management system on new drug development by James M. Yun
INTRODUCTION The development of new drugs, from basic research to targeted N D A submission, requires long-term commitments of resources and facilities. Starting in 1975, Hoechst-Roussel Pharmaceuticals Inc. initiated a new product planning system that utilized a time-sharing program referred to as the Cyphernetics Project Management System (CPMS). The objective of this system was to expedite completion of these long and costly projects. With this computerized system, a project planning section was established to coordinate planning with the various departments having general functional responsibility. The coordinating function also included dealing with individual researchers and management. While management was interested in project monitoring and control, the scientific and technical personnel were concerned with committing themselves to relatively fixed schedules. To make planning productive in this environment, all potential users were involved at the initiation of the project and each developmental stage was submitted for their approval. The CPMS system was developed to meet our specific needs, which included special report formats, frequency of updates, and a proper means of disseminating information. In addition, new product planning chaos were designed which could be drawn by 'COMPLOT from the CPMS system. These charts depict graphically the development of the project from start to new drug approval, indicating all the activities, their duration.
Hoechst- Roussei Pharmcnuuticals, Inc., Somewilk. NJ QB876. Mr. Yun's present addmss is Director oJRescorch Planning Administration Atrdue Frderick Co.. 50 Wcrrhington St.. Nomvlk. CT 06856.
interactions, major milestones and a time scale, grouped by department. At present we have completed inputting our high priority projects into the system and have experienced favorable acceptance by our users.
SYSTEM DEVELOPMENT In 1974, a survey of eight major pharmaceutical companies was made to determine the extent of use of PERT/CPM methods in new product planning. In addition, we were interested in the size of the group that administered the planning and the location of this group in the organization. A study of software packages which would serve Hotchst-Roussel's New Product Planning needs was included with the survey. The findings showed that two of the companies had been using a computerized PERT/ CPM since the late sixties, while three others used handdrawn charts and the rest were in an investigative stage. This survey also indicated that for effective use of a computerized project planning system i n a multidisciplined environment, the following attributes were required:
-
The system must be simple and flexible.
-
The system must offer positive benefits for the users.
-
The system must be cost effective in performance.
The report should be of minimal size, easy to read and comprehend, and cover all essential information.
DRUG INFORMATION JOURNAL
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
October/Dccembcr 76 p l l l
-
It must be easy to install and update.
2. CRITICAL PATH REPORT - Indentifies:
In addition, the general network concept was evaluated so that the user could master it without much training.
a.
All activities from start to finish, department, duration, early start, early finish and final target date;
In current planning methods there are distinctive differences among PERT (Project Evaluation and Review Technique), CPM (Critical Path Method) and MPM (Methods des Potentiels Metra): CPM is activity oriented; PERT is mostly event oriented; MPM is activity oriented but requires only one potential duration estimate. MPM was chosen for its simplicity for user input, since pessimistic, optimistic and most likely duration estimates used in classical CPM were found t o be unnecessarily cumbersome. Use of the MPM estimate of duration was recognized as the most practical for long term projects, as additional time estimates are meaningless. After a careful systems feasibility study, it was decided to go with a commercial time-sharing system* rather than in-house development. The development of an in-house system, even without graphic capabilities, would take at least two t o three years by a n experienced programmer/ systems analyst, with no guarantee that the system would be effective. The commercial system offered basic criteria which could be revised for pharmaceutical new product planning needs.
b.
The department which is responsible for the activities;
c.
Total duration of the critical activities;
d.
Successor activities.
3. MULTIPROJECT DEPARTMENTAL REPORT a.
All the multiproject activities are sorted by an early start date.
b.
Slack or float date is indicated fos rescheduling.
c.
Critical activities are indicated for special attention.
d.
Identifies the early start date and finish date on these activities.
4. MANAGEMENT
EXCEPTION
REPORT
Reports and Formats
Notes:
Some major modifica'tions in the various reports and report formats were made to meet the needs of pharmaceutical organizations for original CPMS packages. The basic working documents of the CPMS consist of major reports such as Status Report (PSSR), Critical Path Report, Multiproject Departmental Report, Management Exception Report and New Product Planning Chart.
a.
Critical path and noncritical activities that are delayed or finished early
b.
Problem areas which require attention from management.
1. STATUS REPORT - This report is a communication medium between users and Project Planning; it includes:
-
5. NEW PRODUCT PLANNING CHART A total graphic illustration of a project from start to finish, in which: a.
All the critical paths are indicated;
b.
The length of the band indicates duration, early start, and early finish date;
a.
Activities in process
b.
Activities completed
C.
All interrelated activities are shown;
C.
Activities due to start
d.
All the departmental activities are grouped into pertinent horizontal columns;
d.
Activities due to finish e.
e.
Indication of early start and early finish
Major events/ milestone shown as flags;
f.
Duration of each activity
f.
Weekly, monthly and annual scales are shown at the top of chart.
g.
Status and comment.
Cyphernetirs Corporation. 175 Jackson Ploza, Ann Arbor, MI 48106.
activities
are
The computer generates the status report on a monthly basis, covering two months' activities. For each.activity, the report includes the early start date, finish date, duration and status comment. The users revise this information and return the report to Project Planning by the end of the month for use in the next update. All
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
reports are reviewed at project committee meetings or, in the absence of a meeting, by the project chairman. The information update is done using a DASI 300 terminal. Direct access to a terminal eliminates keypunching and excessive coding by clerical or professional personnel. All information is keyed in at the terminal with immediate corrections in logic if required, and the printout is verified before final processing. The density of space and other changes in format can be controlled through this terminal before the final printout. During initial system development, frequent changes in activities and logic had to be made as requested by the users, but after these initial changes the update procedure included only the status changes, which do not require extensive recalculation of the critical path. After updating, the new monthly report is generated for dissemination. Quarterly, new product planning charts are generated from which blue print copies are made for distribution. These charts average 36 X 48 inches in size and serve to give a total picture of the project. A particular effort is made to keep the volume of the reports to a minimum. Management and each department receives only the reports related to its responsibility. PLANNING PHARMACEUTICAL RESEARCH
The new product planning system is a useful guide, when properly administered. It requires training of both users and management in order to maximize its benefits. As mentioned by Faust et aL.1 the managers of pharmaceutical firms differ in their views on planning: "some feel that unfettered scientificeffort patterned after academia will prove most rewarding in the long term. Others, however, prefer a more controlled operation, convinced that planning data, often highly quantified and sophisticated, will aid decision-making and insure greater productivity and output." It is true in pharmaceutical organizations that research personnel in scientific disciplines have difficulty with fixed schedule planning which requires "black and white" decisions in areas predominantly "gray." Management, on the other hand, must have a means of precipitating decisions within reasonable time periods. The planner's responsibility is to provide balance in this variable environment for the benefit of both users and management. Once a compound shows promising efficacy and safety and a viable IND and NDA submissions are anticipated, project planning not only is an aid in decision making and time control, but offers corresponding cost control. Robert B. Helm estimated the cost to develop a single product to be $7.5 million.2 If we consider all the compounds which have failed, the cost increased up to $24.4 million dollars, with an after tax cost of $12.2 million. Clymer estimated the discovery period to be three to five years, and eight years for development.' This was recognized by Hoechst-Roussel's management when they decided to institute the project planning system. To assure effective administration and favorable acceptance by the users, Project Planning followed a step
by step introduction: an initial survey stage as described before, a pilot stage (experimental use with three pilot projects), and finally the full scale input. PROJECT IMPLEMENTATION
Hoechst-Roussel's Management Committee instituted an ad hoc New Product Planning Committee to develop the new product planning system to suit our needs. The committee consisted of representatives from marketing, medical, m a n u f a c t u r i n g , q u a l i t y a s s u r a n c e , pharmacology, chemistry, pharmaceutical development, toxicology and drug regulatory affairs. During the pilot stage, three compounds which were unique as to the nature of the study, development or origin were developed. These three samples showed some indication of the durations, complications, and interrelationships in such projects. Durations were estimated by each respective department head, some on the basis of previous experience and others by educated guesses from avaiable data. At each stage, we met with the New Product Planning Committee, reviewed the development and obtained approval for further steps. During this pilot stage, we developed some printout formats and observed carefully the users' response on the update procedure. After five months with these pilot trials, Management approved the pilot program and authorized Project Planning to input additional projects. By the end of the year, all major projects had been processed and the projects were placed under the control of the project committees. During the developmental stage, some of the intrinsic benefits of PERT planning were demonstrated. These included parallel planning of activities, which is the performance of related functions simultaneously. In comparison to conventional serial planning, a substantial time saving on the overall target can be realized. For example, interim statistical analysis of case reports can be scheduled along with clinical trials and the data can be processed steadily, avoiding the last minute rush of accumulated data. By the time the clinical trials are completed, all that remains to be done is the summarizing of the total data. Substantial time can be saved if the project chairman looks carefully at which activities are in the critical path, and how improvements can be made by shifting and starting activities. Needless to say, Management has to support these decisions, but the data are available for their consideration. To expedite the project, the project chairman and management examine the critical path activities to determine whether additional resource allocations may reduce their *duration. With the Multiproject Departmental Report, the department director will identify the number of projects his department is handling per period, and the critical activities which must be given higher priority. With this report, he can also plan manpower allocation for both long-term and short-term needs. The new product planning charts help both the project chairman and departmental directors. The chart shows all the activities for the compound, interactions and critical path activities on one sheet. The user can identify
DRUG INFORMATION JOURNAL
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
Octobcr/December 76 pi13
the current status and possible problem areas. The departmental directors identify their specific activities interacting with other related activities to expedite the project. As noted above, Project Planning updates the status report monthly, using the status updates sent by the users. The report, issued at the beginning of the month, includes the activities on the specific project (per department) over a two-three month interval. The user checks the report, makes any necessary changes, and retusns it by the end of the month. If all the information is correct and on schedule, he returns the report as is, with his signature. Project Planning reviews all the revised P S S R s and adds only corrections of newly started or finished activities to the status file. The project is updated with the new information, and new Status Reports (PSSR), Departmental and Management Exception Reports are generated at the beginning of the month.
The objective in establishing a new product planning system lies in integrating communication, assuring milestones and targets, and defining the interrelationship of activities. Careful observation indicates that these goals have been met and the system offers individual departments a more realistic appraisal of present problems and opportunities. Better judgements concerning current operations and long-term projections are easier to make. With clear milestones, surprises can be avoided with sufficient attention by the pertinent departments. The planner’s responsibilities lie in balancing the creative and innovative spirit against administrative demands and planning overkill.
If the project committee meeting is scheduled, the project chairman refers to the PSSR as part of the meeting agenda, and discusses items scheduled or in process. After discussion of each department’s activities the PSSRs are collected at the meeting for updating. T h e Management C o m m i t t e e reviews t h e Management Exception Report and takes necessary measures on the activities with serious problems, or activities which require major decisions for further progress.
Faust, E. F., and Ackerman, G.: Program/project management a t Hoffmann-LaRoche, Res. Manage., pp. 3842, 1972.
DISCUSSION AND CONCLUSION
BIBLIOGRAPHY
Helms, R. B.,ed.: Drug Development and Marketing. The American Enterprise Institute for Public Policy Research, Washington, D.C. 1975, pp. 67. Clymer, H.:The changing cost and risks of pharmaceutical innovation, pp. 115-116. In: The Economics of Drug Innovation, Washington D.C., American University, 1970.
At present, Project Planning at Hoechst-Roussel is coordinating a dozen projects with this system. We are also rescheduling some of the projects to be adaptable to our research environment and to meet the corporate goals. Continuous training for the user is essential for understanding the application of the system to current and long-term planning. User acceptance in our multidisciplined environment is favorable, evidenced by requests for subnetwork planning as a means of expediting research procedures. This indicates recognition of the benefits gained through their experience. Although we are unable, at present, to make precise quantitative measurement of system performance, there are significant advantages derived from it. Through the status update, each department head can schedule the immediate task within the period to meet deadlines. With the Multiproject Departmental Report, long-range activity requirements on a specific project or multiprojects can be forecasted. This will help forecast manpower and resource requirements to accomplish the specific project. Among the hidden benefits will be integration of communication through the interactions shown on the new product planing charts. On some of the projects a substantial amount of time was saved for NDA targets, e.g., by re-scheduling longterm chronic toxicity studies to start at an earlier date. This of course, requires a management decision, and the system provides the essential data for their decision making.
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
INTRODUCTION The development of new drugs, from basic research to targeted N D A submission, requires long-term commitments of resources and facilities. Starting in 1975, Hoechst-Roussel Pharmaceuticals Inc. initiated a new product planning system that utilized a time-sharing program referred to as the Cyphernetics Project Management System (CPMS). The objective of this system was to expedite completion of these long and costly projects. With this computerized system, a project planning section was established to coordinate planning with the various departments having general functional responsibility. The coordinating function also included dealing with individual researchers and management. While management was interested in project monitoring and control, the scientific and technical personnel were concerned with committing themselves to relatively fixed schedules. To make planning productive in this environment, all potential users were involved at the initiation of the project and each developmental stage was submitted for their approval. The CPMS system was developed to meet our specific needs, which included special report formats, frequency of updates, and a proper means of disseminating information. In addition, new product planning chaos were designed which could be drawn by 'COMPLOT from the CPMS system. These charts depict graphically the development of the project from start to new drug approval, indicating all the activities, their duration.
Hoechst- Roussei Pharmcnuuticals, Inc., Somewilk. NJ QB876. Mr. Yun's present addmss is Director oJRescorch Planning Administration Atrdue Frderick Co.. 50 Wcrrhington St.. Nomvlk. CT 06856.
interactions, major milestones and a time scale, grouped by department. At present we have completed inputting our high priority projects into the system and have experienced favorable acceptance by our users.
SYSTEM DEVELOPMENT In 1974, a survey of eight major pharmaceutical companies was made to determine the extent of use of PERT/CPM methods in new product planning. In addition, we were interested in the size of the group that administered the planning and the location of this group in the organization. A study of software packages which would serve Hotchst-Roussel's New Product Planning needs was included with the survey. The findings showed that two of the companies had been using a computerized PERT/ CPM since the late sixties, while three others used handdrawn charts and the rest were in an investigative stage. This survey also indicated that for effective use of a computerized project planning system i n a multidisciplined environment, the following attributes were required:
-
The system must be simple and flexible.
-
The system must offer positive benefits for the users.
-
The system must be cost effective in performance.
The report should be of minimal size, easy to read and comprehend, and cover all essential information.
DRUG INFORMATION JOURNAL
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
October/Dccembcr 76 p l l l
-
It must be easy to install and update.
2. CRITICAL PATH REPORT - Indentifies:
In addition, the general network concept was evaluated so that the user could master it without much training.
a.
All activities from start to finish, department, duration, early start, early finish and final target date;
In current planning methods there are distinctive differences among PERT (Project Evaluation and Review Technique), CPM (Critical Path Method) and MPM (Methods des Potentiels Metra): CPM is activity oriented; PERT is mostly event oriented; MPM is activity oriented but requires only one potential duration estimate. MPM was chosen for its simplicity for user input, since pessimistic, optimistic and most likely duration estimates used in classical CPM were found t o be unnecessarily cumbersome. Use of the MPM estimate of duration was recognized as the most practical for long term projects, as additional time estimates are meaningless. After a careful systems feasibility study, it was decided to go with a commercial time-sharing system* rather than in-house development. The development of an in-house system, even without graphic capabilities, would take at least two t o three years by a n experienced programmer/ systems analyst, with no guarantee that the system would be effective. The commercial system offered basic criteria which could be revised for pharmaceutical new product planning needs.
b.
The department which is responsible for the activities;
c.
Total duration of the critical activities;
d.
Successor activities.
3. MULTIPROJECT DEPARTMENTAL REPORT a.
All the multiproject activities are sorted by an early start date.
b.
Slack or float date is indicated fos rescheduling.
c.
Critical activities are indicated for special attention.
d.
Identifies the early start date and finish date on these activities.
4. MANAGEMENT
EXCEPTION
REPORT
Reports and Formats
Notes:
Some major modifica'tions in the various reports and report formats were made to meet the needs of pharmaceutical organizations for original CPMS packages. The basic working documents of the CPMS consist of major reports such as Status Report (PSSR), Critical Path Report, Multiproject Departmental Report, Management Exception Report and New Product Planning Chart.
a.
Critical path and noncritical activities that are delayed or finished early
b.
Problem areas which require attention from management.
1. STATUS REPORT - This report is a communication medium between users and Project Planning; it includes:
-
5. NEW PRODUCT PLANNING CHART A total graphic illustration of a project from start to finish, in which: a.
All the critical paths are indicated;
b.
The length of the band indicates duration, early start, and early finish date;
a.
Activities in process
b.
Activities completed
C.
All interrelated activities are shown;
C.
Activities due to start
d.
All the departmental activities are grouped into pertinent horizontal columns;
d.
Activities due to finish e.
e.
Indication of early start and early finish
Major events/ milestone shown as flags;
f.
Duration of each activity
f.
Weekly, monthly and annual scales are shown at the top of chart.
g.
Status and comment.
Cyphernetirs Corporation. 175 Jackson Ploza, Ann Arbor, MI 48106.
activities
are
The computer generates the status report on a monthly basis, covering two months' activities. For each.activity, the report includes the early start date, finish date, duration and status comment. The users revise this information and return the report to Project Planning by the end of the month for use in the next update. All
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
reports are reviewed at project committee meetings or, in the absence of a meeting, by the project chairman. The information update is done using a DASI 300 terminal. Direct access to a terminal eliminates keypunching and excessive coding by clerical or professional personnel. All information is keyed in at the terminal with immediate corrections in logic if required, and the printout is verified before final processing. The density of space and other changes in format can be controlled through this terminal before the final printout. During initial system development, frequent changes in activities and logic had to be made as requested by the users, but after these initial changes the update procedure included only the status changes, which do not require extensive recalculation of the critical path. After updating, the new monthly report is generated for dissemination. Quarterly, new product planning charts are generated from which blue print copies are made for distribution. These charts average 36 X 48 inches in size and serve to give a total picture of the project. A particular effort is made to keep the volume of the reports to a minimum. Management and each department receives only the reports related to its responsibility. PLANNING PHARMACEUTICAL RESEARCH
The new product planning system is a useful guide, when properly administered. It requires training of both users and management in order to maximize its benefits. As mentioned by Faust et aL.1 the managers of pharmaceutical firms differ in their views on planning: "some feel that unfettered scientificeffort patterned after academia will prove most rewarding in the long term. Others, however, prefer a more controlled operation, convinced that planning data, often highly quantified and sophisticated, will aid decision-making and insure greater productivity and output." It is true in pharmaceutical organizations that research personnel in scientific disciplines have difficulty with fixed schedule planning which requires "black and white" decisions in areas predominantly "gray." Management, on the other hand, must have a means of precipitating decisions within reasonable time periods. The planner's responsibility is to provide balance in this variable environment for the benefit of both users and management. Once a compound shows promising efficacy and safety and a viable IND and NDA submissions are anticipated, project planning not only is an aid in decision making and time control, but offers corresponding cost control. Robert B. Helm estimated the cost to develop a single product to be $7.5 million.2 If we consider all the compounds which have failed, the cost increased up to $24.4 million dollars, with an after tax cost of $12.2 million. Clymer estimated the discovery period to be three to five years, and eight years for development.' This was recognized by Hoechst-Roussel's management when they decided to institute the project planning system. To assure effective administration and favorable acceptance by the users, Project Planning followed a step
by step introduction: an initial survey stage as described before, a pilot stage (experimental use with three pilot projects), and finally the full scale input. PROJECT IMPLEMENTATION
Hoechst-Roussel's Management Committee instituted an ad hoc New Product Planning Committee to develop the new product planning system to suit our needs. The committee consisted of representatives from marketing, medical, m a n u f a c t u r i n g , q u a l i t y a s s u r a n c e , pharmacology, chemistry, pharmaceutical development, toxicology and drug regulatory affairs. During the pilot stage, three compounds which were unique as to the nature of the study, development or origin were developed. These three samples showed some indication of the durations, complications, and interrelationships in such projects. Durations were estimated by each respective department head, some on the basis of previous experience and others by educated guesses from avaiable data. At each stage, we met with the New Product Planning Committee, reviewed the development and obtained approval for further steps. During this pilot stage, we developed some printout formats and observed carefully the users' response on the update procedure. After five months with these pilot trials, Management approved the pilot program and authorized Project Planning to input additional projects. By the end of the year, all major projects had been processed and the projects were placed under the control of the project committees. During the developmental stage, some of the intrinsic benefits of PERT planning were demonstrated. These included parallel planning of activities, which is the performance of related functions simultaneously. In comparison to conventional serial planning, a substantial time saving on the overall target can be realized. For example, interim statistical analysis of case reports can be scheduled along with clinical trials and the data can be processed steadily, avoiding the last minute rush of accumulated data. By the time the clinical trials are completed, all that remains to be done is the summarizing of the total data. Substantial time can be saved if the project chairman looks carefully at which activities are in the critical path, and how improvements can be made by shifting and starting activities. Needless to say, Management has to support these decisions, but the data are available for their consideration. To expedite the project, the project chairman and management examine the critical path activities to determine whether additional resource allocations may reduce their *duration. With the Multiproject Departmental Report, the department director will identify the number of projects his department is handling per period, and the critical activities which must be given higher priority. With this report, he can also plan manpower allocation for both long-term and short-term needs. The new product planning charts help both the project chairman and departmental directors. The chart shows all the activities for the compound, interactions and critical path activities on one sheet. The user can identify
DRUG INFORMATION JOURNAL
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
Octobcr/December 76 pi13
the current status and possible problem areas. The departmental directors identify their specific activities interacting with other related activities to expedite the project. As noted above, Project Planning updates the status report monthly, using the status updates sent by the users. The report, issued at the beginning of the month, includes the activities on the specific project (per department) over a two-three month interval. The user checks the report, makes any necessary changes, and retusns it by the end of the month. If all the information is correct and on schedule, he returns the report as is, with his signature. Project Planning reviews all the revised P S S R s and adds only corrections of newly started or finished activities to the status file. The project is updated with the new information, and new Status Reports (PSSR), Departmental and Management Exception Reports are generated at the beginning of the month.
The objective in establishing a new product planning system lies in integrating communication, assuring milestones and targets, and defining the interrelationship of activities. Careful observation indicates that these goals have been met and the system offers individual departments a more realistic appraisal of present problems and opportunities. Better judgements concerning current operations and long-term projections are easier to make. With clear milestones, surprises can be avoided with sufficient attention by the pertinent departments. The planner’s responsibilities lie in balancing the creative and innovative spirit against administrative demands and planning overkill.
If the project committee meeting is scheduled, the project chairman refers to the PSSR as part of the meeting agenda, and discusses items scheduled or in process. After discussion of each department’s activities the PSSRs are collected at the meeting for updating. T h e Management C o m m i t t e e reviews t h e Management Exception Report and takes necessary measures on the activities with serious problems, or activities which require major decisions for further progress.
Faust, E. F., and Ackerman, G.: Program/project management a t Hoffmann-LaRoche, Res. Manage., pp. 3842, 1972.
DISCUSSION AND CONCLUSION
BIBLIOGRAPHY
Helms, R. B.,ed.: Drug Development and Marketing. The American Enterprise Institute for Public Policy Research, Washington, D.C. 1975, pp. 67. Clymer, H.:The changing cost and risks of pharmaceutical innovation, pp. 115-116. In: The Economics of Drug Innovation, Washington D.C., American University, 1970.
At present, Project Planning at Hoechst-Roussel is coordinating a dozen projects with this system. We are also rescheduling some of the projects to be adaptable to our research environment and to meet the corporate goals. Continuous training for the user is essential for understanding the application of the system to current and long-term planning. User acceptance in our multidisciplined environment is favorable, evidenced by requests for subnetwork planning as a means of expediting research procedures. This indicates recognition of the benefits gained through their experience. Although we are unable, at present, to make precise quantitative measurement of system performance, there are significant advantages derived from it. Through the status update, each department head can schedule the immediate task within the period to meet deadlines. With the Multiproject Departmental Report, long-range activity requirements on a specific project or multiprojects can be forecasted. This will help forecast manpower and resource requirements to accomplish the specific project. Among the hidden benefits will be integration of communication through the interactions shown on the new product planing charts. On some of the projects a substantial amount of time was saved for NDA targets, e.g., by re-scheduling longterm chronic toxicity studies to start at an earlier date. This of course, requires a management decision, and the system provides the essential data for their decision making.
Downloaded from dij.sagepub.com at East Tennessee State University on June 3, 2015
