Pharmaceutical discovery as a complex system of decisions:

The pharmaceutical discovery process

Before exploring the predictive performance of experimentation strategies in pharmaceutical discovery, the process needs to be understood. Figure 1 provides an overview of a typical drug discovery process as found in a major R&D operation[1]. Once a biological target is validated during “Target Identification & Validation,” the pharmaceutical drug discovery process aims to find a therapeutic agent with positive effect on this scientifically and commercially interesting target. It proceeds in essentially two stages: screening and optimization.

First, during “High Throughput Screening” (HTS) a lead molecule is selected from a diverse compound collection, constituting a chemical library. A typical major drug company will have hundreds of thousands to millions of compounds in its collection, typically valued at $50—140 million or more (Young et al., 1997). Alternatively, combinatorial chemistry has come into recent use to create large collections of candidate compounds screened for their effect on the biological target (Thomke et al., 1998). The candidate compound or “Hit” coming out of this screening process will be modestly potent. Therefore, in a second stage the candidate compound will be optimized by synthetically adding or removing parts using medicinal chemistry. This proceeds in two steps: lab-based—in-vitro—characterization or Hit-to-Lead (H2L), and animal-based—in-vivo—Lead Optimization (LO). This results in a candidate compound becoming more and more complex in structure as a result of the discovery process. Essentially, “Hit” structures serve as initial starting points to be optimized into “drug-like” leads (Oprea et al., 2001). The final product is a new molecular entity (NME) to be transferred into clinical development for further testing in humans.

The practitioner literature (Macdonald & Smith, 2001; Oprea, 2002; DeWitte, 2002) develops a scientific viewpoint on a number of experimental designs available to run this screening and optimization drug discovery process. It can be summarized with regard to the level of front loading and parallelization that is used in the discovery process (Figure 1), the latter indicating the shape of the discovery funnel.

The level of parallelism can be measured by the number of candidate classes[2] retained for a drug target during the various stages of the discovery process, which reflects the shape of the discovery funnel. Typically in PharmaCo one to five classes showing activity against a target are selected in HTS screening, and one to three classes are further analyzed in H2L and carried through to LO. A narrow funnel, then, would be characterized by the couple (HTS, H2L/LO) = (1, 1). Conversely, a broad funnel would be characterized by the couple (HTS, H2L/LO) = (5, 3). The main cost driver is the number of classes carried through to LO.

The level of front loading can be measured by the number of variables—P and/or B[3]—considered at differing stages during Discovery; the most front-loaded strategy uses P and B from HTS on, the least front-loaded strategy only uses P during the entire process, disregarding B.

Finally, exploratory case-study analysis showed that along the discovery process scientists use surrogate measures for the variable they are ultimately interested in.

This is because scientific methods or surrogate markers for an ultimate variable of interest become more accurate as one proceeds in the discovery phases, which explains the funnel shape above. As an example, bio-availability in humans—b in Figure 2—is first measured in HTS using a rough decision rule set, to get an initial idea about this critical compound property based on virtual bioinformatics data only. Then, in H2L a more refined in-vitro method is used, and the final decision in LO is made using animal tests.

Now, for this study we define the predictive power of this surrogate marker chain, given by the correlation of the successive measurement methods. The higher the correlation between the various methods in the chain, the tighter the chain, and the better its predictive power will be. Within PharmaCo, a tightness level of 70% for all variables studied can be assumed.

Discovery experimentation strategies

The three front-loaded experimental strategies for discovery research considered for subsequent comparative analysis are summarized in Figure 3. The arrows indicate which property classes are used in the prototype compound optimization cycles to converge to an NME. A vertical arrow indicates that only biological activity (P) or potency and selectivity is optimized, an arrow along the diagonal indicates that both biological activity and drug likeness or bioavailability properties (B) are used in multi-factorial optimization cycles.

In the old paradigm, applied in the industry some time ago, discovery research was only concerned about biological activity or potency and selectivity of a candidate compound. Drug-likeness properties were only taken into account from pre-clinical research on, which is a time- and effort-consuming process (Oprea, 2002). This strategy was not entirely successful, since within the industry only one in ten candidate drugs succeeded through clinical trials to reach the market (Drews, 1998).

In the front-loaded paradigm, both biological activity and pharmacokinetic properties are optimized simultaneously from the Hit-to-Lead (H2L) stage on. Here, confirmed hits series from HTS are prioritized and analyzed for problems. Progressive increments in biological activity and bio-availability are obtained by addressing the appropriate molecular determinants that define the desired compound characteristics.

Also, this paradigm is more generous in promoting selected candidate compounds through the H2L and LO process. This argues against a too narrow idea funnel and means that more candidates will reach LO status. Although this is an expensive recommendation, it does prevent candidate eliminations due to tests that mimic reality with statistical inaccuracy, the latter compounded to the number of tests conducted (DeWitte, 2002). The idea, then, is to build up knowledge through to in-vivo of a series of candidates instead of going for a fast attrition based on “in-silico” or in-vitro tests with low predictive power, and running LO with a small subset of “winning” candidate series of compounds.

Finally, early front loading takes the front-loaded paradigm even further, selecting the most promising classes based on the fullest multi-factorial picture available using virtual “in-silico” screening technology during HTS (Oprea, 2002).

Generalizing, our theory-building effort leads us to define residual ambiguity as all non-characterized solution variables of a candidate solution concept, their causal relationships and/or related problem-solving mechanisms at a point in time during the emergence of the solution to an innovation problem. From the above, it will be clear that front-loaded experimentation strategies aim for more extensive candidate characterization or lower levels of residual ambiguity than old-paradigm strategies at the end of discovery before transferring the therapeutic candidate into clinical development.

Now, to quantitatively evaluate predictive performance of discovery experimentation strategies, an analytical framework needs to be developed. This is the subject of the next section.

Predictive performance

Predictive performance can be intuitively defined as the accuracy with which compounds are promoted to therapeutic candidate or NME[4] status, as confirmed by later clinical development. To build a quantitative model of experimentation strategy variables—levels of front loading and parallelism- influencing predictive performance—a more formal notation is required.

Therefore, we distinguish between H⁺ and H^—, being numbers of compounds declared respectively as active or inactive by the discovery experimentation strategy. The former is a set of therapeutic candidates that tested positive throughout the optimization process. During this process, a - virtually infinite - number of candidates were eliminated from further optimization.

Then, active candidates declared NMEs by the discovery experimentation strategy will subsequently be confirmed to be positive or negative in clinical development. We need then to distinguish between C⁺ and C^— as the number of NMEs respectively passing the latter development phases or not.

To evaluate the predictive performance of a discovery experimentation strategy, the following question must be answered: “What is the probability that a candidate is really active, denoted as ?” To answer this question, we must start from the universe of potential therapeutic agents H and examine how the experimentation strategy improves the odds of finding an active compound. This representation of uncertainty about parameters using conditional probabilities is called Bayesian inference. It models the experimentation strategy as a learning process that modifies one’s initial probability statement about the prevalence p or p (C⁺) prior to observing the data during experimentation to updated or posterior knowledge incorporating both prior knowledge and the data at hand (Congdon, 2001).

The positive predictive value, then, denoted p (C⁺|H⁺) or p⁺, is read as the probability that a compound will actually pass clinical development p (C⁺) given that it has been declared active p (H⁺) by the experimentation strategy. Similarly, the probability that a compound will not pass clinical development p (C^-), given it has been declared inactive p (H^-) by the experimentation strategy, is called the negative predictive value, and denoted p (C^—|H^-). The fraction of active compounds that were mistakenly declared inactive by the experimentation strategy is called p^— or p (C⁺|H^-); the negative predictive value is denoted as 1—p^— (Parmigiani, 2002). Hence, predictive performance of an experimentation strategy can be measured using two criteria ?:

An experimentation strategy featuring high positive and negative predictive performance, then, has a high value for p⁺ and a low p^-. High values for p⁺ indicate high levels of clinical confirmation of the earlier discovery decision. Low levels of p— indicate a low level of missed opportunities for commercialization[5]. The transition from p to p⁺ and p^- models the learning about the true status of the universe of potential compounds H. Using the Bayesian logic set out above, it quantifies how inferences about the universe of potential compounds H are updated in the light of new evidence, provided by the experimentation strategy.

Business performance

To formally state the problem of an R&D manager having to choose between various experimentation strategies, the optimal business decision for a strategy d^* can be modelled as follows (Müller, 1999: 99; Parmigiani, 2002):

where

U(d) is the expected utility of an experimentation strategy d, an element of the universe of possible experimentation strategies D. The utility function u(d,?,y) is in our case specified by solving a decision tree of the outcomes y of the various (H, C) combinations. Ordering these combinations in a decision tree, and based on cost assumptions for each branch of the tree, a financial outcome can be calculated and used for comparison to make the best decision d^*.

Figure 4 gives a decision tree representation of the best choice d^* (depicted as a rectangle) to be made between the three discovery experimentation strategies, building on p⁺ and p^- derived above. Overall, the depicted tree[6] represents the two-step decision to take a therapeutic agent to market.

The best choice at the decision node is the experimentation strategy option corresponding to the branch that leads to the maximum value (see a.o. Jensen, 2001). Solving a decision tree in Figure 4 leads to the following utility or business value U(d) for an experimentation strategy d:

Calculating U(d), then, leads to the best business choice between various experimentation strategies; the strategy delivering the maximum business value will be preferred.

Monte Carlo simulation study rationale

Various reasons led us to conclude that computer-based simulation is the preferred research methodology to study the impact of various experimentation strategies on predictive and business performance using the Bayesian framework developed above.

First, the dynamics of problem-solving behavior are not analytically tractable while they have to be represented using discontinuous nonlinear systems, which are generally hard to describe in closed form (Mihm et al., 2003).

Second, some of the variables like or their derived probabilities like p(C^-|H⁺) or p(C⁺|H^-) among others are unobservable in real life, thereby resisting real-life experiments and empirical research (Masuch & Lapotin, 1989).

And finally, a Monte Carlo simulation-based research methodology allows us to vary underlying assumptions and to search virtually for optimal experimentation approaches in very high-dimensional spaces without disrupting practice (Parmigiani, 2002).

Generating a virtual search space

To run a simulation study, a virtual cohort of candidate therapeutic agents or chemical compounds needs to be generated and used to calculate the probability density functions cited above. More specifically, we chose to model a virtual compound having three fundamental virtual properties: Potency (P), Bioavailability (B), and Toxicity (T).

Now, as depicted in Figure 5, the extant search space is the total number of virtual compounds it contains. It is generated in several steps. First, classes get assigned a value for (P), (B), and (T) through random sampling of the exponential distributions we defined for these properties. Then, around these class values, reference compounds are randomly generated using a normal distribution. This is done while, in reality, classes are investigated by scientists in HTS and H2L by using a set of reference compounds describing the classes. Finally, virtual compounds are randomly generated around reference compounds using a normal distribution.

The simulation experiment, then, uses this virtual search space as input to the discovery screening and optimization experimentation process. Before analyzing predictive and business performance of the three experimentation strategies—old paradigm, front-loaded (FL) paradigm, Early front loading (EFL), we need to explain how the experimentation process was emulated in the simulation.

Adaptive systems emulation of the discovery process

Using a simulation methodology, a formal model of the discovery research process needs to be designed. Among garbage can models (Cohen et al., 1972) and physical symbol systems (Newell & Simon, 1976; Masuch & Lapotin, 1989) of organizational decision making, massively parallel connectionist (Rumelhart et al., 1986) and evolutionary computation models like adaptive systems (Holland, 1992, 1998; Mitchell, 2001), we argue that the latter adaptive system (Holland, 1992) is the best artificial reconstruction of the mental modeling process during pharmaceutical discovery research since it mimics best the “selection and optimization” search for an NME in a vast, discontinuous multi-factorial solution space.

According to Mitchell (2001), this adaptive search method 1) initially generates a set of candidate solutions; 2) evaluates the candidate solutions according to some fitness criteria; 3) decides on the basis of this evaluation which candidates will be kept and which will be discarded; and 4) produces further variants by using some kind of operators on the surviving candidates.

Therefore, an annotated[7] adaptive systems model served as the theoretical paradigm underlying the simulation study, in which each experimentation strategy used in a pharmaceutical discovery context is modeled to execute an adaptive plan.

Referring to Figure 6 and using Holland’s (1992) notation, this adaptive plan t starts from the universe of potential compounds representing the extant search space and initial uncertainty about the environment. Only an extremely small fraction of this universe is composed of candidate compounds with therapeutic effect. Through the execution of successive “noisy” selection and optimization loops, the total set of chemical structures attainable at moment t, ?(t), is modified by a set of operators O(t) based on information I(t) received by t at time t. This way, the adaptive plan improves the fit of the solution with the target.

The screening and optimization process on a validated target starts in HTS at the level of the chemical classes to end at candidate compound level. The top-performing classes and compounds are selected using payoff information on two or more properties. I(t) varies depending on the experimentation strategy used; or only biological activity information is used to find the best performer; or a multi-objective function combining payoff information on both biological activity and bio-availability dimensions is used to find the best-performing compounds in the search space. All experimentation strategies are modeled to only deal with chemical class-related performance in HTS and H2L, and with chemical compound-related performance in LO. Both front-loaded and early front-loaded strategies use multi-objective optimization, but start doing so at different moments in the process. The chosen multi-objective function conservatively takes the minimum of both (P) and (B) detector values as the calculated performance value for the chemical class or candidate compound, depending on the discovery research phase. At all stages a compound’s toxicology profile (T) needs a minimum level to pass.

The conceptual model specified above formed the basis to build a computer simulation program in Excel Visual Basic Applications (VBA). Theory building results using simulation-based experiments will be discussed in the following paragraphs.

Simulation model behavior

All experiments were executed within a Monte Carlo design. Two-way ANOVA screening experiments (n=100) showed that the simulation method used only provides meaningful results within an operating window determined by 1) the ruggedness of the solution landscape; and 2) the number of compounds selected in LO; (p). As to ruggedness, to get meaningful results standard deviations were not allowed to move beyond SD=1. Also, when (p) was allowed to move below a certain threshold, results again became meaningless. Therefore, simulation experiments were conducted at p=15 out of 24,000 initial virtual compounds leading to a p(H⁺) = 0.000625. This was the lowest possible value, which fits an industry average of about five compounds being advanced to clinical development out of 10,000 synthesized compounds (Furness, 2003).

Then, selected simulation runs were repeated 200 times to obtain distributions of performance variables. One-way ANOVA was used on these performance data series to test for statistical significance of differences in predictive and business performance of experimentation strategies. Results of these Monte Carlo simulation experiments indicate that the level of front loading and the number of parallel solution concept explorations used significantly influence predictive and business performance.

In the following, outputs of the simulation runs are discussed in greater detail.

Influence of the level of front loading on predictive and business performance

Simulation experiments probing for the influence of the level of front loading on predictive and business performance were conducted assuming a (HTS, H2L) = (5, 2) funnel shape and a surrogate marker chain tightness of 70%. Both conditions reflect respectively best current operational reality and scientific constraints.

(1) Predictive performance. Figure 7 indicates that positive predictive performance of old paradigm, front-loading, and early front loading differ significantly [F(2, 596)=13,79; p<.01], although the actual differences in mean scores are quite small. Also, this result needs to be moderated by the fact that the effect size, calculated using eta squared, was 0.044, indicating a small to medium impact of the independent variables on the outcome variable. Levene’s test showed non-homogeneity of variance across the three groups—which we consider to be normal considering the different experimentation strategy types—implying the need for studying differences between groups at the p<.01 level of significance. Both Tukey’s HSD and Scheffe’s post-hoc comparisons indicated that old paradigm (M=0.974; SD=0.05) is significantly outperformed by front-loaded paradigm (M=0.992; SD=0.026), and early front loading (M=0.993; SD=0.036) at the p<.01 level of significance for positive predictive performance. Front-loaded paradigm and early front loading did not differ significantly. Also, negative predictive performance did not differ significantly across discovery experimentation strategies.

(2) Business performance. Financial assumptions used to perform this analysis include the following: product revenue R was assumed to build up linearly to one billion dollars a year in a period of seven years, a conservative estimation of a typical industry average blockbuster product (Duyck, 2003). C_D and C_Clin amount to $20 and $400 million dollar respectively. These numbers represent typical project costs excluding the contribution for attrition, which is usually taken into account when specifying development costs (Kennedy, 1997; Duyck, 2003).

There was a statistically significant difference at the p<.01 level for the different experimentation strategies. Referring to Figure 8, old paradigm, front-loaded paradigm and early front loading differed significantly [F(2, 596)=12.9; p<.01] on business performance. This result needs to be moderated by the fact that the effect size, calculated using eta squared, was 0.041, indicating a small to medium impact of the independent variables on the outcome variable. Levene’s test showed non-homogeneity of variance across the three groups—which I consider to be normal considering the different experimentation strategy types—implying the need for studying differences between groups at the p<.01 level of significance.

Both Tukey’s HSD and Scheffe’s post-hoc comparisons indicated that old paradigm (M=2241.9; SD=137.8) is significantly outperformed by front-loaded paradigm (M=2287.3; SD=78.2), and early front loading (M=2290; SD=94) at the p<.01 level of significance for business performance. Front-loading and early front loading did not differ significantly, although the difference does amount to $2.7 million in favor of early front loading using the above-mentioned assumptions.

Influence of discovery exploration funnel on predictive and business performance

Simulation experiments probing for the influence of varying shapes of the discovery exploration funnel on predictive and business performance of early front-loaded experimentation strategies were conducted assuming a surrogate marker chain tightness of 70%, which reflects best current scientific constraints.

(1) Predictive performance. Referring to Figure 9, positive [F(3, 769)=23,84; p<.01] and negative [F(3, 796)=21.93; p<.01] predictive performance for the various front-loaded funnel shaping strategies differed significantly, although the actual differences in mean scores are quite small. Effect size, calculated using eta squared, was 0.084 and 0.077 for positive and negative predictive performance respectively, indicating a medium impact of the independent variables on the outcome variables. Levene’s test showed non-homogeneity of variance across the three groups—which I consider to be normal considering the different experimentation strategy types—implying the need for studying differences between groups at the p<.01 level of significance.

Both Tukey’s HSD and Scheffe’s post-hoc comparisons for positive predictive performance indicated that a (1; 1) and a (5; 1) funnel shaping strategy both differ significantly from a (5; 2) and (5; 3) strategy at the p<.01 level of significance. A (5; 2) and a (5; 3) strategy do not differ significantly from each other.

Both Tukey’s HSD and Scheffe’s post-hoc comparisons for negative predictive performance indicated that a (1; 1) and a (5; 1) funnel shaping strategy both differ significantly from a (5; 2) strategy and (5; 3) strategy at the p<.01 level of significance. Also, (5; 2) and (5; 3) strategies differ significantly from each other at the p<.01 level of significance.

(2) Business performance. A one-way between-group analysis of variance was conducted to probe for the impact of various front-loaded funnel shapes on the business performance of the experimentation strategy used. There was a statistically significant difference at the p<.01 level for the different funnel shaping strategies. Financial assumptions used to perform ANOVA were discussed above.

Referring to Figure 10, business performance [F(3, 769)=7.9; p<.01] for the various front-loaded funnel shaping strategies differed significantly. This result needs to be moderated by the fact that the effect size, calculated using eta squared, was 0.031, indicating a small to medium impact of the independent variables on the outcome variable. Levene’s test showed homogeneity of variance across the three groups. Still, differences between groups were studied at the p<.01 level of significance.

Both Tukey’s HSD and Scheffe’s post-hoc comparisons for business performance indicated that a (5; 1) funnel shaping strategy (M=2157.5, SD=75.6) outperforms a (1; 1) strategy (M=2134.8, SD=99.1) by $22M and both a (5; 2) strategy (M=2120.9, SD=91.5) and (5; 3) strategy (M=2121.3, SD=79.3) by $36M, at the p<.01 level of significance. A (1; 1) strategy does not differ significantly from the others and a (5; 2) and (5; 3) strategy do not differ significantly from each other.

Validity considerations and limitations

Computer simulations are without doubt reliable, but the important question is whether a specific simulation model is an acceptable representation of the corresponding real system, given the goal of the simulation model (Kleijnen et al., 2001). Models for simulation purposes cannot be shown to be true or valid in any absolute sense. What can be said is that the model can be valid under certain assumptions. In this study, key assumptions were formulated to reduce the complexity of representing the innovation process without endangering the fulfilment of the research goal of theory development. In summary, three simplifying assumptions were made concerning the representation of the solution landscape, and for optimization and selection conducted during the innovation process.

First, the solution landscape was represented using three compound properties aggregated into reference compounds and chemical classes. This oversimplification of reality was necessary to make the implementation of the conceptual model possible in a VBA environment. Second, the innovation environment was represented as a mechanistic process of optimization stages concluded by a number of decision gates, where candidate solution concepts were promoted to the next stage or terminated if they did not fulfill the selection criteria. Respecting the garbage can philosophy, the complex scientist optimization behavior was conservatively reduced to a simple multi-factorial function, taking the minimum of the (P) and (B) values as an input to a search for maximum performance in an extant search space. White box validation (Pidd, 1992) of this process with PharmaCo scientists showed that the model behaves in a reasonable fashion, depicting a familiar universe of organizing the discovery research process, and confirmed the face validity of this complexity reduction of reality, provided it served the purpose of theory development.

However, considering the complexities of the probabilistic modelling of the innovation process and the hard-to-unravel nature of the model’s inner working, face validation was insufficient to claim internal validity of the simulation model. Therefore, validation ex negatio (Masuch & Lapotin, 1989) of key assumptions was done, showing that these assumptions do little harm to the model’s predictive power. This is why comparative performance conclusions about experimentation strategies were checked for robustness by varying the most basic parameters in the model. Thus, simulation results were shown to be robust for changes in solution landscape ruggedness up to a certain level, down to a certain level for varying numbers of the set of compounds declared active in LO, and for changes in the marginal probability p(H).

Finally, external validity must be gained through empirical observation of the model’s predictions. “The behavior of the ‘real’ system is observed under specified conditions and the model is then run under conditions which are as close as possible to these. If the model is valid in a black box sense, then the observations of the model should be indistinguishable from those of the ‘real’ system” (Pidd, 1992: 106). Recent experience at PharmaCo[8] suggests external validity of the findings that front loading outperforms other experimentation strategies on positive predictive performance. Further longitudinal research data on compounds declared active by discovery and succeeding clinical development at PharmaCo and other pharmaceutical companies would be required to gain sufficient empirical support for the model’s predictions. It should be noted here that the model’s predictions on negative predictive performance are not verifiable in practice, since the key conditional probability involved is not observable in practice.