Counting experiment

Fra mn/fys/epf
Hopp til: navigasjon, søk

The root macro counting_experiment.C can be used to analyse the results of counting experiments in the complete absence of systematic uncertainties and sideband measurements of the predicted background and background counting rates [math]s[/math] and [math]b[/math]. Median expected results (p-values, confidence and significance levels, as well as CLs) for both true and false signal hypotheses are computed with several methods as are the results for [math]n[/math] observed candidates.

The code is very simple and is not protected against zero background rate - it's a very good approximation to simply give a background rate much smaller than the signal rate in such ideal cases.

The results of testing [math]n=0[/math] with [math]s=3[/math] and [math]b\lt \lt 3[/math] (e.g. [math]b\sim 0[/math]) is shown below (this documents the simple argument list as well). Both [math]CL_{s+b}[/math] and [math]CL_{s}[/math] are [math]\sim 5\%[/math], the well-known result that the 95% CL upper limit for zero observed candidates and no background expected is 3 (before the Poisson discreteness is broken by e.g. sideband measurements and systematic uncertainties- in such cases the limit may be as low as 2.3).

The code was validated against Figure 7 of the famous "Asimov paper" by Cowan, Cranmer, Gross and Vitells (in fact, the code was used to reveal some small discrepancies in this figure which were reported to the authors) and old F77 code used to set limits on the DELPHI and combined HIggs searches at LEP (program called "alrmc" for old DELPHI and LEP collaborators).

For very simple limit-setting "by hand" one can find e.g. 95% CL upper limits by tweaking [math]s[/math] until the confidence (either [math]CL_{s+b}[/math] or the pseduo-confidence [math]CL_s[/math], depending on your taste) is 5% (within some numerical tolerance).


root [0] .L counting_experiment.C
root [1] counting_experiment(
void counting_experiment(int n, double s, double b)
root [1] counting_experiment(0,3,0.0001)
===========================================
Observed results
===========================================
Counts n=0  Test signal rate s=3    Background rate b=0.0001
Generalized chisq: 6
Background hypothesis preferred (delta chisq>0)
   Asymptotic signfificance of exclusion: 2.44949
     o Corresponding p-value for s+b hypothesis (CLs+b): 0.00715294
   CLs+b (p-value for s+b hypothesis): 0.0497821
     o Corresponding exclusion significance: 1.64697
   p-value for b hypothesis: 1
     o Corresponding signal significance: 0
   1-CLb (!=pvalue for b!!): 9.9995e-05
     o Corresponding signal significance: 3.71903
     o CLb: 0.9999
   CLs (Pseudo p-value for s hypothesis): 0.0497871
     o Corresponding exclusion significance: 1.64692
 
==============================================
Expected exclusion results for true background
==============================================
Test signal rate s= 3  Background rate b=0.0001
Asimov expected delta chisq for absence of signal: 5.99794
  o Corresponding expected signficance of exclusion: 2.44907
  o Corresponding expected p-value for s+b hypothesis (CLs+b): 0.0071613
Median expected CLsb: 1
  o Corresponding significance of exclusion: 0
Median expected CLb: 1
Median expected CLs: 1
  o Corresponding significance of exclusion: 0
 
==========================================
Expected discovery results for true signal
==========================================
Test signal rate s= 3  Background rate b=0.0001
Asimov expected delta chisq for true signal hypothesis: -55.856
  o Corresponding signficance of rejection of b hypothesis: 7.47369
  o Corresponding expected p-value for b hypothesis: 3.89896e-14
Median expected p-value for background when signal is present: 1.66654e-13
  o Corresponding significance: 7.28021
Median expected 1-CLb for background when signal is present: 4.16633e-18
  o Corresponding significance: 8.59491
  o Note: This is the convention 'invented' at LEP for 
    CLb=P(-2lnQ>=-2lnQ(obs)|s=0,b)=P(Q<=Q(obs)|s=0,b)
    which is inconsistent with conventional definition
    of p-value for b which is P(Q>=Q(obs)|s=0,b). This
    only matters for counting experiments but there can be
    significant differences between 1-CLb and p-value(b).