# Counting experiment

The root macro counting_experiment.C can be used to analyse the results of counting experiments in the complete absence of systematic uncertainties and sideband measurements of the predicted background and background counting rates and . Median expected results (p-values, confidence and significance levels, as well as CLs) for both true and false signal hypotheses are computed with several methods as are the results for observed candidates.

The code is very simple and is not protected against zero background rate - it's a very good approximation to simply give a background rate much smaller than the signal rate in such ideal cases.

The results of testing

with and (e.g. ) is shown below (this documents the simple argument list as well). Both and are , the well-known result that the 95% CL upper limit for zero observed candidates and no background expected is 3 (before the Poisson discreteness is broken by e.g. sideband measurements and systematic uncertainties- in such cases the limit may be as low as 2.3).The code was validated against Figure 7 of the famous "Asimov paper" by Cowan, Cranmer, Gross and Vitells (in fact, the code was used to reveal some small discrepancies in this figure which were reported to the authors) and old F77 code used to set limits on the DELPHI and combined HIggs searches at LEP (program called "alrmc" for old DELPHI and LEP collaborators).

For very simple limit-setting "by hand" one can find e.g. 95% CL upper limits by tweaking

until the confidence (either or the pseduo-confidence , depending on your taste) is 5% (within some numerical tolerance).root [0] .L counting_experiment.C root [1] counting_experiment( void counting_experiment(int n, double s, double b) root [1] counting_experiment(0,3,0.0001) =========================================== Observed results =========================================== Counts n=0 Test signal rate s=3 Background rate b=0.0001 Generalized chisq: 6 Background hypothesis preferred (delta chisq>0) Asymptotic signfificance of exclusion: 2.44949 o Corresponding p-value for s+b hypothesis (CLs+b): 0.00715294 CLs+b (p-value for s+b hypothesis): 0.0497821 o Corresponding exclusion significance: 1.64697 p-value for b hypothesis: 1 o Corresponding signal significance: 0 1-CLb (!=pvalue for b!!): 9.9995e-05 o Corresponding signal significance: 3.71903 o CLb: 0.9999 CLs (Pseudo p-value for s hypothesis): 0.0497871 o Corresponding exclusion significance: 1.64692 ============================================== Expected exclusion results for true background ============================================== Test signal rate s= 3 Background rate b=0.0001 Asimov expected delta chisq for absence of signal: 5.99794 o Corresponding expected signficance of exclusion: 2.44907 o Corresponding expected p-value for s+b hypothesis (CLs+b): 0.0071613 Median expected CLsb: 1 o Corresponding significance of exclusion: 0 Median expected CLb: 1 Median expected CLs: 1 o Corresponding significance of exclusion: 0 ========================================== Expected discovery results for true signal ========================================== Test signal rate s= 3 Background rate b=0.0001 Asimov expected delta chisq for true signal hypothesis: -55.856 o Corresponding signficance of rejection of b hypothesis: 7.47369 o Corresponding expected p-value for b hypothesis: 3.89896e-14 Median expected p-value for background when signal is present: 1.66654e-13 o Corresponding significance: 7.28021 Median expected 1-CLb for background when signal is present: 4.16633e-18 o Corresponding significance: 8.59491 o Note: This is the convention 'invented' at LEP for CLb=P(-2lnQ>=-2lnQ(obs)|s=0,b)=P(Q<=Q(obs)|s=0,b) which is inconsistent with conventional definition of p-value for b which is P(Q>=Q(obs)|s=0,b). This only matters for counting experiments but there can be significant differences between 1-CLb and p-value(b).