\usepackage{float}

Appendix A — Figures

Authors
Affiliation

A.1 IRR Analyses

(a) Fleiss’ kappa results
(b) Fleiss’ kappa funnel plot
Figure A.1: Fleiss’ kappa results and funnel plot for the inter-rater reliability (IRR) meta-analysis.
(a) Cohen’s kappa results
(b) Cohen’s kappa funnel plot
Figure A.2: Cohen’s kappa results and funnel plot for the inter-rater reliability (IRR) meta-analysis.

A.2 Trial Results

A.2.1 Overall Agreement

A.2.1.1 Estimated Marginal Means (EMMs)

Study Contrast Estimate SE df t.ratio p.value
NCT02395172 Site Inv. - Reader 1 -0.0180 0.0372 2613.316 -0.4836 0.8791\(^{ }\)
NCT02395172 Site Inv. - Reader 2 0.0371 0.0372 2617.056 0.9983 0.578\(^{ }\)
NCT02395172 Reader 1 - Reader 2 0.0551 0.0373 2626.392 1.4773 0.302\(^{ }\)
NCT03434379 Site Inv. - Reader 1 0.0519 0.0621 1262.717 0.8352 0.6812\(^{ }\)
NCT03434379 Site Inv. - Reader 2 0.0579 0.0622 1262.846 0.9302 0.6213\(^{ }\)
NCT03434379 Reader 1 - Reader 2 0.0060 0.0614 1259.941 0.0972 0.9948\(^{ }\)
NCT03631706 Site Inv. - Reader 1 0.2284 0.0406 2507.766 5.6305 0\(^{***}\)
NCT03631706 Site Inv. - Reader 2 0.0644 0.0406 2507.789 1.5859 0.2519\(^{ }\)
NCT03631706 Reader 1 - Reader 2 -0.1640 0.0424 2501.910 -3.8645 3e-04\(^{***}\)
Table A.1: Estimated marginal means (EMMs) between all raters for the objective response rate (ORR) analyses

A.2.1.2 LME Equations

\[ \begin{aligned} \operatorname{\widehat{RSSTRESC}}_{i} &\sim N \left(2.51_{\alpha_{j[i]}} + 0.02_{\beta_{1}}(\operatorname{RSEVALID}_{\operatorname{READER\ 1}}) - 0.04_{\beta_{2}}(\operatorname{RSEVALID}_{\operatorname{READER\ 2}}), \sigma^2 \right) \\ \alpha_{j} &\sim N \left(0, 0.73 \right) \text{, for USUBJID j = 1,} \dots \text{,J} \end{aligned} \tag{A.1}\] Equation: Estimated marginal means for the objective response rate (ORR) for study NCT02395172.

\[ \begin{aligned} \operatorname{\widehat{RSSTRESC}}_{i} &\sim N \left(2.34_{\alpha_{j[i]}} - 0.05_{\beta_{1}}(\operatorname{RSEVALID}_{\operatorname{READER\ 1}}) - 0.06_{\beta_{2}}(\operatorname{RSEVALID}_{\operatorname{READER\ 2}}), \sigma^2 \right) \\ \alpha_{j} &\sim N \left(0, 0.54 \right) \text{, for USUBJID j = 1,} \dots \text{,J} \end{aligned} \tag{A.2}\] Equation: Estimated marginal means for the objective response rate (ORR) for study NCT03434379.

\[ \begin{aligned} \operatorname{\widehat{RSSTRESC}}_{i} &\sim N \left(2.98_{\alpha_{j[i]}} - 0.23_{\beta_{1}}(\operatorname{RSEVALID}_{\operatorname{READER\ 1}}) - 0.06_{\beta_{2}}(\operatorname{RSEVALID}_{\operatorname{READER\ 2}}), \sigma^2 \right) \\ \alpha_{j} &\sim N \left(0, 0.89 \right) \text{, for USUBJID j = 1,} \dots \text{,J} \end{aligned} \tag{A.3}\] Equation: Estimated marginal means for the objective response rate (ORR) for study NCT03631706.

A.2.2 Objective Response Rate Results

A.2.2.1 McNemar’s Tests

$reader1_reader2
        READER 2
READER 1 FALSE TRUE
   FALSE   253   18
   TRUE     19   37

$reader1_site
        SITE INVESTIGATOR
READER 1 FALSE TRUE
   FALSE   259   13
   TRUE     14   42

$reader2_site
        SITE INVESTIGATOR
READER 2 FALSE TRUE
   FALSE   268    5
   TRUE      5   50
Table A.2: Contingency tables for the McNemar’s tests study NCT02395172. 
$reader1_reader2
        READER 2
READER 1 FALSE TRUE
   FALSE   100   11
   TRUE      7   10

$reader1_site
        SITE INVESTIGATOR
READER 1 FALSE TRUE
   FALSE   106    5
   TRUE     11    6

$reader2_site
        SITE INVESTIGATOR
READER 2 FALSE TRUE
   FALSE   105    2
   TRUE     12    9
Table A.3: Contingency tables for the McNemar’s tests study NCT03434379. 
$reader1_reader2
        READER 2
READER 1 FALSE TRUE
   FALSE    62    9
   TRUE      9   66

$reader1_site
        SITE INVESTIGATOR
READER 1 FALSE TRUE
   FALSE    55   16
   TRUE      3   72

$reader2_site
        SITE INVESTIGATOR
READER 2 FALSE TRUE
   FALSE    55   16
   TRUE      3   72
Table A.4: Contingency tables for the McNemar’s tests study NCT03631706.

A.2.3 Survival Analyses

A.2.3.1 Time to Progression (TTP) Analyses

A.2.3.1.1 NCT02395172
Figure A.3: Kaplan-Meier survival plot for time to progression (TTP) for study NCT02395172.
Figure A.4: Log-minus-log survival plot for time to progression (TTP) for study NCT02395172.
Figure A.5: Schoenfeld residuals for the Cox proportional hazards model for time to progression (TTP) for study NCT02395172.
A.2.3.1.2 NCT03434379
Figure A.6: Kaplan-Meier survival plot for time to progression (TTP) for study NCT03434379.
Figure A.7: Log-minus-log survival plot for time to progression (TTP) for study NCT03434379.
Figure A.8: Schoenfeld residuals for the Cox proportional hazards model for time to progression (TTP) for study NCT03434379.
A.2.3.1.3 NCT03631706
Figure A.9: Kaplan-Meier survival plot for time to progression (TTP) for study NCT03631706.
Figure A.10: Log-minus-log survival plot for time to progression (TTP) for study NCT03631706.
Figure A.11: Schoenfeld residuals for the Cox proportional hazards model for time to progression (TTP) for study NCT03631706.

A.2.3.2 Time to Response (TTR) Analyses

A.2.3.2.1 NCT02395172
Figure A.12: Kaplan-Meier survival plot for time to response (TTR) for study NCT02395172.
Figure A.13: Log-minus-log survival plot for time to response (TTR) for study NCT02395172.
Figure A.14: Schoenfeld residuals for the Cox proportional hazards model for time to response (TTR) for study NCT02395172.
A.2.3.2.2 NCT03434379
Figure A.15: Kaplan-Meier survival plot for time to response (TTR) for study NCT03434379.
Figure A.16: Log-minus-log survival plot for time to response (TTR) for study NCT03434379.
Figure A.17: Schoenfeld residuals for the Cox proportional hazards model for time to response (TTR) for study NCT03434379.
A.2.3.2.3 NCT03631706
Figure A.18: Kaplan-Meier survival plot for time to response (TTR) for study NCT03631706.
Figure A.19: Log-minus-log survival plot for time to response (TTR) for study NCT03631706.
Figure A.20: Schoenfeld residuals for the Cox proportional hazards model for time to response (TTR) for study NCT03631706.

A.2.3.3 Duration of Response (DoR) Analyses

A.2.3.3.1 NCT02395172
Figure A.21: Kaplan-Meier survival plot for duration of response (DoR) for study NCT02395172.
Figure A.22: Log-minus-log survival plot for duration of response (DoR) for study NCT02395172.
Figure A.23: Schoenfeld residuals for the Cox proportional hazards model for duration of response (DoR) for study NCT02395172.
A.2.3.3.2 NCT03434379
Figure A.24: Kaplan-Meier survival plot for duration of response (DoR) for study NCT03434379.
Figure A.25: Log-minus-log survival plot for duration of response (DoR) for study NCT03434379.
Figure A.26: Schoenfeld residuals for the Cox proportional hazards model for duration of response (DoR) for study NCT03434379.
A.2.3.3.3 NCT03631706
Figure A.27: Kaplan-Meier survival plot for duration of response (DoR) for study NCT03631706.
Figure A.28: Log-minus-log survival plot for duration of response (DoR) for study NCT03631706.
Figure A.29: Schoenfeld residuals for the Cox proportional hazards model for duration of response (DoR) for study NCT03631706.

A.2.4 Time-to-Event Meta-Analyses

Figure A.30: Forest plot of hazard ratios for TTP from the meta-analysis of time to event outcomes
Figure A.31: Funnel plot of hazard ratios for TTR from the meta-analysis of time to event outcomes
Figure A.32: Funnel plot of hazard ratios for DOR from the meta-analysis of time to event outcomes

A.3 Sensitivity Analyses

Figure A.33: Heatmap of Change in Differences between Raters for DoR across RECIST thresholds, unfiltered data
Figure A.34: Heatmap of Change in Differences between Raters for TTR across RECIST thresholds, unfiltered data