Skip to main content
Download PDF

Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group—European Academy of Allergy and Clinical Immunology Task Force

Clinical and Translational Allergy volume 9, Article number: 20 (2019) Cite this article

Abstract

Introduction

A Task Force was commissioned jointly by the European Academy of Allergy and Clinical Immunology (EAACI) and the Respiratory Effectiveness Group (REG) to develop a quality assessment tool for real-life observational research to identify high-quality real-life asthma studies that could be considered within future guideline development.

Methods

The resulting REal Life EVidence AssessmeNt Tool (RELEVANT) was achieved through an extensive analysis of existing initiatives in this area. The first version was piloted among 9 raters across 6 articles; the revised, interim, version underwent extensive testing by 22 reviewers from the EAACI membership and REG collaborator group, leading to further revisions and tool finalisation. RELEVANT was validated through an analysis of real-life effectiveness studies identified via systematic review of Medline and Embase databases and relating to topics for which real-life studies may offer valuable evidence complementary to that from randomised controlled trials. The topics were selected through a vote among Task Force members and related to the influence of adherence, smoking, inhaler device and particle size on asthma treatment effectiveness.

Results

Although highlighting a general lack of high-quality real-life effectiveness observational research on these clinically important topics, the analysis provided insights into how identified observational studies might inform asthma guidelines developers and clinicians. Overall, RELEVANT appeared reliable and easy to use by expert reviewers.

Conclusions

Using such quality appraisal tools is mandatory to assess whether specific observational real-life effectiveness studies can be used to inform guideline development and/or decision-making in clinical practice.

Introduction: background and objectives

Randomised controlled trials (RCTs) are required to establish the efficacy and explore the safety of asthma treatments, but are insufficient to provide holistic evidence on the benefit/risk ratio of interventions when offered, initiated or used in the heterogeneous asthma patient populations and care settings that exist outside the trial environment. Registration asthma and allergy RCTs are designed to maximize chances of observing efficacy [1]. To do so, they impose a standardised level of care (often more intensive and extensive than is feasible in routine care) and systematically exclude patients presenting with characteristics that could modulate treatment responses such as smoking, obesity, poor adherence, inhaler misuse, poor reversibility, possible overlap with chronic obstructive pulmonary disease (COPD) or other comorbidities [2,3,4,5,6,7]. In this way, registration RCTs guarantee high internal validity to obtain market authorization of the drug but do not provide sufficient evidence to guide decision-making in routine clinical practice, since they exclude a significant proportion of the real-life asthmatic population [8]. Excluded patients may also represent subsets of the population in whom achieving asthma control is more difficult, i.e. those who warrant increased attention and monitoring. Therefore, it is necessary to test the external validity of the findings of the registration RCTs: their generalizability and their applicability to the general asthma population and to specific subgroups usually excluded from RCTs [9]. It is important to find reliable research methods to complement RCTs and broaden the evidence-base available to inform clinical practice decision-making [10,11,12]. Several study designs are available to reach this goal, including pragmatic randomized trials and observational studies, which can be prospective or historical, the latter relying on clinical cohorts, registries or databases [1, 13]. Phase IV comparative effectiveness studies are crucial not only to clinicians, but also to other stakeholders such as guideline developers and healthcare policy makers [14]. They can also generate new hypotheses to be tested in RCTs or other adapted study designs.

In observational research as well as in RCTs, ensuring high-quality methodology is crucial to avoid biases that would compromise the reliability and validity of results [15]. Following the GRADE methodology for evidence appraisal, RCTs are initially considered as providing high levels of evidence while observational studies are set at a lower level [16, 17]. Accounting for the quality of available studies, which determines the risk of biases, levels of evidence can be downgraded for RCTs and upgraded for observational research (Fig. 1) [18]. However, while quality standards for RCTs are well-defined and extensively disseminated, e.g., by the CONSORT statements [19, 20], this is less so in the field of observational comparative effectiveness research.

Fig. 1
figure 1

Adapted from [18]

GRADE assessment of the strength of evidence of individual studies and systematic reviews.

Full size image

A number of frameworks have been proposed to help characterise the extent to which a study design reflects pragmatic aspects or real-life patients and practice [1, 21]. One of these frameworks (developed by respiratory clinicians but more generally applicable) relies on two axes, namely the population’s characteristics and the so-called “ecology of care”, which encompasses the context in which care is delivered (e.g., routine primary care vs. controlled conditions with regular visits to specifically trained healthcare professionals, as in most registration RCTs) (Fig. 2) [1]. This framework was proposed by the Respiratory Effectiveness Group (REG), an academic non-profit organization created in 2012 with the aim of promoting high-quality real-life research in respiratory diseases. Several tools to guide the design and reporting of observational research have also been developed, sometimes using systematic and rigorous processes [22,23,24,25,26,27]. While these provide invaluable insights and recommendations for evidence generation, they have not been specifically designed for the evaluation of published (comparative) effectiveness research.

Fig. 2
figure 2

REG research framework. Reproduced with permission from [1]

Full size image

To address this, REG and the European Academy of Allergy & Clinical Immunology (EAACI) convened a joint Task Force to set and test quality standards for observational comparative effectiveness research (CER) in asthma, and to assess the extent to which the evidence from such studies might complement the results of RCTs. The aim of the current manuscript is to provide an overview of how this Task Force developed a quality assessment tool and applied it to selected PICOT questions for which RCTs provided only limited evidence. The purpose of this paper is not to provide an extensive description of the tool development process, nor the detailed results of the literature review; these will be the topics of additional publications (see [28] for details on the tool's development).

Methods

The initial work comprised two simultaneous processes: [1] the development of a quality assessment tool through a synthesis of Task Force expertise and recommendations within existing relevant publications and [2] the selection of relevant research topics and identification of corresponding articles that would be used to test the tool. Once the tool was finalised (following pilot testing and iterative refinement), it was used to appraise the retrieved literature. Studies achieving sufficiently high quality scores were then assessed to determine the extent to which they offered novel data and insights that could influence clinical practice and inform guidelines. A Task Force activity schedule summary is presented in the Additional file 1: Table S1.

Development and testing of the quality assessment tool

Full details of the REal Life EVidence AssessmeNt Tool (RELEVANT) development and assessment process are detailed in a dedicated article [28]. In summary, a systematic review of related quality tools and recommendations for effectiveness research proposed in the published literature was performed [22,23,24,25,26,27]. Appropriate quality assessment domains and specific sub-items were identified, agreed through Task Force discussions and used to build a first version of RELEVANT. The domains were: background, study design, measures, analysis, presentation of results, discussion and interpretation, conclusions and reporting of possible conflicts of interest including, but not restricted to, study funding. For each domain, items (n = 25 altogether) were divided into primary and secondary/supporting. The resulting assessment grid was then tested by 9 members of the Task Force divided in two groups, each assessing three articles selected by the Task Force leads. For each item, the agreement between raters was calculated before being summarized for all primary items, all secondary items and all (primary and secondary) items combined. Three of the papers reviewed considered the relationship between adherence and outcomes in patients with asthma, and three papers examined the relationship between particle size or device type and asthma outcomes [29,30,31,32,33,34]. Rater comments on the ease of implementation of the RELEVANT tool were collected as free text using a dedicated table. Levels of agreement and rater comments were discussed face-to-face at a Task Force meeting and the tool was refined and formatted to be usable on a web-based platform. The resulting version of the tool was tested on a larger basis by 22 raters distributed into 3 groups reviewing 2 papers each. There were two papers on the relation between adherence and asthma outcomes, two on the relation between drug persistence and healthcare resource utilization and two on the relation between particle size and outcomes [35,36,37,38,39,40]. Additional free-text feedback was captured and used to guide further refinements and to finalise the first version of RELEVANT.

Selection of topics and literature search

A first set of 21 PICOT [(P)opulation selected for the study, (I)ntervention, (C)omparator, (O)utcome measures and (T)ime—duration] [41] questions was identified during a Task Force meeting. The questions were selected based on their relevance for asthma management, limited ability of available RCTs to provide robust answers and potential of observational research to address them. Discussions in a plenary session led to the prioritization of 9 PICOT questions. An online poll was conducted among REG members and EAACI representatives to prioritise the topics and 4 PICOT questions were finally selected to ensure feasibility of delivery. The literature search was conducted using the Medline and EMBASE bibliometric databases to identify asthma effectiveness studies. Search results were then categorised to relevance to the selected PICOT questions (see the Additional file 1 for details on the search strategy, Table S2). The flow diagram shown in Fig. 3 depicts the main steps of the selection process.

Fig. 3
figure 3

Flow diagram for inclusion/exclusion of articles

Full size image

Assessment of the literature

Papers considered of sufficient quality were sent to at least two raters each with a reading grid (see Additional file 1: Table S3) comprising the following items: summary of methods (studied population, intervention, outcomes, setting), summary of results (magnitude of differences/robustness) and possible remaining biases. The list of raters is available in the Additional file 1: Table S4. Each paper was sent to a third rater when there was discordance in the conclusions from the two initial raters. For each PICOT question, a summary table was built, mentioning, for each article, the type of source (e.g. claims database or prospective cohort), the main conclusion, the corresponding level of evidence proposed based on GRADE assessment, the possible impact on clinical practice and whether similar evidence was available from RCTs. Summary tables were presented and discussed during a dedicated plenary session at the REG Summit in June 2016.

Results

RELEVANT: the REal Life EVidence AssessmeNt Tool

RELEVANT is presented in Table 1. For each quality domain, sub-items are categorised as primary or secondary items. The general principle for use of the tool is that a study should only be eligible to inform guideline development (or a similar processes) if all primary items are satisfied. Thereafter, secondary items can be used to further appraise non-essential, but pertinent aspects of published studies. Thus, the tool exists in two formats: [1] a version specific to guideline development, in which secondary items appear to the rater only when all primary quality items are satisfied, and [2] a version for more general quality appraisal in which all primary and secondary items are immediately visible.

Table 1 RELEVANT REG quality assessment tool for observational research
Full size table

The final list of 21 items within RELEVANT was informed by appraisal of inter-rater agreement. Initial agreement was poor (≤ 50%) for 8/25 items when first piloted among Task Force members. Iterative refinement was then based on rater feedback and removal of ambiguity in time language to avoid potential for uncertainty or differing interpretation. This resulted in a marked improvement in concordance between raters, to ≥ 73% for all primary and ≥ 69% for all secondary items when averaged over the three rating groups. Further details on concordance are available in the accompanying methodology paper [28].

PICOT questions

The four selected PICOT questions (numbered following their rank during the selection process) dealt with the influence of adherence/persistence on asthma outcomes, the influence of smoking on asthma outcomes and treatment effectiveness, the impact of the inhaler device on asthma outcomes and the relation between particle size of maintenance therapy and asthma outcomes (Table 2).

Table 2 Selected PICOT questions
Full size table

Selected articles

Altogether, 46 relevant articles were identified: n = 24 for PICOT question 1 [7, 29, 35, 36, 38, 42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59], 7 for PICOT question 2 [34, 39, 60,61,62,63,64], 3 for PICOT question 3 [33, 65, 66] and 12 for PICOT question 4 [32, 40, 67,68,69,70,71,72,73,74,75,76]. Figure 4 shows the results of the quality assessment of all papers by PICOT question and Fig. 5 provides a summary of failed items (Fig. 5a) and domains (Fig. 5b). The detailed quality assessment of all selected articles is presented in the Additional file 1: Table S5.

Fig. 4
figure 4

distribution of papers by quality rating within selected PICOT questions. TBC refers to a minority of papers that got only one rating, so that final rating remained to be consolidated

Full size image
Fig. 5
figure 5

proportion of failed items by domains (a) and items b of the quality assessment tool (see Table 1 for details on the items’ labelling)

Full size image

Literature analysis

A summary of the literature analysis is presented in Tables 3, 4 and 5. Study results are presented in a concise way due to space limitations and since more details would actually be out of the scope of the present manuscript. Nonetheless, the main limitation of currently available clinical or claims database studies was found to be the paucity or lack of detailed clinical data regarding: [1] asthma control as assessed by validated questionnaires, [2] severity as assessed by symptoms and lung function, [3] comorbidities, as evidenced by either symptoms suggestive of comorbidities such as rhinitis or gastro-oesophageal reflux disease, or physician diagnoses of comorbidities. As a consequence, authors used available proxy measures, including: [1] short-acting beta-agonist (SABA) use and exacerbations as markers of control; [2] inhaled corticosteroid (ICS) dose and add-on treatments for severity, and [3] prescribed therapies in place of comorbidity diagnoses. In most retained studies, these factors were used to match and/or adjust analyses, a required feature to increase the robustness of results. Use of such statistical approaches allowed the level of evidence to be raised to moderate but not high as residual confounding by hidden factors could not be ruled out; for instance, detailed data on lifestyle behaviours (such as smoking, diet and physical activity) are frequently lacking and body mass index is absent in several databases. The results of observational studies considered of sufficient quality (all 11 primary items fulfilled) suggest several conclusions regarding the corresponding PICOT questions.

Table 3 Summary table of literature analysis, PICOT question 1: influence of adherence to ICS therapy on asthma outcomes
Full size table
Table 4 Summary table of literature analysis, PICOT question 2: influence of device type for ICS therapy on asthma outcomes
Full size table
Table 5 Summary table of literature analysis, PICOT question 3: influence of smoking on asthma outcomes in patients receiving ICS therapy
Full size table

Influence of adherence to ICS therapy on asthma outcomes (Table 3) [38, 42,43,44,45,46,47,48,49]

All papers retained for analysis following RELEVANT-based quality assessments reported an association between lower adherence to maintenance therapy and poorer control, as assessed by SABA use or markers of exacerbations (e.g. oral corticosteroid treatment or hospital contacts: emergency room visits, hospitalizations). More specific findings relating to drug classes were reported: oral leukotriene receptor antagonist (LTRA) therapy appeared to be more effective than ICS in obtaining control in non-adherent patients; the opposite was true for adherent patients [38]. Some aspects included in the PICO question could not be addressed: although several papers used the medication possession ratio (MPR; the ratio of the period covered by drug dispensations by the total observation period) to assess level of adherence [36, 47, 50, 51, 77], they did not determine a minimal threshold of adherence below which asthma outcomes worsen. In addition they were not designed to assess whether a “dose–response” relationship could be described.

Influence of device type for ICS therapy on asthma outcomes (Table 4) [34, 39, 60,61,62,63]

Papers dealing with device choice suggested that limiting the number of different inhaler device types prescribed to an individual patient was associated with better asthma outcomes, while switching devices outside the context of a dedicated medical visit was associated with a greater risk of a loss of control [60, 62]. One study found superiority of a metered dose inhaler (MDI) over a dry powder inhaler (DPI) for the administration of ICS/LABA fixed-dose combination [61]. However, another study by the same group found breath-actuated metered dose inhalers and DPIs to be superior to MDIs for the administration of ICS for several asthma outcomes [34].

Influence of smoking on asthma outcomes in patients receiving ICS therapy (Table 5) [33, 65]

A prospective cohort study did not find differential treatment effects of an ICS-LABA fixed dose combination for patients with different smoking behaviours, but asthma patients with a current smoking status had poorer outcomes overall [33]. Another study suggested that extra-fine ICS could be more suitable to control airways inflammation in smokers and ex-smokers than larger particle ICS alternatives [65].

Influence of ICS particle size on asthma outcomes (Tables 6, 7) [32, 40, 67,68,69,70,71,72]

Several analyses, mostly by the same group [32, 40, 67,68,69,70,71], were performed in various populations (children, adults) using various databases (UK, US) and found superiority of extra-fine versus larger particle ICS administered alone or as part of an ICS-LABA fixed dose combination in asthma [32, 40, 67,68,69,70,71,72]. Accordingly, economic analyses using these results suggested extra-fine ICS was a dominant strategy as compared with larger particle ICS alternatives [32, 69].

Table 6 Summary table of literature analysis, PICOT question 4: influence of ICS particle size on asthma outcomes
Full size table
Table 7 PICOT question 4: influence of ICS particle size on asthma outcomes: summary of results of matched database studies
Full size table

Discussion

The REG-EAACI Task Force on quality standards in asthma comparative effectiveness research defined target quality standards and developed a quality assessment tool—RELEVANT—to appraise published observational effectiveness studies. The definition of standards and development of the tool aimed to provide a simple assessment grid for paper-based or web-based use, specifically designed for the evaluation of published effectiveness research. They were based on previous initiatives in this area [22,23,24,25,26,27] and iterative testing and refinement. RELEVANT follows a two-step process: first assessing primary, critical items; then appraising secondary, beneficial quality items. When used in the context of decision-making or guidelines development, it is strongly suggested to stop the assessment and discard the study if any one primary criterion is not satisfied. One major challenge in that respect is to distinguish between a methodological flaw and insufficient reporting. The latter can be overcome (e.g. by contacting the authors for clarifications), but may also be improved through awareness of the RELEVANT tool.

Four PICOT questions were selected through a poll among the Task Force and members of REG and EAACI, and related observational studies identified through a systematic literature search among bibliometric research databases. Retrieved records were then quality appraised using the RELEVANT tool. Several of the papers were assessed as being of sufficiently good quality to allow an increase in their level of evidence from low to moderate following a GRADE-like process. For all PICOT questions of interest, assessed observational studies yielded results with possible impact on clinical practice in areas where similar evidence from RCTs is lacking. Altogether, this suggests that RELEVANT could become a useful tool to appraise the evidence from observational studies as part of a guideline development process as well as offering wider utility for more general literature reading and appraisal, and to inform research development.

Relevant

A quality assessment instrument needs to offer ease of use and sufficient robustness to ensure confidence in its grading. This requires the items to be clear, easily understood and rated similarly by different raters. While the detailed tool development process is the topic of a dedicated paper [28], it can be emphasized here that the final tool was considered easy to use by members of the REG and EAACI networks who were involved in the final phase. Reaching a 100% inter-rater agreement is probably not realistic considering the heterogeneity of methodology reporting in published papers, but fair levels of agreement were obtained in the implementation phase (≥ 73% for primary items and ≥ 69% for secondary ones averaged over the three raters).

PICOT questions

An interesting observation permitted by this process was the high number of PICOT questions (n = 21) identified during the first round of selection by the Task Force and its REG and EAACI correspondents. This outlines the evidence gaps that are still present regarding asthma treatment, despite the considerable research interest for this common but still insufficiently controlled disease. This also contrasts with the high number of RCTs assessing the efficacy of various components of asthma care, including both medications and non-pharmacological approaches such as education or integrated care. This discrepancy illustrates that classical RCTs are not sufficient to answer all questions regarding real-life effectiveness of therapeutic strategies, and need to be complemented by dedicated studies designed to reflect better the range of patients and care settings that exist in routine clinical practice [1, 9, 13, 14].

Literature search: rationale and quantitative results

The initial steps of the process confirmed that conducting a search specific to observational effectiveness research is challenging given the lack of standardised and specific terminology/nomenclature and requires the input of specialists of bibliometric research databases. Despite this, the number of published articles available for each of the PICOT questions was surprisingly low given the long period that was scrutinized (10 years), especially for PICOT questions 2 (influence of device type on the effectiveness of maintenance ICS therapy, n = 7 papers) and 3 (influence of current active smoking on the effectiveness of ICS-containing maintenance therapy regimen, n = 3 papers). This illustrates that observational effectiveness research in asthma is still infrequently performed or reported despite the clear need for real-life data to complement classical RCTs by answering different study questions including to, better assess the extent to which their results can be generalised, and identify the best target populations. Inhalation technique is known to be poor overall with at least 50% of the patients making manipulation and/or inhalation errors, which is associated with poor control of asthma and chronic obstructive pulmonary disease (COPD) [6, 7, 78,79,80]. This has led to the development of new devices aiming at facilitating the use of inhaled therapy [81]. However, none of these devices can be considered as ideal [82], making it necessary to determine whether some are more effective than others in real-life populations, as opposed to RCTs populations in which patients are specifically educated to use studied devices properly. Regarding smoking asthmatics, in vitro studies showed that smoking decreases the sensitivity of inflammatory cells to corticosteroids, at least in part through oxidative stress-induced decreases in the cofactor histone deacetylase 2 [83, 84]. RCTs also showed a decreased clinical effect of ICS in smoking asthmatics [4]. How this translates within real-life populations is largely unknown. Altogether, real-life effectiveness studies might be useful to better identify subpopulations with specific responses to therapy, allowing more effective treatment individualisation.

The number of articles published in the 10-year period of interest regarding the influence of adherence and particle size on the effectiveness of maintenance ICS therapy was higher than for the two previous topics mentioned above. However, it remained insufficient given the importance of these issues. More specifically, although poor adherence has been associated with both poor inhaler technique and poor disease outcomes [78, 85], the level of adherence required to achieve better control is difficult to determine from RCTs. RCT-reported adherence is believed to be markedly higher than in routine care, although even relatively small differences can be associated with marked differences in outcomes [86]. The search strategy retrieved 24 articles on this topic. While it has been shown with beclomethasone dipropionate that smaller particle size allows a decrease in the nominal dose required to achieve a given physiological effect [87], whether this translates into an improved benefit-risk ratio is unknown, and could be influenced by inhaler technique: the efficacy of extra-fine particle aerosols seems to be less dependent on inhalation technique [88, 89]. Only real-life studies where patients use inhalers with their “natural” technique could help answering this question. Twelve such studies were found to inform PICOT question 4.

Quality assessment

A common limitation of analysed papers was the lack of precise clinical data available to ensure that patients of compared groups were 100% comparable at baseline. To minimize the impact of confounding (bias), most studies used statistical strategies including matching (exact or propensity score-based) and adjustments for variables differing between groups during the baseline period. These strategies could not, however, account for variables that were entirely absent from the databases. For instance, in most studies lung function data (e.g., FEV1) or questionnaire-measured level of asthma control were not available; so proxies for control were used although they were not necessarily strictly concordant with questionnaire results, e.g., due to time frame differences. In addition, some studies did not report sufficient detail on baseline population characteristics, making it difficult to assess their quality with a high level of confidence. These points reflect the lack of 100% concordance between raters, as outlined above. To deal with this difficulty, a third rater intervened in all cases of discordance between the two initial raters. Ultimately, 48% (see Fig. 4) of eligible papers were considered of sufficient quality to be analysed further in terms of their results and how these could inform current knowledge and guideline recommendations.

There was a marked “research team” bias since 16 articles (34.8%) came from a single research team (references [32, 34, 40, 58, 60,61,62,63,64,65, 67,68,69,70,71, 76] for the 4 PICOT questions, respectively), among which only 3 (18.8%) [58, 64, 76] were rejected based on quality assessment (vs. 52% of all articles). To avoid possible reviewing bias raters of a given paper could not belong to or collaborate with the research team(s) of the authors. Thus, this observation suggests the importance of experience in the field of observational research to increase the likelihood of high-quality methods.

Literature analysis

As already outlined, evidence from real-life effectiveness research can complement RCTs in several ways [1, 9]. First, by determining whether results of RCTs are applicable to broader populations receiving usual care, or to populations excluded from RCTs. Second, by exploring whether some subgroups may respond better than others, or could be at increased risk of side-effects. Third, by raising hypotheses on treatments effects or differences unsuspected in RCTs. The PICOT questions selected by the Task Force addressed several of these general issues: PICOT questions 1 and 3 (adherence and smoking) dealt with specific populations usually excluded from RCTs, i.e., poorly adherent patients and smokers, while the two other PICOT questions (2 and 4) dealt with possible differences between treatments that could not be convincingly demonstrated in RCTs. Corresponding hypotheses are that some devices may be associated with better outcomes due to improved ease-of-use or patient’s preference, leading to better adherence (PICOT question 2), and that extra-fine particles might be more effective to administer asthma maintenance treatments due to improved lung deposition and distribution to the distal airways (PICOT question 4).

Influence of adherence to ICS therapy on asthma outcomes (PICOT question 1)

Overall the observational research literature analysed here shows that poor adherence is associated with poor outcomes [38, 42,43,44,45,46,47,48,49]. However, the studies did not define thresholds of adherence which are reliably associated with improved outcomes. Thus, further analyses specifically designed to address this issue are needed. Results from observational studies suggesting that LTRA may be more effective than ICS in obtaining control in non-adherent patients (while the opposite is true for adherent patients [38]) are in line with those of a pragmatic randomized trial, which found similar effectiveness of these two drug classes in a real-life setting [90]. These data contrast with that of classical RCTs, which found greater efficacy of ICS [91]. This discrepancy might relate to differences in adherence, which is expected to be greater in classical RCTs’ patients than in real-life populations, in which it could be superior with an oral drug than with an inhaled treatment. The reluctance of some patients to take corticosteroids might also play a role.

Influence of smoking on asthma outcomes in patients receiving ICS therapy (PICOT question 3)

One study found that asthma control was impaired by smoking [33], which confirms previous evidence [92]. However, this study did not report differential treatment effects in patients with different smoking-related behaviours, contrary to results of previous RCTs [4] and in vitro studies on smoking-induced mechanisms of decreased sensitivity to corticosteroids [83, 84]; this discrepancy might be the consequence of intricate factors influencing the effects of maintenance therapy including adherence to treatments. In addition, the findings from this study need to be interpreted with caution due to several limitations (high drop-out rate, no adjustment nor matching, few clinical data available) that prevent elevation of the evidence quality level provided by this article. The results of another study suggest a beneficial effect of extra-fine versus fine particle ICS in smokers and ex-smokers [65]. However, although patients were matched and analyses were adjusted, differential treatment effect was only an exploratory objective of the study. Overall, this topic clearly requires additional research to disentangle the influence of smoking versus other factors modulating asthma control and treatment effects.

Influence of device type for ICS therapy on asthma outcomes (PICOT question 2)

Regarding PICOT question 2, the main findings from observational studies were that mixing devices in a single patient or switching devices without proper support are associated with impaired asthma control [60, 62]. This finding is especially important in the context of constrained economic resources, leading some healthcare systems to recommend treatment substitutions by less costly generic (same device) or hybrid (different device) alternatives, whenever available. Regarding direct devices comparisons, findings are contradictory; one study concluded that an MDI is associated with better outcomes than a DPI (in patients receiving an ICS/LABA combination [61]), while the other study by the same research team suggested that a breath-actuated inhaler or a DPI were more effective than a MDI (to deliver ICS) [34]. The first of these findings was rather unexpected as DPIs were developed to alleviate the need for patients to coordinate the timing of actuation and breath intake [6, 81], which is a common inhaler technique error among MDIs users [78]. It could be the result of wider patient preference for MDIs leading to increased adherence and subsequently better outcomes than for DPIs. This hypothesis could not, however, be tested in the study. The second finding could relate to improved ease-of-use and/or adherence of BAIs and DPIs (compared to MDIs); this needs to be further assessed in future studies that minimise potential confounding factors. The apparent discrepancy between these studies could also relate to differences between the open inclusion of all DPI devices in the first study, all of which have unique characteristics, compared to the restriction to a single, specific DPI in the second study [34, 61]. Again, despite the constantly growing number of available inhaler devices, each of which offers potential advantages or drawbacks as compared to their predecessors and alternatives, only a few studies have been specifically designed to compare clinical outcomes between different devices. This is only partial due to the limited number of pharmacological agents administered using different devices.

Influence of ICS particle size on asthma outcomes (PICOT question 3)

Several studies (most performed by a single research group) found that asthma outcomes were better in real-life patients receiving ICS or ICS-LABA combinations delivered by extra-fine rather than fine particle formulations [32, 40, 67,68,69,70,71,72]. These studies followed strict processes to minimize severity biases, including matching and adjustment strategies. However, as always in observational database research, it is not possible to totally exclude residual confounders that would be detected only if full clinical data were available. Importantly, the selection and quality assessment processes used here were subsequently applied to a recently published systematic review and metaanalysis on this topic [93].

Strengths and limitations

This is the first initiative to develop and use a quality-assessment tool specifically designed to aid appraisal of observational asthma research. Although RELEVANT could be used in other fields, testing was performed using articles reporting results from effectiveness studies in asthma. The Task Force systematically reviewed previous quality assessment tools for observational research, selected items and revised the incorporated quality domains and items based on pilot and extended pilot testing. This provided an opportunity to improve item selection and formulation and to reduce and refine the items classification into primary and secondary items to aid inter-rater agreement.

The final literature analysis served as a validation process confirming the applicability of the tool and its relevance for quality assessment with the aim of informing guidelines and clinicians. Some limitations need to be acknowledged. First, the process could be applied only to a limited number of PICOT questions considering the high burden of each set of analyses. To limit the impact of a biased selection of questions, a consensus on prioritisation was reached not only within the Task Force but also through a poll among REG and EAACI members. Similarly, a large number of REG and EAACI members contributed to the literature analysis, to limit reviewing biases. Systematic literature appraisal is time- and resource-consuming. Ultimately the use of standardised tools such as RELEVANT should facilitate this activity but the process still requires the involvement of experienced reviewers. The limited number of high-quality real-life comparative effectiveness research studies retrieved here underlines the need for a more extensive “real-life research culture”.

Opportunities for future research and initiatives

Comprehensive quality assessment tools are now available, not only for efficacy trials (e.g., CONSORT Statement) [94], but also for their pragmatic counterparts (CONSORT Statement extension) [95], for observational studies in epidemiology (e.g., STROBE statement) [96] and, more specifically, for pharmacoepidemiology and pharmacovigilance studies (EMA-ENCePP checklist for study protocols) [22]. Another useful initiative called SPIRIT has recently published recommendations for describing clinical trials protocols [23]. Quality criteria and minimal datasets requirements for observational studies are also the topic of the UNLOCK initiative [24]. For meta-analyses, the QUOROM (quality of reporting of meta-analysis) reporting guideline and the PRISMA (preferred reporting items for systematic reviews and meta-analyses) guidance seeks to improve the reporting of key information [19, 97]. RELEVANT now provides a quality assessment tool specifically developed for real-life effectiveness research, which can be applied to determine whether and how study results can be used to inform guidelines or clinical decision-making. It may also help guide the development of observational research protocols and study dissemination. We obviously welcome input on the RELEVANT tool from the broader community to improve its usefulness.

Conclusions

The EAACI-REG Task Force developed and tested RELEVANT, a quality assessment tool for real-life comparative effectiveness observational research. The tool was specifically used to evaluate literature relating to PICOT questions pertinent to current evidence gaps around existing asthma interventions, but can also be used in other areas of medicine. It is hoped that the availability of this tool will assist the expansion of high-quality real-life effectiveness research in the field of respiratory medicine and allergology. Importantly, several results of the literature analysis conducted using the RELEVANT tool could lead to changes in clinical practice.

References

  1. Roche N, Reddel HK, Agusti A, Bateman ED, Krishnan JA, Martin RJ, et al. Integrating real-life studies in the global therapeutic research framework. Lancet Respir Med. 2013;1(10):e29–30.

    PubMed  Google Scholar 

  2. Lisspers K, Teixeira P, Blom C, Kocks J, Ställberg B, Price D, et al. Are pharmacological randomised controlled clinical trials relevant to real-life asthma populations? A protocol for an UNLOCK study from the IPCRG. NPJ Prim Care Respir Med. 2016;26:16016.

    PubMed  PubMed Central  Google Scholar 

  3. Price DB, Swern A, Tozzi CA, Philip G, Polos P. Effect of montelukast on lung function in asthma patients with allergic rhinitis: analysis from the COMPACT trial. Allergy. 2006;61(6):737–42.

    CAS  PubMed  Google Scholar 

  4. Chalmers GW, Macleod KJ, Little SA, Thomson LJ, McSharry CP, Thomson NC. Influence of cigarette smoking on inhaled corticosteroid treatment in mild asthma. Thorax. 2002;57(3):226–30.

    CAS  PubMed  PubMed Central  Google Scholar 

  5. Peters-Golden M, Swern A, Bird SS, Hustad CM, Grant E, Edelman JM. Influence of body mass index on the response to asthma controller agents. Eur Respir J. 2006;27(3):495–503.

    CAS  PubMed  Google Scholar 

  6. Molimard M, Raherison C, Lignot S, Depont F, Abouelfath A, Moore N. Assessment of handling of inhaler devices in real life: an observational study in 3811 patients in primary care. J Aerosol Med. 2003;16(3):249–54.

    CAS  PubMed  Google Scholar 

  7. Giraud V, Allaert F-A, Roche N. Inhaler technique and asthma: feasability and acceptability of training by pharmacists. Respir Med. 2011;105(12):1815–22.

    PubMed  Google Scholar 

  8. Travers J, Marsh S, Williams M, Weatherall M, Caldwell B, Shirtcliffe P, et al. External validity of randomised controlled trials in asthma: to whom do the results of the trials apply? Thorax. 2007;62(3):219–23.

    PubMed  Google Scholar 

  9. Price D, Chisholm A, van der Molen T, Roche N, Hillyer EV, Bousquet J. Reassessing the evidence hierarchy in asthma: evaluating comparative effectiveness. Curr Allergy Asthma Rep. 2011;11(6):526–38.

    PubMed  PubMed Central  Google Scholar 

  10. Price D, Bateman ED, Chisholm A, Papadopoulos NG, Bosnic-Anticevich S, Pizzichini E, et al. Complementing the randomized controlled trial evidence base. Evolution not revolution. Ann Am Thorac Soc. 2014;11(Suppl 2):S92–8.

    PubMed  Google Scholar 

  11. Krishnan JA, Schatz M, Apter AJ. A call for action: comparative effectiveness research in asthma. J Allergy Clin Immunol. 2011;127(1):123–7.

    PubMed  Google Scholar 

  12. Carson SS, Goss CH, Patel SR, Anzueto A, Au DH, Elborn S, et al. An official American Thoracic Society research statement: comparative effectiveness research in pulmonary, critical care, and sleep medicine. Am J Respir Crit Care Med. 2013;188(10):1253–61.

    PubMed  PubMed Central  Google Scholar 

  13. Silverman SL. From randomized controlled trials to observational studies. Am J Med. 2009;122(2):114–20.

    PubMed  Google Scholar 

  14. Krishnan JA, Lindenauer PK, Au DH, Carson SS, Lee TA, McBurnie MA, et al. Stakeholder priorities for comparative effectiveness research in chronic obstructive pulmonary disease: a workshop report. Am J Respir Crit Care Med. 2013;187(3):320–6.

    PubMed  PubMed Central  Google Scholar 

  15. Roche N, Reddel H, Martin R, Brusselle G, Papi A, Thomas M, et al. Quality standards for real-world research, Focus on observational database studies of comparative effectiveness. Ann Am Thorac Soc. 2014;11(Suppl 2):S99–104.

    PubMed  Google Scholar 

  16. Guyatt G, Gutterman D, Baumann MH, Addrizzo-Harris D, Hylek EM, Phillips B, et al. Grading strength of recommendations and quality of evidence in clinical guidelines: report from an american college of chest physicians task force. Chest. 2006;129(1):174–81.

    PubMed  Google Scholar 

  17. Guyatt G, Akl EA, Oxman A, Wilson K, Puhan MA, Wilt T, et al. Synthesis, grading, and presentation of evidence in guidelines: article 7 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):256–61.

    PubMed  Google Scholar 

  18. Schünemann HJ, Oxman AD, Akl EA, Brozek JL, Montori VM, Heffner J, et al. Moving from evidence to developing recommendations in guidelines: article 11 in Integrating and coordinating efforts in COPD guideline development. An official ATS/ERS workshop report. Proc Am Thorac Soc. 2012;9(5):282–92.

    PubMed  Google Scholar 

  19. Moher D, Cook DJ, Eastwood S, Olkin I, Rennie D, Stroup DF. Improving the quality of reports of meta-analyses of randomised controlled trials: the QUOROM statement. Quality of reporting of meta-analyses. Lancet. 1999;354(9193):1896–900.

    CAS  PubMed  Google Scholar 

  20. Moher D, Hopewell S, Schulz KF, Montori V, Gøtzsche PC, Devereaux PJ, et al. CONSORT 2010 Explanation and Elaboration: updated guidelines for reporting parallel group randomised trials. J Clin Epidemiol. 2010;63(8):e1–37.

    PubMed  Google Scholar 

  21. Thorpe KE, Zwarenstein M, Oxman AD, Treweek S, Furberg CD, Altman DG, et al. A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers. J Clin Epidemiol. 2009;62(5):464–75.

    PubMed  Google Scholar 

  22. The European Network of Centres for, Pharmacoepidemiology and pharmacovigilance (ENCePP). Guide on methodological standards in pharmacoepidemiology (Revision 1). EMA/95098/2010. http://www.encepp.eu/standards_and_guidances/documents/ENCePPGuideofMethStandardsinPE.pdf. Accessed 2 Mar 2019.

  23. Chan A-W, Tetzlaff JM, Altman DG, Dickersin K, Moher D. SPIRIT 2013: new guidance for content of clinical trial protocols. Lancet. 2013;381(9861):91–2.

    PubMed  Google Scholar 

  24. Chavannes N, Ställberg B, Lisspers K, Roman M, Moran A, Langhammer A, et al. UNLOCK: uncovering and noting long-term outcomes in COPD to enhance knowledge. Prim Care Respir. 2010;19(4):408.

    PubMed  Google Scholar 

  25. Cox E, Martin BC, Van Staa T, Garbe E, Siebert U, Johnson ML. Good research practices for comparative effectiveness research: approaches to mitigate bias and confounding in the design of nonrandomized studies of treatment effects using secondary data sources: the international society for pharmacoeconomics and outcomes research good research practices for retrospective database analysis task force report–part II. Value Health. 2009;12(8):1053–61.

    PubMed  Google Scholar 

  26. Johnson ML, Crown W, Martin BC, Dormuth CR, Siebert U. Good research practices for comparative effectiveness research: analytic methods to improve causal inference from nonrandomized studies of treatment effects using secondary data sources: the ISPOR good research practices for retrospective database analysis task force report–part III. Value Health. 2009;12(8):1062–73.

    PubMed  Google Scholar 

  27. Berger ML, Martin BC, Husereau D, Worley K, Allen JD, Yang W, et al. A questionnaire to assess the relevance and credibility of observational studies to inform health care decision making: an ISPOR-AMCP-NPC Good Practice Task Force report. Value Health. 2014;17(2):143–56.

    PubMed  PubMed Central  Google Scholar 

  28. Campbell JD, Perry R, Papadopoulos NG, Krishnan J, Brusselle G, Chisholm A, et al. The REal Life EVidence AssessmeNt Tool (RELEVANT): development of a novel quality assurance asset to rate observational comparative effectiveness research studies. Clin Transl Allergy. 2019. https://doi.org/10.1186/s13601-019-0256-9

    Article  Google Scholar 

  29. Rust G, Zhang S, Reynolds J. Inhaled corticosteroid adherence and emergency department utilization among Medicaid-enrolled children with asthma. J Asthma. 2013;50(7):769–75.

    CAS  PubMed  PubMed Central  Google Scholar 

  30. Santos DO, Martins MC, Cipriano SL, Pinto RM, Cukier A, Stelmach R. Pharmaceutical care for patients with persistent asthma: assessment of treatment compliance and use of inhaled medications. J Bras Pneumol. 2010;36(1):14–22.

    Google Scholar 

  31. Sadatsafavi M, Lynd LD, Marra CA, FitzGerald JM. Dispensation of long-acting β agonists with or without inhaled corticosteroids, and risk of asthma-related hospitalisation: a population-based study. Thorax. 2014;69(4):328–34.

    PubMed  Google Scholar 

  32. Price D, Small I, Haughney J, Ryan D, Gruffydd-Jones K, Lavorini F, et al. Clinical and cost effectiveness of switching asthma patients from fluticasone-salmeterol to extra-fine particle beclometasone-formoterol: a retrospective matched observational study of real-world patients. Prim Care Respir J. 2013;22(4):439–48.

    PubMed  Google Scholar 

  33. Brusselle G, Peché R, Van den Brande P, Verhulst A, Hollanders W, Bruhwyler J. Real-life effectiveness of extrafine beclometasone dipropionate/formoterol in adults with persistent asthma according to smoking status. Respir Med. 2012;106(6):811–9.

    PubMed  Google Scholar 

  34. Price D, Haughney J, Sims E, Ali M, von Ziegenweidt J, Hillyer EV, et al. Effectiveness of inhaler types for real-world asthma management: retrospective observational study using the GPRD. J Asthma Allergy. 2011;4:37–47.

    CAS  PubMed  PubMed Central  Google Scholar 

  35. Lee T, Kim J, Kim S, Kim K, Park Y, Kim Y, et al. Risk factors for asthma-related healthcare use: longitudinal analysis using the NHI claims database in a Korean asthma cohort. PLoS One. 2014;9(11):e112844.

    PubMed  PubMed Central  Google Scholar 

  36. Elkout H, Helms PJ, Simpson CR, McLay JS. Adequate levels of adherence with controller medication is associated with increased use of rescue medication in asthmatic children. PLoS One. 2012;7(6):e39130.

    CAS  PubMed  PubMed Central  Google Scholar 

  37. Terzano C, Cremonesi G, Girbino G, Ingrassia E, Marsico S, Nicolini G, et al. 1-year prospective real life monitoring of asthma control and quality of life in Italy. Respir Res. 2012;6(13):112.

    Google Scholar 

  38. Tan H, Sarawate C, Singer J, Elward K, Cohen RI, Smart BA, et al. Impact of asthma controller medications on clinical, economic, and patient-reported outcomes. Mayo Clin Proc. 2009;84(8):675–84.

    PubMed  PubMed Central  Google Scholar 

  39. Voshaar T, Kostev K, Rex J, Schröder-Bernhardi D, Maus J, Munzel U. A retrospective database analysis on persistence with inhaled corticosteroid therapy: comparison of two dry powder inhalers during asthma treatment in Germany. Int J Clin Pharmacol Ther. 2012;50(4):257–64.

    CAS  PubMed  Google Scholar 

  40. Barnes N, Price D, Colice G, Chisholm A, Dorinsky P, Hillyer EV, et al. Asthma control with extrafine-particle hydrofluoroalkane-beclometasone vs. large-particle chlorofluorocarbon-beclometasone: a real-world observational study. Clin Exp Allergy. 2011;41:1521–32.

    CAS  Google Scholar 

  41. Guyatt G, Rennie D, Meade M, Cook D. Users’ guides to the medical literature: a manual for evidence-based clinical practice. 3rd ed. McGraw Hill. 2008. http://jamaevidence.mhmedical.com/book.aspx?bookID=847. Accessed date 30 Jul 2017

  42. de Dalcin PTR, Grutcki DM, Laporte PP, de Lima PB, Viana VP, Konzen GL, et al. Impact of a short-term educational intervention on adherence to asthma treatment and on asthma control. J Bras Pneumol. 2011;37(1):19–27.

    Google Scholar 

  43. Friedman HS, Navaratnam P, McLaughlin J. Adherence and asthma control with mometasone furoate versus fluticasone propionate in adolescents and young adults with mild asthma. J Asthma. 2010;47(9):994–1000.

    CAS  PubMed  Google Scholar 

  44. Campbell JD, Allen-Ramey F, Sajjan SG, Maiese EM, Sullivan SD. Increasing pharmaceutical copayments: impact on asthma medication utilization and outcomes. Am J Manag Care. 2011;17(10):703–10.

    PubMed  Google Scholar 

  45. Williams LK, Pladevall M, Xi H, Peterson EL, Joseph C, Lafata JE, et al. Relationship between adherence to inhaled corticosteroids and poor outcomes among adults with asthma. J Allergy Clin Immunol. 2004;114(6):1288–93.

    CAS  PubMed  Google Scholar 

  46. Taegtmeyer AB, Steurer-Stey C, Price DB, Wildhaber JH, Spertini F, Leuppi JD. Predictors of asthma control in everyday clinical practice in Switzerland. Curr Med Res Opin. 2009;25(10):2549–55.

    CAS  PubMed  Google Scholar 

  47. Laforest L, Licaj I, Devouassoux G, Chatté G, Belhassen M, Van Ganse E, et al. Relative exposure to controller therapy and asthma exacerbations: a validation study in community pharmacies. Pharmacoepidemiol Drug Saf. 2014;23(9):958–64.

    PubMed  Google Scholar 

  48. Laforest L, Licaj I, Devouassoux G, Chatte G, Martin J, Van Ganse E. Asthma drug ratios and exacerbations: claims data from universal health coverage systems. Eur Respir J. 2014;43(5):1378–86.

    PubMed  Google Scholar 

  49. Björnsdóttir US, Sigurðardóttir ST, Jonsson JS, Jonsson M, Telg G, Thuresson M, et al. Impact of changes to reimbursement of fixed combinations of inhaled corticosteroids and long-acting β-agonists in obstructive lung diseases: a population-based, observational study. Int J Clin Pract. 2014;68(7):812–9.

    PubMed  PubMed Central  Google Scholar 

  50. Taylor A, Chen L-C, Smith MD. Adherence to inhaled corticosteroids by asthmatic patients: measurement and modelling. Int J Clin Pharm. 2014;36(1):112–9.

    CAS  PubMed  Google Scholar 

  51. Herndon JB, Mattke S, Evans Cuellar A, Hong SY, Shenkman EA. Anti-inflammatory medication adherence, healthcare utilization and expenditures among medicaid and children’s health insurance program enrollees with asthma. PharmacoEconomics. 2012;30(5):397–412.

    PubMed  Google Scholar 

  52. Sawicki GS, Strunk RC, Schuemann B, Annett R, Weiss S, Fuhlbrigge AL, et al. Patterns of inhaled corticosteroid use and asthma control in the childhood asthma management program continuation study. Ann Allergy Asthma Immunol. 2010;104(1):30–5.

    CAS  PubMed  PubMed Central  Google Scholar 

  53. Bukstein DA, Murphy KR, Katz LM, Ramachandran S, Doyle JJ, Stern LS. Outcomes among a young population of pediatric asthma patients using controller therapies: results from a retrospective database analysis. Pediatr Asthma Allergy Immunol. 2007;20(4):211–22.

    Google Scholar 

  54. Lasmar L, Camargos P, Champs NS, Fonseca MT, Fontes MJ, Ibiapina C, et al. Adherence rate to inhaled corticosteroids and their impact on asthma control. Allergy. 2009;64(5):784–9.

    CAS  PubMed  Google Scholar 

  55. de Santos PM, D’Oliveira A, de Noblat LACB, Machado AS, Noblat ACB, Cruz AA. Predictors of adherence to treatment in patients with severe asthma treated at a referral center in Bahia, Brazil. J Bras Pneumol. 2008;34(12):995–1002.

    Google Scholar 

  56. Klok T, Kaptein AA, Duiverman EJ, Brand PL. It’s the adherence, stupid (that determines asthma control in preschool children)! Eur Respir J. 2014;43(3):783–91.

    PubMed  Google Scholar 

  57. Guest JF, Davie AM, Ruiz FJ, Greener MJ. Switching asthma patients to a once-daily inhaled steroid improves compliance and reduces healthcare costs. Prim Care Respir J J Gen Pract Airw Group. 2005;14(2):88–98.

    Google Scholar 

  58. Price D, Chisholm A, Hillyer EV, Burden A, von Ziegenweidt J, Svedsater H, et al. Effect of inhaled corticosteroid therapy step-down and dosing regimen on measures of asthma control. J Allergy Ther. 2012;4:2.

    Google Scholar 

  59. Klok T, Kaptein AA, Duiverman EJ, Brand PL. Long-term adherence to inhaled corticosteroids in children with asthma: observational study. Respir Med. 2015;109(9):1114–9.

    PubMed  Google Scholar 

  60. Price D, Chrystyn H, Kaplan A, Haughney J, Román-Rodríguez M, Burden A, et al. Effectiveness of same versus mixed asthma inhaler devices: a retrospective observational study in primary care. Allergy Asthma Immunol Res. 2012;4(4):184–91.

    CAS  PubMed  PubMed Central  Google Scholar 

  61. Price D, Roche N, Christian Virchow J, Burden A, Ali M, Chisholm A, et al. Device type and real-world effectiveness of asthma combination therapy: an observational study. Respir Med. 2011;105:1457–66.

    PubMed  Google Scholar 

  62. Thomas M, Price D, Chrystyn H, Lloyd A, Williams AE, von Ziegenweidt J. Inhaled corticosteroids for asthma: impact of practice level device switching on asthma control. BMC Pulm Med. 2009;9:1.

    PubMed  PubMed Central  Google Scholar 

  63. Price D, Thomas V, von Ziegenweidt J, Gould S, Hutton C, King C. Switching patients from other inhaled corticosteroid devices to the easyhaler(®): historical, matched-cohort study of real-life asthma patients. J Asthma Allergy. 2014;7:31–51.

    PubMed  PubMed Central  Google Scholar 

  64. Kemp L, Haughney J, Barnes N, Sims E, von Ziegenweidt J, Hillyer EV, et al. Cost-effectiveness analysis of corticosteroid inhaler devices in primary care asthma management: a real world observational study. Clin Outcomes Res. 2010;2:75–85.

    Google Scholar 

  65. Roche N, Postma DS, Colice G, Burden A, Guilbert TW, Israel E, et al. Differential effects of inhaled corticosteroids in smokers/ex-smokers and nonsmokers with asthma. Am J Respir Crit Care Med. 2015;191(8):960–4.

    CAS  PubMed  Google Scholar 

  66. Telenga ED, Kerstjens HAM, Ten Hacken NHT, Postma DS, van den Berge M. Inflammation and corticosteroid responsiveness in ex-, current- and never-smoking asthmatics. BMC Pulm Med. 2013;22(13):58.

    Google Scholar 

  67. van Aalderen WMC, Grigg J, Guilbert TW, Roche N, Israel E, Martin RJ, et al. Small-particle inhaled corticosteroid as first-line or step-up controller therapy in childhood asthma. J Allergy Clin Immunol Pract. 2015;3(5):721–731.e16.

    PubMed  Google Scholar 

  68. Martin RJ, Price D, Roche N, Israel E, van Aalderen WMC, Grigg J, et al. Cost-effectiveness of initiating extrafine- or standard size-particle inhaled corticosteroid for asthma in two health-care systems: a retrospective matched cohort study. NPJ Prim Care Respir Med. 2014;24:14081.

    PubMed  PubMed Central  Google Scholar 

  69. Colice G, Martin RJ, Israel E, Roche N, Barnes N, Burden A, et al. Asthma outcomes and costs of therapy with extrafine beclomethasone and fluticasone. J Allergy Clin Immunol. 2013;132(1):45–54.e10.

    CAS  PubMed  Google Scholar 

  70. Price D, Thomas M, Haughney J, Lewis RA, Burden A, von Ziegenweidt J, et al. Real-life comparison of beclometasone dipropionate as an extrafine- or larger-particle formulation for asthma. Respir Med. 2013;107(7):987–1000.

    PubMed  Google Scholar 

  71. Price D, Martin RJ, Barnes N, Dorinsky P, Israel E, Roche N, et al. Prescribing practices and asthma control with hydrofluoroalkane-beclomethasone and fluticasone: a real-world observational study. J Allergy Clin Immunol. 2010;126(3):511–518.e1-10.

    PubMed  Google Scholar 

  72. Allegra L, Cremonesi G, Girbino G, Ingrassia E, Marsico S, Nicolini G, et al. Real-life prospective study on asthma control in Italy: cross-sectional phase results. Respir Med. 2012;106(2):205–14.

    PubMed  Google Scholar 

  73. Popov TA, Petrova D, Kralimarkova TZ, Ivanov Y, Popova T, Peneva M, et al. Real life clinical study design supporting the effectiveness of extra-fine inhaled beclomethasone/formoterol at the level of small airways of asthmatics. Pulm Pharmacol Ther. 2013;26(6):624–9.

    CAS  PubMed  Google Scholar 

  74. Paggiaro P, Patel S, Nicolini G, Pradelli L, Zaniolo O, Papi A. Stepping down from high dose fluticasone/salmeterol to extrafine BDP/F in asthma is cost-effective. Respir Med. 2013;107(10):1531–7.

    PubMed  Google Scholar 

  75. Müller V, Gálffy G, Eszes N, Losonczy G, Bizzi A, Nicolini G, et al. Asthma control in patients receiving inhaled corticosteroid and long-acting beta2-agonist fixed combinations. A real-life study comparing dry powder inhalers and a pressurized metered dose inhaler extrafine formulation. BMC Pulm Med. 2011;11:40.

    PubMed  PubMed Central  Google Scholar 

  76. Israel E, Roche N, Martin RJ, Colice G, Dorinsky PM, Postma DS, et al. Increased dose of inhaled corticosteroid versus add-on long-acting β-agonist for step-up therapy in asthma. Ann Am Thorac Soc. 2015;12(6):798–806.

    PubMed  Google Scholar 

  77. Laforest L, Yin D, Kocevar VS, Pacheco Y, Dickson N, Gormand F, et al. Association between asthma control in children and loss of workdays by caregivers. Ann Asthma Immunol. 2004;93(3):265–71.

    Google Scholar 

  78. Giraud V, Roche N. Misuse of corticosteroid metered-dose inhaler is associated with decreased asthma stability. Eur Respir J. 2002;19(2):246–51.

    CAS  PubMed  Google Scholar 

  79. Molimard M, Raherison C, Lignot S, Balestra A, Lamarque S, Chartier A, et al. Chronic obstructive pulmonary disease exacerbation and inhaler device handling: real-life assessment of 2935 patients. Eur Respir J. 2017;49(2):1601794.

    PubMed  Google Scholar 

  80. Sanchis J, Gich I, Pedersen S. Systematic review of errors in inhaler use: has patient technique improved over time? Chest. 2016;150(2):394–406.

    PubMed  Google Scholar 

  81. Dekhuijzen PNR, Vincken W, Virchow JC, Roche N, Agusti A, Lavorini F, et al. Prescription of inhalers in asthma and COPD: towards a rational, rapid and effective approach. Respir Med. 2013;107(12):1817–21.

    CAS  PubMed  Google Scholar 

  82. Roche N, Scheuch G, Pritchard JN, Nopitsch-Mai C, Lakhani DA, Saluja B, et al. Patient focus and regulatory considerations for inhalation device design: report from the 2015 IPAC-RS/ISAM workshop. J Aerosol Med Pulm Drug Deliv. 2017;30(1):1–13.

    PubMed  Google Scholar 

  83. Ito K, Lim S, Caramori G, Chung KF, Barnes PJ, Adcock IM. Cigarette smoking reduces histone deacetylase 2 expression, enhances cytokine expression, and inhibits glucocorticoid actions in alveolar macrophages. FASEB J. 2001;15(6):1110–2.

    CAS  PubMed  Google Scholar 

  84. Boardman C, Chachi L, Gavrila A, Keenan CR, Perry MM, Xia YC, et al. Mechanisms of glucocorticoid action and insensitivity in airways disease. Pulm Pharmacol Ther. 2014;29(2):129–43.

    CAS  PubMed  Google Scholar 

  85. Normansell R, Kew KM, Mathioudakis AG. Interventions to improve inhaler technique for people with asthma. Cochrane Database Syst Rev. 2017;3:CD012286.

    PubMed  Google Scholar 

  86. Vestbo J, Anderson JA, Calverley PM, Celli B, Ferguson GT, Jenkins C, et al. Adherence to inhaled therapy, mortality and hospital admission in COPD. Thorax. 2009;64(11):939–43.

    CAS  PubMed  Google Scholar 

  87. Busse WW, Brazinsky S, Jacobson K, Stricker W, Schmitt K, Vanden Burgt J, et al. Efficacy response of inhaled beclomethasone dipropionate in asthma is proportional to dose and is improved by formulation with a new propellant. J Allergy Clin Immunol. 1999;104(6):1215–22.

    CAS  PubMed  Google Scholar 

  88. Usmani OS, Biddiscombe MF, Barnes PJ. Regional lung deposition and bronchodilator response as a function of beta2-agonist particle size. Am J Respir Crit Care Med. 2005;172(12):1497–504.

    PubMed  Google Scholar 

  89. Newman SP, Pavia D, Garland N, Clarke SW. Effects of various inhalation modes on the deposition of radioactive pressurized aerosols. Eur J Respir Dis. 1982;63(suppl):57–65.

    Google Scholar 

  90. Price D, Musgrave SD, Shepstone L, Hillyer EV, Sims EJ, Gilbert RFT, et al. Leukotriene antagonists as first-line or add-on asthma-controller therapy. N Engl J Med. 2011;364(18):1695–707.

    CAS  PubMed  Google Scholar 

  91. Yang D, Luo H, Wang J, Bunjhoo H, Xu Y, Xiong W. Comparison of inhaled corticosteroids and leukotriene receptor antagonists in adolescents and adults with mild to moderate asthma: a meta-analysis. Clin Respir J. 2013;7(1):74–90.

    CAS  PubMed  Google Scholar 

  92. Godard P, Huas D, Sohier B, Pribil C, Boucot I. Asthma control in general practice: a cross-sectional survey of 16,580 patients. Presse Med. 2005;34(19 Pt 1):1351–7.

    CAS  PubMed  Google Scholar 

  93. Sonnappa S, McQueen B, Postma DS, Martin RJ, Roche N, Grigg J, et al. Extrafine versus fine inhaled corticosteroids in relation to asthma control: a systematic review and meta-analysis of observational real-life studies. J Allergy Clin Immunol Pract. 2017;6:907–15.

    PubMed  Google Scholar 

  94. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, et al. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Intern Med. 2001;134(8):663–94.

    CAS  PubMed  Google Scholar 

  95. Zwarenstein M, Treweek S, Gagnier JJ, Altman DG, Tunis S, Haynes B, et al. Improving the reporting of pragmatic trials: an extension of the CONSORT statement. BMJ. 2008;337:a2390.

    PubMed  PubMed Central  Google Scholar 

  96. Vandenbroucke JP, von Elm E, Altman DG, Gøtzsche PC, Mulrow CD, Pocock SJ, et al. Strengthening the reporting of observational studies in epidemiology (STROBE): explanation and elaboration. Ann Intern Med. 2007;147(8):W163–94.

    PubMed  Google Scholar 

  97. Liberati A, Altman DG, Tetzlaff J, Mulrow C, Gøtzsche PC, Ioannidis JPA, et al. The PRISMA statement for reporting systematic reviews and meta-analyses of studies that evaluate healthcare interventions: explanation and elaboration. BMJ. 2009;339:b2700.

    PubMed  PubMed Central  Google Scholar 

Download references

Authors’ contributions

NR, JC, NP, JK, GB, AC, LB, MT, EVG, MB, JQ, DP, GC all participated to the literature appraisal, design and testing of the tool, analysis and interpretation of data and manuscript writing, and approved the final version. NR and JC elaborated and coordinated the project. All authors read and approved the final manuscript.

Acknowledgements

The authors would like to thank Katy Gallop and Sarah Acaster who contributed to the literature search and Zoe Mitchell who designed the web-based version of the tool.

Competing interests

JC discloses prior Respiratory Effectiveness Group funding related to this study, but has no other conflicts of interests associated with this paper. NP reports grants from Gerolymatos, personal fees from Hal Allergy B.V., personal fees from Novartis Pharma AG, personal fees from Menarini, personal fees from Hal Allergy B.V., personal fees from Mylan, outside the submitted work. JK has research funding from the U.S. National Institutes of Health and the U.S. Patient Centered Outcomes Research Institute paid to the University for investigator-initiated research. JB does not serve on advisory boards or have other potential conflicts of interest. GB has, within the last 5 years, received honoraria for lectures from AstraZeneca, Boehringer-Ingelheim, Chiesi, GlaxoSmithKline, Novartis and Teva; he is a member of advisory boards for AstraZeneca, Boehringer-Ingelheim, GlaxoSmithKline, Novartis, Sanofi/Regeneron and Teva. AC and GChave no conflicts of interest to declare in relation to this paper. LB has no perceived COI. MT nor any member of his close family has any shares in pharmaceutical companies. In the last 3 years he has received speaker’s honoraria for speaking at sponsored meetings or satellite symposia at conferences from the following companies marketing respiratory and allergy products: Aerocrine, GSK, Novartis. He has received honoraria for attending advisory panels with; Aerocrine, Boehringer Inglehiem, GSK, MSD, Novartis, Pfizer. He is a recent a member of the BTS SIGN Asthma guideline steering group and the NICE Asthma Diagnosis and Monitoring guideline development group. EVG reports grants and personal fees from ALK ABELLO, grants and personal fees from Bayer, grants and personal fees from BMS, grants and personal fees from GlaxoSmithKline, grants and personal fees from Merck Sharp and Dohme, personal fees from PELyon, outside the submitted work. MB reports grants from GlaxoSmithKline, TEVA, Chiesi, Genentech, outside the submitted work. JQ’sresearch group has received funding from The Health Foundation, MRC, Wellcome Trust, BLF, GSK, Insmed, AZ, Bayer and BI for other projects, none of which relate to this work. Dr Quint has received funds from AZ, GSK, Chiesi, Teva and BI for Advisory board participation or travel. DP has board membership with Aerocrine, Amgen, AstraZeneca, Boehringer Ingelheim, Chiesi, Mylan, Mundipharma, Napp, Novartis, Regeneron Pharmaceuticals, Sanofi Genzyme, Teva Pharmaceuticals; consultancy agreements with Almirall, Amgen, AstraZeneca, Boehringer Ingelheim, Chiesi, GlaxoSmithKline, Mylan, Mundipharma, Napp, Novartis, Pfizer, Teva Pharmaceuticals, Theravance; grants and unrestricted funding for investigator-initiated studies (conducted through Observational and Pragmatic Research Institute Pte Ltd) from Aerocrine, AKL Research and Development Ltd, AstraZeneca, Boehringer Ingelheim, British Lung Foundation, Chiesi, Mylan, Mundipharma, Napp, Novartis, Pfizer, Regeneron Pharmaceuticals, Respiratory Effectiveness Group, Sanofi Genzyme, Teva Pharmaceuticals, Theravance, UK National Health Service, Zentiva (Sanofi Generics); payment for lectures/speaking engagements from Almirall, AstraZeneca, Boehringer Ingelheim, Chiesi, Cipla, GlaxoSmithKline, Kyorin, Mylan, Merck, Mundipharma, Novartis, Pfizer, Regeneron Pharmaceuticals, Sanofi Genzyme, Skyepharma, Teva Pharmaceuticals; payment for manuscript preparation from Mundipharma, Teva Pharmaceuticals; payment for the development of educational materials from Mundipharma, Novartis; payment for travel/accommodation/meeting expenses from Aerocrine, AstraZeneca, Boehringer Ingelheim, Mundipharma, Napp, Novartis, Teva Pharmaceuticals; funding for patient enrolment or completion of research from Chiesi, Novartis, Teva Pharmaceuticals, Zentiva (Sanofi Generics); stock/stock options from AKL Research and Development Ltd which produces phytopharmaceuticals; owns 74% of the social enterprise Optimum Patient Care Ltd (Australia and UK) and 74% of Observational and Pragmatic Research Institute Pte Ltd (Singapore); and is peer reviewer for grant committees of the Efficacy and Mechanism Evaluation programme, and Health Technology Assessment. NR - Dr. Roche reports grants and personal fees from Boehringer Ingelheim, grants and personal fees from Novartis, personal fees from Teva, personal fees from GSK, personal fees from AstraZeneca, personal fees from Chiesi, personal fees from Mundipharma, personal fees from Cipla, grants and personal fees from Pfizer, personal fees from Sanofi, personal fees from Sandoz, personal fees from 3 M, personal fees from Zambon, outside the submitted work. The authors declare that they have no competing interests

Availability of data and materials

The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request.

Consent for publication

Not applicable.

Ethics approval

Not applicable.

Funding

This study was funded by the Respiratory Effectiveness Group (REG), and the European Academy of Allergy and Clinical Immunology (EAACI).

Publisher’s Note

Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.

Author information

Authors and Affiliations

  1. Pneumologie, Hôpital Cochin, 27 rue du Faubourg Saint Jacques, 75014, Paris, France

    Nicolas Roche

  2. Center for Pharmaceutical Outcomes Reasearch, Skaggs School of Pharmacy and Pharmaceutical Sciences, University of Colorado, Anschutz Medical Campus, Aurora, CO, USA

    Jonathan D. Campbell

  3. University of Illinois, Champaign, IL, USA

    Jerry A. Krishnan

  4. Department of Respiratory Medicine, Ghent University Hospital, 9000, Ghent, Belgium

    Guy Brusselle

  5. Syneos Health, 10 Bloomsbury Way, London, WC1A 2SL, UK

    Alison Chisholm

  6. Department of Respiratory Medicine and Allergology, Skane University Hospital, 221, 85, Lund, Sweden

    Leif Bjermer

  7. University of Southampton, University Road, Southampton, SO17 1BJ, UK

    Mike Thomas

  8. Claude-Bernard Lyon1 University, 43 Boulevard du 11 Novembre 1918, 69100, Villeurbanne, France

    Eric van Ganse

  9. University of Groningen, Hanzeplein 1, 9713 GZ, Groningen, Netherlands

    Maarten van den Berge

  10. Medical University - Sofia, Faculty of Public Health, 8 “Bialo more” str, 1527, Sofia, Bulgaria

    George Christoff

  11. Imperial College London, South Kensington Campus, London, SW7 2AZ, UK

    Jennifer Quint

  12. National and Kapodistrian University of Athens, Fidippidou 41 Str, Goudi, Athens, 11527, Greece

    Nikolaos G. Papadopoulos

  13. Observational and Pragmatic Research Institute, 60 Paya Lebar Road, Paya Lebar Square, #05-33/34, Singapore, 409051, Singapore

    David Price

Authors
  1. Nicolas Roche
    View author publications

    You can also search for this author in PubMed Google Scholar

  2. Jonathan D. Campbell
    View author publications

    You can also search for this author in PubMed Google Scholar

  3. Jerry A. Krishnan
    View author publications

    You can also search for this author in PubMed Google Scholar

  4. Guy Brusselle
    View author publications

    You can also search for this author in PubMed Google Scholar

  5. Alison Chisholm
    View author publications

    You can also search for this author in PubMed Google Scholar

  6. Leif Bjermer
    View author publications

    You can also search for this author in PubMed Google Scholar

  7. Mike Thomas
    View author publications

    You can also search for this author in PubMed Google Scholar

  8. Eric van Ganse
    View author publications

    You can also search for this author in PubMed Google Scholar

  9. Maarten van den Berge
    View author publications

    You can also search for this author in PubMed Google Scholar

  10. George Christoff
    View author publications

    You can also search for this author in PubMed Google Scholar

  11. Jennifer Quint
    View author publications

    You can also search for this author in PubMed Google Scholar

  12. Nikolaos G. Papadopoulos
    View author publications

    You can also search for this author in PubMed Google Scholar

  13. David Price
    View author publications

    You can also search for this author in PubMed Google Scholar

Corresponding author

Correspondence to Nicolas Roche.

Additional file

Additional file 1: Table S1.

Taskforce Activity Schedule Summary. Table S2. Literature review search terms: used to identify a list that would include asthma observational comparative effectiveness studies. Table S3. Reading grid used by reviewers to summarize selected articles. Table S4. List of REG and EAACI contributors. Table S5. Literature Review Assessment Overview: All papers.

Rights and permissions

Open Access This article is distributed under the terms of the Creative Commons Attribution 4.0 International License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted use, distribution, and reproduction in any medium, provided you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons license, and indicate if changes were made. The Creative Commons Public Domain Dedication waiver (http://creativecommons.org/publicdomain/zero/1.0/) applies to the data made available in this article, unless otherwise stated.

Reprints and permissions

About this article

Check for updates. Verify currency and authenticity via CrossMark

Cite this article

Roche, N., Campbell, J.D., Krishnan, J.A. et al. Quality standards in respiratory real-life effectiveness research: the REal Life EVidence AssessmeNt Tool (RELEVANT): report from the Respiratory Effectiveness Group—European Academy of Allergy and Clinical Immunology Task Force. Clin Transl Allergy 9, 20 (2019). https://doi.org/10.1186/s13601-019-0255-x

Download citation

  • Received:

  • Accepted:

  • Published:

  • DOI: https://doi.org/10.1186/s13601-019-0255-x

Keywords

Clinical and Translational Allergy

ISSN: 2045-7022

Contact us