User acceptance is a prerequisite to reaping the potential benefits of any new technology. In health care, patient satisfaction is often used as an indicator of the service performance and reflects patient’s values and expectations [1]. When there is a match between the care expected and received, patients are satisfied [2]. In other words, satisfaction is influenced not only by experiences but also by prior expectation [3].

The Treatment in Morning vs. Evening (TIME) [4] study is a clinical trial evaluating morning dosing of antihypertensive medication versus evening dosing to assess whether one provides greater cardiovascular protection. The study data and all follow ups are captured entirely online, using internet technology. The online methodology used in the TIME study is novel and it is unclear how acceptable this use of technology will be to study participants. In order to maintain or improve participation rates and minimise waste in clinical research it will be necessary to evaluate any new methods or innovations [5].

There have been several patient satisfaction surveys developed to evaluate telemedicine, or information technology in healthcare, including: Telemedicine Perception Questionnaire (TMPQ), Telemedicine Satisfaction and Usefulness Questionnaire (TSUQ) and Telemedicine Satisfaction Survey (TSQ). These questionnaires have all been designed specifically for use in telemedicine [6-9], and none were deemed suitable for use in assessing the TIME study. Problems with the aforementioned questionnaires include lack of measures of reliability or validity and limited evidence of practical application. Patient selection criteria were not always clearly specified so it is possible sampling bias may have be an issue [10,11]. It was identified by the authors that a suitable questionnaire was required that could be used to gather feedback for online-only clinical studies using a similar methodology as the TIME study.

Purpose

The research objective was to create an online questionnaire capable of capturing clinical study participant feedback to evaluate the novel methodology used in the TIME study. Using similar methods to those reported in the development of telemedicine surveys, we devised and tested a questionnaire for the evaluation of specific components of the TIME study website.

Literature review

A broad search of previous published literature on questionnaire development for similar applications was performed. The outcomes of this review were used to inform the questionnaire development process below.

Questionnaire ease of use

Care was taken to ensure that the questionnaire was not burdensome to the respondents, increasing the likelihood of complete and accurate responses [12]. Potential questions were compiled and reviewed for inclusion by TIME study staff. Questions were then prioritised for inclusion if they appeared relevant to the trial technology and then edited to a reduced set of items. Correspondence was personalised and questions were kept short, interesting and user friendly. The survey also asked for reasons for not taking part. All of these are features that have been found to have a positive impact on response rates [13].

Questionnaire readability

A number of reports have documented the extent of poor health literacy and its impacts on health status [14-16]; therefore, materials were designed with careful attention to readability. To ensure the questionnaire was suitable for a wide potential audience of trial participants, we aimed to adopt the recommended readability level between US 6th to 8th grade (11 to 14 years of age) English [17,18], measured using the Flesch-Kincaid reading ease scale [19].

Questionnaire language

Some questions were phrased positively and some negatively to minimise the “halo” effect, where responders may have an overall attitude towards the system and respond to all items consistently without reading the individual statements carefully [12]. All questions were worded to avoid generalities or ambiguity and no emotional or persuasive terms were used. Questions were reviewed by study research staff familiar with the trial data collection process to ensure that they captured the participant experience. A Likert-type scale (from strongly disagree to strongly agree) was used for survey responses [20]. Exceptions to this were a statement reflecting the previous participation in clinical research (question 1: “I have taken part in other clinical studies”), where a Yes, No or Not Sure options was required. Additionally, exceptions were made for two questions which were free text and allowed responders to give additional feedback as to why they decided to take part, or not take part, in the study. Prior to the questionnaire being sent to participants, all theme headings were removed and the order of the questions was randomised (all participants received the same questionnaire once randomised). No compensation or incentives were offered for completing the survey.

TIME online methodology

TIME study is conducted online, so there is a perception that participants are familiar with web-based technology. An online questionnaire was considered an appropriate and efficient method of gathering feedback. A web based questionnaire was also deemed the most suitable as it would not require additional software to be installed. The questionnaire was developed using Microsoft C# .NET and responses were stored in the TIME study SQL Server database. An encrypted Uniform Resource Locator (URL), unique for each individual, was emailed to TIME study participants who had been randomised and to any participants who had not been randomised but had consented to being contacted. This encrypted URL took the individual directly to the questionnaire when clicked.

Questionnaire validation

To validate the questionnaire we tested face validity, content, and construct validity [21]. Face validity was addressed by receiving feedback from study personnel and academic staff. The members of staff included the principal investigator, the trial physician, the senior software developer, the trial statistician and the study administrative assistant. Where differences of opinion occurred, the majority opinion was taken. Content validity was achieved by ensuring the questionnaire covered all relevant themes, including reported advantages and disadvantages of online clinical study methodology identified through the literature review. Construct validity was investigated by examining the results of the testing procedure and comparing these with published findings. The reliability of the questionnaire was tested using internal consistency. Internal consistency is used to assess the consistency of results across items within a questionnaire. The index of internal consistency used was Cronbach’s Alpha [22], which is the average of all possible split-half estimates. Test-retest reliability was not measured at this point, as it requires that the same subjects are exposed a second time to the questionnaire. This will, however, be assessed at the end of TIME to evaluate the consistency of the questionnaire from one time point to another. Participants will be re-sent the same questionnaire to measure consistency of answers between the trial start and finish.

Questionnaire development

The initial questionnaire was created during December 2015. Principal component analysis was used through iterations of questionnaire development to identify the extent that questions were relatable to intended themes and that the questions were appropriately grouped. The authors wished to identify how easy participants found the study website to use, what participants views were about the study being conducted entirely online with no nurse or clinician contact, and what participants thought of their personal data being used in this manner. These 3 themes were named: Functionality, Personal Contact, and Trust. Cronbach’s Alpha statistic was calculated to check the internal consistency of the questions within each theme and ensure questions were correctly associated to the theme. Cronbach Alpha scores above 0.7 are preferable [23], but for an exploratory study, a value above 0.6 is considered tolerable [24]. Questions that were answered inconsistently, or that did not relate to the intended theme, were considered redundant and were reworded or removed. This iterative process went through four rounds of development before the final questionnaire was felt to be performing as required. Each round involved the questionnaire being responded to by between 121 and 297 newly enrolled patients from January 2015 to July 2016.

Three questions were not analysed in the questionnaire development (question 1, question 16, and question 17). These were either free text or required a Yes/No/Not known response, and so were not suitable for data analysis. These three questions were included to collect additional information as to why participants had decided to take part or not in the TIME study, and whether this had been their first experience of participating in a clinical study.

The final version of the 14 question TIME evaluation survey (Figure 1) was emailed to 263 participants, 149 (57%) of whom responded, between July 2016 and January 2017. The mean age of respondents was 64 (median 67) and 72 were male (48%). 103 (69%) participants indicated that they had never taken part in a clinical study before, 41 (27%) indicated that they had, and 6 (4%) participants were unsure or did not know.

Figure 1: Rescoring FTND-Q4 “How many cigarettes do you smoke a day?” appropriately

The Cronbach Alpha of the themes for the final questionnaire were: functionality 0.764, personal contact 0.690, and trust 0.668. Due to the Cronbach Alpha scores being below 0.7 for personal contact and trust a correlation matrix (Table 1) was calculated to detect redundant items, and to give confidence that questions pertaining to specific themes were sufficiently related to one another. There were no redundant questions in the final version of the questionnaire. The readability for the survey achieved 82.3 (age 11 to 12) using the Flesch-Kincaid reading ease scale (using https://readable.io/text/). The frequency distribution of the questionnaire responses support an assumption of validity (frequency distribution shown in Table 2). The possible total scores ranged from 14 to 70 for the questionnaire (the greater the score, the more positive the overall impression of the study website). The mean scores for the three themes were: functionality, 3.99; personal contact, 3.89; and trust, 4.03; suggesting an overall positive perception of the study methodology. Table 3 shows the frequency distribution of the total score as the percentage of responses that fell within the questionnaire range. 99% of the questionnaire responses had a total score above the median score of 41, which indicates an overall positive impression of the study website and the themes contained within the questionnaire [21]. Table 4 lists the common themes yielded by analysis of the free text responses.

Table 1: Correlation Matrix of Survey Questions.

Table 2: Questionnaire items and frequency distribution (n and %) of responses (n=149).

Table 3: Distribution of Total Scores (n=149).

The Functionality theme (Figure 2) was associated with whether patients went on to be randomised (t-test statistics=4.8, d.f.=147, p=<0.0001), such that lower scores were associated with a reduced likelihood of randomisation. Participants who were more comfortable with the lack of personal contact (Figure 3) offered by the TIME study were also more likely to be randomised (t-statistic=2.51, d.f.=147, p=0.0132). Concerns relating to privacy and protection of patient data (Figure 4) were not statistically associated with being randomised (tstatistic= 0.64, d.f.=147, p=0.5254).

Figure 2: Plot for Functionality Theme (%).

Figure 3: Plot for Personal Contact Theme (%).

Figure 4: Plot for Trust Theme (%).

The results from this questionnaire indicate that participant satisfaction was positive. Patients are concerned about privacy and confidentiality but the overwhelming majority of responders to the survey (Table 4) indicated altruistic attitudes to research specifically or took part for “the greater good” in general. Lack of face-to-face contact and a reduced sense of intimacy have been suggested as a disadvantage; however, in general, participants were comfortable and enthusiastic about the study being conducted entirely online. The concern of participants about privacy and confidentiality of their medical data, use and reliability of the required technology, and the generally positive attitude towards home telecare are in line with those reported in the published literature [6,9,21,25-29] and, therefore, support the construct validity of the questionnaire. Test-retest reliability will be conducted on the questionnaire at the end of the TIME study.

Table 4: Free Text Responses for Taking Part (n=149).

One weakness of any survey, whether self-administered, conducted by telephone or face-to-face, is that the instrument itself may promote change. Respondents can easily fall into role selection when answering questionnaires (i.e. respondents are likely to anticipate what the interviewer expects of them in an interpersonal situation and act accordingly) since no one is present to observe the role they are taking and challenge it. Specifically for self-administered surveys, a problem may arise if respondents misunderstand the questions or wish for clarification of some items [12]. It is not known whether these factors played a part in the responses received.

An issue highlighted by the literature as affecting survey responses is perceived anonymity and privacy. The anonymity of subjects is imperative, since individual’s responses to survey items may be affected by their fear of identification. It is important to ensure anonymity and to assure participants that their responses will only be presented in aggregate form without identifying them. It is not known whether fear of identification would have played a part in the responses we received, or if responders even considered this possibility.

The developed questionnaire has several other limitations. This study was unable to specifically contact individuals who had not completed consent or had asked not to be contacted when they did consent. Individuals with concerns over privacy or who have anxiety using technology are unlikely to have signed up to the TIME study and therefore could not complete the questionnaire. Data on why individuals did not consent or decided not to take part would have been invaluable and likely added more insight into the more meaningful changes needed to increase participation in the study. Given the small sample size, it is unknown how representative the questionnaire is of the general TIME study population. Improvements are necessary to render the questionnaire re-usable and generalizable to other research studies that also use information technology.

The questionnaire underwent several iterations in attempts to improve questions and make them more relatable to the themes the investigators wished to evaluate. This took longer than anticipated and ultimately reduced the data collected in the final questionnaire as study recruitment was drawing to an end. The prolonged development also prevented further improvements aimed at increasing the internal consistency of the trust and personal contact themes. On-going development of the questionnaire will be necessary to strengthen its validity.

Electronic data capture systems have been proven to improve data quality over time. Their digital nature, and online availability, can provide fast and easy insight into outcomes such as adherence to treatment protocols and patient satisfaction [30]. The developed questionnaire is an example of how web-based technologies can be used to evaluate clinical study technology. The questionnaire successfully identified the views of TIME study participants around the themes chosen by the investigators.

Participation rates in clinical studies have failed to improve over the last decade despite concerted efforts by researchers. Evaluation and feedback on existing studies will be essential to ensure that participants are not dissuaded from future research and that the technologies used meet with their expectations. The results of evaluation questionnaires such as described above can be used to improve trial procedures and electronic data capture tools. On-going development and assessment of participant satisfaction questionnaires will be necessary to improve the design and implementation of future technology-based clinical research applications.

Ethical approval and consent to participate

Ethical approval has been obtained from the Tayside Committee on Medical Research Ethics. MREC reference: 11/AL/0309. UKCRN ID: 17071. TIME is registered as ISRCTN: 18157641. The trial is performed in line with Good Clinical Practice guidelines and International Society of Pharmacoepidemiology (ISPE) Good Pharmacoepidemiology Practice Guidance. All participants have consented to taking part in the TIME study.

Consent for publication

All listed authors fulfil the requirements for authorship and agree to submission of the manuscript in its current form.

Availability of supporting data

Anonymised data from the study can be made available to bona fide researchers on application.

Competing interest

There are no conflicts of interest.

The TIME study has been funded by a grant from the British Heart Foundation, Greater London House, 180 Hampstead Road, London, NW1 7AW

Authors contributions

David Rorie conceived the idea. The initial draft of the manuscript was created by David Rorie. Thomas MacDonald, Robert Flynn and Amy Rogers reviewed and edited the manuscript. All listed authors fulfil the requirements for authorship and agree to submission of the manuscript in its current form.

Medicines Monitoring Unit, University of Dundee.