Assessing the content validity of the Manchester–Oxford Foot Questionnaire in surgically treated ankle fracture patients: a qualitative study

A qualitative deductive design was used to investigate patients’ response processes of the MOXFQ. Individual interviews were conducted using cognitive interviewing based on the theoretical framework of the 4-step model by Tourangeau (Fig. 2), which addresses the stages respondents undergo when replying to survey questions [33‐35]. The first step, the comprehension, focuses on how the respondent understands the question. Step two, the retrieval of relevant information, involves the information that the respondent needs to recall, and the strategy used to recall the information to answer the question. In step 3, labeled judgement, the cognitive effort used to evaluate relevant information to accurately answer the question is addressed. The last step, the response, is asking about the choices that the respondent makes when selecting a suitable response.

The license to use the Norwegian version of the MOXFQ was obtained from Clinical Outcomes at Oxford University Innovation (Additional file 1). The translation and cross-cultural adaptation process has been performed by a third party and followed a minimum methodology with at least two forward translations and one backward translation. The instrument includes 16 items distributed over three domains: (1) the pain domain (five items); (2) the walking/standing domain (seven items); and (3) the social interaction domain (four items) (Table 1). The items have a recall period of four weeks and are intended to be completed with responses given on a 5-point Likert scale, where zero indicates least problems and four indicates greatest problems. A score for each domain is presented on a scale from 0 to 100 where a higher score denotes greater severity [14]. The scores reflect the impact of foot and ankle problems on the specific domains. An overall score, referred to as the index score, is calculated in the same matter, except the sum score is obtained from all 16 items [36]. The index score aims to measure the overall impact of foot and ankle problems on health-related quality of life (HRQOL).

Patients who were surgically treated at the Stavanger University Hospital for a unilateral ankle fracture classified as AO/OTA 44 [37] and at least 18 years of age were potential eligible candidates. The exclusion criteria were: less than 4 weeks or more than 18 months since surgery; other major concomitant injuries; patients with cognitive impairment or inability to give informed written consent; inability to speak, read or write fluently in Norwegian; major illnesses, other injuries or surgeries affecting impact of the ankle fracture on HRQOL 12 months prior or after the ankle fracture; other dominant musculoskeletal disorders. Purposive sampling [38] was conducted to ensure heterogeneity in the respondent group concerning age, sex, time since surgery, fracture characteristics, and pre-operative physical status assessed with the American Society of Anesthesiology (ASA) score [39] (Additional file 2). The post-operative routines for surgically treated ankle fracture patients often include partial weight bearing, and follow-up appointments at two and six weeks. Patients are usually allowed weight bearing as tolerated (WBAT) after the consultation at six weeks. The selection of respondents was stratified into three groups: (1) early-phase recovery, i.e., less than three months after surgery; (2) middle-phase recovery, i.e., from 3 to 12 months after surgery; and (3) late-phase recovery, i.e., more than 12 months after surgery. The interviews were conducted in a quiet, separate facility next to the hospital or in the respondents’ home, if preferred by the respondent.

A semi-structured interview-guide (Table 2) was constructed in conjunction with a research team consisting of clinical researchers with background in qualitative research, validation of PROMs, and fracture treatment. Two pilot interviews were conducted to test and accommodate the interview-guide, resulting in modest changes, e.g., improving the wording to augment the interview’s intent. These interviews were not included in the analyses. Some questions regarding retrieval and judgment of relevant episodes were rewritten or removed for increased clarity, e.g., that the relevance of each question was directed toward the respondent’s own experience. Minor adjustments to the order and wording of the questions asked were also implemented as the interviews were conducted and analyzed (“Generally, what are your thoughts on the relevance of these questions considering your ankle fracture?”). To capture all important aspects of the response process, the interviewer adjusted his approach during the data collection, especially with use of follow-up questions, such as “how did you proceed in choosing your response?”.

The main researcher (MQN) conducted all the interviews. Initially, the interviewer administered the paper version of the MOXFQ to the respondent, asking the respondents to follow the instructions and independently complete the measure. The interviewer registered any facial expressions, body language or verbal cues indicating difficulty completing the form. After completing the questionnaire, the respondents were asked about the comprehension of the wording in the instructions. The interviewer then proceeded to explore their initial thoughts on the questionnaire (“In general, what are your thoughts on the questionnaire?”) [40], and the relevance and understandability of each item based on their experiences after treatment of the ankle fracture (“what are your thoughts on this question? What do you think the question is asking you?” [41]. They were asked about the appropriateness of the recall and response options [41]. Furthermore, they were asked about the length and time it took to complete the MOXFQ, and their thoughts on the questionnaire’s format. Comprehensiveness was assessed by prompting the respondents about other relevant situations pertaining to the domains of the MOXFQ. Additionally, the respondents were asked if there were other topics that they felt were relevant to their experience with an ankle fracture, but not covered by the instrument [42]. Lastly, they were given the opportunity to add additional comments. Interviews lasted on average 37 min (range 18–60). The data sampling was an iterative process, where the data collection and analyses were reviewed by the research team as the sampling progressed. The interviews continued until no new topics were identified following three consecutive interviews.

In the deductive qualitative analysis, the initial code scheme was based on Tourangeau’s 4-step model of cognitive interviewing [43]. Each step was defined as a category. They were then divided into sub-categories based on the different parts of the questionnaire, i.e., instructions, items, and response categories. Texts that described respondents’ thought processes upon responding on the questionnaire were highlighted as meaning units. The meaning units were condensed and labeled with a code. The codes were sorted according to the appropriate predetermined categories. The labeling and categorization of codes was performed consecutively with the interviews, and the findings were discussed in the review team. The interviews were audio-recorded with a digital voice recorder (Olympus WS-853) and transcribed verbatim using NVivo by the main researcher.

In total, 17 respondents (65% females) were interviewed (Table 3). Respondents’ age ranged from 27 to 76 years. All respondents had at least 10 years of education. There were office workers and manual workers. Some of the respondents were retired. Respondents had uni- and multi-malleolar fractures. Additionally, one respondent had an open fracture. The fractures were mainly treated with open reduction and internal fixation. One respondent received a fibula nail, and one was treated with syndesmosis screws. The ASA score ranged from one to three. Some of the respondents presented with several comorbidities, e.g., one respondent had diabetes mellitus, hypertension, asthma, and morbid obesity. Another suffered from polyarthritis, hemochromatosis, and hypothyroidism.

Item 7, that asked about the avoidance of standing for a long time due to pain, had one missing response. During the interview, the respondent had to stop and think about it for some time when filling out the questionnaire because “I don’t stand that much, you know, so I haven’t thought about it” (r5). She was asked to elaborate on her thought process and the reason for the missed response: “… because I was thinking of work. But then I don’t stand that much at work. Like when I’ve been to airports, then I’ve avoided standing for longer periods. I would rather go sit down … It has just become a part of me, I think … I think it was just forgetfulness…” (r5). The same respondent also had some remarks on item 13 since the question addressed limitations in both work and everyday activities due to pain, and she struggled to pick a suitable response option: “… since I have an office job, it doesn’t prevent me from doing my job. But daily activities … Maybe you should split them?” Respondent 3 also presented the same reluctancy when replying to this item.

Respondents selected multiple responses on some items. For instance on item 6, which included hard and rough surfaces in the question, the respondent selected one response per adjective describing the surface: “… hard surfaces I don’t mind. I walk well on asphalt. But if there’s a slope, or especially a steep downhill, then you get this bend and you must sort of limit yourself a little…” (r15). Another item that had multiple response options selected was item 15. This respondent was still early in the recovery phase and found it difficult to pick one response option due to the use of the word usually in the question, with reference to some days the pain would be better than other days: “When it starts to hurt, then it really hurts” (r12).

To the best of our knowledge, this is the first qualitative study to explore aspects of relevance, comprehensiveness, and comprehension, i.e., the content validity, of the MOXFQ when applied to an ankle fracture population by investigating the response process. The study utilized a deductive approach to examine if the existing theory concerning life impacts for patients with a hallux valgus deformity could be accommodated to an ankle fracture population. Some crucial aspects were highlighted when applying the MOXFQ to this context. Firstly, pain was an important but difficult concept in the follow up after surgical treatment of ankle fractures and respondents perceived pain differently during the course of recovery. The items that measured pain and pain-related limitations in mobility were relevant for patients that were allowed WBAT. However, the subscale intended to measure limitations in social interactions was perceived as not relevant. Secondly, the index summary score was not comprehensive in measuring the overall impact of ankle fractures on HRQOL. Lastly, the study identified ambiguous wording that should be addressed to improve comprehension and relevance of some items when applied to an ankle fracture population.

The MOXFQ intends to measure two constructs of pain: pain in general, and pain when walking or standing. Pain is well known to have a great impact on a person’s overall quality of life [44] and is often used in the assessment of treatment results, both clinically and in research studies [45, 46]. In the work of developing an instrument to measure overall quality of life by the World Health Organization (WHO) group, six broad domains were identified: physical, psychological, level of independence, social relationships, environment, and spirituality/religion/personal beliefs [47]. In this multi-dimensional construct, pain and discomfort were encompassed in the physical domain. However, pain itself is not considered a unidimensional concept. For instance, the McGill Pain Questionnaire assesses quality and intensity of pain [48]. Additionally, patients with chronic pain report poorer overall quality of life compared to patients with acute pain [44]. Therefore, it is understandable that the majority of items in a questionnaire developed to assess a hallux valgus deformity condition consist of questions pertaining to pain. However, the pain in patients suffering from an ankle fracture is of an acute nature. One could argue that patients with these two conditions would experience some of the same challenges and limitations in the recovery phase. For instance, there are over 130 different procedures recommended for the hallux valgus correction [49], but similar post-operative restrictions in weight bearing would apply to both groups. Additionally, the pain experienced by the patients in early stages after the procedure would typically be related to the surgery itself. One would also expect the remission of pain and disability within the first one to two years after surgery [50]. In other words, using MOXFQ for patients in the middle-phase recovery, i.e., between 3 and 12 months after surgery, seems reasonable. In early stages, before WBAT is allowed, one could risk measuring limitations in mobility or in social interaction due to post-operative treatment regimens, and not limitations due to pain as the domains intended. Also, if used earlier than four weeks after surgery, the recall period would include the pre-operative phase, which can be confusing for the respondents. In late phases, ceiling effect (lower scores indicating less disability) could be a potential threat to the instrument’s content validity since patients perhaps no longer associate their main restrictions with pain.

Overall, the instrument seems capable of capturing key elements of the construct pain and pain when walking or standing. However, the suitability of the items that covered limitations in social interaction was less convincing. Particularly items 9 and 10, where the questions focused on being limited due to people’s view on the appearance of the injured foot or the use of footwear. Although a few would find item 10 more relevant than item 9, e.g., some might experience swelling resulting in the need to use bigger shoes, or prominent osteosyntheses with ensuing discomfort when using hiking boots, the recurrent notions for these items were “… That’s my business! So that I don’t care about” (r14). This might be attributed to the development of the instrument, where the instrument’s items were based on patients with a hallux valgus deformity scheduled for surgery, and not an ankle fracture population. A main difference between the two groups is the medical indication for treatment, i.e., hallux valgus correction procedures are mainly performed for the relief of chronic pain, while the treatment of ankle fractures is an emergency or urgent procedure. Another important difference is the sex distribution between the groups. Though, biological factors probably play a role in the development of the hallux valgus deformity, most likely the pain and disability associated with the condition are accentuated by less physiological favorable female footwear, demonstrated by the higher proportion of females undergoing this procedure [51]. In contrast, both sexes would be affected in an ankle fracture population. Additionally, one could not neglect the potential cultural difference: “… I believe that we have sensible footwear in Norway. Nobody cares if we wear hiking shoes downtown” (r9).

As 14 of 16 items include pain or pain-related limitations, a summary score labeled the MOXFQ-index was constructed to measure the overall life impact on HRQOL [36]. Where the MOXFQ subscales are clear on the construct they intend to measure, the MOXFQ-index was proposed to measure a more general construct. However, respondents in this study reported the lack of questions addressing other physical symptoms, e.g., swelling and stiffness, and psychological factors, e.g., worrying about complications or fear of falling again. If compared to another study assessing the life impact of ankle fractures [52], the MOXFQ-index lacked constructs to cover the entire aspect of life impacts in patients suffering from ankle fractures, e.g., psychological and financial factors, and medication taking. One could claim that the items of the MOXFQ-index to be relevant in assessing one of the aspects of overall quality of life, i.e., pain-related life impact of ankle fractures. However, the instrument would not be comprehensive in assessing general life impact due to the absence of items addressing other physical and psychological factors. Even if it is simpler to use an index score for the assessment of patients in clinical practice or analyses in a research setting, one should be wary of these limitations when applied to an ankle fracture context.

Some of the items received multiple responses, resulting in uncertainty in the interpretation of the respondents’ answers and complicates the scoring. If two responses have been selected, the user manual for the MOXFQ instructs to select the most severe response option when calculating the scores. Items where multiple responses were selected may indicate some degree of ambiguity in the questions, e.g., item 6 in the walking/standing domain concerning rough and hard surfaces, or item 13 in the social interaction domain. The wording of the latter item addresses both work and everyday activities, leading to discussions on which activity to consider when replying to the question. These questions are referred to as double-barreled questions where “respondents must answer two questions with one answer” [53], and should be avoided since they could potentially have adverse effect on validity [54]. A suggestion from one of the respondents was to split the question to enhance the intent of the question. Item 15 from the pain domain faced a similar challenge with the use of the word usually. One respondent (r3) from the late recovery phase interpreted the question as if it asked about pre-injury status. Another respondent from early recovery phase explained that the pain varied too much during the last weeks to be able to pick only one option. This reflects one of the challenges when changing the context of use for an instrument, exemplified in this case by applying an instrument intended for patients with stable conditions undergoing elective surgery to patients being treated for emergent or urgent conditions. On the other hand, being too specific in the wording of a question renders it less relevant for many patients, e.g., item 8, which focuses on the need to use secondary transportation methods. The interviews consisted of respondents living in cities and on the countryside, and a repeating voice from the latter group was the need to use a car irrespective of the injury or pain.

The findings of this study entailed patients suffering from an ankle fracture. However, it would be fair to believe that the implications made in this study would also account for patients presenting with similar fractures in the foot or distal tibia undergoing emergent or urgent surgery. Future studies should focus on further validation of the MOXFQ’s measurement properties in a similar context, e.g., validation of the validity, reliability, and responsiveness.

There are no published studies on the translation and cross-cultural adaptation of the Norwegian version of the MOXFQ. The lack of documentation of the translation and cross-cultural adaption might affect the validity of the instrument.

The research team consisted of five researchers with expertise in different fields. The main researcher was a resident orthopedic surgeon and Ph.D. candidate and received guidance from two R.N. Ph.D. professors with extensive experience within this field (AB, MMI). The team also included two consultant orthopedic surgeons with Ph.D. with experience in PROM validation and some experience in qualitative research (AP, KH). The broad expertise of the team contributed to increase the dependability of the study, aiding in decision-making during the analysis process. However, since the main researcher conducted all interviews, there was a risk of introducing bias to the data collection, interpretation, and creation of text due to the orthopedic background. Being aware of this, the interview-guide, transcripts, predetermined categories, and codes were continuously discussed in the research team during the analyses as part of the process to maintain confirmability.