How to assess the METHODS section of a research article (Part 2 of evaluating a research paper)

Hi everyone, Welcome back So, in this second video in the series on evaluating a research paper, we will discuss the methods section To be honest, no, in a brief YouTube video of 20-25 minutes, we cannot make you an expert in judging the methods section of a research paper For that we need detailed discussions on various topics like study designs, sampling, randomization, etc. etc And indeed, in the future, we will have detailed videos on all those topics In this particular video, we will have an overview of the major points that you should remember when you are going through the methods section of a manuscript So, the first thing that you have to pay attention to is what is the design of that particular study Here is a brief reminder about the major study types that we frequently encounter: descriptive study, cohort study, case control study, cross sectional, experimental, qualitative and all So, usually, in the title itself, the authors may have mentioned Or, after going through the initial paragraphs of the methods section, you can know that this particular study is a cohort study, case control study, like that So, you can make some impression about what all conclusions are possible from a design like this. And also, after that, you have to have a look at whether the design the authors have used, whether that is appropriate for the kind of research question they have In the previous video, we discussed about, like, how to assess whether the authors have raised a proper question and also that particular question and this design, whether both of them are appropriate to each other This particular aspect you have to assess in depth So, like, for the usual kind of questions, what are the most appropriate research designs that the authors should use. For a prevalence study If it is a prevalence study, the appropriate design to assess the prevalence of a condition in a particular population is to use a cross sectional design For example, see the title of this particular study: a cross sectional study of prevalence of depression, anxiety and stress among professional cab drivers in New Delhi So, basically, they wanted to study the prevalence, and they have rightly used the cross sectional design So, this is it For a prevalence study, it should be of a cross sectional design Now, if the authors are planning to study the incidence of, say, a particular disorder in a particular population So, for that, the kind of design that they should use is a cohort design What exactly happens in a cohort study, I will briefly summarize for you Like, what you are doing is, we have two groups And this indicates, basically, exposure So, this particular group of people have been exposed to a risk factor while this particular group has not been exposed So, then what do we do is, usually we follow them up for an amount of time to see, like, how many number of, like, people develop the condition These blue people, they have developed, the blue colored people, they have, like, developed the outcome, whatever outcome you’re studying, for example, developing stroke or developing depression or whatever And also, in the group that did not have the exposure, in that group also you see how many people developed the condition So, by comparing the number of people in these two groups, number of people who develop the condition in the exposed group and in the non-exposed group, comparing the two – these two – Suppose in the exposed group, the outcome is higher, in that case, you can say that this particular exposure is a risk factor for this particular outcome For example, like, this is a study titled “Postpartum depression in a cohort of women from a rural area of Tamil Nadu, India: incidence and risk factors So here they have the incidence, and also risk factor we’ll, like, discuss this here and the next subsection also So we will be seeing about the risk factors So, what these authors did here is, they wanted to study the postpartum depression in a cohort of women from Tamil Nadu So what they did was, like, they had, like, two groups of women, one group of women had, say, one risk factor, like, psychosocial stressors in the family, poverty, things like that, while the other group did not have that particular risk factor So after that, they followed these two, like, groups of women for some time In this particular study, they followed up for about three months And after that, they saw, in this group that had the exposure, say, like, poverty, in that particular group, how many women developed postpartum depression and also in the group that did not have that risk factor, how many people, how many women, developed postpartum depression So they get the incidence in the two groups and also, like, by assessing the prevalence

in, sorry, the incidence in groups that had and did not have the exposure, they can study whether they can find out whether this particular risk factor contributed, factor is actually a risk factor to the development of postpartum depression To identify the risk factor, as we discussed already, the cohort design is usually used. We already saw that, how it was used to identify the risk factors for postpartum depression I also told you that in that particular study, the follow up was for a brief period of about three months Usually cohort studies follow up for a much longer period For example, see this one: the absence of postive psychological eudemonic well being as a risk factor for depression: a 10 year cohort study So, basically, what they did was, they had two groups: one group of people had positive psychological well being; the other group did not have positive psychological well being And then they followed up these two groups of people for a period of 10 years and they compared the, like, what is the rates of development of depression in the group with and without this particular, like, positive psychological well being. So that is a cohort study of identifying whether this particular factor, absence of that, is a risk factor for the development of depression More commonly, people also use the case control design for the identification of risk factors So, in this particular design, what usually happens is, for example: depression as a risk factor for ischemic heart disease in men: population based case control study So, in kind of, this kind of design, what happens is, we have, like, two groups Here, actually, the illness, the outcome, has already appeared So, here, to start with, we have two groups: one group of men have already developed the ischemic heart disease, the case status They are already cases of ischemic heart disease, while we have another group of men who do not have ischemic heart disease So, like, the case status is already there In the cohort study, you know that, like the case status develops afterwards, like, postpartum depression developed later But here, when we are selecting itself, the sample already has the caseness – one of the groups has already developed ischemic heart disease Now, what we do is, we go back in time and check whether, like, what is the rate of the history of depression in these two groups So, like, we can see here that in this particular group, the patients, the people, who subsequently developed ischemic heart disease, these particular people, this many number of people already have a history of depression, while in the other group, the people who do not have ischemic heart disease, this many number of people have a history of depression So, then, comparing these two rates, using various statistical methods, you can see, like, whether depression, a history of depression is actually a risk factor for development of ischemic heart disease So, this is how the authors should be doing If they are using a case control design to identify, to find out the risk factors Now, if the authors had wanted to assess the prognosis of a particular condition, then the design they should have used is a, again, a cohort design, as we saw already So, here what happens is, for example, here is a study: a cohort study of the prevalence and impact of comorbid medical conditions in pediatric bipolar disorder So, in a study like this, what the authors do is, again, they already have a cohort, like, they have two groups of children with pediatric bipolar disorder. One of the groups, the members already have comorbid medical conditions, while the other group, they do not have comorbid medical conditions. Then these two groups are followed up over a period of time to, like, asses how many people develop the relapses to mania or depression in each of these groups, like that So, then, suppose, in the group with the comorbid medical disorders, the rate of relapse is high, then you can say that they have a, like, worse prognosis, like that So, cohort design is the best design, the design that the authors should be using if their objective is to identify the prognosis, the influence of certain factors on the prognosis of the condition under study Now, obviously, if the aim is to find the treatment effect of a particular agent or an intervention, then the optimal design for that the authors should have used is controlled trial So, like this, you check what were the objectives, what is the design the authors used, and the design is the best one, the appropriate one for that particular objective This is how you evaluate the appropriateness of the design Now, one thing you have to pay attention to is, like, whether they have described the

methods in sufficient detail Especially, like, the description of the methods should be such that if you want to replicate the study, if you want to repeat the study on your own, all the information that you require for that should be there in the methods section That is how you judge whether the methods is sufficiently elaborate, it is complete And especially if you are doing the analysis, critique of a journal article for your journal club or something, in that case, what, or maybe, if you are doing the peer review for a journal, in that case, one of the tools that you can use to assess whether the reporting is complete is to use the checklists, various checklists that are available in this website: Equator Network For each of the designs, randomised trials, observational studies, etc., etc., they have different different checklists So, what all items should be reported in the methods section, it will be available in the checklist So you can compare their list with whatever the authors have, like, included in their article So, that way also you can check for the completeness of the report Now, another aspect that you have to pay attention to in the methods section is the kind of tools theyave used Whether those tools are sufficiently reliable and valid Especially in psychiatry and psychology, people will be using, the authors will be using, uh, lots of rating scales, to assess various variables, to, various qualities So, like, especially in India and all, if the scale is in English, and especially if this is a self-rated scale, then the authors have to translate that scale into the local language and ensure that it is sufficiently valid and all. So, like, especially in some studies, the authors may mention like this: the fact that the Hindi version is not validated should be considered when interpreting these results. This they mentioned in the limitations in the discussion section So, you can see, like, whether in the method section, they have described the details, so, how they did the validation, whether they did not do the validation and all. And if that tool is not something that is sufficiently validated for that particular population, you can suspect, like, how dependable or reliable the findings they got are And also, whether the tools are appropriate. Even if they have used valid tools, whether those tools actually are the appropriate ones for that particular purpose Many a time, authors make mistake in this particular aspect also For example, see this study Especially epidemiological studies, many in Psychiatry, like, they do this mistake. Prevalence of depression, anxiety and stress among adule population: Results of Yazd health study. So they want to study the prevalence. That is an epidemiological study So the tool they used is DASS Depression, Anxiety and Stress Scale was used That they have mentioned And in the same paragraph, subsequently they’re saying that it is a short screening tool So with a screening tool, you cannot make a diagnosis of depression or anxiety or stress like that So, like, if you want to assess the prevalence, after using a screening tool, you have to use either a diagnostic clinical interview or one of, some of those diagnostic tools like SCAN and all that we have in psychiatry So, if it is not done, you cannot say that the authors used the right tool You cannot say that this is the prevalence of depression So, these kinds of mistakes, whether it is there, that you have to assess Also, you have to check the methods section to see whether the authors have, like, there are any biases in their methods Many a time, it is there. Like, as we will see in the coming examples, many a time, the authors may mention that for this particular bias, there is a possibility But many a time, you have to find that out for yourself There are lots and lots of biases actually; we will be discussing only the major ones This is an old article. It came in 1979 in Journal of Chronic Diseases. That one lists nearly 50 biases that can be there in clinical research, especially in the biomedical arena So we will see only the, some of the major ones Especially, one is selection bias, which is very common in even published literature What that exactly means is, like, when you are doing a study, you won’t be studying the entire population, right? You will not be studying the whole population who has that particular disorder or something What you’ll be doing is, you’ll be studying only a selected sample And after that, like, how you do that is, like, you pick one patient and you pick another patient, you pick another patient, you pick another patient and after that you assess that particular sample But anyway, but your intention will not be to study what happens in this sample Your intention is basically to… to generalize whatever you find in this sample to the population

from which they were picked That is how this entire inferential statistics, or your P value, everything works based on this, like, part only So, like, you have to select some sample from the, your population and you study them and whatever findings you get, you want to generalize back to the population But what can happen is, the sample that you selected, th at may not be fully representative of the population So, when that happens, we can say that you have a selection bias For example, this is a study that we published in the journal that I edit: Indian Journal of Psychological Medicine: Substance use related emergencies in a tertiary care general hospital setting: observations and discussion And in this particular article, the authors themselves say in their discussion section that there might be a selection bias in the sample, as patients with more severe medical comorbidities are less likely to be referred for psychiatric evaluation So, what happened in this study is, in their center, they have an emergency setting, the casualty setting There, lot of people with substance-use-disorders-related complications are brought So, many of those patients will be referred to the psychiatric team in the emergency So, they analysed the register that they have in that psychiatric emergency department and what all patients came there with substance-related, substance-use-related emergencies. They, that is what this particular study is about But the authors are rightly pointing out that there is a possibility of selection bias because patients with more severe medical comorbidities, they may not have been referred – like, for example, somebody is unconscious, somebody has severe aspiration pneumonia and all – in those situations, those people may not have been referred to the psychiatric, like, emergency team who are waiting there So, like, again, I am coming back to this particular diagram. Like, if you want to, like, generalize whatever findings they got in the sample, like, whatever group was referred to them, the psychiatric emergency team, they want to be able to generalize their findings to all the people who come to the emergency department, that is their aim. But unfortunately, we cannot do that because many people are missing – as I told you, like, those who have severe complications, they may have been taken directly to the ICU Some of them may be already dead and they may be handled accordingly and all So, you cannot, like, correctly generalize back, whatever findings you got, to the entire population, entire group of people who come to that emergency department. You can only refer, like, generalise, to, like, people with less severe complications who are likely to be referred to psychiatry So, that is what is selection bias means The sample from which you are collecting the information is not representative of the entire population There can also be recall bias Again, let me show you an example Especially in psychiatry and psychology, you may be asking for a lot of history from the past – what happened in your childhood and all But many people, they may not remember or depending on whatever psychological psychiatric condition they have now, based on that also, their recall may differ So, that is what is recall bias This is one study, again, published in the same journal: gender specific correlates of alcohol use among college students in Kerala, India So, what they tried to find was, they studied some college students and they asked them about various, like, historical data also, from their childhood and all. For example, they asked whether the sample had any symptoms of ADHD – attention deficit hyperactivity disorder Whether they had hyperactivity in the childhood, whether they had inattention in childhood and all Whether those, those people who had such symptoms, whether there is a higher chance of alcohol use – that is what they studied to, wanted to study But again, in the discussion section, they specifically mention that there is a possibility of recall bias. Because the subjects were asked to retrospectively recollect childhood symptoms, limiting its validity So, many people may not remember exactly, correctly, what kind of, like, whether they had hyperactivity in the childhood, whether they had inattention in the childhood and all. So, because of that, as the authors rightly point out, there is a possibility of recall bias Again, many authors may not specifically mention that there is a possibility of this kind of bias. As the reader, as the reviewer, as the researcher, as a student, you have to reach at your own judgement using the kind of skills that you are learning from this presentation Another problem that will be present is called confounding Again, I will first explain that with the help of an example Initially, there were some studies which found that those who drink coffee – they have a higher chance of developing various cancers So many people were afraid; they even thought about leaving coffee forever and all

But subsequently, some other studies came which actually showed that it is not the coffee that is the culprit here But, most of those who drink coffee excessively, they also are smokers So basically, it, the cancers were result of the cigarettes not coffee itself So here, basically, in the initial studies, we can say that the cigarettes were a confounding factor The authors did not consider the confounding effect of the cigarettes. Because of that, they falsely assumed that the coffee was the carcinogenic agent So this is what is called confounding Let me show you an example from psychiatry In this study, what they assessed was psychiatric admissions of low income women following abortion and childbirth So, they had two groups of women: one group have had a normal childbirth while the other group had abortion So, the authors studied whether there is a difference in the rates of development of various psychiatric disorders, the rates of hospitalization in psychiatric facilities, etc., etc And they found some odds ratios. And they found that the women who had abortion, they were at higher odds of developing various disorders like depression, bipolar disorder, and all And also, higher odds of getting admitted to psychiatric facilities So it, like, you may feel that abortion leads to development of psychiatric illness. Because they have higher odds. The ladies who had abortion had higher odds of developing psychiatric illness. But again, here is, like, in this particular study, as pointed out by some of the subsequent authors, there were a possible, there was the possibility of some confounding factors For example, factors like financial concerns, worries about the relationship, lack of a relationship – Some women may have these and these factors may lead to abortion. Because they may not feel secure about their financial abilities to raise the kid, whether their relationship with their partner will be stable enough to grow the child, and all. So, when these concerns are there, because of that, they may have decided to get an abortion And also, these kinds of factors, the financial concerns, lack of a relationship, these are also risk factors for development of psychiatric illness also So confounding factors like thess may have led to both abortion and psychiatric illness Like, in the previous example, we saw, no? Like, those cigarettes. They lead to cancer and along with cigarettes, you drink coffee or so So the association between coffee and cancer came. Like that, the real association may be actually with these kind of factors and psychiatric illness, not with abortion as such . So that is what confounding means So, again, many authors may mention that there is a possibility of confounding, like this But many authors may not do that; you may have to make a decision for yourself In this particular study, as I told you earlier, they had found the odds ratios Like, these are the odds of this particular group developing these particular outcomes But, actually, these kinds of odds ratios do not take into account the possibility of confounders What the authors should have done is, they should have assessed, studied, this particular, like, whatever confounding variables you can guess based on the previous research, based on theory, based on your own experience, and all You have to assess them. You have to study their roles also. Then you have to statistically adjust for the roles of the confounding factors Only then you can say that your, like, variable, in this case, abortion, it has this many, this much, kind of, degree of risk, confers this kind of risk to development of this particular outcome For example, this is another study: medical comorbidity in women and men with schizophrenia So, here, like, the authors have indeed, rightly, identified some of the possible confounds Then they, like, statistically adjusted for that as I told you. You can see here, in their article, the odds ratios were adjusted for age, gender, residence, non-mental health care utilization using logistic regression This is the statistical method they used For example, like, the medical comorbidity, no? That is, even if people have a medical comorbidity, it may not be simply because of their gender, simply because of their schizophrenia It may be because of other reasons also Those who are of older age, there is higher risk of medical comorbidity People from certain areas, maybe areas with higher pollution and all, they are at higher risk for medical comorbidities and all, like that So, the authors assessed those factors also. Then they statistically controlled for those factors and after that they are giving you the adjusted odds ratios So, that is how the authors should be adjusting for the possible confounding factors Even after this, there may be confounders that they did not know about, that did not analyze So, even then, you cannot be very sure, but at least you can be confident that this many confounding factors were taken care of I told you that even if you do a logistic regression, we cannot be sure that all confounders have been taken care of Because, many confounders, the authors may not have thought about, they may not to have measured it. So they cannot statistically adjust

for that. So, the best method the authors can use to protect their study from confounding is called randomization. All of you know about that from randomized controlled trials and all So, what happens in randomization is, suppose this is your sample So, randomly, this sample is divided into two groups Randomly means it is completely by chance You cannot predict that this particular patient is more likely to end up in this group The chances of one particular sample, member of the sample, one particular patient, ending up in either of this group is, like, equal So, for that, people usually use various techniques including, like, a computer generated random number tables like this For example, in this one, you can see that the first patient goes to the intervention group, second to the control group, third to the intervention, fourth to the control, and 5, 6, 7 to the intervention, like that So, from this kind of table, it is completely random that you are allotting the, from the sample, members of the sample, to the two groups. So, both of them may be intervention groups of one of them may be control group Whichever way, the chances of your sample ending up in either of these two groups should be equal So, after that, what you do is you do the intervention, be it a placebo intervention or the active intervention to both these groups. And after that you study the outcome in the two groups. Suppose it is a drug and a placebo Then how many people improve in this intervention group, how many people improve in this placebo group, and you statistically, like, compare the numbers and you may find out that statistically, there is a possibility that, like, more people improve in the intervention group, then you say that the medicine is effective So that is how randomization helps So, suppose, again, there is a difference in the age or difference in gender or residence and all, as we saw earlier, but, like, because of randomization, like, it is more likely that the confounding variables will be equally distributed in the two groups Like, that is how you protect against randomization, sorry, protect against confounding, using randomization Now, in the randomization studies, in their method section, you will have this CONSORT flow diagram So that you have to pay a lot of attention to Because here, they will have given a lot of useful information: how many patients, what number of people were screened initially? Then how many of them are not included? And what exact reasons? For each reason, how many people were excluded? All this specific information you can get How many people were randomized into both groups? Then, after that, if, suppose one or some of the patients did not receive the intervention in either of the groups, how many of them did not receive? If no, they did not receive, why they did not receive? And the follow up, like, how many people came for follow up in both the groups? If somebody did not turn up for a follow up, like, whether they completed the intervention or whether they, like, discontinued the intervention? If the reason is available, the reason is also mentioned, like that So, if you pay, especially for drug trials and all, you pay a lot of attention here, because, especially in the follow-up rates, there may be lot of difference between the two groups So that may explain your, the findings you got in your statistical analysis, not the real difference because of the medicine and all. So, all these areas How many people were analyzed? How many were excluded from the analysis in both the groups? If they were, people were excluded, for what reason? All this kind of information you can get from this table So you pay a lot of attention to this table, the kind of numbers that they have in both the sides of the flowchart But, I told you that randomization, RCT, the risk of confounding is less At the same time, it is not, even a randomized design is not fully safe from having biases. Even in RCTs also, we can have some biases, like, selection bias.. that we discussed earlier. And also other kinds of biases, especially performance bias This is an interesting study about the, like, selection bias in clinical trials with antipsychotics So, what the authors did was, in their center, there are lots of patients of schizophrenia. Many of them are taken up for various clinical trials, various efficacy trials of various antipsychotics So, the authors wanted to assess whether the people who are selected for the randomized control trials, they are representative of the, all the schizophrenia patients they have in their center You may remember this diagram. Like, only if, like, the sample is representative of the population, only then you can generalize your findings back to the entire population So these authors wanted to know whether, like, the findings that the RCT is in their center generate, based on the kind of sample that is selected, whether that can be generalized to all the schizophrenia patients. For that, the kind of their patients who are selected for randomised controlled trials have to be representative of all the schizophrenia patients that they get And interestingly, what they found was that it was not the case The patients who are selected for RCTs, they were younger, had a more recent onset of illness,

and had experienced fewer psychotic episodes in the past That is, because, like, when RCTs are there, they have a lot of exclusion critieria So, they may exclude people who have a past history of noncompliance, history of, may be, not coming for follow up, etc. etc So, because of that, like, this kind of selection bias appeared there in the study So, like that, selection bias can be there in various RCTs also. When you are assessing the quality of an RCT, you have to use your own judgment to reach a conclusion Performance bias What, what that means, like, usually there is a blinding. In drug trials and all, the patient does not know whether the patient is taking, getting, a placebo or the active medication The treating clinician may not know whether his or her patient is on a, on a placebo or the active agent But, in, like, some areas, like, your psychotherapy, counseling and all, it may not be possible to, like, blind. The clinician will know, when the person is giving a psychotherapy, counseling, the clinician will know that I am giving the active intervention only The patients who are getting thatwill also recognize that they are getting some structured therapy intervention So, here, blinding is not that easy So, because of that, in trials of these kind of non-pharmacological interventions, there is a possibility of performance bias. Here is a sample study: a randomised trial to assess the efficacy of a psychoeducation intervention on caregiver burden in schizophrenia So here, what they found was, what they did was, like, they gave a psychoeducation intervention, many sessions of psychoeducation to caregivers of patients with schizophrenia, and then, they wanted to find out whether if they give this kind of psychoeducation intervention sessions, then whether there is any, that leads to a decrease in the burden reported by the caregivers But in their discussion section, the authors make this point: we cannot guarantee that, like, there may have been a performance bias – because knowing the assignment to intervention or control has not biased the perception of caregivers to the questions posed in the assessment scales – that is, I will explain Like, the authors asked the caregivers, like, how is your burden and all But the caregivers already know, these particular people have spent so much time with me Suppose that if the caregiver belonged to the intervention group, the caregiver has received so many sessions of psychoeducational intervention. They will know that these people have spent spent so much time with me, if I still report a lot of burden, they may not feel happy and things like this So, I will give a better rating. This kind of performance bias may be there This is what the authors are explaining, that a possibility cannot be excluded If you are doing the risk of bias assessment in a particular study, this is a good tool that you can use: Cochrane risk of bias tool Using that, you can grade each study on various biases: selective reporting, incomplete outcome data, like that CONSORT diagram that I showed you earlier, from that you can get this information I won’t go into the details, but if you can use this particular tool, you can give one objective rating of the amount of bias that particular study had overall Also, you have to think about the generalizability Again, we discussed about that earlier. Like this particular, like, about the selection bias and all when I was telling you, no, whether it can be generalized back to that particular population. Not only that You have to also consider whether that particular study the findings are generalizable to your patients: that is more important to you, right? Whether the sample they had, whether the kind of exclusion inclusion criteria that they used, whether all that applies, in such a sample that they studied after so much filtering and all So, the findings will apply to your patients or not. That kind of decision you have to make for yourself And also, the ethical aspects Mostly the journals, good journals will not publish journals that, articles that had ethical issues But for yourself you have to check and ensure whether the study was pre-approved by an institutional ethics committee Whether the authors have got informed consent Or if it is a study on children, whether they have got the assent and all So that kind of aspects also, as you are judging the quality of the method section you have to pay attention to So that is all about the methods section Again, we will have further videos on the result section and discussion section soon And those of you who are new to this channel can have a look at some of the recent videos that we uploaded We will meet again within a week Thanks for watching