Introduction
The following questions pertain to a experiment carried out by a software development team working alongside scientist to build a platform that responds to data change. This experiment introduced a new software that designers hope will be easy for scientist to use while fostering interchangeability of data. The software developed is called DAC-MAN but will be referred to simply as ‘the software’ in order to allow the evaluation to focus on the research methods employed instead of the research outcome.
1. Provide the citation and attach a pdf of the article
User Research
Paine, D., Ghoshal, D., & Ramakrishnan, L. (2020). Experiences with a Flexible User Research Process to Build Data Change Tools. Journal of Open Research Software, 8. https://doi.org/10.5334/jors.284
2. What is the abstract of the article?
Abstract:
Scientific software development processes are understood to be distinct from commercial software development practices due to uncertain and evolving states of scientific knowledge. Sustaining these software products is a recognized challenge, but under-examined is the usability and usefulness of such tools to their scientific end users. User research is a well-established set of techniques (e.g., interviews, mockups, usability tests) applied in commercial software projects to develop foundational, generative, and evaluative insights about products and the people who use them. Currently these approaches are not commonly applied and discussed in scientific software development work. The use of user research techniques in scientific environments can be challenging due to the nascent, fluid problem spaces of scientific work, varying scope of projects and their user communities, and funding/economic constraints on projects.
In this paper, we reflect on our experiences undertaking a multi-method user research process in the Deduce project. The Deduce project is investigating data change to develop metrics, methods, and tools that will help scientists make decisions around data change. There is a lack of common terminology since the concept of systematically measuring and managing data change is under explored in scientific environments. To bridge this gap we conducted user research that focuses on user practices, needs, and motivations to help us design and develop metrics and tools for data change. This paper contributes reflections and the lessons we have learned from our experiences. We offer key takeaways for scientific software project teams to effectively and flexibly incorporate similar processes into their projects. (Paine et al., 2020)
3. Was the study experimental or non-experimental? Explain, tell us what made that clear
The research conducted by the team would be classified as quasi-experimental for its inclusion of some elements that make up an experimental study. However, it is not a true experiment. If I was to classify this definitively as experimental or non-experimental, I would select experimental. This study includes the introduction of a treatment. The treatment being the software program they developed to aid scientist in dealing with data-change. However, the experiment only somewhat covers the basis needed to be a experimental study due to its lack of a control group. Initial interviews were conducted on twelve scientist and six utilized the new software. This seems like an identifiable control group but their is no other mention of the other six individuals results. Instead, the experiment utilized the feedback from the six scientist who utilized the software to improve the platform. This adds to my affirmation that it is, at best, a quasi-experimental study. Additionally, a pre-test was given and the participants were selectively chosen, further alienating it from experimental requirements. An argument may come up concerning how the data was empirically obtained by observing the users interaction with the interface and thus, may be considered non-experimental. The introduction of a treatment (the new software program) should emphasize the difference between observing something newly introduced and something in its natural place.
4. Was the research qualitative or quantitative? Again, explain.
This was an example of qualitative research. The study was conducted utilizing interviews that coincided with observation and notetaking. The study was carried out with one small quantitative variable being accounted for and that was a record of how many of the assigned task the user could get done in an hour. Outside of this, there were no timetables or quantifiable numbers to account for. Instead, the researchers utilized semi-structured interviews to pinpoint the problems within both their software and the scientist work-flow.
5. What was the population studied?
The population of individuals targeted for this study includes people in scientific fields that work with data with the potential to change. This is intentionally established to include a large number of possible users that could provide multiple perspectives on how to implement the platform. Electing to target a large population allowed the researchers to obtain data on how the platform would be used within multiple fields.
6. What sample was used for this study?
The sample that received the treatment within this study included six scientist whom “had experience working with scientific datasets, tools, and computing systems” (Paine et al., 2020). Two of the participants were computer scientist, two were astrologist, one was an earth science specialist and the final was an electromagnetic spectrum expert. The sample also represented an array of how the platform would be utilized with some experts (the astrologist) computing with large numbers while others, (the earth scientist) working with comparatively smaller ones.
7. What was the method of measurement?
The experiment measured how smoothly the software they introduced could be implemented within the scientist current workflow. This was done to the benefit of the research team who is looking to improve the software and identify friction points in its use. The measurement methods within this experiment fall mostly in line with an achievement test scenario. Here, the researchers are testing participants knowledge based on some limited instruction. Another hallmark of achievement test is found within the list of task to accomplish that the researchers provided participants with. Other metrics measured included ease-of-use, documentation evaluations, and platform reliability.
8. What was the method of analysis?
This research was analyzed using qualitative analysis. The questions answered included, How can this software be incorporated? What can we do to improve the software? Why would this be useful to scientist? ect. The data is collected after direct observation of individuals using the software but the data analysis may be considered in line with grounded theory principles. The Researchers did not formulate a hypothesis prior to conducting the test. Instead, they generated theories of how they could improve their platform only after conducting the usability test. Throughout the test, they looked for common problems, or themes, that they could address. This is in line with Grounded theory research.
9. What was the conclusion of the study?
The research team was able to conclude that the usability test was instrumental l in helping them improve their software. The developers affirmed that being able to witness scientist from multiple fields allowed them to adapt the platform to each specialist individual needs. Developers also concluded that “Teams need to thing about the who, what, when, where, and why as they incorporate user research techniques” (Paine et al., 2020). This quote emphasizes the implementation of qualitative research.
10. Why is this study useful to you? Explain in detail
This study was an excellent example on how to conduct a usability test in a flexible manner with a more specific goal in mind. I found the study to be enlightening as it really combined my current academic studies in this research course with courses I have taken in the past that pertained to usability. This study can surely be referenced as an example of how to pursue constant improvement when building a software platform. The choices they made differ from that of other usability principles I have read about but that is the nature of research. One aspect I found useful within this text I have yet to discuss is their section discussing how to eliminate bias by paying attention to the timing of your questions and trying to avoid assumptions. When doing these kind of test, it is easy to impose your will on how to utilize the application. That is to say, you designed it and want it to be used in a way you intended. But it is important to remember not to do so when conducting the test and the researchers cover that base well.
11. What would be the next logical step in extending this study?
The most simple answer, and the one with the most relevance in regard to usability testing, is to make the improvements that were identified as necessary and proceed to test again. I do think that they could conduct a test that falls more in-line with a true experiment. This could be done by assigning task to be completed with scientist current software and assigning the same task to be completed given the new software. This would allow the experiment to be conducted with a control group.
References
Paine, D., Ghoshal, D., & Ramakrishnan, L. (2020). Experiences with a Flexible User Research Process to Build Data Change Tools. Journal of Open Research Software, 8. https://doi.org/10.5334/jors.284
Patten, M. L., & Newhart, M. (2018). Understanding research methods: an overview of the essentials. Routledge.
First off, thank you for always being so on top of your assignments each week. When I woke up this morning and saw that I had an email saying you commented on my post from last week, I thought “I’m have to go catch up to Benjamin!”. No pressure for commenting on my stuff next week…but thanks for now.
I enjoyed your article review. You have a really great understanding of the text and did well with incorporating what you’ve learned into your response to the article. I agree that interviewer bias may be hard to avoid when listening to the scientists’ responses to the software program. I had two thoughts as I was reading about the study:
1.) Would the researchers have gotten better feedback with a larger experimental group? The scientists’ specialties were pretty diverse and it makes me wonder if the feedback was limited because of it.
2.) Could the interviewers eliminate bias by having non-scientists complete the interviews at the end? This may keep leading questions out of it. Or maybe participants could respond to a questionnaire with their thoughts?
Software development is always a challenging process. It is more so when you are designing a software solution for a niche thing like data change in scientific fields. I can see how they greatly benefited from a usability test. Since the people who write software and the people who will use it are not from the same background having it tested by people who will then use it is a good idea especially in the case of a software like this that is intended to be a specific thing and not a catch all generalized product. I also can see how doing more in depth study on their core use case was the right way to develop this type of software.