“To Tech or Not to Tech?” A Critical Decision-Making Framework for Implementing Technology in Sport

Excitement is the most appropriate response when taking an optimist's view of technology in sport. In this section, we outline just a handful of benefits practitioners can expect from successful technological implementation.

Technology can improve measurement precision and automate the process so that practitioners do not have to manually record data as they did in the past. For example, understanding the match demands to which athletes are exposed is foundational in the sport sciences.^15–17  Football is no exception, and time-motion analyses to understand football players' physical outputs (eg, total distance run in a match, time spent at different speeds) and physiological responses have been performed for decades.^18–20  Before the technological advances that are commonplace today, these time-motion analyses were performed using tape-recorded commentaries, video recordings, and film analyses. All of these notational analysis processes were extremely time consuming, often limiting researchers' ability to examine more than a small number of players in a defined number of matches.²¹  Technological advances, both through wearable devices²²  and optical tracking systems,²¹  now provide these physical output measures to researchers and practitioners in near real time.^23,24  Although these systems are not without error and vary among technological providers and systems,²⁵  many provide more accurate physical output data than estimates derived from manual notational analyses based on video and can supply these data on all players simultaneously and in near real time. In this way, technology and its efficiency have off-loaded weeks and months of practitioner and researcher time. If used wisely, this regained time allows a deeper dive into the information and may inform practice more thoroughly. For example, advances in optical tracking and wearable technology have now allowed a better understanding of how physical exertion in football players relates to such contextual variables as player position, stage of play, and teams or players being in or out of possession.^26–28 

The prevalence of chronic injuries in volleyball players is known to be high.^29,30  Of these injuries, jumper's knee (ie, patellar tendinopathy) is the most common. The injury mechanism is fundamentally an overload of the extensor mechanisms of the knee joint.³¹  With the introduction of IMUs that measure athlete jump counts, the sport-specific load requirements of the knee extensors could be quantified for the first time without manual video annotation of jump counts for all players.³² 

One author (K.M.) used his dual roles as a researcher and volleyball coach to scientifically evaluate such a technology and implement simple heuristics to inform decision making in coaching. As a researcher, he performed validation work on a wearable IMU for measuring jump count³²  to ensure that he could rely on the jump-count data being provided by the IMU software. Specifically, the jump counts from the accelerometers related very closely to the jump counts extracted from manual video notation, so he was comfortable that the errors were few enough to reasonably inform practice. As a coach, measuring jump loads for all players in training and matches facilitated a better understanding of the position-specific training and match demands and individualized athlete load management throughout the season. After a retrospective assessment of training and match demands, he prospectively planned and prescribed individual jump loads. This prospective prescription may have improved the capacity of the players' tendons to withstand the sports demands and prevent the development or flare-ups of such chronic injuries as jumper's knee. In his case, this informed decision-making process helped to mitigate the prevalence of overuse injuries (zero practice sessions or games missed due to overuse injuries) in the team's volleyball players and culminated in a national championship.

Although not all stories end in a championship, technological implementation allows for sport-specific and movement-specific load quantification that can inform practitioners' workload, recovery, and return-to-sport decisions. A multitude of factors affect the onset of any injury, yet an informed approach to load is certainly a beneficial addition to any injury-mitigation strategy.

Performance is multifactorial and requires adequate physical, mental, technical, and tactical expertise to compete at the elite level. The contribution of each element depends on the sport's demands and the characteristics of the individual athlete. Furthermore, sport is dynamic and ever changing as athletes pursue multiple phases over the course of a single season, including training, competition, and recovery. Sleep, recovery, nutrition, social factors, and lifestyle can all affect athletes' responses to and outcomes in training and performance.^33,34 

Technology allows for the rapid collection and analysis of data from many of these areas. The ability to integrate data streams enables practitioners to better understand how one factor affects another by providing a holistic perspective of the athlete. It also permits information to be shared across disciplines, blending injuries with training load, medicine with physiology, and physical with technical and tactical performance. Where limitations once existed in storing and processing vast amounts of information safely and in a time-efficient manner, technological advancements have reduced many of the challenges involving costs, computing speed, and intelligence tools. Although pitfalls still exist (see the next section), technology allows practitioners and teams to provide holistic perspectives on athlete performance when using a strategic approach.

The data life cycle may be summarized broadly as plan → collect → analyze → communicate. Each subsequent step relates closely to the intended use of the information, as determined by a thoughtful plan underpinning the technology's implementation. A problem at any stage of this life cycle can be fatal for the successful use of any technology. Failed technological implementations can have lasting ramifications, so considering the following potential pitfalls is important.

Underlying all stages of the data life cycle is a belief that the data are trustworthy enough to collect and interpret. However, some of the bold promises made by technology companies may not actually be true. In these instances, failed promises may result in poor data quality (eg, measurement error is too large) that challenges a practitioner's ability to interpret any signal amid the noise. The failures could also stem from black-box algorithms that summarize the data and produce unactionable, uninterpretable outputs.

Scientists have explored technological devices in an attempt to better understand the validity and reliability of the data from these emergent technologies. As such, different technologies are known to have inherent limitations: for example, the ability of GPS technology to accurately measure high-speed running velocities,^22,35,36  the sensitivity of heart-rate variability measurement,³⁷  and the subsequent requirement of rigid, standardized testing procedures or the effectiveness of wrist-based sleep monitoring compared with the criterion standard of polysomnography testing.³⁸  Although these examples include some technological devices that have published validity-related evidence, it is important to note 2 items. First, none of these devices and the data they provide are perfect; all come with inherent measurement error. Second, most consumer devices have little scientific evidence for their accuracy, validity, and reliability,¹⁴  so a prudent practitioner should approach any new technological device with a healthy dose of skepticism.

The best technologies are useless if they are not implemented in a way that informs decision making or changes practice. Given that technological implementation may require sizable investments of financial and human resources, understanding the burdens on time and staff resources that implementation will require is crucial. If the burden on staff is too great, practitioners may be stretched beyond their skill sets, be forced into uncomfortable situations, or have unrealistic time constraints placed on them, negatively affecting the feasibility and quality of data collection. If the staff are not educated about the potential benefits, lack the desire to collect the data appropriately, and do not believe the technology will provide useful information, the investment is flawed before it begins.

A high-tech solution and precise data collection do not inherently mean that the right data are being collected. What constitutes the right data in a particular setting depends on several contextual and organizational factors, such as the questions being asked by different practitioners within the organization and how the organization's decision-making structure allows data to inform decisions.

Even if the right data are being collected, it is vital to understand what type of data the technology will provide. Many technologies come with a software package that delivers a dashboard or printable report of the data collected. It is important to consider whether these standard reports or dashboards analyze the data in a way that reflects the user's needs and corresponds to the original plan. The analyze portion of the data life cycle implies that the technology analyzes the data in the way you need or that you can access and analyze the data yourself in accordance with those needs if the technology does not provide the answers you are seeking. In these instances, it is important to understand whether a technology provides access to the raw data so that you can perform the appropriate analyses in accordance with your original plan. If the technology does not provide such access and only reports summary findings based on proprietary algorithms, the ability of researchers and practitioners to analyze data in the ways they need may be compromised.

Even when technology is introduced and data are collected consistently in an applied environment, the data have just made it through part of the data life cycle—they must still be analyzed and disseminated in accordance with the plan. Technology itself does not inherently communicate to decision makers. Although some technological devices are accompanied by software tools that provide reports or dashboards that summarize the underlying data, the message delivered to decision makers must be readily interpretable by the end user, which can include high-performance team members, coaching staff, or management, and answer the specific questions that were planned. What appropriate communication and dissemination look like, therefore, depend on the intended use of the technology and the context in which it is implemented. Understanding end users' requirements, interests, and necessary decisions is crucial so that information can be tailored into a clear, concise message. Crucially, these steps must be taken in each environment and are not accomplished simply by having purchased a given technology.

Asking the right questions before jumping into a new technological investment can help guide practitioners and researchers to the vision of the technology while avoiding some of the common pitfalls. Unsurprisingly, several frameworks for integrating technology into sport have been proposed in the literature,^39,40  and we strongly encourage readers to explore and critically think through other frameworks in addition to those presented here. In our critical decision-making framework, we pose 4 questions, all of which should be answered affirmatively before arriving at a decision to purchase a given technology (Figure 1). Each of these 4 questions, important follow-up questions, sources of evidence for finding appropriate answers, and key take-home messages are discussed in the following sections, detailed in Figure 1, and summarized in the Table.

New technologies arrive on the market daily, many of which may pique the interest of curious and intelligent individuals. These technologies often come with bold claims, savvy marketing, and grand promises. In the sport sciences, these claims may include accurate injury prediction or “1-stop shops” for understanding an athlete's fatigue and recovery status. We propose that the first question practitioners should ask when encountering a technology and engaging with such claims is would the promised information be helpful? The ability to extract new information can be exciting, yet this does not mean the information will help to inform decision making for practitioners in their specific contexts.

To answer this question, individuals should consider what specific question will be answered or which decision will be informed. This is an extremely important point, as technology must inform practice to be useful. When a new technological opportunity is considered, at least 1 key decision should be informed by the available information or 1 question should be answered. This is the premise of the first (and most important) stage of the data life cycle: planning. To plan effectively, practitioners must understand their specific contexts so they address relevant and pertinent questions that end users need to answer. Ideally, a need may already have been identified within the practice for more information to be collected, in which case investing in a technology that provides that specific data makes more sense. Considering 2 contexts from earlier in this article, wearable technology may provide insight for team-sport practitioners, but this looks different in soccer, where GPS data may be most valuable to understand athlete distances and speeds, than volleyball, where an accelerometer that provides information about jump counts and distances may be deemed less important (and not measurable with GPS indoors). Note that in each of these instances, a need is a question that must be answered or a decision that must be informed, rather than a specific technology that the organization believes is in itself necessary.

• Start with the end in mind. Understand the decision you want to inform and which data from the technology you will need to extract and communicate in order to inform that decision.

• Explore existing paths. Is there an existing data stream you can use instead or a better alternative?

When researchers speak about technology, the discussion typically includes aspects of measurement error, reliability, responsiveness, and validity. Beyond all these technical terms, practitioners need to know whether they can take information from a given technology and be confident in making a decision based on the evidence it provides. We believe the principles of unified validity theory can help guide researchers and practitioners in trying to answer this question of trustworthiness.

Grounded in the work of Loeveringer⁴¹  and Cronbach and Meehl,⁴²  Samuel Messick posited and promoted a unified view of validity theory.^43–45  In place of validity types, the following definition of unified validity theory was proposed by Messick and adopted in the Standard for Educational and Psychological Testing: “Validity is an integrated evaluative judgment of the degree to which empirical evidence and theoretical rationales support the adequacy and appropriateness of interpretations and actions based on test scores or other modes of assessment.”⁴⁶ 

Unpacking this definition reveals 3 primary ways in which unified validity theory differs from other common and more traditional views of validity:

1. Validity is about claims and inferences that can be made, not about measures.

2. Validity evidence has multiple sources, and the aim is an integrated evaluative judgment (Figure 2).

3. Validity invites consideration of the consequences associated with technology and the data it provides.

Unified validity theory provides practitioners and scientists with a lens to look through as they consider different measures. When we look through this lens, we see that thinking about measurements from technology is similar to thinking about science. The answers to most of our questions are more nuanced than yes versus no or valid versus invalid. Instead, we use terms such as it depends, to a degree, or in this specific context. This sets the stage for a practitioner considering a new technology to evaluate the technology on a continuum in terms of his or her specific context.

As an integrative, evaluative judgement, practitioners should examine all the available sources of evidence regarding a given technology and the specific metric it provides. Some answers may be found in the peer-reviewed literature, or the practitioner may have to pilot data internally. Reaching out to colleagues who have already implemented these technologies may offer opportunities to discuss their internal validity-related evidence.

The consequences of testing are a final consideration that unified validity theory emphasizes and that practitioners should carefully consider. Implicit and explicit consequences are inherent when measuring and testing something by implementing a technology. Athletes and practitioners will consider the quality being measured as important, athletes may train to improve that given quality, and decisions may be based more heavily on the provided data than on other pieces of information. These intended and unintended consequences can be positive or negative but should be considered carefully.

This process and evaluation must be performed on each of the different metrics that a practitioner hopes to use to inform decision making. Returning to our volleyball example, the IMU provides a more accurate measure of jump count than does video notation. However, this same IMU also provides measures of jump height and ground reaction forces. Although these measures may theoretically be linked to overuse injuries and performance, they should each be further investigated. In this case, video notation would not be the appropriate comparison measure, and more advanced biomechanical analyses and equipment would be preferable.

Ultimately, no technology, or the data it provides, is perfectly trustworthy. The practitioner faces this question: Given all the information at his or her disposal, are the limitations of the technology minimal enough that it can still inform decision making?

• Evaluate continually. View trust in one's data as an ongoing endeavor to judge how trustworthy the technology and data are on a spectrum using all the available sources of validity-related evidence.

• Consider the consequences of testing. What potential consequences, intended and unintended, could introducing the technology have for athletes and practitioners?

• Pilot test where possible. If practitioners can gain early access to the technology before purchasing, they can conduct preliminary analyses of the data before purchasing.

• Partner where appropriate. When in-house expertise is not sufficient for examining certain aspects of the data, collaborate with a research laboratory, university, or third-party company to facilitate analyses of the trustworthiness of the technology.

A 360° view of an athlete's training, recovery, lifestyle, and so on is a major potential benefit of technological implementation. If a technology and the data it provides are deemed trustworthy enough to be added to this holistic athlete view, the next step is to understand how the data will be extracted, integrated with other data sources, and analyzed in a meaningful way.⁴⁷  Different technologies provide different levels of granularity to the data, and the means of data extraction can vary from manual downloading of files (eg, spreadsheets) to application programming interfaces that allow automated data extraction. The extracted data may also need additional cleaning before analysis and modification to be integrated with other data sources. It is vital to understand how much time will be needed to extract and clean the data; although these processes (eg, manual downloads and spreadsheet data management) may be common, they limit scalability and may preclude successful and sustainable implementation of a given technology.

A well-known truism in the data-science realm is that data scientists spend most of their time cleaning and preparing data for analysis. Given the complexity and challenges inherent in combining data from disparate technologies, practitioners must consider whether they have the expertise on their team or at their disposal to create a system that brings their data together. This may be accomplished through in-house data-science personnel, third-party athlete-management systems, or external consulting agencies. Without this expertise, it is very challenging to combine data sources to create a holistic athlete profile.

Once the data are collected and combined, analysis presents its own challenges. Over the last few decades, intensive longitudinal data have become increasingly common in elite sport settings. These data present a specific set of challenges and assumptions inherent to repeated-measures data, and in many instances, more sophisticated analyses are recommended to deal with these challenges.^48,49  At least in the workload injury field, the authors of a methodologic review⁵⁰  identified that statistical approaches to adequately address these challenges when investigating the question of how workload data relate to injury risk have not been applied to many intensive longitudinal data sets. Statistical in-house or outsourced expertise can help ensure that the statistical approaches applied are appropriate for the data complexity.

To understand the demands of accessing, cleaning, integrating, and analyzing the data that a new technology will provide, it is highly prudent to ask the company for a free trial and access to their data streams. Discussions with other practitioners in the industry who already use the technology may also be fruitful to paint a realistic picture of the data-management demands.

• Plan ahead: Obtaining data samples from prospective companies ahead of time helps to ensure that the processes and systems can be tested and evaluated before a technology is introduced.

• Educate: By training practitioners in basic principles of data collection,⁵¹  many of the data-cleaning challenges can be proactively prevented.

• Automate: Software solutions (eg, Alteryx [Irvine, CA], Fivetran [Oakland, CA], Matillion [Manchester, UK]) and open-source coding platforms (eg, R [Vienna, Austria], Python [Wilmington, DE]) can enable data scientists to streamline and automate many processes, thereby reducing the amount of time required to manually input, download, and edit data.

• Audit: Set up data-audit or quality-control checks to ensure the data are clean and appropriately combined and then respond appropriately when you find mistakes and outliers.

The fourth and final question to consider is whether a technology can realistically be implemented in your specific sporting context. In the research arena, we know that injury-prevention protocols that are effective in a controlled trial setting may fail to deliver the same results in a real-world environment because the effects are largely dependent on the adoption and implementation of the program.^43–45  In the same way, even a near-perfect technology may fail in an environment where it encounters an implementation problem.

Implementation failures may occur anywhere along the data-science pipeline. The challenges in data analysis were largely addressed in question 3, but implementation challenges are especially pivotal to consider at the data-collection and data-dissemination steps.

Implementation challenges in data collection stem from increased practitioner or athlete burden. The demand on staff and athletes alike to collect data can be significant. It is not uncommon for the rollout of a new technology to be one more item in a long list of responsibilities for staff. Athletes are also not always burden free when it comes to the implementation process. It is important to consider what the athletes will be asked to do, the collective burden of technology and data collection on athletes as a whole, and how the athletes will perceive the new technologies. One must consider the possible ramifications of this increased staff and athlete burden and whether the technology's potential benefits outweigh the cost of implementation.

The flexibility, mentality, and willingness of the people in an environment to adopt new practices can determine whether implementation challenges in data-informed decision making arise. Certain sports have already embraced technology, whereas others may be deemed resistant to technological innovations. It is therefore essential to consider if members of your sporting culture will accept a specific technology in their environment. For numerous reasons, an organization may not want the data or may not be keen to have the answers that the data could provide. Also, deep-rooted doubt in the reliability or validity of the data may prevail, despite the best available evidence. Many sporting cultures resist changing the way things have always been done, so technological implementation may be seen as changing or modifying their sport.

The successful implementation of any technology requires careful consideration of the time and resources required from practitioners and athletes, the process and procedures that need to be in place to minimize the burden from the technology, and the communication and decision-making channels whereby the analyzed data will be delivered and used to inform practice.

• Understand and improve the implementation context. Consider the burden and challenges on practitioners and athletes at all points across the data pipeline. Educating practitioners on the rationale and benefits associated with technology and empowering them in their roles may facilitate “buy-in” and more successful implementation.

• Look for invisible monitoring opportunities, which impose virtually no burden on athletes. The data collection is automated, and nothing else is needed from the athletes. These invisible opportunities, whatever they may be, will still require data aggregation and integration from staff and require an environment for informed decision making, but they may enhance implementation success when athlete buy-in is the primary barrier.

• Build technological implementation into existing and new routines and tasks to make it obvious that technology is being integrated and something new is being introduced.

The final step, which should only be considered seriously once all 4 previous questions have been answered affirmatively, is whether the technology is worth the investment. This is essentially a cost-benefit analysis comparing the expected net performance effect with the financial burden that the technology carries in the overall context of the other 4 questions in the framework.

Although our focus in this article was predominantly on guiding researchers and practitioners to critically evaluate whether they should purchase a new technology, we believe it would be a mistake not to briefly address several key considerations for when that technology is introduced.

The following concrete recommendations may facilitate buy-in and increase the probability that technological implementation succeeds: (1) undersell and overdeliver; (2) allow athletes to provide the equivalent of informed consent before data collection; (3) try to include multiple key stakeholders in data-evaluation sessions so that the technology does not isolate support staff or create a “secret society”; (4) continue to evaluate signal-to-noise ratios and provide clarity on accuracy and reliability; (5) allow complex analyses to take place behind closed doors but present simplified, clean data that support important messages to coaches and athletes; and (6) do not use data in ways that contribute to political agendas or undermine the integrity of colleagues.

Integrating advanced technology into high-performance sport can be challenging. Emotions, egos, time constraints, and unrealistic expectations can make it difficult to gain approval as well as purchase and implement new technology. Many young practitioners may purchase technology in an attempt to win favor from upper management as they work to encourage enthusiasm, hope, and belief in players and stakeholders. Without being fully aware of the power of the placebo (that is, the excitement of buying something new), they may convince sports administrators to allocate significant funds for a speculative purchase. We hope that the critical framework provided in this article will help to prevent these types of poor decisions, yet we believe the placebo, or belief, effect is an important aspect of technological implementation to leverage.

Interestingly, it has been reported that some of the earliest placebo researchers examined the influence of fake advanced medical technology for treating pain. For those who believed new technology could take away pain, the sophisticated device that supposedly harnessed the power of special metals worked equally well when the devices looked like they were made of metal but were actually made of wood.⁵²  Introducing advanced technology into a high-performance program may have similarly positive effects if it is sold and implemented the right way, with a collective belief among practitioners and athletes that the technology can have a positive effect. In contrast, organizational differences of opinion can dilute the power of the belief effect. When the support team is divided on the efficacy of using new technology, their conversations and attitudes can ultimately undermine buy-in from athletes. Technological implementation should ultimately be a collective team effort whereby stakeholders engage throughout the stages of the decision-making framework and as the technology is implemented.

Technology may help organizations reach their grandiose vision or drag companies down because of its pitfalls. We hope this critical framework will empower practitioners and organizations to make informed, wise decisions about whether a technology should be implemented. However, we acknowledge that this stepwise approach is an oversimplification and over-regimentation of the technological-evaluation process. At times, a technology may simply cost too much, in which case the 4 questions are irrelevant. Several questions may be investigated at once, such as when a company provides a free trial. Although the framework may be applied differently than described in this stepwise presentation, we caution that all 4 questions are critical to answer affirmatively before an investment is made.

Three fundamental principles underpin this type of a framework: (1) proactivity allows practitioners to start with the end in mind and to plan ahead in considering how to solve implementation challenges across the data pipeline; (2) critical thinking informs how practitioners evaluate the trustworthiness of technology and its data, as well as the intended and unintended consequences of introducing the technology; and (3) collaboration, specifically internal collaboration, underpins the success of communication and data-informed decisions and external collaboration can be essential for piloting technologies where appropriate (eg, outsourcing validity-related, data management, or analytical work that is beyond an organization's current capabilities). Each of these principles bodes well in critically thinking about technology and even more broadly for practitioner and organizational excellence.

The decision “to tech or not to tech” is critical but complex. It should be made through a careful evaluation of evidence related to the technology and the environment in which it will be deployed. Ultimately, it is a question that every organization and practitioner will face in today's technologically driven age. Much like the technologies in question, no person or company will make all perfect decisions, but a thoughtful framework and critical approach can help them hit the target more often than not.