Real-time Delphi

Last updated

Real-time Delphi (RTD) is an advanced form of the Delphi method. The advanced method "is a consultative process that uses computer technology" [1] to increase efficiency of the Delphi process.

Contents

Definition and idea

Gordon and Pease [2] define the advanced approach as an innovative way to conduct Delphi studies that do not involve sequential "rounds" and consequently lead to a higher degree of efficiency with regard to the time frame needed to perform such studies. Friedewald, von Oertzen, and Cuhls [3] underline that aspect by writing, in "a Real-Time-Delphi, the participants do not only judge twice but can change their opinion as often as they like when they see the aggregated results of the other participants". So, here it becomes clear that the Real-Time Delphi approach requires real-time calculation and provision of group responses. Friedewald et al. [3] further state that the Real-Time Delphi method has beneath its explorative and predictive elements also normative and communicative elements. These latter are investigated by Bolognini, [4] who explores the potential of computer-based Delphi as a communication technique for electronic democracy.

Comparative studies of von der Gracht and colleagues have revealed that Real-time Delphi studies are comparable to the outcome of conventional round-based Delphi surveys. [5]

History

The basic idea of a real-time, therefore computer-based (usually web-based), Delphi approach originates in a paper published by Turoff back in 1972 about an online Delphi conference conducted in the United States. [6] The conference was characterized by remote locations of participants, an online tool to access and give judgments, anonymity of the participants, continuous operations and analysis of results (i.e. participants were able to see given answers of the other participants in real-time), as well as asynchronous participation (i.e. participants could independently login and logout how often and when they desired). The stated aspects are some of the key characteristics of Real-Time Delphi studies, which shows that the original idea of conducting such studies can be traced back to the respective year. Today, nevertheless, technological innovations and advanced computer aided design possibilities (e.g. high-speed internet connections, high definition graphic, and advanced processor performance) facilitate more sophisticated studies in this context. [7] The general idea to develop a faster advanced form of Delphi studies by using ideas and basic concepts of Turoff, was initiated by the U.S. Defense Advanced Research Projects Agency (DARPA), which awarded a grant in 2004 to develop an approach to improve "speed and efficiency of collecting judgments in tactical situations". [2] A small software company named Articulate Software in San Francisco was awarded an innovation research grant to develop what DARPA was asking for. [2] Adam Pease, principal consultant and CEO of Articulate Software, published the findings and methodology together with Theodore Gordon in 2006. [2]

Based on the findings in this seminal paper, Heiko von der Gracht developed a second Real-time Delphi platform until 2008. In subsequent years, von der Gracht and colleagues used the platform extensively for prospective research studies. In 2011, von der Gracht and Gordon presented a joint paper at WFS Annual Summit 2011, which summarizes the lessons learned of 40 RTD studies across the two platforms. [8]

In 2017, Angenheyster and colleagues published a study, which compares various Real-time Delphi platforms. Analysis criteria include among others visual appearance, data output, user friendliness, administration, and types of questions. [9] A cross-disciplinary study by Beiderbeck et al. focused on new directions and advancements of the Delphi method, including real-time formats. The authors provide a methodological toolbox for designing Delphi surveys including among others sentiment analyses of the field of psychology. [10]

Differences between conventional and real-time Delphi method

The question arises how a Real-Time Delphi study differs from a Conventional Delphi study. The basic framework is to think of a Delphi study which is conducted in form of an online questionnaire. However, a Conventional round-based Delphi study conducted via the internet is called "Internet Delphi". The basic difference to Internet Delphi is that the process of a Real-Time Delphi is not characterised by single iterated rounds. In fact, real-time calculation and provision of responses are the key characteristics of Real-Time Delphis. Various other labels for Real-Time Delphi can be found in literature and many authors are not completely aware of the differences: [11] "Electronic Delphi", "Computer Delphi", "Computer-aided Delphi", and "Technology Delphi". However, it is important to truly understand the design and process a researcher has chosen to find out whether real-time calculations and provisions have been applied or not.

The typical Real-Time Delphi process can be described in the way that participants get access to an online questionnaire portal for a certain time frame, within which they are allowed to log in and log out as often as they want. Whenever they login, they will see all their quantitative and qualitative answers of previous sessions and they can change all answers as desired within the given period of time. Besides their own answers they will see the ongoing – hence, real-time – responses of other participants, and with regard to metric assessments the group as a whole will be visualised in terms of median, average, and interquartile range (IQR). It has to be pointed out that the numerical visualisations as well as the qualitative inputs change in the course of other participants changing their responses. [2] Consequently, a participant can find out to what extent his own responses from an earlier point of time are still within the group opinion (i.e. IQR). The core innovation, then, of Real-Time Delphi studies is the real-time calculation and provision of results. [12]

Methodological advancements

The core methodological innovation of Real-Time Delphi studies are the absence of iterated rounds and the real-time calculation and provision of group responses. Whereas Conventional Delphi studies are characterised by repeating sequential rounds, the Real-Time Delphi approach is characterised by a continuous round-less procedure leading to a reduced time frame needed to conduct such studies. [13] Consequently, conducting large-scale studies of huge complexity in a relatively short period of time becomes possible. [13] Another core methodological innovation is the fact that experts may not only judge once or twice, depending on the number of rounds, as it has been usual in a Conventional Delphi study. During a Real-Time Delphi, experts can independently reassess their responses as often as they want. [3]

Hartman and Baldwin [1] discuss further advantages of the Real-Time Delphi approach: First, the number of experts participating in the real-time study can be increased due to a higher degree of automation during and improved possibilities for analysis after the study. Additionally, the Internet provides the possibility to invite a worldwide expert panel to participate in the study. Second, the degree of interaction among the experts can be increased due to the fact that they can immediately react on others' comments. Additionally, the time frame between giving own answers and getting insights into others' responses is very short, which encourages stronger cognitive examination with the respective issue in question. Hartman and Baldwin [1] argue that with the help of this procedure the validity of results is maximised.

In order to conduct a Real-Time Delphi study, computer software – usually web-based – is needed for facilitating real-time calculations and visualisation of results. It is generally proposed in the existing literature that the experts participating in the study see not only their own answers but also the median and interquartile range of all given responses immediately after answering a quantitative question. [2] Besides the quantitative assessment a qualitative judgment of participants can be shown which serves as a justification for their numerical assessment of the question. Additionally, it can be shown to the expert how many respondents have already given their answers. To examine the qualitative arguments of others participants can click on a button, and a "reasons window" opens, which shows the statements of other participants to underline their point of view. So, the respective legitimisations given by others may cause a respondent to recapture his own point of view. [2] In the next step the expert can change his own answer, add new arguments to underline his point of view, or leave his answer unchanged. In addition, the respondent will be shown an attention indicator, a so-called "flag", [13] if his answer is within or outside the interquartile range or significantly different from the median. This application helps to see and understand immediately the own assessment and to think about reasons for the deviation from group opinions or else a high degree of consent. The respondent's attention will be called by highlighting questions with a high degree of deviation with a different colour and by asking him to give further reasons for his deviation from the group opinion.

After operating a question in the described procedure, the participant can continue to the next question or press a "save"-button in the program, which leads to an immediate update of the median, interquartile range, and given arguments, and then leave the program. A second advantage of the round-less approach is the fact that, in order to take part in the study, participants can login and logout with their personalised account as often as they want during the time frame provided. Their already given answers will be saved and recalled when they login the next time. So, by design of the study, there are no explicit single rounds to answer the questions. [2] Updating and playing back the information to the other participants follow immediately in succession to the process of answering. Here, it becomes clear that the process of answering can be synchronous or asynchronous and a worldwide expert panel can be reached, which is one of the major advantages of web-based tools. Turoff and Hiltz [11] argue that the issue of asynchronous interaction is probably one of the least understood characteristics of Real-Time Delphis. Zipfinger [14] points out two advantages of asynchronous participation of the experts: First, they can login to the portal whenever they want; therefore, one could argue that the degree of convenience of taking part is increased for the participant due to a 24-hour availability of the portal. Second, panellists can contribute to whatever aspects in the questionnaire they want, especially when having gone through each question at least once. [11] Here, a substantial aspect of Real-Time Delphis becomes obvious: Turoff and Hiltz [11] explain that a Real-Time Delphi study offers a design of structured communication which allows every individual to choose the sequence and speed to contribute to the problem solution process. So, in comparison to face-to-face discussions, the Real-Time Delphi approach gives room for individuality and different cognitive abilities of the participants. [11]

A further advantage is the fact that the administrator of the study can set an arbitrary time frame in which participants have to log in and take part in the questionnaire. So, whenever the researcher or administrator of the study is satisfied with the existing answers (i.e. in terms of quantity and quality), he can declare the study to be ended and close the online tool (i.e. "freeze" the responses). [2] [13]

The key features of Delphi studies, such as anonymity, controlled feedback and group response, are also met in the context of Real-Time Delphi studies. However, the issue of iteration is, by design of the technique, not valid for Real-Time Delphi studies anymore. Instead of answering each question a first time and getting a second sheet with the group responses in the second round, the Real-Time Delphi already shows the second screen (i.e. group responses) immediately after answering each question. Having answered each question at least once, the participant can usually control which question to reassess from a "consensus portal", which serves as a kind of control panel to access single questions again. So, on the one hand, the procedure differs from a Conventional Delphi and, on the other hand, the iteration into single rounds is missing.

Having asked the question how the accomplishment of a Real-Time Delphi study differs from conducting a usual Delphi study, Gordon and Pease [2] point out that a Real-Time Delphi study can be implemented via a site on the Internet or in any other network (e.g. intra-company network, local area network) and is, therefore, not conducted in paper-and-pencil form any more.

As with all Delphi studies, the process of defining and selecting experts is still extremely important. [2] The Conventional Delphi study is then divided into several steps of response round, analysis through the facilitator, playing back the information, next response round, and so on. However, the Real-Time Delphi study is, after granting access to the online tool, rather a self-running process.

The basic strengths of a Real-Time Delphi study are its efficiency and applicability to all Delphi topics (i.e. common problem sets, decision making issues, cross impact studies, etc.). [2] Figure 2 illustrates that the process of a Real-Time Delphi differs. Important is to point out that the number of interventions of the facilitator needed during the response phases (i.e. after opening the online tool) are usually less. Having developed the online tool in advance, the intermediate analysis done by the facilitator of the study is rather uncomplicated in comparison to the Conventional Delphi. The overall shortened time period needed to conduct a real-time study underlines that the approach can be regarded as generally more efficient. [2]

Gordon and Pease [2] point out that a Real-Time Delphi study is applicable for a wide range of possible circumstances under which the consultation of experts is necessary. On the one hand, the authors give the example of a "small group operating synchronously in a conference room with laptop computers connected wirelessly to the web site where the software resides, with anticipated completion of the exercise in say 20 min.". On the other hand, it can be thought of a larger panel of experts operating asynchronously from remote locations within a longer period of time.

The greatest weakness of the real-time approach is that it is missing a wholly integrated, scientifically founded concept. The real-time Delphi idea is still a very new concept, which requires further research and application to become a tool for full-scale operations. [2] Especially the editing of the alpha (i.e. the first) inputs of respondents, the real-time presentation of group results, and the tracking of progress over time should be integrated in a kind of administrator package to make the accomplishment of a Real-Time Delphi less difficult. [2]

Examples of real-time Delphi applications

Numerous examples for real-time Delphi applications can be found. Among them, The Millennium Project conducted by Glenn, Gordon, & Florescu in 2009 provides a context for global thinking and improved understanding of global issues, opportunities, challenges, and strategies. More information on the project can be found on www.millennium-project.org

Another stream of projects based on real-time Delphi studies was conducted by the former Institute for Futures Studies and Knowledge Management of EBS University of Business and Law in Germany, among them the T&L2030 Series of five different studies together with PricewaterhouseCoopers: Transportation & Logistics 2030 Series. Further illustrative examples in academic journals include cases from logistics, [15] social business, [16] professional services, [17] retail, [18] and automotive. [19]

Notes

  1. 1 2 3 Hartman, F. T., & Baldwin, A. (1995). Using Technology to Improve Delphi Method. Journal of Computing in Civil Engineering, 9(4), 244-249.
  2. 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 Gordon, T. J., & Pease, A. (2006). RT Delphi: An Efficient, “Round-less”, Almost Real Time Delphi Method. Journal of Technological Forecasting and Social Change,73(4), 321-333.
  3. 1 2 3 Friedewald, M., von Oertzen, J., & Cuhls, K. (2007). European Perspectives on the Information Society (EPIS) (Delphi Report Deliverable 2.3.1). European Techno-Economic Policy Support Network (ETEPS Net).
  4. ↑ Maurizio Bolognini (2001), Democrazia elettronica: Metodo Delphi e politiche pubbliche (Electronic Democracy: Delphi Method and Public Policy-Making) (in Italian), Rome: Carocci Editore, ISBN   978-88-430-2035-5 . The author argues that computer-based Delphi can be especially relevant in the context of e-democracy, not only for the roundless approach, the real-time updating of statistical response, or the large number of panellists, but for the possibility to establish split panels corresponding to different groups (such as policy-makers, experts, and citizens), which the administrator can give different tasks and privileges, depending on the issues and the type of decision-making process.
  5. ↑ Validating an Innovative Real-Time Delphi Approach – A methodological comparison between real-time and conventional Delphi studies. In: Technological Forecasting & Social Change. 78, 9, 2011, p. 1681–1694, doi:10.1016/j.techfore.2011.04.006.
  6. ↑ Turoff, M. (1972). Delphi Conferencing: Computer-Based Conferencing with Anonymity. Technological Forecasting and Social Change, 3, 159-204.
  7. ↑ Häder, M. (2002). Delphi-Befragungen. Ein Arbeitsbuch. Wiesbaden: Westdeutscher Verlag.
  8. ↑ New Frontiers in Delphi Research: Experiences with Real-Time Delphi in Foresight. In: Conference Proceedings of World Future 2011, p. 129-160, 10. July 2011, via researchgate.
  9. ↑ Real-Time Delphi in practice — A comparative analysis of existing software-based tools. In: Technological Forecasting & Social Change . Volume 118, May 2017, Pages 15-27, doi:10.1016/j.techfore.2017.01.023.
  10. ↑ Preparing, conducting, and analyzing Delphi surveys: Cross-disciplinary practices, new directions, and advancements. In: MethodsX . Volume 8, 2021, 101401, doi:10.1016/j.mex.2021.101401.
  11. 1 2 3 4 5 Turoff, M., & Hiltz, S. (1995). Computer based Delphi processes. In M. Adler & E. Ziglio, E. (Eds.), Gazing into the Oracle: The Delphi Method and its Application to Social Policy and Public Health (pp. 56-88). London: Jessica Kingsley Publishers.
  12. ↑ Monguet, J., Ferruzca, M., Gutiérrez, A., Alatriste, Y., Martínez, C., Cordoba, C., Fernández, J., et al. (2010). Vector Consensus: Decision Making for Collaborative Innovation Communities. Communications in Computer and Information Science (Vol. 110, pp. 218–227). Viana do Castelo, Portugal: Springer. doi:10.1007/978-3-642-16419-4_22
  13. 1 2 3 4 Gordon, T. J. (2007). Energy forecasts using a “Roundless” approach to running a Delphi study. Foresight, 9(2), 27-35.
  14. ↑ Zipfinger, S. (2007). Computer-Aided Delphi: An Experimental Study of Comparing Round-Based with Real-Time Implementation of the Method. Linz: Trauner Verlag.
  15. ↑ Energy-Constrained and Low-Carbon Scenarios for the Transportation and Logistics Industry. In: International Journal of Logistics Management . Vol. 27, No. 1, 2016, p. 142-166, doi:10.1108/IJLM-12-2013-0150.
  16. ↑ Heading Toward a More Social Future - Scenarios for Social Enterprises in Germany. In: Business & Society . Vol 55, No. 1, 2014, p. 56-89, doi:10.1177/0007650314523096.
  17. ↑ The Future of the Foresight Professionals: Results from a Delphi Study. In: Futures (journal) . Volume 71, August 2015, Pages 132-145, doi:10.1016/j.futures.2015.03.005.
  18. ↑ Delphi-based strategic issue management: crafting consumer goods supply chain strategy. In: International Journal of Physical Distribution & Logistics Management . Vol. 44, Issue 5, 2014, p. 373-391, doi:10.1108/IJPDLM-09-2012-0289.
  19. ↑ Heading Towards a Multimodal City of the Future? Multi-Stakeholder Scenarios for Urban Mobility. In: Technological Forecasting & Social Change . Volume 89, November 2014, Pages 201-221, doi:10.1016/j.techfore.2013.08.036.

Related Research Articles

Usability testing is a technique used in user-centered interaction design to evaluate a product by testing it on users. This can be seen as an irreplaceable usability practice, since it gives direct input on how real users use the system. It is more concerned with the design intuitiveness of the product and tested with users who have no prior exposure to it. Such testing is paramount to the success of an end product as a fully functioning application that creates confusion amongst its users will not last for long. This is in contrast with usability inspection methods where experts use different methods to evaluate a user interface without involving users.

Usability Capacity of a system for its users to perform tasks

Usability can be described as the capacity of a system to provide a condition for its users to perform the tasks safely, effectively, and efficiently while enjoying the experience. In software engineering, usability is the degree to which a software can be used by specified consumers to achieve quantified objectives with effectiveness, efficiency, and satisfaction in a quantified context of use.

The Delphi method or Delphi technique is a structured communication technique or method, originally developed as a systematic, interactive forecasting method which relies on a panel of experts. The technique can also be adapted for use in face-to-face meetings, and is then called mini-Delphi or Estimate-Talk-Estimate (ETE). Delphi has been widely used for business forecasting and has certain advantages over another structured forecasting approach, prediction markets.

Scenario planning, scenario thinking, scenario analysis, scenario prediction and the scenario method all describe a strategic planning method that some organizations use to make flexible long-term plans. It is in large part an adaptation and generalization of classic methods used by military intelligence.

An electronic meeting system (EMS) is a type of computer software that facilitates creative problem solving and decision-making of groups within or across organizations. The term was coined by Alan R. Dennis et al. in 1988. The term is synonymous with group support systems (GSS) and essentially synonymous with group decision support systems (GDSS). Electronic meeting systems form a class of applications for computer supported cooperative work.

All telecommunications service providers perform forecasting calculations to assist them in planning their networks. Accurate forecasting helps operators to make key investment decisions relating to product development and introduction, advertising, pricing etc., well in advance of product launch, which helps to ensure that the company will make a profit on a new venture and that capital is invested wisely.

<i>The Wisdom of Crowds</i> 2004 book by James Surowiecki

The Wisdom of Crowds: Why the Many Are Smarter Than the Few and How Collective Wisdom Shapes Business, Economies, Societies and Nations, published in 2004, is a book written by James Surowiecki about the aggregation of information in groups, resulting in decisions that, he argues, are often better than could have been made by any single member of the group. The book presents numerous case studies and anecdotes to illustrate its argument, and touches on several fields, primarily economics and psychology.

Strategic foresight is a planning-oriented discipline related to futures studies. In a business context, a more action-oriented approach has become well known as corporate foresight.

Futures techniques used in the multi-disciplinary field of futurology by futurists in Americas and Australasia, and futurology by futurologists in EU, include a diverse range of forecasting methods, including anticipatory thinking, backcasting, simulation, and visioning. Some of the anticipatory methods include, the delphi method, causal layered analysis, environmental scanning, morphological analysis, and scenario planning.

Technology forecasting attempts to predict the future characteristics of useful technological machines, procedures or techniques. Researchers create technology forecasts based on past experience and current technological developments. Like other forecasts, technology forecasting can be helpful for both public and private organizations to make smart decisions. By analyzing future opportunities and threats, the forecaster can improve decisions in order to achieve maximum benefits. Today, most countries are experiencing huge social and economic changes, which heavily rely on technology development. By analyzing these changes, government and economic institutions could make plans for future developments. However, not all of historical data can be used for technology forecasting, forecasters also need to adopt advanced technology and quantitative modeling from experts’ researches and conclusions.

Synchronous conferencing is the formal term used in computing, in particular in computer-mediated communication, collaboration and learning, to describe technologies informally known as online chat. It is sometimes extended to include audio/video conferencing or instant messaging systems that provide a text-based multi-user chat function. The word synchronous is used to qualify the conferencing as real-time, as distinct from a system such as e-mail, where messages are left and answered later.

Group decision-making is a situation faced when individuals collectively make a choice from the alternatives before them. The decision is then no longer attributable to any single individual who is a member of the group. This is because all the individuals and social group processes such as social influence contribute to the outcome. The decisions made by groups are often different from those made by individuals. In workplace settings, collaborative decision-making is one of the most successful models to generate buy-in from other stakeholders, build consensus, and encourage creativity. According to the idea of synergy, decisions made collectively also tend to be more effective than decisions made by a single individual. In this vein, certain collaborative arrangements have the potential to generate better net performance outcomes than individuals acting on their own. Under normal everyday conditions, collaborative or group decision-making would often be preferred and would generate more benefits than individual decision-making when there is the time for proper deliberation, discussion, and dialogue. This can be achieved through the use of committee, teams, groups, partnerships, or other collaborative social processes.

Harold Adrian Linstone was a German-American mathematician, consultant, futurist and University Professor Emeritus of Systems Science at Portland State University and a specialist in applied mathematics.

Absolute probability judgement is a technique used in the field of human reliability assessment (HRA), for the purposes of evaluating the probability of a human error occurring throughout the completion of a specific task. From such analyses measures can then be taken to reduce the likelihood of errors occurring within a system and therefore lead to an improvement in the overall levels of safety. There exist three primary reasons for conducting an HRA; error identification, error quantification and error reduction. As there exist a number of techniques used for such purposes, they can be split into one of two classifications; first generation techniques and second generation techniques. First generation techniques work on the basis of the simple dichotomy of 'fits/doesn't fit' in the matching of the error situation in context with related error identification and quantification and second generation techniques are more theory based in their assessment and quantification of errors. 'HRA techniques have been utilised in a range of industries including healthcare, engineering, nuclear, transportation and business sector; each technique has varying uses within different disciplines.

Robust decision-making (RDM) is an iterative decision analytic framework that aims to help identify potential robust strategies, characterize the vulnerabilities of such strategies, and evaluate the tradeoffs among them. RDM focuses on informing decisions under conditions of what is called "deep uncertainty", that is, conditions where the parties to a decision do not know or do not agree on the system model(s) relating actions to consequences or the prior probability distributions for the key input parameters to those model(s).

Adaptive learning, also known as adaptive teaching, is an educational method which uses computer algorithms as well as artificial intelligence to orchestrate the interaction with the learner and deliver customized resources and learning activities to address the unique needs of each learner. In professional learning contexts, individuals may "test out" of some training to ensure they engage with novel instruction. Computers adapt the presentation of educational material according to students' learning needs, as indicated by their responses to questions, tasks and experiences. The technology encompasses aspects derived from various fields of study including computer science, AI, psychometrics, education, psychology, and brain science.

Corporate foresight has been conceptualised as a set of practices, a set of capabilities and an ability of a firm. It enables firms to detect discontinuous change early, interpret its consequences for the firm, and inform future courses of action to ensure the long-term survival and success of the company.

An online interview is an online research method conducted using computer-mediated communication (CMC), such as instant messaging, email, or video. Online interviews require different ethical considerations, sampling and rapport than practices found in traditional face-to-face (F2F) interviews. Online interviews are separated into synchronous online interviews, for example via online chat which happen in 'real time' online and asynchronous online interviews, for example via email conducted in non-real time. Some authors discuss online interviews in relation to online focus groups whereas others look at online interviews as separate research methods. This article will only discuss online interviews.

With the application of probability sampling in the 1930s, surveys became a standard tool for empirical research in social sciences, marketing, and official statistics. The methods involved in survey data collection are any of a number of ways in which data can be collected for a statistical survey. These are methods that are used to collect information from a sample of individuals in a systematic way. First there was the change from traditional paper-and-pencil interviewing (PAPI) to computer-assisted interviewing (CAI). Now, face-to-face surveys (CAPI), telephone surveys (CATI), and mail surveys are increasingly replaced by web surveys.

Communication in Distributed Software Development is an area of study that considers communication processes and their effects when applied to software development in a globally distributed development process. The importance of communication and coordination in software development is widely studied and organizational communication studies these implications at an organizational level. This also applies to a setting where teams and team members work in separate physical locations. The imposed distance introduces new challenges in communication, which is no longer a face to face process, and may also be subjected to other constraints such as teams in opposing time zones with a small overlap in working hours.

References

This page is based on this Wikipedia article
Text is available under the CC BY-SA 4.0 license; additional terms may apply.
Images, videos and audio are available under their respective licenses.