Challenges and Research Directions in Big Data

Big Data has opportunities for diagnostic, prognostics, and decision making. This presentation describes challenges associated with Big Data that need to be resolved for these opportunities to be fully realized. The challenges include how to best

(1) extract information from high-order tensor data structures, which arise from spatially resolved photoluminescence, spectral imaging, and color video

(2) integrate information from a variety of types, such as process flowsheet structure, causal relationships between variables, and real-time sensor data

(3) manage the effects of uncertainties, disturbances, faulty sensors, variation in operator practices, and machine drift

(4) address nonlinear dynamic operating conditions of simultaneous discrete and continuous nature such as startup, shutdown, and product changeovers

Several examples are provided where wrong conclusions or poor performance were obtained during the seemingly reasonable application of data analytics, which illustrate some common misconceptions and the potential pitfalls of applying methods naively. Some specific research directions motivated by the challenges are outlined.