Publication

Data management is essential for a successful field operational test or naturalistic driving study. The complexity of data management tasks and thus the resources needed are often underestimated. This deliverable, “D3.3: Data management in euroFOT”, is a public deliverable of subproject 3 (SP3: “Data Management”) of euroFOT. Its purpose is: 1) to describe the data management solutions (the data acquisition system, database and storage, and the base of analysis tool) used in euroFOT; 2) to summarise how we came to choose those solutions; 3) to summarise the quality assurance checks performed throughout the whole data management chain; 4) to share our experiences and the lessons learned. The main contributions from D3.3 to euroFOT and a wider community include: •Five state-of-the-art data acquisition systems that have been successfully used for FOT: 1) CTAG Datalogger II (CAN-only logger), 2) CEESAR videologging system (a combined CTAG Datalogger II and video logger based on Nexcom VTC3300, with radar, eye tracker and lane tracker), 3) SAFER-euroFOT DAS (an integrated CAN and video logger based on Nexcom VTC6100, with external accelerometer and eye tracker), 4) BMW DAS (a CAN and video logger based on network attached storage, with two radar sensors), and 5) Daimler DAS (a CAN, video and audio logger); •Three implementations of web-based DAS-vehicle monitoring tools for quality assurance; •The required infrastructure and services for data storage, including pre-processing and uploading tools. The solutions chosen vary depending to the expected total size of data. The video data are stored as files in file management systems. The non-video data are generally stored in relational databases; Oracle, MySQL, and MS Access are used. At some partners, the non-video data are also stored as MATLAB files in the file management systems, to allow analysis through database and/or direct with MATLAB. •Three implementations of the base data analysis tool that allows researchers to visualise time-based data from multiple sources and manually or automatically annotate the data; •A set of requirements for the design of data acquisition systems and sensors. Fulfilling this new set of requirements, in a production scale, would further advance the state of the art in data acquisition systems and sensor technologies and their applicability for FOTs; •47 lessons learned. Working methods that proved positive include: using common templates for gathering requirements; having specialised/focused groups to enable joint efforts in performing specific tasks; separating the handling of incoming disks/data from pre-processing stage; using modular approach for developing the base data analysis tool. Common DAS-hardware issues include: broken compact flash and SD cards, failing connectors and loose cameras. Recommendation made include: use industrial grade compact flash cards and SD cards; give extra attention when installing connectors and cameras; and finally be open to the potential benefits of using database features like partitioning and compression to enhance the standard relational database, using proper data warehousing, or even to other technologies (e.g. column-oriented databases, Hadoop database (HBase) or data warehouse for Hadoop (Hive)) for the management and analysis of FOT data of the scale of this project or larger projects.