Description
"nA key challenge to reproducible data collection is capturing and organizing metadata without constraining flexibility and improvisation. u201cBlueskyu201d is a data collection framework designed to solve this problem. It communicates with hardware through a high-level interface and performs data collection, capturing data and rich metadata for streaming, live analysis.n n The project is developed at the National Synchrotron Light Source II --- a Dept. of Energy X-ray user facility. The X-ray beam is used by internal scientific staff and external visitors from academia and industry. These users employ bluesky in a broad range of experiments, ranging from well-defined, established techniques to ad hoc, improvised experiments.n n Bluesky expresses an experimental procedure as an iterable. Each element in the iterable specifies a granular step: u201cMove motor X; read detector Y; ...u201d. Bluesky supervises the execution of each step, handling common supervisory tasks: monitoring for problems, recovering from interruptions, safely cleaning up. In hardware control, the devil is in the details. Bluesky handles many of these details for the user, separating them from the scientific logic of the experimental procedure.n n While executing the procedure, bluesky collates all measurements and metadata into Python dictionaries with a flexible schema. These data u201cdocumentsu201d are created in a streaming fashion during the experiment and dispatched to user-defined functions. They can be printed, plotted, written into a database, broadcast as JSON documents, or fed into a real-time processing pipeline.n n Bluesky employs Python language features not as commonly used in the scipy community: generators, coroutines, and the asyncio event loop. For example, generators provide a parsimonious syntax for expressing sequential steps of an experiment. Coroutines can express adaptive logic, such as spacing measurements adaptively in response to the local slope, concentrating on regions of high variability. The asyncio event loop manages hardware control, data collection, visualization, and light-weight analysis in a single process. Wherever possible, bluesky relies on core language features and built-in data structures, avoiding a proliferation of special classes or a sprawling vocabulary.n n By integrating cleanly with the scipy stack, bluesky empowers scientists to build sophisticated experimental control logic. While currently deployed for X-ray experiments at NSLS-II, bluesky is developed in the open and should be useful for scientific experiment control in any context. It is thoroughly documented at nsls-ii.github.io/bluesky. The source code is available at github.com/NSLS-II/bluesky.n nTalk co-authers: Thomas Caswell, Brookhaven National Lab, Kenneth Lauer, Brookaven National Lab."