The Bluesky Project Contributors
Bluesky originated at National Synchrontron Light Source II
("Giant X-ray beam")
NSLS-II is a "User Facility"
Build software that enables collaboration and specialization
Bluesky is designed in service to data analysis
When analyzing data we want....
Bluesky may be used from IPython or from graphical user interfaces
Bluesky is a bridge to the open-source ecosystem
Figure Credit: "The Unexpected Effectiveness of Python in Science", PyCon 2017
Bluesky is written in Python, which is very popular
Figure Credit: Stack Overflow Blog https://stackoverflow.blog/2017/09/06/incredible-growth-python/
Bluesky is designed for the long term
Bluesky has individually useful core components
Other facilities have adopted Bluesky piecemeal, adapting, extending, or replacing components to meet their requirements.
List of facilities known to use Bluesky (1 of 2)
List of facilities known to use Bluesky (2 of 2)
We can learn a lot from particle physics, astronomy, and climate science....but we have some unique problems too.
What changed to make data problems harder?
"Big data is whatever is larger
than your field is used to."
A spot check for data volume at an NSLS-II Project Beamline so far...
What changed to make data problems easier?
HPC is becoming more accessible.
One inviting example: jupyter.nersc.gov
Also: Commodity cloud-based tools
...which is not a new idea, but ease-of-use matters.
meta_data_in_37K_fname_005_NaCl_cal.tif
What's the problem?
What do we need to systematically track?
Experimental Data
Analysis needs more than "primary" data stream:Sample Data
Bureaucratic & Management Information
both technical and sociological
for an end-to-end data acquisition and analysis solution that leverages data science libraries
Technical Goals
Sociological Goals
Bluesky is designed for Distributed Collaboration
Bluesky is designed for Distributed Collaboration (cont.)
Layered design of Python libraries that are:
Looking at each component, from the bottom up....
Device Drivers and Underlying Control Layer(s)
You might have a pile of hardware that communicates over one or more of:
Ophyd: a hardware abstraction layer
trigger()
, read()
, and set(...)
.
Bluesky: an experiment specification and orchestration engine
Mix and match (or create your own) plans...
...and streaming-friendly viz...
...and streaming-friendly analysis
High Throughput
trigger
,
read
, save
, ...).
kickoff
("Go!") — complete
("Call me when you're done.") — collect
("Read out data
asynchronously.").Suitcase: store in any database or file format
DataBroker takes the hassle out of data access.
Keep I/O Separate from Science Logic!
Interfaces, not File Formats
The most important aspect of the Bluesky architecture are the well-defined protocols and interfaces.
Interfaces enable:
Interface Example: Iteration in Python
for x in range(10):
...
class MyObject:
def __iter__(self):
...
for x in MyObject():
...
Interface Example: numpy array protocol
import pandas
import numpy
df = pandas.DataFrame({'intensity': [1,1,2,3]})
numpy.sum(df)
Interfaces in Bluesky
Work openly
Build a lasting collaboration
Automated tests are essential
They enable people to try new ideas with confidence.
Good, current documentation is essential.
It convinces people that it might be easier to learn your thing than to write their own.
Minimalist and Extensible
Bluesky emits documents, streamed or in batches
This is an area of very active development is Bluesky.
Coordinated efforts underway at:
Feedback Paths
Scales of Adaptive-ness
below bluesky
&
ophyd
in bluesky
plans,
but without
generating event
providing feedback on a per-event
basis
providing feedback on a per-run / start
basis
providing feedback across many runs
asynchronous and decoupled feedback
Docs with theory and examples:
bluesky/bluesky-adaptive
Bluesky Queue Server
Bluesky's first target was users coming from SPEC
New Capability: Editable Control Queue
Separation between user app and queue server
Various institutions are building graphical user interfaces on Bluesky.
pyStxm at Canadian Light Source (Russ Berg)
GUI for SAXS at Australian Synchrotron (Stephen Mudie)
GUI for COSMIC at Advanced Light Source (Xi-CAM Team)
Finally, various one-off solutions developed by beamline and/or Controls staff at NSLS-II
We intend to guide a systematic refactor of these onto components from bluesky-queueserver and bluesky-widgets.
A new project aimed at sharing GUI components built on Bluesky interfaces
bluesky/bluesky-widgets
Goals
Examples of integrating Data Broker search into existing software...
Model can be manipulated from IPython terminal
Search Data Broker from napari (N-dimensional image viewer)
Search Data Broker from PyFAI (powder diffraction software)
Search Data Broker from Xi-CAM
Proof of concept:
In this scan, each step is determined adaptively in response
to local
slope.
The system is designed to make fast feedback easy to write.
LCLS's Skywalker project builds on this to automatically deliver the photon beam to a number of experimental hutches at LCLS.
A stream of images from a linear detector is reconstructed into a volume using tomopy (APS).
It took one TomoPy developer and one Bluesky developer less than 20 minutes to write this.
A Gaussian is fit to a stream of measured data using the Python library lmfit (from U. Chicago / APS).
The Xi-cam 2 GUI / plugin
framework from
CAMERA
has adopted Bluesky's Event Model
for its internal data structures.
Real-time Data Analysis at APS
Data is streamed from APS to Argonne Leadership Compute Facility. Results are immediately visualized at APS.
玻璃钢生产厂家上饶特色玻璃钢雕塑江苏知名玻璃钢雕塑服务介绍呼和浩特玻璃钢动物马雕塑江苏装饰商场美陈怎么样玻璃钢雕塑平面图芜湖玻璃钢雕塑雕塑天筑玻璃钢雕塑多少钱岳阳玻璃钢雕塑定制黎城玻璃钢雕塑厂家上饶玻璃钢浮雕人物山水雕塑福建主题商场美陈有哪些郑州广场标识校园玻璃钢雕塑商场美陈广宣检查鹤壁肖像校园玻璃钢雕塑温州玻璃钢雕塑摆件售价西城春节商场吊饰美陈玻璃钢仿铜雕塑厂家地址徐州设计玻璃钢雕塑方案四川步行街玻璃钢雕塑供应商园林人物玻璃钢雕塑摆件打造厂家四川走廊商场美陈制造山西大型玻璃钢雕塑制作金昌仿真玻璃钢雕塑订做浙江省玻璃钢雕塑制品厂家甘南玻璃钢卡通雕塑定做绍兴组合式玻璃钢花盆玻璃钢草帽雕塑徐州玻璃钢人像雕塑贵州玻璃钢校园雕塑商场美陈培训香港通过《维护国家安全条例》两大学生合买彩票中奖一人不认账让美丽中国“从细节出发”19岁小伙救下5人后溺亡 多方发声单亲妈妈陷入热恋 14岁儿子报警汪小菲曝离婚始末遭遇山火的松茸之乡雅江山火三名扑火人员牺牲系谣言何赛飞追着代拍打萧美琴窜访捷克 外交部回应卫健委通报少年有偿捐血浆16次猝死手机成瘾是影响睡眠质量重要因素高校汽车撞人致3死16伤 司机系学生315晚会后胖东来又人满为患了小米汽车超级工厂正式揭幕中国拥有亿元资产的家庭达13.3万户周杰伦一审败诉网易男孩8年未见母亲被告知被遗忘许家印被限制高消费饲养员用铁锨驱打大熊猫被辞退男子被猫抓伤后确诊“猫抓病”特朗普无法缴纳4.54亿美元罚金倪萍分享减重40斤方法联合利华开始重组张家界的山上“长”满了韩国人?张立群任西安交通大学校长杨倩无缘巴黎奥运“重生之我在北大当嫡校长”黑马情侣提车了专访95后高颜值猪保姆考生莫言也上北大硕士复试名单了网友洛杉矶偶遇贾玲专家建议不必谈骨泥色变沉迷短剧的人就像掉进了杀猪盘奥巴马现身唐宁街 黑色着装引猜测七年后宇文玥被薅头发捞上岸事业单位女子向同事水杯投不明物质凯特王妃现身!外出购物视频曝光河南驻马店通报西平中学跳楼事件王树国卸任西安交大校长 师生送别恒大被罚41.75亿到底怎么缴男子被流浪猫绊倒 投喂者赔24万房客欠租失踪 房东直发愁西双版纳热带植物园回应蜉蝣大爆发钱人豪晒法院裁定实锤抄袭外国人感慨凌晨的中国很安全胖东来员工每周单休无小长假白宫:哈马斯三号人物被杀测试车高速逃费 小米:已补缴老人退休金被冒领16年 金额超20万