Description
Francesc Alted - New Trends In Storing Large Data Silos With Python [EuroPython 2015] [20 July 2015] [Bilbao, Euskadi, Spain]
My talk is meant to provide an overview of our current set of tools for storing data and how we arrived to these. Then, in the light of the current bottlenecks, and how hardware and software are evolving, provide a brief overview of the emerging technologies that will be important for handling Big Data within Python. Although I expect my talk to be a bit prospective, I won't certainly be trying to predict the future, but rather showing a glimpse on what I expect we would be doing in the next couple of years for properly leveraging modern architectures (bar unexpected revolutions ;).
As an example of library adapting to recent trends in hardware, I will be showing bcolz (https://github.com/Blosc/bcolz), which implements a couple of data containers (and specially a chunked, columnar 'ctable') meant for storing large datasets efficiently.