Evgeniy Polyakov: "History, experience, mistakes and successes in the process of creating a truly scalable storage systems"
History, experience, mistakes and successes in the process of creating a truly scalable storage systems
We create distributed storage systems in Reverbrain.com. We do it many years. This is my personal main development area for about 10 years. I made zillions of errors there, but now I know for sure that complex systems do fail. Simple systems will fail too, but at least you will be able to fix things, to recover your data and to automate all necessary things. Very simple systems, trivial systems will not work at all. For the context, one of our Elliptics clusters hosted 53+ billions of keys in 2013, for comparison, Facebook had about 450 billions (250 billions in other sources) photos those times. To make such a storage one we had to work hard, but we created remarkable storage system - it works well with couple of servers and scales easily to hundreds of nodes.
I will tell the story of my adventures during learning and development of the very different storage systems, starting from block network devices (dst, drbd, AoE) upto cloud SaaS (Elliptics). There will also be distributed filesystems (POHMELFS for instance) and many other storages. I do expect tricky questions, we may talk about almost any popular storage, CAP, consistent hashing, DHT, sharding, NoSQL and other buzzwords.
Evgeniy Polyakov
Russia. Moscow
CEO
Reverbrain.com
Present: CEO Reverbrain.com - distributed storage system Elliptics, scalable distributed search index Greylock
Past: head of the filesystem development group in Yandex (Elliptics hosts Yandex music, maps, panoramas, photos, mail, market, disk and much more other services)
Past past: linux kernel (network raid, w1, netlink, dst, netchannels, pohmelfs), hacking (dns poisoning, earned NYT article)
Past past past: embedded linux kernel hacker (ppc405/440 embedded linux, audio/video capturing)
Past past past past: MIPT DPQEPresent: CEO Reverbrain.com - distributed storage system Elliptics, scalable distributed search index Greylock
Past: head of the filesystem development group in Yandex (Elliptics hosts Yandex music, maps, panoramas, photos, mail, market, disk and much more other services)
Past past: linux kernel (network raid, w1, netlink, dst, netchannels, pohmelfs), hacking (dns poisoning, earned NYT article)
Past past past: embedded linux kernel hacker (ppc405/440 embedded linux, audio/video capturing)
Past past past past: MIPT DPQE