Memory via PCIE-SSD as the authoritative data source

i've blogged a few times about Fusion-IO devices and my experiences with SSD. I've been thinking about the implications of large in-memory databases that retain persistence and transactional capabilities. Turns out the game may be changing long-term to an in-memory database model forscalability. I.e. Fusion-IO demonstrated technology that extends memory to Fusion-IO devices and supports up to 19.2 TB mapped as permanent memory across an 8-way redundant network of commodity hardware using a technology known as auto-commit-memory – see https://www.fusionio.com/press-releases/fusion-io-breaks-one-billion-iops-barrier/. Via high-speed PCIE, the line between storage and memory is crossed, memory = storage and storage = memory.

Huge memory that is both partitioned and replicated with hundreds of servers in a cloud and an abstraction layer for transforming queries via web-services to the cloud and yu’ve solved the scalability problem. Easily, you could support petabytes of storage between hundreds of servers that together have all the memory in memory at the same time with redundancy and persistence to boot.

The authoritative data for future RDBMS looks to be in memory-data accessed via the cloud with a controller layer to parlay queries.  SQL databases become merely snapshotsin time. If you factor the growth rate in flash density, PCIE 3.0 with bandwidth higher than DDR3 and the approach to commoditize PCIE in the same way 2.5 SAS slots are (see the Dell R720 offering with hot-swap PCIE), then we may be easily looking at memory-mapped 100TB machines in the next couple of years. That along with controller-layer and partitioning schemes to divide up large datasets among peer servers of the and you have scalability through an interface that runs the query against in-memory representations of tables/indexes rather than on-disk versions with peer-to-peer memory mirroring through messaging rather directed from the stored databases.

I'm still getting upto speed on HADOOP, etc, but memory shared between multiple servers seamlessly as authoritative is probably a key enabler.

Sounds pretty radical, but the times – They are a-Changin – Bob Dylan –- https://www.youtube.com/watch?v=vCWdCKPtnYE

Bob