Thoughts on Hyperconvergence, Containers and 100TB SSDs

In the search for the holy grail of ‘compute and data co-location’, Hyperconvergence seems an interesting "avenue". Especially, with containers and the high storage-to-compute ratio offered by the forcecoming 100TB SSDs.

First, lets clear off the question of what do I mean by Hyperconvergence. The term is overloaded and I will be out of my ligue trying to define it. I am thinking of Hyperconvergence as a clustered homoginious hardware that "supplies" both storage and compute under a unified management software stack. Currently, Hyperconvergence seem to facilitate mainly VDI related use cases. In this blog post I am raising the question of whether hyperconvergence as defined above is a good fit for data centric workloads with varying degree of compute intensity.

A fundamental building block of hyperconverged systems is, well..., the hypervisor. Hypervisors or server virtualization have always carried a performance penalty when it comes to I/O intensive workloads. With containers, however, one can isolate the resources used by the storage  system without paying the notorious I/O penalty of virtualization. That is, unless one is willing to dedicate a fixed portion of the storage devices to the storage cluster part of the hyperconverged system (which is less flexible, but might make sense) and use passthrough (e.g. with SRIOV). Thus, replacing the traditional virtualization with containers the performance penalty is pretty much gone.

Another question is what should be the storage to compute ratio in such a homoginious hardware environment. This is where the forcecoming 60 and 100 TB SSDs come into play. Those 'mini tape library' in a 2.5" / 3.5" form factor' bring a lot of flexibility: Using the  exact same hardware one can choose between tens of GBs to tens of TBs drives. When multiplied by an ‘almost standard’ 12 bay servers one gets a very wide storage-to-compute ratio range.

Having high storage-to-compute ratio suggests that the hyperconverged cluster can now also serve as a massively scalable storage system such as an object store (server NAS is typically not an object store).  While price-wise this is probably not a good idea, I thought I’d play with the idea some more. and so we have aone happy hyperconverged system running an isolated object store with compute capacity. Have we found our holy grail yet? Do we know the airspeed velocity of an (African) unladen swallow?Well, not quite. To enable the data and compute co-location it seems  that we need several things:

  1. The data has an actual location. Well, this is a somewhat ‘blurred’ definition. It basically means that there is enough data in one place which makes sense to process independently of other data that may reside elsewhere. This is probably not true in “heavily striped” clustered file systems/block storage systems such as GPFS or Ceph. It is also not true for EC data (including systematic codes as the ‘stripes’ are typically small. It is, however, true for object stores that do not use EC.
  2. The data is not encoded in a way that prevents it from being processed on its  own. E.g. it must not be encrypted or encoded using a non-systematic code.
  3. The data location must be exposed to the ‘task scheduler’ –  whatever it might be - so that tasks can get scheduled where their data reside.
  4. The process needs to be able to directly access the data (locally).

Condition 1, exhibits a storage / networking tradeoff: Doing E.C. means we can pay less for storage capacity (while maintaining data redubndancy) on the expanse of moving the data if we wish to process it. Having said that, object stores seem to fulfill conditions 1 and 2, and in some cases condition 3 (e.g. Openstack Swift list_endpoints middleware that exposes the internal location of objects replicas). Condition 4, however, seems more diffcult: Object stores, being built to be offered as a service, are not tailored to allow local access to data. Moreover, Judging at least from Openstack Swift there seem to be some inherent problems in fulfilling local access. The main issue IMO has to do with access control. Openstack Swift as a two-tier architecture:

  • A client-facing tier, where authorization is enforced.
  • A ‘storage’ tier where the data is kept.

Allowing the client to directly access the storage tier, would thus break authorization.

The only system I am aware of that complies with all four requirements is Hadoop HDFS, esp. when used with a large chunk size. Still, the ‘POSIX nature’ of HDFS seems to constrain its ability to scale  massively.

To conclude: Hyperconvergence (as defined above) may be used for implementing data services infrastructure when used with containers, chosen storage-to-compute ratio and running some object store variant (Oh, yes, and putting the $/GB question aside for a couple of years). I think I will call it data-centric hyperconvergence.

Add new comment