Mapping Technology Trends to Enterprise Product Innovation

Scope: Focusses on enterprise platform software: Big Data, Cloud platforms, software-defined, micro-services, DevOps.
Why: We are living in an era of continuous change, and a low barrier to entry. Net result: Lot of noise!
What: Sharing my expertise gained over nearly two decades in the skill of extracting the signal from the noise! More precisely, identifying shifts in ground realities before they become cited trends and pain-points.
How: NOT based on reading tea leaves! Instead synthesizing technical and business understanding of the domain at 500 ft. 5000 ft., and 50K ft.

(Disclaimer: Personal views not representing my employer)

Wednesday, November 12, 2014

Part V: Nuts-and-bolts of a Scale-out Distributed Storage System

This post covers Elasticity.


The configuration of a distributed system is constantly changing. The workload access characteristics shift the hotspots to different parts of the namespace.

Elasticity represents the capability of the system to automatically manage changes in cluster configuration and workload. The actuators to accomplish elasticity are:

  • Re-sharding: Splitting the namespace for better load balancing across the nodes
  • Re-mapping:
    • Re-assigning Shard-to-node mapping: This is typically for a subset of the namespace, and could be a result of nodes added/deleted, newly created shards due to splitting, and other configuration changes.
    • Re-assigning replica-to-node mapping: Changes in access can also affect the replica placement
    •  Re-assigning namespace-to-caching servers
  • Auto-scaling: This is the eventual goal, where the system can dynamically model the load and spin up/spin down service instances, nodes, network routers on the fly.


No comments:

Post a Comment