We are currently conceiving of a data processing system which would analyze large amounts of data (terabytes to petabytes at rest) whereby which the storage and compute are separated. We would attempt to be smart about bringing the minimum amount of data over the wire, but network performance will still be a major factor. In AWS today, network performance is excellent and it is possible to move 40GBit/sec from S3 to an EC2 instance with high concurrency. There are now 100Gbit/sec instances and conceivable those could be saturated with data coming from S3.
I know there are lots of upgrades and 100GBit/sec top of rack is becoming common, but I'm out of touch with modern datacenters. What kind of performance could we nominally expect in the average enterprise datacenter today?
No comments:
Post a Comment