Cloud storage solutions needs to be separated into two markets, the consumer and business markets. In this blog we compare the end point cost of cloud storage provided in the consumer market.
When we do narrow Cloud Storage down to the price point, we get a crude comparison of granny smith apples, to google apples, to microsoft apples, to fuji applies, etc.
What the basic end price point now tells consumers is that they can now use the cloud to share data across machines and locations, that they can dump their physical backup drives and that over all, it is nearly time to migrate your digital lives and consumer cloud services. What we are wondering is this - Is the current storage solution pricing a loss leader to other objectives and is this based on realistic demand patterns ?
Note that for Amazon AWS S3 - we summed the storage and data upload for the same storage amount for this comparison. Akamai, Amazon AWS, Dell, Google, HP, IBM, Microsoft Azure, Oracle, Rackspace, Salesforce, Yahoo. The Skills Framework for the Information Age (SFIA) provides a common reference model for identifying professional skills. Wikibon is a professional community solving technology and business problems through an open source sharing of free advisory knowledge. New data center deployment philosophies that allow data to be shared across the enterprise rather than stove-piped in storage pools dedicated to particular types of applications. This research discusses the key trends in flash and disk technologies in detail and the potential impact of new application design philosophies. CXOs and senior enterprise business leaders will need to restore their faith that IT can and will improve its dismal record of delivering on complex deployments. IT senior management will need to change and reorganize IT infrastructure management, slim down infrastructure staffing, automate and orchestrate deployment of IT resources, and move rapidly towards a shared data philosophy enabled by flash storage deployment for almost all enterprise data. All-flash array vendors and other flash architectures will need to focus on developing scale-out architectures and maximize the potential for data sharing. The end-point of the adoption of flash-only persistent storage is a completely "electronic data center", with no mechanical components except for a few pumps and fans. When magnetic drum technology was first introduced, access time to data was as fast as the compute technology of its day. The history of storage array development has been the application of technologies to mitigate this disparity in cycle time. The attempts to mitigate the mechanical problems have been significant, but the impact of disk storage limitations on application design has also been stark.
The bottom line is that current applications are dependent on very complex infrastructure software to mitigate the ever increasing gap between compute and persistent storage cycle times.
Has very low maintenance (not yet passed on by vendors), and very low power & space costs. The bottom line is flash is not “perfect”, and disk is currently better at sequential writes.
Buffers are a partial solution, not nearly as good to (say) a PC enduser as all-flash storage as Apple has shown with the success of the all-flash “Air” products.
Separate buffers on each disk is a sub-optimum way of designing buffers, both for IO average latency and more important IO latency variance (jitter). Disk technology is used in consumer PCs and enterprise storage in servers and storage arrays. The bottom line is that there is no development of performance hard disk disks and some limited potential innovation in capacity disks but not sufficient to stop the deployment of the electronic data center. All mobile devices have used only flash and ARM technology, and increasingly PCs are flash-only. The bottom line is that data centers are able to take advantage of the huge flash volumes for the consumer market. There has been much discussion about replacing flash with some other type of Non-volative Memory (NVM) technology. Understanding exactly how the technologies work and how they can be produced in volume and at high yield; MRAM is in production on low-density chip technology, and may make some inroads into DRAM, but other technologies are still at the university stage and are a long way from volume production. Developing sufficient demand to enable the investment necessary to deploy them on the latest chip technology (and therefore attain the performance gains); the catch 22 of new technologies, the challenge of securing investment to deploy on the latest technology to increase the volume and lower cost in the long run. Flash has a strong future path laid out to 3D technology and has shown that investment has managed to solve the problems that the pundits for new NVM technologies said were impossible to solve. MRAM is in limited production, and has a potential role as an alternative for capacitor-protected DRAM. The bottom line, as Wikibon has stated before (See Note 2 in the linked reference), is that flash will be the only NVM technology of importance for enterprise computing for at least the next five years (end of this decade) and very probably for the next decade.
Disk drives can crash, and, as the name implies, they crash catastrophically from a user perspective. Flash drives “wear”, that is the more times you write data to a flash cell, the higher the probability that a cell will fail. The bottom line is that flash drives are much more reliable that disk drives, and will continue to increase in reliability. De-duplication and compression have been around for a long time, introduced by Microsoft on PCs and NetApp on arrays and now provided by most disk-storage array vendors. The storage controller overhead for data reduction processing and management is very high, and in traditional two-controller array architectures can itself lead to significant performance impacts, especially when combined with other storage array software such as replication.
Permabit has introduced the advanced data reduction technology for traditional storage arrays, and have recently announced partnerships with EMC, NetApp and others. The reason that flash can support de-duplication and compression is that the access density for flash is much higher, 100s of times higher than mechanical disk drives.
The bottom line is that flash can support data reduction techniques such as de-duplication and compression as standard, an important contribution in the journey to the electronic data center. Another potential example is the development of applications that analyze large amounts of data in real-time and then adjust the operational systems in real time. The biggest constraint in moving to this type of environment will be persuading senior storage administrators and DBAs, who have grown up with the current constraints of existing arrays, to utilize completely new ways of managing production storage. The bottom line is that management has a significant role in driving attitude change in IT management and IT vendors (software & hardware), necessary for the full deployment of the electronic data center. The level of data sharing that can be achieved, by building large scale-out flash architectures to increase the amount of data sharing; the purple line shows the increase from just over 1 to a factor of 6 as the shared data model is adopted in data centers.
Work which does produce high de-duplication rates will be migrated to their storage arrays.
The potential for shared copy data is greater that de-duplication and compression, as can be seen in Figure 2. To achieve higher levels of data sharing, the architectures of all-flash arrays and the business practices of data sharing will need to evolve. Figure 1 below shows the projection of the true cost of raw HDD and SSD technology, with all the costs (except for infrastructure personnel costs) included. The bottom line is that the era of the electronic data center is cost justified now, that data sharing is a key metric to optimize, and with speed to deploy advantages it will be a major factor in establishing business differentiation. Whenever a new technology is introduced, the initial imperative is to minimize the cost of migration by avoiding having to make any changes to applications or the infrastructure supporting them. This initial implementation of flash allowed confidence to grow in flash as a medium for persistent enterprise storage. A number of hybrid architecture storage arrays have been introduced with much higher amounts of flash and where the master copy of data is written through to a disk back-end. This approach leads to more consistent IO response times and higher IOPS but still maintains disk as the ultimate storage.
Because of the foundation of hybrid storage arrays linked to disk drives, it is not a foundation for a modern shared storage architecture. These architectures are limited by the dual controller in both function that can be deployed and in the amount of de-duplication that can be achieved. This is the subject of the next section, and is key to the adoption of the electronic data center.
Adopting a shared data philosophy within the data center mandates that the flash persistent storage arrays supporting this deployment have a number of architecture attributes.
To allow sharing of data across an enterprise, the storage arrays must scale beyond the traditional dual-controller architecture. A scale-out architecture must allow access by all nodes to all the metadata describing the data, snapshots of data and applications accessing the data.
With traditional disk arrays, snapshot management is a challenge in production workloads, especially large-scale production workloads.
With all-flash arrays, the access density of flash allows snapshot copies of data to have potentially the same performance as the original and enables multiple snapshot copies to be made of the original and derivative snapshots. Full-performance, space-efficient writeable snapshots with full metadata support are key to data sharing.
The management of snapshots must include a full-function catalog describing when copies were taken, how presented, and when they were used by other applications (this function can be within the array or can be provided by external software with APIs to the array). Advanced snapshot technologies will completely alter the way that high-performance systems are backed up, allowing specific RPO and RTO SLAs to be adopted dynamically for different applications, and eliminating the requirement for de-duplication appliances. Quality of service should ideally include minimum and maximum performance on an application basis to map directly to application SLAs. Quality of service should allow a view by physical use of shared data as well as the normal logical use by LUN or equivalent, so as to provide management tools to enable full performance management.
The bottom line is that scale-out architectures with rich metadata support, consistent latency, advanced snapshot technologies and advanced quality of service are a prerequisite for fourth generation all-flash storage arrays. Figure 1 shows the worldwide All-Flash array revenues by vendor, taken from a recent report from IDC. The data shows that the leading scale-out architectures are EMC (23% share) and SolidFire (7% share) making scale-out architecture about one third of the marketplace.
No current all-flash array meets all the requirements laid out in the “Ideal Scale-out Shared Data Architectures” section above. EMC is well positioned with the XtremIO scale-out all-flash array with a fundamentally good data reduction-led (de-duplication & compression) architecture, with good metadata management and data sharing characteristics.
SolidFire also has a solid scale-out architecture with good QoS functionality, focused initially on storage service providers. Wikibon believes that general purpose dual-controller all-flash array vendors such as Pure Storage and Nimbus will be restricted to IT enterprises that do not accept the benefits of shared data, unless they move to a scale-out architecture. The bottom line is that the scale-out all-flash sector is set to explode, and enterprises embracing the benefits of a shared data philosophy will benefit greatly from lower IT costs and improved application functionality. Wikibon is developing a case study of a financial enterprise managing more than $150 billion in assets that is rapidly completing its journey to the electronic data center. Wikibon has written about other trends in flash architectures, in particular with Server SAN or Hyperscale architectures. This research shows that flash will become the lowest cost media for almost all storage from 2016 and beyond, and that a shared data philosophy is required to maximize the potential from both storage cost and application functionality perspectives. The continual reduction in flash costs driven by consumer demand for flash - this is inexorable as mobile phones, tablets, phablets and wearable devices continue to roll-out, along with new markets such as the Internet-of-things. New Scale-out flash array architectures that allow physical data to be shared across many applications without performance impact. New data center deployment philosophies that allow data to be shared across the enterprise, rather than stove-piped in storage pools dedicated to particular types of application. Wikibon recommends that deployment of a data sharing philosophy is an important step in the deployment of the electronic data center.


Is the timescale for migration of all mission critical workloads to an electronic data center infrastructure less than 2-years? What are the most important applications that will use the electronic data center to improve enterprise productivity and revenue?
For example data demand patterns in telecom data usage which the models may be based upon may not represent the usage patterns in storage. Business game-changing potential waits for enterprises that develop and deploy new data-rich applications that can profoundly improve productivity and revenue potential. Within the electronic data center, electronic storage will allow logical sharing of the same physical data to be fully enabled. Improved magnetic heads reduced the size of magnetic “blob”, so tracks could be closer, with more data per track. Read buffers and smart algorithms attempt to maximize the probability that data required by applications would be found in DRAM.
As Wikibon has pointed out in previous research, applications have been written and designed with small working sets that fit into small buffers.
This complex software is spread between the controllers of storage arrays and ever more complex databases from Oracle, IBM, Microsoft, and others that protect applications. The complexity and resultant cost of deploying additional database, storage array software and storage management software to enable moderate or large-scale applications is very high. However, this research will show that flash is cheaper now than disk for active data and will become cheaper than disk for almost all applications in the data center.
The disk market for mobile devices (laptops, phones, wearable devices, etc.) is rapidly dwindling as flash takes over, and bulk data requirements are provided in the cloud. For example, HGST is planning to introduce helium drives with Shingled Magnetic Recording (SMR).
The first introduction of flash in a large-scale consumer product was for the Apple iPod in September 2005, powered by ARM chip technology. Flash technology has improved at a faster rate than Moore’s Law, with more than 50% improvement in annual price performance and density.
The proponents of these technologies usually start presentations at IEEE meetings with the premise that flash has reached the end of the line and cannot be developed further. Flash, warts and all, is the persistent storage medium for the foreseeable future in the electronic data center. Most PC users have experienced the impact of a disk crash, and recovery takes a long time and is often incomplete. The average maintenance cost of disk drives is about 18% of the acquisition price per year. Wikibon research shows that there is a compelling case to used compression & de-duplication on the flash storage component within hybrid arrays, but few applications can support it on disk-based arrays.
The overhead on storage and array controllers is still high, and flash storage arrays will have to be designed to reduce the overhead as much as possible. Deeply imbedded in disk storage practice is avoiding applications accessing the same data on the same drives and avoiding the same database tables being accessed from multiple applications in database environments.
Space-efficient snapshots capture the blocks that have changed since the last snapshot, in the same way as a traditional differential backup, and allow rapid rollback to previous versions of data. Using space-efficient snapshots together with high-performance metadata, a logical copy of data can be made available to other applications in seconds, but sharing the same physical data where it has not been changed. This type of system is much easier to implement if the data is shared between operational and analytic applications.
The biggest constraint for storage vendors with new types of flash arrays will be persuading them that the architecture and metadata management of storage arrays has to be completely different for this new environment. The all-flash storage vendors often argue that all data can produce data reduction rates of five or higher and show “call home” figures that prove that their arrays achieve that.
In six years time, the cost of data center storage will be 40 times lower than today, will be flash only, will have higher data transfer and IO rates, and IO densities that are thousands of times greater than today. EMC announced its first flash SSD drives in early 2008, less than three years after the first large-scale consumer introduction into the Apple nano in September 2005.
As an example, Tintri introduced a “flash-first” design that writes all data to flash, and then trickles down storage to high-capacity disk as the number of accesses to data decrease over time. Wikibon predicts that this percentage will increase dramatically over the next few years, as the value of a shared data philosophy permeates IT infrastructure management.
They have been successful in this space but will have to provide a broader portfolio to break into enterprises which follow a shared data philosophy.
It is too early to judge FlashRay against the criteria in the “Ideal Scale-out Shared Data Architectures” section. Functions such as atomic writes will allow flash to become an extension of DRAM, and improve the functionality of large scale database driven applications. This is probably the biggest challenge in educating senior storage executives to reverse the storage management principles of a lifetime. This again will need significant re-education of senior application analysts and architects as to what is now possible with new technologies. As part of our benchmarking we recently decided to run a test designed to validate our tooling and automation scalability as well as the performance characteristics of Cassandra. It shows that flash (the blue line) will become a lower cost media than disk (the red line) for almost all storage in 2016, as scale-out storage technologies enable higher levels of data sharing, and lower storage costs.
These application that combine operation with in-line analytics can only be realized on flash-only shared-data environments.
CEOs should be asking IT about definitive plans to move to an electronic data center, the timescale of the migration, and information about the first application(s) to employ the electronic storage to radically improve enterprise productivity and revenue. Compute and persistent storage were balanced, with ten microseconds for compute and ten microseconds for access to persistent storage (See Note 3 of linked reference for more detail).
The technology to fabricate magnetic heads was the same fundamental technology used to fabricate memory and logic chips. Battery and capacitance protect DRAM write buffers, allowing high burst rates of data to be written with low latency. NoSQL databases offer apparent relief for some applications but in most instances just move the complexity from the database back to the programmer. The disappointment felt by business leaders in the failure of IT to meet its potential to decrease business costs can be placed squarely in this complexity, caused almost exclusively by the failure of mechanical persistent storage to keep up with compute and network technologies. It will show that flash is much better as a foundation for the next generation of big data applications and will become the foundation for the electronic data center. Desktop PCs are a sharply declining market with multiple alternatives including VDI and PC cloud services from Google and Microsoft.
This overlaps the tracks and is expected to eventually enable data densities as high as 3 trillion bits per square inch.
It had far less capacity than the “classic” hard-disk alternative iPod but allowed smaller and lighter devices, needed less power and less recharging and was much more resilient when dropped. The technologies are exciting and include MRAM, RRAM (HP’s Memristor is in this category), PCM & racetrack technologies.
With atomic writes, the access time between disk and flash is reduced by 10,000 times, from a best of 1 milliseconds to 100 nanoseconds for a line write of 64 bytes (See Note 3 in the link for a technical deep dive).
However, “wear leveling" software has advanced dramatically and is now implemented in the controllers of solid state drives.
For example, a clone of the current production environment is made for the members of the development team. They are also a very useful type of snapshot for taking logical copies of the same data; however on traditional disk storage the performance characteristics and metadata management largely made these copies useless for active disk-sharing process workflows. For example, the development team at a European investment management company gets a copy of the production database and then publishes the latest full version of this application and data to all the developers, testers and QA. The key question now is when all disk drives systems will be more expensive than flash drives. The data in Figure 2 below was generated using the data and assumptions from that study, adding in the cost of power, space and maintenance, and adding in the potential improvement in practical de-duplication rate that can be achieved for all workloads. As was discussed earlier, the uplift of storage arrays over raw technology is very high at the moment (10-15x).
Most importantly, it will remove IO as a constraint to application innovation, and allow real-time deployment of big data systems. EMC with the VNX and others have re-written their controller software to facilitate the adoption of larger amounts of flash storage. Wikibon believes that EMC will grow faster than the overall rate of the all-Flash array industry.
This will be another way of improving the productivity of users and organizations, and further integrating big data analytics with operational systems. Only if you call wear leveling and failure prediction in flash drives reliable, yes - it is reliable to assume that flash drive will fail inevitably at some point because those cells do wear out at a very predictable rate.
The number of database calls within a business transaction have been severely limited by the high variance of disk-based storage; this is necessary in order to reduce operational and application maintenance complexity and cost. Previous Wikibon research has shown the design of application suites such as SAP and many others are also constrained by disk-based storage, and apparently “simple” projects combining multiple landscapes and integrating business processes are in reality difficult, time-consuming and risky. The magnetic disk drive is the last mechanical device in the path of the electronic data center. The high-speed enterprise disks are a rapidly declining market as flash dominates with lower costs of IO performance and lower costs of maintenance by storage administrators and DBAs. SMR is designed for continuous writing and erasing rather than random updates, and especially suitable for write-once-read-never (WORN) applications. The “Classic” iPod with a 1.8 inch hard drive and up to 160 GB of space was finally withdrawn in September 2014 as cloud service offerings made the higher capacities irrelevant. With relatively little additional investment, flash technologies can be used in the enterprise datacenter. History teaches us that only two successful major volume memory technologies have been introduced in the last fifty years - DRAM & flash. RRAM and PCM technologies have military potential in niche usage, and these technologies may have uses in micro-sensors for the Internet-of-things. Where disks are bound together in RAID groups, the data is not lost if a single disk fails. Wear leveling is now a problem that NAND flash suppliers have solved and will continue to improve. Flash drives can be loaded in hours, compared with the days or months that it takes in a typical migration from one disk-based array to another.
Full clones are invariably used if multiple copies are needed for (say) application developers, and often only a subset of the data is used.
The developers have gone from an IO constrained to a processor constrained development environment - they needed faster processors to improve development time.
The assumptions used in the chart are detailed in Table Footnotes-1 in the Footnotes below. This ratio will drop significantly over the next five years but will not matter to the final outcome because flash arrays will have the ability to match or exceed these drops in uplift ratio.
These drives were used to release storage volumes that are a bottleneck in applications, especially in databases. This trend will also result in many storage services to migrate from the SAN array to the server.
Writes stress a data store all the way to the disks, while read benchmarks may only exercise the in-memory cache.


All these technologies help if applications are designed in a certain way, with small working sets and limited functionality. Large applications are still designed as a series of modules with loosely coupled databases. Disk vendor CEO claims that there is not enough money to build enough flash capacity to replace disk drives are simply false - the consumer flash market's virtual elimination of consumer disk is proof positive. However, Wikibon believes that unless a new NVM technology is adopted in consumer products in volume, the continued investment in flash technologies driven by consumer demand will make volume adoption of any NVM alternative very unlikely, and make their adoption in enterprise computing economically impossible.
SSDs are now offered with longer life-time guarantees than disk drives (5-10 years and expected to increase).
The write off-period for flash arrays will be longer and less risky that current disk-drive arrays, which will contribute to the early deployment of the electronic data center.
They have gone from a subset of production to a full production copy, so testing and QA is far more effective.
The use of these drives was enhanced by the introduction of tiering software such as EMC's FAST VP that allowed automatic migration of volumes and parts of volumes to flash. The complexity of dealing with occasional very high latencies still exists within this flash implementation, and it is not suitable as a basis for new application design.
This switch will require changes to the functionality of databases and file-systems and is not going to be adopted quickly, but rather as a slow 10-year migration. The benchmark results should be reproducible by anyone, but the Netflix cloud platform automation for AWS makes it quick and easy to do this kind of test.The automated tooling that Netflix has developed lets us quickly deploy large scale Cassandra clusters, in this case a few clicks on a web page and about an hour to go from nothing to a very large Cassandra cluster consisting of 288 medium sized instances, with 96 instances in each of three EC2 availability zones in the US-East region. However, the rate magnetic data could be read or written (bandwidth) has only improved by less than the square root of the improvement of the data density (~10-12% per year). A whole major industrial market and supply chain has been built around “standard” disks from Seagate, HGST, and others housed in disk shelves and surrounded by proprietary software on Intel processors.
There will be sufficient flash capacity available to enable the deployment of the electronic data center. In arrays such as the IBM XIV, recovery times are reduced by spreading all the data over all the disks, which improves recovery time at the cost of doubling the storage required. All-flash arrays will dominate new storage arrays in the early part of the evolution of the electronic data center.
Using an additional 60 instances as clients running the stress program we ran a workload of 1.1 million client writes per second.
The physics of spinning the media faster soon reached a plateau, as the outer edge of disk drives approached the speed of sound.
The proprietary software from EMC, NetApp, HP, IBM, and others has been the “glue” that has allowed high functionality with very high uplifts (10-15:1) on the “Fry’s Price”, the base cost of a single disk drive. Often the data copies are a subset - for example, developers may be given a small subset of the production database.
The characteristics of flash mean that the overhead is near zero, and the time to produce a new development copy is reduced by factors of 10 or more. The value of this software has decreased as flash prices have dropped much faster than disk. Data was automatically replicated across all three zones making a total of 3.3 million writes per second across the cluster. Disks became smaller, heads moved across the magnetic media from one track to another, but nothing has had much impact on the time to access data. The reason is the elapsed time taken to make these copies, as well as the physical space taken. Some customer quotes are included in the "Case Study: Implementing The Electronic Data Center" section below. The entire test was able to complete within two hours with a total cost of a few hundred dollars, and these EC2 instances were only in existence for the duration of the test. Magnetic disks are limited by rotation speed, and access times are measured in milliseconds, while compute cycles are measured in nanoseconds. There was no setup time, no discussions with IT operations about datacenter space and no more cost once the test was over.To measure scalability, the same test was run with 48, 96, 144 and 288 instances, with 10, 20, 30 and 60 clients respectively.
The load on each instance was very similar in all cases, and the throughput scaled linearly as we increased the number of instances.
Our previous benchmarks and production roll-out had resulted in many application specific Cassandra clusters from 6 to 48 instances, so we were very happy to see linear scale to six times the size of our current largest deployment. The time taken by EC2 to create 288 new instances was about 15 minutes out of our total of 66 minutes.
The rest of the time was taken to boot Linux, start the Apache Tomcat JVM that runs our automation tooling, start the Cassandra JVM and join the "ring" that makes up the Cassandra data store. For a more typical 12 instance Cassandra cluster the same sequence takes 8 minutes.The Netflix cloud systems group recently created a Cloud Performance Team to focus on characterizing the performance of components such as Cassandra, and helping other teams make their code and AWS usage more efficient to reduce latency for customers and costs for Netflix.
The other instance type we commonly use for Cassandra is an M2 Quadruple Extra Large (m2.4xl) which has eight (faster) CPUs, 68GB RAM and two disks of 800GB each, total 26 units of CPU power, so about three times the capacity. In this case we were particularly interested in pushing the instance count to a high level to validate our tooling, so picked a smaller instance option.
Three ASGs are created, one in each availability zone, which are separate data-centers separated by about one millisecond of network latency. EC2 automatically creates the instances in each availability zone and maintains them at the set level. If an instance dies for any reason, the ASG automatically creates a replacement instance and the Netflix tooling manages bootstrap replacement of that node in the Cassandra cluster.
It is also possible to efficiently double the size of a Cassandra cluster while it is running. Each new node buddies up and splits the data and load of one of the existing nodes so that data doesn't have to be reshuffled too much. If a node fails, it's replacement has a different IP address, but we want it to have the same token, and the original Cassandra replacement mechanisms had to be extended to handle this case cleanly. We call the automation "Priam", after Cassandra's father in Greek mythology, it runs in a separate Apache Tomcat JVM and we are in the process of removing Netflix specific code from Priam so that we can release it as an open source project later this year. We have already released an Apache Zookeeper interface called Curator at Github and also plan to release a Java client library called Astyanax (the son of Hector, who was Cassandra's brother, and Hector is also the name of a commonly used Java client library for Cassandra that we have improved upon). We are adding Greek mythology to our rota of interview questions :-)Scale-Up LinearityThe scalability is linear as shown in the chart below. Each client system generates about 17,500 write requests per second, and there are no bottlenecks as we scale up the traffic.
Each client ran 200 threads to generate traffic across the cluster.Per-Instance ActivityThe next step is to look at the average activity level on each instance for each of these tests to look for bottlenecks.
A summary is tabulated below.The writes per server are similar as we would expect, and the mean latency measured at the server remains low as the scale increases.
The response time measured at the client was about 11ms, with about 1.2ms due to network latency and the rest from the Thrift client library overhead and scheduling delays as the threads pick up responses.
The write latency measured at each Cassandra server is a small fraction of a millisecond (explained in detail later). Average server side latency of around one millisecond is what we typically see on our production Cassandra clusters with a more complex mixture of read and write queries. This could be due to a random fluctuation in the test, which we only ran once, variations in the detailed specification of the m1.xl instance type, or an increase in gossip or connection overhead for the larger cluster. Disk reads are due to the compaction processing that combines Cassandra SSTables in the background. Network traffic is dominated by the Cassandra inter-node replication messages.Costs of Running This BenchmarkBenchmarking can take a lot of time and money, there are many permutations of factors to test so the cost of each test in terms of setup time and compute resources used can be a real limitation on how many tests are performed. Using the Netflix cloud platform automation for AWS a dramatic reduction in setup time and cost means that we can easily run more and bigger tests.
This changes with Cassandra 1.0, which has an improved compaction algorithm and on-disk compression of the SSTables.
We ran this test in our default configuration which is highly available by locating replicas in three availability zones, there is a cost for this, since AWS charges $0.01 per gigabyte for cross zone traffic. An estimation of cross zone traffic was made as two thirds of the total traffic and for this network intense test it actually cost more per hour than the instances. The test itself was run for about ten minutes, which was long enough to show a clear steady state load level. Taking the setup time into account, the smaller tests can be completed within an hour, the largest test needed a second hour for the nodes only.Unlike conventional datacenter testing, we didn't need to ask permission, wait for systems to be configured for us, or put up with a small number of dedicated test systems. We could also run as many tests as we like at the same time, since they don't use the same resources. Denis has developed scripting to create, monitor, analyze and plot the results of these tests. For consistent read after write data access our alternative pattern is to use "LOCAL QUORUM". In that case the client acknowledgement waits for two out of the three nodes to acknowledge the data and the write response time increases slightly, but the work done by the Cassandra cluster is essentially the same.
As an aside, in our multi-region testing we have found that network latency and Cassandra response times are lower in the AWS Europe region than in US East, which could be due to a smaller scale deployment or newer networking hardware. The total on disk size for each write including all overhead is about 400 bytes.Thirty clients talk to the first 144 nodes and 30 talk to the second 144. Cassandra client's are not normally aware of which node should store their data, so they pick a node at random which then acts as a coordinator to send replicas of the data to the correct nodes (which are picked using a consistent hash of the row key).
This is useful when replacing an ephemeral memcached oriented data store with Cassandra, where we want to avoid the cold cache issues associated with failed memcached instances, but speed and availability is more important than consistency. However an immediate read after write may get the old data, which will be eventually consistent.To get immediately consistent writes with Cassandra we use a quorum write.
Two out of three nodes must acknowledge the write before the client gets its ack so the writes are durable.
In addition, if a read after a quorum write also uses quorum, it will always see the latest data, since the combination of two out of three in both cases must include an overlap. Since there is no concept of a master node for data in Cassandra, it is always possible to read and write, even when a node has failed.The Cassandra commit log flushes to disk with an fsync call every 10 seconds by default.
This means that there is up to ten seconds where the committed data is not actually on disk, but it has been written to memory in three different instances in three different availability zones (i.e. The chance of losing all three copies in the same time window is small enough that this provides excellent durability, along with high availability and low latency. The latency for each Cassandra server that receives a write is just the few microseconds it takes to queue the data to the commit log writer.Cassandra implements a gossip protocol that lets every node know the state of all the others, if the target of a write is down the coordinator node remembers that it didn't get the data, which is known as "hinted handoff".
For use cases that need a global view of the data, an extra set of Cassandra nodes are configured to provide an asynchronously updated replica of all the data written on each side.
There is no master copy, and both regions continue to work if the connections between them fail. In that case we use a local quorum for reads and writes, which sends the data remotely, but doesn't wait for it, so latency is not impacted by the remote region. Cassandra scales linearly far beyond our current capacity requirements, and very rapid deployment automation makes it easy to manage.



Cloud storage app for blackberry torch
Centos owncloud php 5.4 2005


Comments

  1. 15.07.2014 at 22:33:36


    Backup software supports virtual machines (VMs), so if the company is using.

    Author: Leyla_666
  2. 15.07.2014 at 17:48:24


    You to combine the free mobile.

    Author: QANQSTER
  3. 15.07.2014 at 23:39:26


    CMS DB on the CMS role server little cost, and see what difference they make this video.

    Author: NIKO_375
  4. 15.07.2014 at 13:19:46


    Online and CrashPlan can have you this limit, Dropbox will justCloud on a daily basis, getting.

    Author: turkan