So in ensuring performance, it really is an application specific thing, so it depends on what your application is all about. In some applications, it is the response time for the client that is super-important. In some other applications, it may not be individual response time or in addition to individual response time, it may be throughput for the application. Depending again, on the application characteristic, you may be interested in other metrics such as how many false positives are there, how many false negatives are there in the service that you're providing. So in order to get good performance, the first thing that you have to do before you try to deploy it or anything like that, you have to have a very good model of the workload that is going to be hitting your server so that you can use that model to do performance stress testing. You don't want to identify performance bottlenecks after deployment. So ideally what you want to do is if you have a robust and complete workload model, you can do that stress test ahead of time so that you can identify bottlenecks in your design and try to fix that before you deploy it in the field. Another best practice is endurance, meaning that your application is not done once and then you're not going to revisit it again. So you're going to be thinking about how to continually improve. The Gmail that you may be using today is not the same that you used yesterday or day before yesterday. Continuously, these applications that you're using or services that you're using are going to be refined and so if you are the developer of the application, you have to design your service with that kind of endurance in mind. So designing for testing and verification are super-important. So you have to have built-in mechanisms for independently measuring the health of the components. You have an application that are built off several loosely coupled components, and you should have an ability to independently measure the health of the components, and you have to have automated mechanisms for gathering and reporting on the health and performance statistics so that these kinds of things are going to be useful in terms of finding out what is going right, what is not going right. This may be something that you may have observed on your own machines that when you install some service, it'll say, "Well, can we gather from statistics from you every once in a while?" Which may be an annoyance for many of the things that you want to do. But certainly from a service provider's point of view, that is the way we can find out whether the application that the service that we're providing is good and performing well and so on. So you want mechanisms that allow gathering and reporting on these health and performance statistics of the deployed service and you have to design for future evolution. So which means that you shouldn't have to take down, look, once again Gmail as an example. It is not like Gmail says, "I'm going to shut it down for two days and then I'm going to upgrade it before I can let you use it." So incrementally, you are adding features and these features become available the next time you launch that particular application, and you want to do that seamlessly without having to disrupt the current user of that service. Similarly, when you add these new features in the field, you want to have incremental testing of the upgrades that you're adding so that you can ensure that this evolution of new features is not in any way negatively impacting the correct working of the service. As much as possible, you want automated maintenance of the service so that there is no manual intervention. I've shown you pictures of a data center called darkroom, so nobody is inside. So you cannot go in and put a monitor and start debugging things. So these are all the best practices as an app developer trying to provide a service in order to ensure that your services are scalable. As you add more nodes, you expect more performance, so that should be a goal. It should be reliable and available, it should have good security properties, and it should perform well, and it should evolve easily from one generation to the other.
