Building Applications in the Cloud Concepts, Patterns, and Projects

Putting your company's applications in the cloud will no doubt save your IT department time and money -- but it's very often a more challenging proposition than you might expect. With this book, blogger Chris Moyer (http://blog.coredumped.org) has created a handy reference of patterns and techniques to properly leverage the benefits the cloud can provide. Readers who have been handed a project and told to "make it scale" can flip through the patterns provided here to get ideas and see what fits the project at hand. Use this book to get a lay of the land and guide your cloud project to successful scalability. **This is an early version of the manuscript -- updates to the chapters currently accessible will be posted as soon as they are made available.

Chris Moyer is a recent graduate of RIT, the Rochester Institute of Technology, with a bachelor’s degree in Software Engineering. Chris has more than five years experience in programming with a main emphasis on cloud computing. Much of his time has been spent working on the popular boto client library, used for communicating with Amazon Web Services. Having studied under the creator of boto, Mitch Garnaat, Chris then went on to create two web frameworks based on this client library, known as Marajo and botoweb. He has also created large scaled applications based on those frameworks.

 

Chris is currently Vice President of Technology for Newstex, LLC, where he manages the technological development of migrating applications to the cloud, and he also manages his own department, which is actively maintaining and developing several applications. Chris lives with his wife, Lynn, in the New York area.

 

Preface     xv

About the Author     xx

 

Introduction     1

What Is Cloud Computing?     1

The Evolution of Cloud Computing     2

   The Main Frame     3

   The PC Revolution     4

   The Fast Internet     5

   The Cloud     6

   HTML5 and Local Storage     7

   The Dawn of Mobile Devices     9

Threading, Parallel Processing, and Parallel Computing     10

How Does Cloud-Based Development Differ from Other Application Development?     11

What to Avoid     13

Getting Started in the Cloud     14

Selecting a Cloud Pattern     16

Implementing a Cloud Pattern     17

 

PART I:  CONCEPTS

 

Chapter 1  Fundamentals of Cloud Services     21

Origins of Cloud Computing     21

What Is a Cloud Service?     23

   Compute     24

   Storage     24

   Connectivity     24

The Legacy Pattern     25

Just Because It’s in the Cloud Doesn’t Mean It Scales     26

Failure as a Feature     27

Consistency, Availability, and Partition Tolerance     28

   Consistency     29

   Availability     30

   Partition Tolerance     30

Eventual Consistency     31

Summary     32

 

Chapter 2  Making Software a Service     33

Tools Used in This Book     34

   Signing Up for Amazon Web Services     34

   Installing boto     35

   Setting Up the Environment     36

   Testing It All     38

What Does Your Application Need?     39

Data Layer     40

   Introducing the AWS Databases     41

Application Layer     47

   Using Elastic Load Balancing     47

   Adding Servers to the Load Balancer     49

   Automatically Registering an Instance with a Load Balancer     51

HTTP and REST     53

   The Header     53

   The Body     57

   Methods     58

Authorization Layer     62

Client Layer     64

   Browser-Based Clients     65

   Native Applications     66

Summary     67

 

Chapter 3  Cloud Service Providers     69

Amazon Web Services     69

   Simple Storage Service (S3)     71

   CloudFront     77

   Simple Queue Service (SQS)     80

   Elastic Compute Cloud (EC2)     83

   Elastic Block Storage (EBS)     88

   Elastic Load Balancing (ELB)     91

   SimpleDB     93

   Relational Database Service (RDS)     95

   Simple Notification Service (SNS)     102

   Virtual Private Cloud (VPC)     106

Google Cloud     108

   AppEngine     108

   Google Storage     111

Rackspace Cloud     112

   CloudFiles     112

   CloudServers     113

   CloudSites     113

Summary     114

 

PART II:  PATTERNS

 

Chapter 4  Designing an Image     117

Prepackaged Images     119

   Overview     119

   Reasons for Usage     119

   Description     120

   Implementation     120

   Example     122

   Summary     125

Singleton Instances     127

   Overview     127

   Reasons for Usage     127

   Description     128

   Implementation     128

   Example     128

   Summary     130

Prototype Images     131

   Overview     131

   Reasons for Usage     131

   Description     132

   Implementation     132

   Example     133

   Summary     135

 

Chapter 5  Designing an Architecture     137

Adapters     139

   Overview     139

   Reasons for Usage     140

   Description     140

   Implementation     140

   Example     141

   Summary     146

Facades     147

   Overview     147

   Reasons for Usage     148

   Description     148

   Implementation     148

   Example     149

   Summary     152

Proxies and Balancers     153

   Overview     153

   Reasons for Usage     153

   Description     154

   Implementation     154

   Example     155

   Summary     158

 

Chapter 6  Executing Actions on Data     159

Queuing     161

   Overview     161

   Reasons for Usage     162

   Description     162

   Implementation     163

   Example     163

   Summary     170

Command      173

   Overview     173

   Reasons for Usage     173

   Description     174

   Implementation     174

   Example     175

   Summary     179

Iterator     181

   Overview     181

   Reasons for Usage     181

   Description      182

   Implementation     182

   Example     183

   Summary     185

Observer      187

   Overview    187

   Reasons for Usage      188

   Description     188

   Implementation     188

   Example     189

   Summary     191

 

Chapter 7  Clustering     193

The n-Tier Web Pattern     195

   Overview     195

   Reasons for Usage     196

   Description     196

    Implementation     197

   Example     198

   Summary     210

Semaphores and Locking     211

   Overview     211

   Reasons for Usage     211

   Description     212

   Implementation     212

   Example     213

   Summary     218

Map/Reduce     219

   Overview     219

   Reasons for Usage      220

   Description     220

   Implementation     220

   Example     222

   Summary     226

 

PART III: PROJECTS

 

Chapter 8  A Simple Weblog     229

Storage     229

   Creating a Domain     231

   The User Object     232

   The Post Object     234

   The Comment Object     237

Application     240

   A Brief Introduction to WSGI     241

   The DB Handler     243

   The User, Post, and Comment Handlers     248

   Spam Classification     249

Presentation     253

   Setting Up the HTTP Proxy     254

   Posts     255

   Comments     266

Deploying     272

   Starting the Base Instance     272

   Installing the Software     273

   Installing the Application     275

   Installing Apache      276

   Bundling the Image      277

   Creating the Proxy     279

Summary     279

 

Chapter 9  A Weblog Using Marajo     281

Initializing the Environment     282

   handlers     283

   resources     283

   static     283

   templates     283

Creating the Resources     284

Creating the Handlers     286

Configuring the Application      287

Creating the Templates      288

Running the Application     289

Creating Custom Templates     289

   The List Template     289

   The Edit Template     292

Summary     296

 

Glossary     297

Index     307

 

Extreme Programming Installed Preface After a few months working as a developer in a small start-up company migrating existing services to the cloud, I started realizing that there was way too much work to be done just by myself. I started looking around for other developers like myself that could assist me, or replace me if I were to find a better and more exciting opportunity elsewhere. I quickly realized that there are so few people that actually fully comprehend the level of complexity it requires to develop a cloud-based application, and almost all these people were happy with their current companies. I began to create a series of blog posts about working with cloud-based platforms, now still available athttp://blog.coredumped.org"http://blog.coredumped.org, but soon realized that I could quite literally spend an entire year writing up everything there is to know. This documentation would be better placed in a reference book than simply scattered throughout several blog posts, so I decided to write this book. What Is the Purpose of This Book? This book isn't designed as a tutorial to be read through from cover-to-cover. It's not a guide for how to build an application for the cloud, but instead it's designed as a reference point for when you have specific questions. When your boss hands you a new project and tells you to make it scale, check the patterns discussed in this book to see what fits. When you work on a project and you find a specific problem that you don't know how to handle, pick this book up. If you're trying to start on a new project, and you have a perfect idea, but you don't know how to scale it, pick this book up. If you're trying to modify an existing project to scale in the cloud, pick this book up. If you don't know what kinds of applications you can build with cloud computing, pick this book up. This book doesn't invent many new patterns but simply shows you the tricks and new techniques that you need to consider while running them in the cloud. Although you can use any patterns discussed in this book in any sort of clustering environment, they're designed to take full advantage of the services provided by cloud computing. How Should This Book Be Used? This book is divided into three parts. Everyone should read Part I, "Concepts," for a basic understanding of cloud computing. In Part II, "Patterns," you can skip through to the patterns you're most interested in. If you've never developed any sort of cloud-based application, you may want to go over the example applications in Part III, "Projects," so that you can see exactly what kinds of applications are best suited for this type of system. Part I, "Concepts" Part I is designed to give you a general concept of how to develop in the cloud. It's designed to be read from start to finish and is broken into different key chapters important to development. Chapter 1, "Fundamentals of Cloud Services"Provides a basic set of fundamental ideals when working with cloud-based solutions. This is an absolute must read for any developer beginning with this book. Chapter 2, "Making Software a Service"Provides a basic set of instructions for providing Software as a Service (SaaS). It includes details on why this is a good idea and some basics as to how to properly construct your SaaS. Chapter 3, "Cloud Service Providers"Provides some specific examples of services offered by cloud providers. Part II, "Patterns" Part II functions more like a reference manual and provides you with a problem and the pattern that solves that problem. Chapter 4, "Designing an Image"Includes basic patterns for use in building your basic image that is the basis for the rest of your application. Chapter 5, "Designing an Architecture"Includes the patterns used for interacting with external systems, not systems offered by your cloud provider. Chapter 6, "Executing Actions on Data"Includes the patterns used to execute code segments against your data. Chapter 7, "Clustering"Includes the patterns used within a basic framework designed to take advantage of multiserver deployments. Part III, "Examples" Part III includes examples of real-world applications of the patterns provided throughout the book. These chapters all use the same overall hello world tutorial, but in two different ways. Chapter 8, "A Simple Weblog"Details how to build a simple weblog from scratch, not using any existing frameworks. Chapter 9, "A Weblog Using Marajo"Details how to build a weblog using the Marajo cloud-based Web framework. Conventions Used in This Book Throughout this book you occasionally see bolded words. These words represent a new term, followed by the definition. If you find a term in the book that you don't know, see the Glossary for a full listing of definitions. Words listed in italics highlight key important ideas to take away from the section. These are usually used to highlight important keywords in a topic, so if you're skimming over a section looking for something specific, this should help you find exactly what you need. Where to Begin The first question for most people now is, where do you start? How do you quickly begin developing applications? What if you don't want to go through and read about all these things you could do and simply want to get into the meat of how things work? By picking up this book, you're already on the right track. You already know that you can't simply go to a cloud provider and start launching servers and expect to get exactly what you want out of them. People who just pick up a cloud provider and don't do enough research beforehand typically end up with lots of problems, and usually end up blaming the cloud provider for those problems. This is like buying a stick-shift car without first knowing how to drive it and then complaining to the dealership for selling it to you. If you don't first do some research and preparation, you shouldn't be surprised when you have problems with the cloud. If you're not a developer, you probably would be better suited to using a third party to manage your cloud, but if you're reading this book, I'm going to assume that you're interested in more then just "let that guy handle it." Copyright Pearson Education. All rights reserved.