Friday, July 31, 2009

Day 1 at #oscon 2009

The first day of OSCon 2009 covered a number of tutorials and I decided to attend Google App Engine tutorial for the first half of the day. Google App Engine API follows CGI model of web development, i.e., it uses stdin and stdout files and assumes stateless applications. There is limit of 10MB response size and 30 requests per second, but it does not allow streaming. The tutorial started pretty slow and we spent first hour just installing the SDK and tutorial. The Google App Engine SDK is available from http://code.google.com/appengine/downloads.html. I downloaded Mac image and then dragged the image to my hard drive. I then double clicked the app icon for Google Appe Engine SDK, which installed the SDK under /usr/local/google_appengine. Once the SDK is installed, you have to install Google App Engine tutorials from http://code.google.com/p/app-engine-tutorial/.



After installing SDK and tutorial, I copied all files named tutorial? under the SDK. The rest of session covered those tutorials one by one, though we ran out of time in the end and completed only upto tutorial7. In order to install first tutorial, I went into tutorial1 directory, e.g.

cd /usr/local/google_appengine/tutorial1

Then started local app server as follows:
python ../dev_appserver.py .


When I pointed my browser to the http://localhost:8080, I was able to see “Hello World!”.


Next, I registered myself to http://appspot.com. After registering, I received an SMS message for confirmation and was able to fully register after entering the confirmation number. Next, I created an application-id on Google App Engine. You can only create 10 app-ids and you cannot delete app-ids, so be careful with ids. Also, you can also use your own domain instead appspot.com. For my testing purpose, I chose the id “shahbhat”.



Next, I changed app.yaml inside my local tutorial1 directory that describes how your application is configured. You may also notice index.yaml, which describes list of indices in the database, though Google App Engine can figure out what queries are being used and creates indices automatically. I changed application name in app.yaml to “shahbhat”, e.g.

application: shahbhat

I then pushed my application to the Google App Engine by typing
python ../appcfg.py update .

I was then able to go to http://shahbhat.appspot.com/ and see my application, Voila. You can also see your application usage from http://appengine.google.com/dashboard?app_id=shahbhat (you will have to change app_id parameter in your application).



Unfortunately, a lot of people had problems getting to that state so we wasted another half hour in break where other folks sort out configuration and deployment issues.
Next, I went through another tutorial to turn on authentication by setting:

login: required

in app.yaml file. Next I added caching by adding expires options in the app.yaml. I was also able to use curl to test my applications and see headers to verify caching, e.g.
curl --include http://localhost:8080

Which showed following when caching was not configured:
Cache-Control: no-cache

When I configured the caching to 2d, I was able to see:

Cache-Control: public, max-age=172800



The Google App Engine SDK also includes development that you can view by going to:
http://localhost:8080/_ah/admin


The Google App Engine supports Django based templates, e.g.
#!/usr/bin/env python

import os
from google.appengine.ext.webapp import template

def main():
template_values = {"foo" : [1,2,3]}
template_file = os.path.join(
os.path.dirname(__file__), “index.html”)
body = template.render(
template_file, template_values)
print “Status: 200 OK”
print “Content-type: text/html”
print
print body

if __name__ == ‘__main__’:
main()


In addition, Google App Engine supports WSGI standard (PEP 333), e.g.

import os
import wsgiref.handlers

from google.appengine.ext import webapp
from google.appengine.ext.webapp import template

class IndexHandler(webapp.RequestHandler):

def get(self):
template_values = {"foo" : 1}

template_file = os.path.join(os.path.dirname(__file__), "index.html")
self.response.out.write(template.render(template_file, template_values))


def main():
application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
wsgiref.handlers.CGIHandler().run(application)


if __name__ == ‘__main__’:
main()

Other tutorials included authentication APIs such as
create_login_url(dest_url)
create_logout_url(dest_url)
get_current_user()
is_current_user_admin()



The SDK also includes decorator to add authentication automitcally using
from gogole.appengine.ext.webapp.util import login_required
...
@login_required
def get(self):


Finally, we went over datastore APIs for persistence support, e.g.
import os
import wsgiref.handlers

from google.appengine.ext import webapp
from google.appengine.ext.webapp import template
from google.appengine.ext import db

class ToDoModel(db.Model):
description = db.StringProperty()
created = db.DateTimeProperty(auto_now_add=True)
foo = db.FloatProperty(default=3.14)
bar = db.IntegerProperty()
baz = db.BooleanProperty(default=False)
N = db.IntegerProperty()
l = db.ListProperty(str, default=["foo", "bar"])


class IndexHandler(webapp.RequestHandler):

def get(self):
todo = ToDoModel(description = “Hello World”, bar=1, baz=True)
todo.put()

def main():
application = webapp.WSGIApplication([('/', IndexHandler)], debug=True)
wsgiref.handlers.CGIHandler().run(application)

if __name__ == ‘__main__’:
main()


You can view the model by going to http://localhost:8080/_ah/admin/datastore. The data store supports a number of types such as string, boolean, blob, list, time, text. However, there are some limitations,
e.g. TextProperty can only store upto 500 bytes and Google App Engine will create index if needed, however it won’t create index on TextProperty. For each row, the datastore assigns a numeric id and UUID based key,
though you can provide your own key. Also, a row cannot exceed 1MB.
Unfortunately, we ran out of time At this time, so I had to go to http://code.googlecom/appengne/docs for further documentation. Overall, I thought it was good introduction to Google App Engine, but I was disappointed that instructor wasted a lot of time with setup that could have been used to cover rest of the tutorials.




For the second half of the day, I attended session on “Building applications with XMPP”. This was interesting session that showed usage of XMPP for a number of usecases such as IM, gaming, real-time social networking, , monitoring, etc. The session started with history of XMPP (Extensible Messaging and Presnce Protocol), its Jabber roots and its use of streaming XML. A number of factors contributed to the popularity of XMPP such as open source, XML, federeated network, low latency, etc. The XMPP is also very extensible and supports audio/video via jingle, geo location, TLS, SASL, etc. XMPP architecture is based on client server, where servers are decentralized and federated. XMPP identifies a user with jabber id that looks like email address and consists of local part, domain and resource, e.g. alise@wonderland.lit/TeaParty, where domain is mandatory, but local-part and resource are optional. There are three types of XMPP message stanzas, i.e., presence, IQ, message. The presence-stanza is asynchronous, but IQ stanza requires response. As opposed to web architecture that is based on short lived connections and stateless architecture, XMPP uses one long lived session and events are received asynchronously.



Next, the tutorial showed how to build a javascript client for jabber using sleekxmpp and Strophe library (alternatively you can use twistedword). The example used Bosh protocol to wrap XMPP protocol with HTTP protocol. Unfortunately, there was a lot of fast typing and I could not follow all that. I am waiting for the presenters to post the slides online so that I can use those examples in my own applications.



Blog copy:
Day 1 at #oscon 2009

Thursday, July 30, 2009

Cut the scope and make your life easy

I have been developing software for over twenty years and in every project you have to grapple with iron triangle of schedule/cost/functionality or sometime referred to as cost/quality/schedule or cost/resources/schedule. In my experience, curtailing the scope produces better results than adding more resources or extending deadline. In addition, slashing the scope also produces other side effects such as reducing the complexity of the software, easier learning curve for users, less training/support cost and better communication among team members.


You can reduce the scope by focusing on essential features using Pareto principle (80-20 rule) and companies like like Apple or 37Signals produce great products that are not only more useful but are much simpler to use. However, this is not easy as project manager or product owner have to say NO. Too often, I see project managers say YES to anything to please upper management and users. In the end, the team is overwhelmed and under stress. Also, a big pile of features where all features are of same importance (priority) is biggest reason for death-march projects.



Working with a small number of features reduces complexity such as essential complexity, cyclomatic complexity or accidental complexity because your codebase is smaller. Though, you still have to apply good software engineering principles such as domain driven design, unit testing, refactoring, etc, but maintenance becomes easier with smaller codebase. When you have a small codebase you have fewer bugs as they are no bugs for zero code. Fewer bugs means less support cost when some user complains of a bug or when system crashes in the middle of the night.


With a small set of features, the user interface becomes simpler, which in turn provides better usability to the users. Often, I have seen users get confuse when they have to work with a complex software that has a lot of features. This often is remedied by providing training or adding support that adds a lot more overhead to the projects. Again, better user interface does not come free automatically with a small set of features, but the usability problem becomes easier with fewer features.


Finally, small number of features and small code means your team size will remain small so communication among team members becomes easier. I like to work with team with size of 5 plus/minus 2, as number of communication links increase exponentially when you add more members. Also, with smaller teams that are colocated, you have better
Osmotic communication that Alistair Cockburn talks about. At Amazon, we have “2-Pizza” teams, i.e., teams are small enough to have team lunch with just two pizzas. Another factor when building teams is whether they are cross functional (vertical) or focus on single expertise such as systems, database, UI, etc. I prefer working with cross functional teams that focus on a single service or an application as communication and priorities within a single team is much easier to manage than between different teams.


In nutshell, reducing scope not only helps you deliver the software in time and delight your users but prepares you better to maintain and support the software. The complexity is number one killer for the software and results in buggy and bloated software. You should watch out when someone says “Wouldn’t it be cool if it did X?” kind of feature requests and often I see developers see this as a challenge or an opportunity to learn or apply new technology. However, each new feature takes a toll on your existing features, software maintenance and your team.




My other weblog: Cut the scope and make your life easy