Cluster computing on the cloud (by the numbers)

I was recently involved in some consulting work with a small aircraft company and had to look up computing costs on privately owned clouds. I've always pooh-poohed this in the past because it seemed too expensive and cumbersome. The numbers have, however, gotten to a point where a second look is required. The following numbers are for the Amazon cloud and the rest of the vendors are competitive with this:

Usage of 16 cores costs $2.4 / hour and the first GB of data transfer is free.

If you're a bit more serious and are thinking long term, a yearly lease of 16 cores costs $4148 (up front) + $ 0.54 / hr and $0.12/GB. These costs can go down if you're thinking even more big time.

So let's crunch some numbers here... Assuming we need 16 cores at 60% utility over an entire year, the cost comes to $7k + data transfer. For a 3 year lease, it is $15k + data transfer.

96 such cores over 3 years thus cost $90k + data transfer. Unless you're running calcs that you shouldn't be running, data transfer won't be more than $10k, so we're looking at $100k over 3 years.

Compare this with your own 96 core cluster that probably is really useful for only 3 years - It costs $25k not including cooling, power, space and system administration. The last two items are definitely non-trivial. At a place like Stanford (or anywhere else, for that matter) you have to move heaven and earth to carve out space and a system admin costs $200k/year (including benefits - So even assuming that maintaining your cluster constitutes less than 5% of his/her responsibility, we're looking at non-trivial numbers). All said and done, it won't be unfair to put the real cost at $50k over 3 years.

Factor of 2 ? wow! Even if you just consider the cost of your cluster, it is just a factor of 4.

Clouds are here to stay... even from a HPC perspective

A Sense of Scale

Apple's recent announcement of it's cash pile and the California government's much maligned money problems got me thinking about the following numbers:

California:

Population: 39 Million
GDP : $ 2 Trillion
Budget deficit: $10 Billion
Total debt: $ 360 Billion

Apple Inc:

Employees: 60 K
Revenue: $ 110 Billion
Profit : $ 26 Billion
Cash Reserves: $ 100 Billion

Stanford University:

Staff+Students: 20 K
Revenue: $3.8 Billion
Net Inflow: $4.0 Billion
Total Endowment: $ 18 Billion

 

To be continued....

Energy use and GDP

Per capita electricity use / day
USA : 40 units (of which 36% is residential)
EU : 20 units (35% is residential)
China : 7 units (15% is residential
India : 2.5 units (!) (20% is residential)
World : 8 units (27% is residential)

Take a look at this very interesting graph..  
A good straight-line fit gives us a GDP of 40 cents for every unit of energy (total, not just electricity) consumed.  I know there are some implicit, non causal relationships, but this is a wonderful number to remember.. say for India to reach a per capita GDP of $10000, the annual per capita energy use could be close to 25000 units. Another way of looking at it (a Nobel winning Stanford Prof says this): For (roughly) every dollar spent on energy, you typically generate 4 dollars of GDP.  Quite remarkably, this number is more or less the same across all type of economies including for the big bad US of A. Americans account for 25% of the world's energy consumption and account for 29% of its GDP. In other words, there are no shortcuts, at least so far.


India better be building one nuclear reactor every month for the next 20 years ?

Daily energy use of the world : 360 billion units
Global power consumption : 15 Tera watts
Guy with a plan:  http://www.stanford.edu/group/efmh/jacobson/Articles/I/susenergy2030.html

Some numbers related to Energy

Energy Densities (Amount of energy released when the given substance completely combusts with oxygen at standard atmospheric conditions)

Coal :          10-24 MJ/kg
Wood :        14 MJ/kg

Gasoline*:    44.2 MJ/kg (31.7 MJ/litre)
Natural gas: 53.9 MJ/kg (0.038 MJ/litre)
LNG          : 53.9 MJ/kg (22 MJ/litre)
Propane     : 50 MJ/kg (25 MJ/litre)
Biogas       : 28 MJ/kg  (7 MJ/litre)
Hydrogen : 119 MJ/kg (8.1 MJ/litre)

Lead-acid batteries : 0.125 MJ/kg
Li-Ion batteries : 0.432 MJ/kg
Alkaline batteries : 0.594 MJ/kg

* Diesel and Kerosene are within 3% of this number

Let's switch to convenient energy units i.e.,  1 unit = 1 kilo watt - hour = 3.6 MJ

So a kg of coal contains roughly 6 units of energy and a litre of Gasoline contains about 8.75 units of energy.

Note that the energy density of the best possible battery is about 100 times less than that of an equivalent mass of gasoline. Of course, electrical energy conversion is much more efficient than thermal energy conversion, but this fact is only going to buy us a factor of 2. In addition to being heavy,  battery electrodes are quite expensive as well. Even though things are improving, you should get a sense of what electric cars are up against.

If you assume that gasoline costs $3.50 / gallon, the price you are paying per unit of energy content comes to 11 cents. This is quite interesting because we pay around 10 cents / unit of residential electricity.

Cost / unit of energy content  (at wholesale prices) :

Coal  :  0.4 cents / unit
Natural gas : 1.5 cents / unit
Crude oil : 5 cents / unit

Note that so far, we've been just talking about energy content in fuels and haven't considered conversion losses. If we take everything into account and go by the wikipedia definition:

Levelized Energy Cost (LEC) is an economic assessment of the cost of the energy-generating system including all the costs over its lifetime: initial investment, operations and maintenance, cost of fuel and cost of capital,  this is what we get

 The chart shows cost / Mega watt - hr, so divide the last column (which is in dollars) by 1000 to get cost / unit of electricity. So according to this chart, conventional sources cost less than 10 cents / unit and land-based wind energy is competitive. Solar energy continues to be off the charts expensive even with all the recent advances.

Coal and natural gas based energy is dirt cheap and the big 3 energy consumers (USA, China and India) have enough reserves to last out this century at least as far as electricity use is concerned. Transportation fuel may be a bit more complicated with the approach of peak oil (and such), but I think gas to liquid technologies will come to the fore. Note that if more stringent emission restrictions are brought in on fossil fuels, wind energy has a good chance to be very viable and Solar can become borderline competitive.

Overall,  the cost of energy continues to be amazingly low considering the fact that the best cyclist in the planet can only give you 0.3 units of energy over an hour and animal labor isn't that much different. Yes! I compared the present situation with the stone age to make you feel better!

10 Myths about India (International edition): Call centers are very important to India's growth

When the topic of India gets brought up in chatting with my colleagues (and students) or friends (especially the long drawn discussions at the dog park), I frequently start off with "No, in reality, it is not like that.." So I thought I should put some of my points in one place (not in any particular order) to decrease entropy generation:

Myth #2: Call centers are very important to India's growth: Not really!. Out of India's $ 76 billion information technology (IT) industry, services such as call centers, tech support and medical transcription constitute a small fraction of the revenue. India has a healthy and growing IT industry that offers high quality software and consulting services on par with the best in the world. Call centers, on the other hand, account for about 1% of GDP and about 0.1% of the workforce, and since this sector does not involve a high degree of technical skill (I am appreciative of good communication skills) competition from other low cost markets (such as the Philippines) is growing. The state of the IT industry, overall, is better because it is not just dependent on low cost services.  The IT industry has changed some business approaches and social practices and has been very beneficial, but it still measures only up to 7% of  GDP and only recently have some companies started to focus on creating original, innovative products rather than rely on services. In comparison, the engineering industry (including mining and assorted stuff) contributes about 20-25% of the GDP (by some measures) and though traditional industries aren't as "world class" as IT services, specific sectors like automotives are getting there in patches.

10 Myths about India (International edition) : The education system in India should be really good

When the topic of India gets brought up in chatting with my colleagues (and students) or friends (especially the long drawn discussions at the dog park), I frequently start off with "No, in reality, it is not like that.." So I thought I should put some of my points in one place (not in any particular order) to decrease entropy generation:

Myth #1: The education system in India should be really good : You're very likely to hear this in the US and outside of the Indian elite, probably only in the US. My take on this myth might be good enough even for the Indian edition. While literacy rates in India have increased at an appreciable rate (from 12% when the British left us to about 76% currently and I am indeed proud of this), serious issues remain. The definition of basic literacy is downright depressing. A good fraction of primary and secondary schools are run by just one teacher (simultaneously teaching many classes while being absent half the time) and despite all the efforts, only 16% of students reach high school and only 8% graduate. This is absymally low for a rapidly growing economy with an acute shortage of skilled labor. It is true, however, that the fortunate few millions that end up in good private schools receive better training in mathematics and science than the average high school student in the US. So we have that going for us.

About 6% (of the total college age population) reach college and on average, the quality of education and employability of graduates continues to be very poor, and by all measures, is declining by the day. Adam Smith (in my book, a top 10 human being of all time) once said "…if an equal proportion of people were educated at the public expence, the competition would soon be so great, as to sink very much their pecuniary reward.” While this problem could be true in advanced economies (present day Britain, especially), this is not at all India's concern. In India, the issue is that with growth in the higher education sector, quality has taken a hit beyond imagination. Take the example of the southern state of Tamil Nadu - This is one of the most progressive and prosperous states in India (I would rate them in the Top 3 by both measures), engineering colleges have mushroomed in the last 10 years.  About 20 years ago, one in 80 aspirants could get into engineering colleges.. and these days about 20% (50000) of seats remain vacant. With the explosion in opportunities has arisen a tremendous shortfall in lecturers (quality and quantity wise) and while there are some very good teachers around, a good majority (not all, bless those souls) of lecturer posts get taken up by fresh college graduates and many of them do so because they can't find other decent options. Some reports that I have read suggest that companies think that less than one in 10 engineering graduates are "employable". Access to education is one of the most important aspects of a growing economy and while the increase in quantity has done short term good, the alarming decline in quality doesn't bode well for the future.

Secondary education is about quantity and Higher education is about quality. At the present moment, India is well behind the curve in both aspects. Changes can't and won't happen overnight, but I think things will get worse before getting better.

All this said, at the very top (IITs), the quality of education is still very good (but declining according to various sources) and comparable to the Ivy leagues and Oxbridge (in technical rigor, but much less well rounded). However, less than 1 in 200 aspirants make it to the original 6 IITs) and just the cream in many other universities is comparable. Every year, more than 100,000 Indians leave for higher education to the US (an equal number to the rest of the world) and a majority of these folks rely more on hard work, survival instincts and adaptability than background knowledge to make it to solid jobs, all the way to the top of the middle class. In fact, the average household income of Indians in the US is the highest among any group and is twice that of the US average. Of course, at the highest end of this spectrum, the best of the best Indians have made amazing contributions in founding big corporations and produced significant achievements in science and engineering.

In summary, the Indian education system has always faced incredible challenges (even at the present moment, only at par with the likes of Pakistan and Bangladesh on average) and is exhibiting growing pains. You probably have to be aware of the fact that it is indeed the top 5% of the crop that comes to the US. To boot, a majority of this group is successful not because of what they learnt in the class room, but rather off of it. I am, however, confident that with the growing awareness of India's acute problems and increasing pressure on the government to perform, things will improve in a few years.

A day in the life of a stupid NRI

"It is my goddamn right to be in this bus and you damn better not ask me to get off it !!!" I was about to scream at the top of my lungs.

How did it come to this? Less than 30 minutes ago I was speeding home in an ultra-modern chauffeured SUV and with the air conditioning in full blast, one that was giving us less than 11 miles to the gallon... and now I am about to trade blows with the conductor of a crowded bus,  a ticket of which costs Rs. 9 (17 cents) for a 23 km journey.

It all started yesterday when my cousin sent another such vehicle to pick me up from home (I accepted the offer as I needed to clear my mind) to be dropped off at his remote retreat in a jungle about a 4 hour drive away. At 33, he is quite the self made man and having started off from scratch after college, he now runs three different businesses and recently started two more to fund his own racing team. I spent the afternoon and evening at an even more remote location with some of his interesting friends; the type who make a trip to London to receive a photography prize at the Natural History Museum in the dead week between two car races. Having grown up in the upper middle (which in India means the top 1-2%) class in Coimbatore, this is the India I am familiar with, especially in my stints home the past decade. Entrepreneurship in Coimbatore - the small city with big money - is perhaps more ingrained than in Silicon Valley (although you could substitute the word 'innovation' with 'imitation', 'high tech' with 'manufacturing' and 'billion dollar ideas' with 'million dollar banality') and enterprising people abound. Things have been so good lately (for the fortunate 5%, about a third of whom depend on money pumped from their progeny in the United States) that real estate and apartment prices are now much higher than in Bangalore. If I were to buy an suitable apartment in a good residential location, I am looking at shelling out upwards of $175,000 (not including about $25000 worth of furnishing) in a city where the median annual income of a middle class family is probably less than $5000 (when adjusted for a PPP factor of 3, this doesn't look too bad, but it also takes apartment prices to Silicon Valley levels). I read, think and theorize a good deal about the plight of the common Indian and while (I think) I have a reasonable understanding of the general difficulties, I must say that I have probably never experienced the daily grind and with the skyrocketing inequality, I have been wanting to do something about it.

This morning, after a short walk in the forest, I was ready to get back home and my cousin was insistent that I should be chauffeured home again. On the way home, however, I convinced the driver that I should be dropped off three fourths of the way and that I would find the rest of my way home using public transport, and that too of the cheapest variety. So I got off at the bus terminus in some place X and needed to get to place Y to catch a connecting bus home. I was about to wander off in search of the ticket counter when the driver told me that there was no such thing as a ticket booth and that I was supposed to buy my ticket on the bus. So I got on a bus that was on it's way from X to Z via Y as per the sign on the side of the bus.  Z was about 45 km and Y was roughly half way.

 I got on it, fit myself into a seat that has a leg room of about one third of that in the local flights (or one half of that in Southwest Airlines) I had taken a few weeks ago. To enrich the experience, I even got myself a Rs. 5 portion of hot Sundal (fried masala chickpeas for the uninitiated) wrapped in a cone shaped piece of newspaper, served off an open bucket. When I was about to gleefully consume it, the bus conductor started shouting "If you're just planning to go to Y, you better get off the bus. Preference is for those traveling to Z". Some people started complaining and the conductor was like "Preference is for Z, all those planning to get off at Y better get the heck out". More protests, more of the same. Shit got real when the conductor asked me where I was going and (somewhat more politely) told me that it would be a good idea to get off. I was kinda confused and was unsure whether this was because the bus wasn't planning to stop in Y or whether he was screwing the good few Y people who were comfortably seated because the Z people were more lucrative customers. I got off with a bevy of frustrated folks getting their displeasure heard more amongst themselves rather than to the conductor albeit in a non belligerent manner. 

So I stood on the pavement and was trying to figure this deal out over my surprisingly delicious Sundal. I did confirm with one of the victims that the conductor did indeed screw us over and before I returned after a failed sojourn to find a trash can (this might be about as bad as locating one in Blighty, but I was not going to litter the place like many of my fellow countrymen or the Neds in Glasgee for that matter), the bus took off. I did find a few of the victims sticking around, but not all.

I carefully wrapped the almost empty sundal packet, stuffed it into my pocket and started to think BIG. BIG, as in the kind of ideas mainly NRIs are capable of. I would take off my Roger Federer hat, ruffle my hair, hide the $1300 worth of electronics deep into my backpack and act mean. If the conductor in the next bus does the same thing, I would stand my ground and yell at him along the lines of

"What do you think you're doing? It is clearly written on the side of the bus that this bus will stop in Y and I know for sure it will stop in Y. I got to my seat a full 10 minutes before the other guys did. It is my goddamn right to be on this bus and you better damn well not ask me to get off it. If you still insist, I want your name and am going to take action. If you want to remove me, you have to do so physically.. not just me but all the other Y people" at which point, there would be more people who join the cause and a minor battle would be won for the common people.

Sure enough, the next bus rolled in and I watched the conductor go  'Only people to Z... Only people to Z... Y people better not get in' for about five minutes. I got in nevertheless and was rehearsing my tirade and made sure I stood right next to the conductor. The bus was getting quite incredibly crowded and the conductor stopped yelling the pleasantries and bus took off a few minutes thereafter. I was the first to buy a ticket and when I said 'One ticket to Y', he gave me a ticket and was rather indifferent about it. The next 7 or 8 takers were all Y people and the rest were pretty much Z folks. Then I got it - He just made sure the longer haul people were seated and didn't care two hoots about what happened after that.

So what did I learn from all this? a) That injustice can come at you in many forms, b) That most people don't complain when screwed, c) You  really can't predict how you might be treated - Your experiences will be slightly different every time unlike in the developed or upper middle class world, d) Be ready to take a stand the next time lest that opportunity slip, e) You just wasted a bunch of your time reading this blog, and most importantly, f) To never shove even an almost fully finished Sundal pack in your pocket - I thought I was pretty careful, but by the time my journey was half complete, my pocket was a mush.

So as the 52 seat bus (crammed with 70 people) took off, it made an unspecified stop; about 10 people got off and 10 got in... admist the crowd, the conductor was skillfully navigating his way and getting tickets to his new clientele; some people made way for an OLD lady and someone offered his seat (gasp); a couple of standing (possibly Y) folks were entertaining a (possibly Z) baby seated on her mother's lap and was probably uncomfortable with the crowd while her mother was quite seriously reading a 'how to speak Hindi clearly' book; someone (possibly one of the victims from the previous incident) started a conversation with me in broken English about the welfare policies of the government and its impact on the local economy; a group of young kids were rhythmically tapping the ceiling to the beats of a catchy song blaring from the speakers of the bus; It all seemed to come together for everyone in bus and life just chugged along. 

Just another day in the ebb and flow of life in this country. Change will come when it comes. Humanity gets by, collaborating when necessary, berating when not, but always with the bottom line in mind.  Not unlike the few missed heart beats that I initially experienced on the "hair pin bends" on the way down from the hills. Sure, one in a million is going to die, but the rest of us just get by juuust fine.

ps: Yes.. you guessed it, the last paragraph was just a (poor) attempt at getting at least a blog post out of the non-event.... and yes, the entire post is supposed to radiate the NRI naivete.