Amazon s3 vs commodity hardware storage vs cloud computing

Rcently I have read quite a lot about cloud-computing. At first I read about Amazon S3, and I was fascinated by the scalability of this service and also by the possibility to have a look at the cost of storage for big players. Then I read Don MacAskill's ideas about Amazon S3; MacAskill is Smugmug's CEO and so he is in a position where data management is really important. I wanted to learn more about this subject, and I found thsi nice article about CapEx vs OpEx at Thinking Out Cloud.
More recently I came across Backblaze's blog, where there are detailed instructions on how to make petabyte storage with commodity hardware. By the way, I really think that these guys should sell their pods, as it seems that there is demand. The people at Blackblaze think that their storage solution is cheaper than Amazon S3.
At Wikibon.org, David Vellante argues that traditional raid might be running out of gas. I cannot follow the discussione because I am a lay-man, but he points to two interesting solutions; TAHOE, the Lovely Accessible File System, and Cleversafe.
As for me, I while stick to Ubuntu ext4 with Sbackup and wait for Btrfs file system.