W O R M S    A G A I N S T    N U C L E A R    K I L L E R S
    _______________________________________________________________
    \__  ____________  _____    ________    ____  ____   __  _____/
     \ \ \    /\    / /    / /\ \       | \ \  | |    | | / /    /
      \ \ \  /  \  / /    / /__\ \      | |\ \ | |    | |/ /    /
       \ \ \/ /\ \/ /    / ______ \     | | \ \| |    | |\ \   /
        \_\  /__\  /____/ /______\ \____| |__\ | |____| |_\ \_/
         \___________________________________________________/
          \                                                 /
           \    Your System Has Been Officially WANKed     /
            \_____________________________________________/

     You talk of times of peace for all, and then prepare for war.

(courtesy Wikipedia)

If you wan to read about the wank worm, check out Suelette Dreyfuss’s intriguing Underground: Tales of Hacking, Madness, and Obsession on the Electronic Frontier.

Take home message: It’s a dark and untrustworthy internet out there. Is your email account for sale? Your credit card? Can you verify that your traffic isn’t being stolen? What about when the hacks move into the real world?

Hell, I hate marketing buzz. Let’s check out ReadWriteWeb’s opinion.

1. Structured Data
2. The Real-Time Web
3. Personalization
4. Mobile Web & Augmented Reality
5. Internet of Things

I’d add to that list…

Webhooks

Yahoo pipes is the harbinger of a whole ream of useful tools that do clever things by updating live in response to changes in the internet, and could be some of the most important technical plumbing the web sees. Check out codepad, Utility Mill, and Scriptlets for a taste of that.

The Semantic Web

See the W3C for some nice background, and some of the design issues opposed to it, such as The Rule of Least Power. Check out spooky services like “open” calais, and zemanta, and also naysayers, who have my sympathies.

Data visualisation

I partially wrote a lecture about that which we didn’t have time for.

the dark net

I partially wrote a lecture about that too which we didn’t have time for either.

Fabrication

Making things over the net is a rapidly expanding project.

code

Well, this ain’t new. The net’s been running on code for a long time. It could be next for you, however. Now code is more powerful than its ever been before, and there are more great tutorials. If you want to do some real fancy tricks online, why not give some more javascript (and jQuery) a burl? Or one of those modern trendy languages, python and ruby?

Data visualisation and aesthetics are one of the crazes, or movements, sweeping the net right now. The arguments they make are potent.  There’s lots of data out there that visualisation can help us to understand. The net itself, forexample, is hard enough to understand without visualising it, (even our local bit of it).

http://your.flowingdata.com/

Where is the public space online?

Public space  is generally considered pretty critical to, among other things, a functional civil society. Consider how many governments fall because of mass street protests. Or the curfews that totalitarian regimes impose upon the use of public space.

Thai political protestors throng in a Bangkok street

devotees of a minority religious grouping make use of Australia's public spaces

As public discourse (even voting) moves online, it is timely to consider what the online equivalents might be of physical public space.

For me, a lot of the internet reminds me of a trip to Jakarta for their Biennale. Massive exhibitions, crammed overwhelmingly into malls. When I asked “why?”, the curators were surprised I’d even bother with the question. “Where else do people hang out?” they’d say in reply.

The thrumming village square atmosphere of your typical Jakarta mall

In some ways, exhibiting your work in a mall is not only a higher profile place than on the street, but cheaper. Apparently, the overlapping systems of gang protectionism, bribery and bureaucracy are so tortuous that simple street murals cost more time and labour on top of the cash- and it’s harder to get sponsorship.

In the internet, where almost every bit of virtual space is provided by for-profit commercial concerns, it’s worth wondering if the virtual world is basically the equivalent of an endless online mall.

And even if it is not yet, as peoples online experience become more consolidated by social networking sites into the hands of a few companies, it may get more so. Sure any of us can get set up a web space for not much cash, but if all we can afford is a little streetside nook, and not prime mall shopspace, will anyone hang around to look at it?

Running your own low-budget webspace is all very well, but it doesn't grant you legitimacy

So where are we in this course displaying our work? In the malls or in the street of the internet conurbation? Do we have to make that choice online, or does the metaphor a poor one for the infinitely remixable world of the net? Are city parks and commons even meaningful or desirable online? Does money actually make a difference, and is it a concern if it does?

Doing it yourself, versus getting yourself done.

We went over the commercial/technical risks of self-versus corporate hosting last time. Let’s look at some legal and political ones, in a bit of historical context.

So, people have been rolling their own sites for a long time. I’ve mentioned Indymedia in class before now, which was one of the attempts to formalise the early success of the citizen journalism movements springing out of the activist dissatisfaction with mass media reporting. Indymedia has had its problems in its esteemed history, but is of particular relevance here because it was strongly associated with Sydney through the Cat@lyst collective (who incidentally taught me how to code in PHP back in 2001 in their weird icecream factory hideout). These guys were hand soldering a lot of their equipment, and pulling stuff out of bins to make the technology cheap and accessible – and to keep it in the hands of people using it. (Compare and contrast with what some argue is a more modern version of that, the commercial service Opera Unite. Does it obviate the need for all this messing about in factories?) More modern appraches that have sprung from the same community include, for example, engagemedia.org, which I’ve banged on about in class every now and again.

But these folks were in turn building upon the efforts of earlier pioneers – the servers they were building were running GNU software, which came attached to strong stances about “freedom” as something which applied to bits and bytes as well as personal rights. And I’ve mentioned this phrase “open source“ before. Just to confirm, all the software we’re running on our servers in this course is “open source”, from jQuery via apache httpd through to wordpress, not to mention Firefox, Chrome and (mostly) safari. Not only may you use it for free, but you may modify it for your own ends also for free i f you have the requisite skills. There’s a kind of massive distributed skillshare out there, rooted in these communities that believe in the principle of providing you with the tools to do it yourself.

Not all the software we use, mind. That’s a whole ‘nother story, what with Dreamweaver, with Yahoo Pipes and the various other cloud hosting options we’ve discussed. But rather, I want to tease out one of the important threads in the online tapestry, one which has zero marketing budget: the community driven, collaborative software that underpins large sections of the web.

And there are whole worlds of co-operation and antagonism between these communities of openness and the bodies of closed, metered resources controlled by Microsoft, Apple and Google and so on.

This one might be closest to home, all you firefox fans out there: Code Rush in the Creative Commons

Who owns the mix?

There’s another kind of intellectual property evolution going on here too, but it’s to do with what some people might call creative works. Music (and piracy thereof). News (And reproduction thereof). Movies (and cheap distribution thereof). Hell, textbooks and postcodes are all up for disputes and negotiations about who gets to distribute them to whom when. As we’ve discussed, the Australian government itself is reconsidering what data its happy to share and what not.

The news is an interesting one, given the proximity of the the renown journalism faculty to us here. Has anyone been following the story of Associated Press versus Google (1, 2)? There’s a really nice exposé on  ABC’s Background Briefing called Who Owns The News? that I recommend you check out. (Also check out the upcoming show on piracy.) Long story short: There’s heaps of unverified citizen journalism sloshing around the internet, or even just copied versions of the official newspapers and whatnot, and traditional media organisations feel they are being driven out fo business by it. They also feel they have a legitimate case that the respectability, research budget and accountability of new organisations is a public good that we can’t afford to lose. Is it a pure problem of the economics of oversupply? The attention economy? (see also the seminal Goldhaber essay about this)

One little bit of the struggle that may amuse you is the CSS Descrambler saga, where America’s DMCA made it illegal to expor the very simple software needed to play DVDs, which resulted in an amazingly inventive variety of methods used to do it.

It’s intersting to me that now that mp3 trading is old hat, newspapers and books are the new contentiousness. Have you seen the The Google Books Settlement furore? (You should have, since I believe Kiaya has already posted it to the friendfeed)

Who owns you?

Here’s one video I couldn’t embed in this blog: Google Opt Out

Most of us don’t run a crumbling media empire whose profits we are anxious to shore up with aggressive prosecution of those who get our content for free. But we might feel a little more ownership of our own data- what others are allow to know of us, and what we are allowed to see of other peoples’.

Who here has run into google Google Analytics? Who uses it on their own site?If you’ve not seen it, here’s a screen grab:

Think about what it might mean that my site (and by extension, google) knows that you last visited my site “between 121 and 364 days ago”. Now, do you remember this video from week 2?

All I had given him was my e-mail and name,” Dakan said. “He knew everywhere I’d lived, every car I had driven, and even someone else in Alabama who was using my Social Security number since 1983. (more here)

OK, so our privacy is compromised – should we get over that because giving it up is so useful?

On the other side of the coin, what you may and may not view on the internet is fascinating. Most recently in Australia’s we had the ACMA censorship scandal. Other countries have more thoroughly documented censorship regimes, e.g.  The american Chilling Effects database. Did anyone partake of the enthusiastic vicarious Iranian revolution by twitter? Is anyone here a scientologist? Then you’ll know about the stoush with Anonymous. Is Conroy’s clean feed a good idea or not?

Or, relatedly, who chooses what information you do get? Has anyone run into the Sourcewatch wiki, which monotor’s the PR industry’s hold on communication? Is that a public benefit of the internet? How about wikileaks? Would it be a good thing if we were involved in internet privacy schemes? Or would that give license to child pornographers?

For example, if you followed the link through to the google spreadsheet map maker from the last post in this series, then you might have found that it made you some horrible javascript that it is NOT obvious how to do nice stuff with, and how to attach javascript widgets to. If anyone wants to follow through a step-by-step rebuild of that into some nicer javascript, then we can do that.

We can also look at other things – Google Apps for Your Domain (or What Does Google Apps for Your Domain Actually Do?)… BBpress installation as an example alternate app install. It’s up to you.

Similarly, there was a lot of talk of technologies that can be used without a motivation as to why to use them. Google maps visualisations might be cool or whatever, but there is also a real need to justify why to use it other than that “it is there”. There has to be some value add to your aesthetic or commercial goal to motivate using it.

Prioritise! The bulk of proposals were for project far more ambitious than could be delivered in the timeframe. That’s fine, but make sure that you are clear on the priority of your goals, and that you have a plausible fallback plan in the face of (inevitable) difficulties.

Finally, a project proposal is a pitch document. Make sure that you are aware of its need to be persuasive and to present your idea as worth investing time in. Don’t get bogged down in the technologies you will use any more than necessary  - they might be how you will implement them ,but they aren’t what makes your project worth doing.

Now, one final some resource that springs to mind from reading the assessments: The form generation tool that google provides – that dumps data straight into a google spreadsheet for you to use, mashup-style. I think that would be really handy for a number of you getting punter surveys created.

hypertext as an idea

This assignment was by its description very focussed on the idea of text, and what hypertext means, and a few folks has trouble distinguishing the idea of hypertext from the idea of graphic design. I recommend diving back into the history of this stuff and work out what is going on with hypertext as an idea, and where it has come from. e.g. the original proposal for the WWW, Doug Engelbart’s 1968 demo of some of the ideas in embryonic form , or even Vannevar Bush’s 1945 techno-utopian screed on technological knowledge diffusion which is credited with spawning lots of these ideas. Or you might want to expand on the list of media-art specific readings that I gave you - the media art history conference re:live has their archives online, and there is reams of hypertext-oriented material in there. Or you might want to look at some alternative vision of what hypertext could be, e.g: hypertextopia.

progressive enhancement

image courtesy AListApart

javascript good practice

Javascript frameworks: I noticed a few people using Spry in the class, spry being a javascript toolkit that comes bundled with Dreamweaver. If you press one of Dreamweaver’s ”insert javascript for me automatically” buttons you’ll get javascript that uses this spry toolkit. Nothing wrong with that, but be aware that this is different to the toolkit that we are using in the tutorials, jQuery. Things that work in one toolkit might not work in the other and vice versa. You are welcome to use Spry if you wish, but I suggest you will have an easier time with jQuery. Firstly, because it’s what the class tutorials use. Also also, the class tutorials use jQuery over Spry for several reasons.

1. jQuery has a lot more active users and a lot more tutorials online that Spry, so you’ll find it easier to get help.
2. Dreamweaver’s automatic Spry code is (to my mind!) unintuitive and, worse, it doesn’t really give you a sense of how it works when dreamweaver automatically builds it for you.

Now, last time when we left off, the site was in what you’d call a colossal mess. Trashed wordpress install, and still no interesting visualisation. So, how to improve things?

We’ll start by doing what I suggested last time – mapping off with a less ambitious dataset, based on LGAs (Local Government Areas) instead of suburbs. Here it is:

I even made a little chart. And look, since I am no longer quite as bored of CDATA as I was last week, I made another other spreadsheet based on percentage of people doing volunteer labour, by LGA.

Now, I’m keen to graph these data sets against each other. And I can, using google spreadsheets “Chart” function.

But first I need to tidy up those names. What’s with the ABS sticking “(C)” and “(A)” all over the place? FFS. Right, a new spreadsheet formula to trim off that crap. This one is a bit ugly:

But it works.

Now… about that chart… Well, we could do one in google spreadsheets, but that’s basically like doing it in excel and about as much fun. So let’s not.

Instead, let’s upload it to ManyEyes. Have you seen this thing? Fun, collaborative mashup, and you do it by copying and pasting. There are a couple of tricks. Like I can’t have dollar signs in the spreadsheet values, but I have to put the dollar sign in the column header. Then I have to select just the rows I want. Here a screenshot fo the spreadsheet after I’ve done that:

Note that I don’t know that I have to do that just because I’m some kind of web genius, in case you were wondering. In fact, I just read those instructions off the manyeyes upload page: (click through to see it yourself – you’ll have to sign on, mind)

Wham. There it is. I did the same for the volunteers doohickey. And then I made a merged version of those two and uploaded it, because it’s more interesting that way. (“Interest” is also why I spelled it “voolunteerism, ok?)

Now this ain’t a statistics class, but, er, there you go, a possibly informative data visualisation. Well, it might be if you click through to a less microscopic version. So it turns out that rich local government areas tend to volunteer an amount close to average, and poor ones volunteer over a wider range. Hm. Damn those rich people, and their… lack of variability? That could have been more interesting. Messing around with statistics until a more fascinating one pops out is left as an exercise, however. Let’s power on through and work out what we can do with the tools at hand.

We’re gonna map this thing.

Now, first I try that yahoo pipes idea the obvious way, and repeat last time – i just send the output of my spreadsheet into the yahoo pipe, and hope the location extractor will do something useful:

Look at that. Dang, no location information in the output. Lame! I might have to read the manual.

Location Extractor can glean location data from some URLs, such as those from maps.yahoo.com, maps.google.com, and mapquest.com.

A wide range of location mark-up is recognized, including GML (Geography Markup Language), Abbreviated GML, W3C Basic Geo, Abbreviated W3C Basic Geo, Simple GeoRSS, Yahoo! Local format, and KML LookAt and Point tags.

Right. Well, clearly, our feed doesn’t look like yahoo local or whatever the crap that is. But I reckon getting our data format in google maps link format should be fine, because it’s easy. I know from playing around, that you can get google maps to link to, say, Armidale, by using a URL like this:

http://maps.google.com/?q=Armidale, New South Wales

I guess that might be documented somewhere, but I just figured it out by typing things into the URL bar until it worked.

Anyway, we could use Google Spreadsheets to turn all those “Armidale”-type cells into “http://maps.google.com/?q=Armidale, New South Wales” type cells, but it’s late in the day, so I’ll show you a little hack to do it without using Yahoo Pipes to save more fiddling around:

Woo! It worked! What does it do? Well, I’m not going into too much detail with this one, since, as I said, you can do it all in google spreadsheets… but the regex tool is a handy thingy for doing translation of strings of letters into other strings of letters (in this case, into strings of letters with “New South Wales” stuck on the end, and the URL builder just shoves stuff into a URL – in this case, the google maps one. Feel free to play around with it. The only trick with this one was I noticed that Yahoo Pipes was occasionally breaking when I tried to use “Local Government Area” as a field name in the CSV import thingy, so I renamed it to “LGA” in the CSV import widget, and then it worked OK. I guess it doesn’t like spaces in the name. Anyway, watch out for that.

Long story short: It works.

But lets get that stuff off the pipes page, by getting the KML export URL:

Now we’ve copied that url, we paste it in to google spreadsheets, right there in the search box.

And the result?

Win. we have a google map of the yahoo pipe of the google spreadsheet of the ABS data.

Not that it’s perfect. I meant, as far as visualisations go, this could be way more interesting – this is more a map of where local governments are than it is of the stats that started the whole thing off. We could pup pop-up speech bubble with the ABS stats into the yahoo pipe output if we wanted, which might be fun. Or we could try to import THIS data set into wordpress. Or maybe we could try to map it differently. If you’re not afraid to get your feet wet in javascript, the handy javascript mapping library Openlayers javascript examples page includes on of resizable map markers. So we could build a map that had, maybe, big markers for really rich LGAs, and really small ones for poor ones. Or colourise them. Or… Lots of options. However, there are a few steps along that path that were a little hairy, so I’m going to give you an alternative geocoding method instead – since more of you seem to want to map specific locations rather than census statistics, it might pay to stick to close to what the class needs.

Basically, it was too much work to get it to geocode properly (it might have been a bit more fun if we controlled the source of data, and say, had a wordpress blog outputting GeoRSS or something).

Fortunately, Pamela Fox has blogged a nice way to geocode your spreadsheets. (She also has some awesome tips for designing maps.) Let’s follow her instructions. First, we add “New South Wales” to the end of all our LGA names, so that google can find ‘em better.

Nice. The rest is all following Pamela’s instructions.

You whack her gadget in there:

set it up:

Then you press the go button (small correction – up above I showed myself geocoding 150 records at once, but it only for 99. If you find it stops half-way through, that’s why. But it’s easy enough to restart.)

And then it goes. Not completely smoothly – it Turn out google has no idea where “Greater Hume Shire” is. And nor do I. I could have fixed that manually (possibly using the plain old paper street directory), but decided to be lazy. I copy and paste and then:

From the same blog post, Pamela also gives links to a handy spreadsheet mapping tool or two. There are, as always, too many alternatives. You might want to reverse geocode, or try this other hand bulk geocoder, or the GeoCommons open source geographic database, and there are semi-commerical services to automate part of it for you, such as umapper. Bah!

The real idea that I’m trying to get across here is that none of this is magical secret powers (well, except regexs), but there is a lot of muddling through. The list of tools out there is really large, and they change. No-one is a ninja at most of them, but be being happy to just try things and see if they work you can get a long way with a lot of them. There are bugs and annoyances. But then you usually get something at the end of the day.

Here are some links for lists of other directions to explore with mashups. I’ve mentioned some before, and some are new.

1. HackU
2. ToyChest
3. Simon Willison’s roundup for the Guardian
4. Freebase appears to have a general purpose mashup engine too.
5. YQL is hot right now, if you want to get just a shade geekier. It supports a massive number of the services we are looking at here and more I haven’t heard of, and if those aren’t enough, you can turn anything, almost, into a YQL data source with a little bit of code. Very handy.
6. google has a Javascript playground! (although I reckon the Openlayers examples page, mentioned above, is cooler)

Now, that’s enough writing from me for now. I’m may have time to do another walk-through of things for this semester… If anyone wants to suggest the topic in the comments, fire away.

One student has asked:

I’m in the process of looking for a webhost and am seemingly overwhelmed with options…

I’ll be hosting a small … portfolio site and am hoping to create custom email accounts. Pretty standard stuff…

Do you have any suggestions for companies that are reliable yet affordable, preferably local?

When looking for hosting there are a few factors to bear in mind:

Any massively very cheap option will suck to a degree. Sites like Dreamhost with millions of subscribers will definitely have at least a few tens of thousands of irate customers, by virtue of raw statistics, but also by virtue of being a bit lame.

THAT said, most commodity webhosting sucks, even if you pay a little more. For real. DO take the time to explore options. Preferable go with personal recommendations from people with similar needs to your own. I’d be wary of skimping now even for a simple site. If what you are hosting gradually grows and you wish to move it you will discover that migrating hosting is a horrible pain and can waste days of your time. If you to host only a small amount of content now, just make sure that it’s on a host that has several incremental plans that allow you to gradually increase your quotas over time. (Dreamhost, for example, used to have the $10/month plan, and the next step up was a $395 plan – I’m still in the process of extricating some enormous websites off there, a year after I started switching. They have some intermediate priced plans now too, but there’s enough stigma attached to them now that I will never host anything on them that will reflect on my professionalism)

As for going local – that’s a laudable option. Its greener, friendlier, there’s a good chance the support will be better… However, I don’t do it, because there are problems too. The economies of scale aren’t here in Australia, and even if they were, Australia is still a nation with an awful network infrastructure by the standard of your more future-focussed economies. Hosting here is, and will be for a long time to come, much more expensive. For most start-ups, prohibitively expensive.

Other random tips that you didn’t ask, but I might suggest anyway:

1. Never sign on to a host that guarantees you so so-many megabytes of data transfer, but does not guarantee you a certain amount of server memory, or a certain number of CPU cycles. The modern internet (at least in hosts in places with good infrastructure like the US) is not limited by bandwidth, but by CPU and memory. (by contrast, in Australia it can be both)
2. Never sign on to a plan that offers FTP access but not SSH or SFTP access. Very unprofessional. FTP is so monstrously insecure that any plan that offers it and not the more secure alternatives clearly is only appropriate for data you don’t mind losing and sites you don’t mind getting hacked. Moreover, SSH offers you the ability to do many very flexible and sophisticated things with your site that FTP does not, and if you end up doing any server-side development later on you will want it. I have banged on about this before, and I will again.
3. Who do I host with? My personal sites is on my fifth host now, and so far they have been the best by a long shot. They are http://www.webfaction.com – UK-based company, with servers in the States. For local hosting for clients who are prepared to pay a premium, I use http://pps.com.au. Much more expensive, but similarly excellent support and mere metres from UTS. There are, of course, many others – I just dont’ have any others that I can recommend as both responsive and full-featured at the moment.

disclaimer: I don’t know the legal terms to express this but… just because I’ve had good experiences with them, doesn’t mean you will… My views in no way reflect those of the UTS. Etc. Choose your host at your own risk, and whomever you go with, remember that your choice will reflect a lot about your professionalism to clients and the wider world. Take care, and get multiple opinions.

Now, the other thing to consider here is whether you want to even have classic hosting.

All the discussion so far has assumed that you want to go with the “classic” model of hosting. Let’s pick apart steps in that:

1. you give an ISP money
2. they give you a user account on a shared computer somewhere (or maybe, if you give them lots of money, they plug in a whole computer just for you)
3. That computer is running some set of open-source tools: Linux, Apache, MySQL and PHP (or for my sites, usually something a bit more fashionable such as nginx, MongoDB and python, or maybe Ruby on Rails… but same general principle)
4. you upload some files to it, maybe your wordpress blog software, or a wiki, or some kind of community forum written using the aforementioned software.
5. Users can sign up to your site, get passwords emailed out to them and post comments to the site.

There are permutations here involving hosting on windows or mac os (if you like paying lots of money for essentially the same product) or hosting on virtual machines, but let’s put those ideas in the same box – basically, it boils down to us, as site maintainers, getting to know one machine very well as it has some files and software on it that makes our self-contained little site site go.

Now, let’s consider this in the light of the tools that we’ve been looking at in the course. Every one of those 5 steps is subject to revision. Facebook, google, yahoo, twitter, amazon… Is this how they do things, with some geeky-lookin’ sysadmin somewhere plugging in stuff for us to play with somewhere? Does the entire world run on webfaction (or even dreamhost?)

Not so much. There’s a key buzzword here that you can take home: cloud computing. Like most buzzwords involved in this course, it’s being used to sell so much stuff that there is no longer any real fixed definition left… but the gist of it is that it marks the new wave of web hosting and web services which de-emphasise the role of your particular piece of hardware somewhere, and concentrate on getting your stuff out there. I like Jason Scott’s definition:

By the cloud, of course, I mean this idea that you have a local machine, a box running some OS, and a vital, distinct part of what you do and what you’re about or what you consider important to you is on other machines that you don’t run, don’t control, don’t buy, don’t administrate, and don’t really understand. These machines are connected via the internet, and if you have a company then these other machines are not machines run by your company, and if you’re a person they are giving it to you without you signing anything accompanied by cash or payment that says “and I mean it“.

At the lowest level, companies like google and amazon lease out their infrastructure in products called, respectively Google App Engine, Force.com and Amazon Web Services. You can’t run classic applications such as wordpress on these, but instead must write custom applications that use the unique quirks of the system they are built with (although not always – see rackspace cloud for a different approach.) Consider also Eucalyptus, an open-source solution that allows you to build your own “cloud” – sadly probably below the budgets of most of us here.)

A level “up”, some cloud providers provide more specialised tools than basic data storing. Talis Platform and Freebase’s Acre, for example, provide “semantic” web applications that can query data in human-comprehensible ways, provides particular tools for customer relation-ship management.

Another level up… there are applications out there that provide services that you would once have had to build your own application for… Why copy your photos onto a web host and code your own photo gallery when there is Yahoo’s Flickr or Google’s Picasa web, or Adobe’s Photoshop.com? Who has built their own map server rather than just using google maps, Bing Maps or even stodgy old MapQuest? For that matter, why even bother installing the slow, expensive and bloated Microsoft office when you can use google docs or Zoho office? Or taking it still further – who here accesses that most primordial of internet services, email, using someone else’s web app? Google mail? Yahoo Mail? Windows Live? Anyone? Who, even,  uses weird services that only make sense on the internet, such as Yahoo pipes, freebase and YQL?

And if you want to host comments on your site, why woudl you run your own blog comment engine if there are systems that allow people to comment on your site and share their comments across blogs, such as  DISQUS Comments and IntenseDebate? Why manage users and give people yet another bloody password if you can leave the data in the hands of your users using OpenID, Yahoo IDs, Google Accounts, Facebook accounts (or all at once using RPX?) Why build your own social networks in general if there are hosted ones such as ning, or if you can integrate with Facebook or Google Friend Connect?

Here’s some advantages to think about with these services:

• in some cases… built in crowdsourcing (how good is gmail’s spam filter?)
• some cloud services have great viral/social media implications (consider if your site posts updates to your friends newsfeeds in facebook)
• built-in scalability (if your site goes up on digg.com and you suddenly have ten thousand people trying to view it at once, what happens?)
• economies of scale (a lot of this stuff is just plain cheaper than even buying your own computer and keeping it under your desk, because you don’t have the buying power of google.)

Now, some disadvantages:

• some of these systems don’t run traditional web software (although traditional web software may suck – and, as I keep saying, you should consider any data you post to dreamhost’s cheap arse accounts to have been donated to the world at large anyway)
• The privacy implications of some services are unknown but dubious: (e.g. Google Privacy Blunder Shares Your Docs Without Permission, Does what happens in facebook stay in facebook?, Lifehacker’s take, New York Times’ take)
• You are at the whim of someone else’s opinion of whether your activity is ok (e.g Facebook email censorship,  13 reasons your facebook account will be disabled, Yahoo rats out Chinese reporter to Beijing, writer gets 10 years jail)
• You are at the mercy of someone else’s data centres and backup regimes
• It despite some people touting its sustainability it may be an econological disaster (although is that just laziness?)
• Sundry other issues

So, to roll your own, or to host in the cloud?

Personally, I take it on a case by case basis. Possessing infrastructure that circumvents pervasive corporate surveillance is probably necessary for a health democracy, but it is time spent rolling my own everything a worthwhile investment? This is a critical question, and we’ll return to it in the final lecture.

Points to consider:

1. is wordpress.com a cloud service?
2. How much do each of these services cost? (Does someone want to calculate the comparative cost of hosting on Amazon EC2 and Dreamhost? How about comparing with the costs of a local Australian host?)

Further reading? Sure. Try:

• Electric Duncan: After the Cloud
• Richard Stallman warns Cloud Computing is a trap
• Tim O’Reilly’s take on the definitions and economics at stake
• The Open Cloud Manifesto
• The Belgian Beer Lovers Guide to Cloud Security
• if all this talk of self-hosted versus cloud hosting doesn’t make much sense yet, perhaps you can get nitty gritty. Compare the lifecycle of a classic PHP page view with a google app engine page view.

Postscript:

Get a slice of history! Geocities closed in the course of the week of this lecture. What does this say about the transitory nature of things hosted online?

Has anyone noticed this section of our course guide?

Sound and video for the web: Issues re video and sound delivery on the web. Overview of compression and techniques for archived media, podcasts and real time streaming.

Of all the course notes, that’s the one that I think has dated most since it was written. So, let’s take a quick dive into why that is, and what it might mean.

Let’s start with the most basic requirement of browser-based media. Can anyone think of a  ”Distributed hypermedia method for automatically invoking an external application providing interaction and display of embedded objects within a hypermedia document”?

That’s the summary of a patent. A patent that has recently won the company holding it $585 million dollars in damages from Microsoft — and many other companies are likely to cough up more follow. There’s some pretty remarkable patents being claimed out there – did anyone see the patent claim on XML? Have you heard the term “patent troll”? “Submarine patent”?

Patents wars are just one of the many kinds wars being waged on- and off-line governing the kinds of media we can consume, and whom and how we pay to do so.

How is this relevant to the question of how to encode your video for the web?

Let me start with the trite summary of the course objective of teaching you how to encode video for the web:

To upload your video to the internet, it’s probably easiest to upload it to vimeo or youtube.

Does that tell you anything about what happens to the bits and bytes of your video? No. But it tells you a lot, implicitly, about the state of media on the internet that the easiest way of dealing with it is to let some third party handle all the manifold problems for you.

To come at it from another angle, let’s look at Ian Hickson’s recent announcement that the latest, hottest HTML 5 standard will include no standard video format. On one hand, Apple reckons the open-source entrant, Ogg Theora, is crap, and on the other, the open source browser, Firefox can’t support H.264 because it requires licence fees and Firefox is a free browser. Spot quiz: what format does Youtube use for its videos?

Still interested in the mechanics of building your own video streaming service? you crazy fellow. OK, I recommend lighttpd’s FLV streaming module. Or maybe give plumi a go. The punchline is that there are a lot of “classic” ideas about how to present your media online, but between the standards battles, the different bandwidths, browser glitches and so on, this is a market that you probably want to enter for very specific ideological or technological reasons (breaking down the youtube monopoly, hating their censorship policies, hating iTunes’ curation, displaying on some device that is beyond mainstream support) rather than because it is in any way convenient. Because there is no way of “just” putting your media out there; there are many encodings for different devices, and many bandwidth problems and so on, and they are all very vulnerable to being solved by corporate economies of scale.

Or, to go deeper into that tricky minutiae, you might want to check out this handy presentation that Fred Rodrigues has offered up to the class:

Or spy on a course by Sean Healy down at RMIT which in many ways is a sister course to this.

A lot of this technical detail, for me, is less interesting than the other battles sweeping the mediascape. Consider how the net changes the economics of  distribution of media. (To read/view: Peter Hirshberg on PC vs TV, Dave Winer on podcasts RSS enclosures)

I’m interested in the massive scale and low marginal cost of digital distribution, that is, rather than the tedious technical details. This may end up being relevant to the livelihoods of lots of us in the room. Have you heard of the idea of  disruptive innovation? (see also here What does it mean for traditional means of disseminating media – how abou the much-heralded death of the CD? (On the other hand, how about the 88 million cassette tapes sold each year in Turkey alone? or the endless resurgence of vinyl in unexpected forms?)

On the other side of the coin, if MP3 trading is so notoriously cheap, how do manufacturers wtill profit by it? Are we heading into a world of iTunes stores?

On the other hand, if information distribution is so very cheap, how do manufacturers value-add? Could you or I build youtube right now if we wanted to without its audience base?  (and someone has already written the code) Consider the earlier class on the web as a social landscape.

What do these technologies do for our aesthetics? Consider:

… there’s evidence that consumers are simply adapting to the MP3′s thin sound. Jonathan Berger, a professor of music at Stanford University, recently completed a six-year study of his students. Every year he asked new arrivals in his class to listen to the same musical excerpts played in a variety of digital formats—from standard MP3s to high-fidelity uncompressed files—and rate their preferences. Every year, he reports, more and more students preferred the sound of MP3s, particularly for rock music. They’ve grown accustomed to what Berger calls the percussive sizzle—aka distortion—found in compressed music. To them, that’s what music is supposed to sound like.

How about our ethics?

Bonus Reading

1. Bower, Joseph L. & Christensen, Clayton M. (1995). “Disruptive Technologies: Catching the Wave” Harvard Business Review, January-February 1995
2. WhatWG blog
3. Pitchfork’s Social History of the MP3