News You Can Bruise

Wed, 27 Aug 2008 02:56:46 GMT

Apropos today's Dinosaur Comics cartoon about abrupt sex changes I emailed Ryan to tell him about a movie I'd seen in India. There were no subtitles but I got the gist of it:

[A] guy was a jerk to women, and ended up getting hit on the head with a statue of Shiva [on second thought, maybe it was just a random statue -LR]. He had a dream where Shiva and Parvati lectured him, and when he woke up he'd been turned into a woman. It was wacky! But then he slept with his best friend or something, and got pregnant! And then the girl he was a jerk to framed him for the murder of his male self! He went to jail! He was about to give birth when we got to the airport and I stopped watching the movie.
Unfortunately it's difficult for me to search for this movie because some subset of these things happens in almost every Bollywood movie. This was just the one that had it all.

Ryan's friend Priya knew the movie! It's called Mr Ya Miss, and Priya comments: "It is the worst movie ever." Also, apparently the blow to the head actually killed the protagonist, and the scene with Shiva and Parvati was a reincarnation interview. I don't think reincarnation is supposed to give you a fully mature new body, but Parvati works in mysterious ways her wonders to perform. And how can you say a movie is the worst ever when the main character gets killed fifteen minutes into the movie? I've been waiting for that for years!

Content-Encoding versus Transfer-Encoding

Tue, 26 Aug 2008 22:23:19 GMT

Today I discovered a "problem" where Apache's mod_deflate changes a resource's ETag value from "foo" to "foo"-gzip when compressing the representation. This problem has been a while in coming, and after some research I'm convinced that Apache's behavior is almost completely correct.[0] But very annoying. Your representations are transparently compressed on the way out, but then the client sends you values for If-Match and If-None-Match that you've never seen before. True transparency is a two-way street.

There are a number of ways to solve the problem. One way would be for Apache to magically strip the "-gzip" off the entity-tags when they come back in, but how is Apache supposed to know that that particular -gzip was one it added? So, as usual, the magic solution fails.

An alternative that appeals to me is to stop treating compression as an aspect of the representation such that having it or not having it means you need to change the ETag. Move it out of Accept-Encoding and Content-Encoding, and into TE and Transfer-Encoding. This is related to what I was saying last year, but I didn't make the connection between "this is part of the transmission, not part of the representation" and "this should be in Transfer-Encoding, not Content-Encoding." Or to quote the RFC, "the transfer-coding is a property of the message, not of the entity."

The problem with this is that Transfer-Encoding describes the transmission from one intermediary in a chain to the next, not the transmission from server to client. Similarly for TE and the transmission from client to server. Intermediaries are my RESTful weak point so I don't fully grasp the implications of this. If the client sends TE: gzip, the proxy can decide to get an uncompressed representation, look at it, and then compress it before sending it back to me. That hurts performance but everything still works, right?

Update: "That is why transfer encoding was invented in the first place."

[0] To be 100% correct it would change the ETag to "foo-gzip", since entity-tags are supposed to be enclosed in quotes.

Tue, 26 Aug 2008 21:27:25 GMT

If you're up on your Anathem gossip you'll know that the book (or, at least, the Advance Reader's Copy) comes with a CD of nice acapella chant music. What has been somewhat obscured by descriptions of this music is that each piece doesn't just have a fancy math-related title, it actually is a piece of math. Usually a description of a cellular automaton, or a proof represented in some bizarre musical Gödel-encoding. And what I didn't know until Mirabai emailed me is that the scores for these songs are being published by their author, David Stutz.

Totally Heavenly Product

Tue, 26 Aug 2008 00:53:28 GMT

More praise from spam weblogs:

I must assert that I’m generally one that does not love shopping online, but my concept on that has changed since I ordered my RESTful Web Services from ebay...
My RESTful Web Services arrived in a week from my seller. The RESTful Web Services was exactly as it was described! I am so at ease about ebay, I’m sure I will go back for more!
I have left more info on RESTful Web Services, a totally heavenly product...

I see nothing special about the RESTful Web Services. >

Crankiness Antidote

Tue, 26 Aug 2008 00:44:34 GMT

So the only thing I post today isn't me bring cranky, check out this excellent summary of the Launch Pad Astronomy Workshop, where writers go to learn about the universe. Yes, this universe.

Tue, 26 Aug 2008 00:04:04 GMT

Is short fiction devalued by being available for free? Hey, here's the elephant in the room that keeps being pointed out and yet remains elephant-shaped: there's way more short fiction than there is market for it. In my writing group we critique 2-3 stories a month. Half of 'em are a rewrite away from publication quality. They're also likely years away from publication, because the market is so small and response times so long.

Why do the markets pay so poorly? That's the natural result of massive oversupply. Why is there so much bad free stuff online? My portfolio shows the problem in miniature. My old crappy stories are online because I didn't know any better. My dynamite new stuff circulates endlessly through snail and electronic mail. Sure, I'm bitter, but it seems pointless to complain about this because the solution is obvious.

Seriously, what am I doing? I don't care about money--there's not enough to care about. If I wanted an audience I'd split my stories into RSS-feed cliffhangers and put them in syndication once they passed writing group muster. All that's left is the feeling that there are dues I ought to be paying. I've got a bad case of professionalism.

Who needs it? Life is too short. I know what position I'm sliding towards and I keep resisting it. Maybe I'll start writing novels instead; those take a lot longer to crank out, at least.

Sun, 24 Aug 2008 22:46:02 GMT

Best "I ♥ NY" shirt ever: one in which the shirt is the same color as the ♥, making it read "I NY".

Sat, 23 Aug 2008 21:02:13 GMT

I was reading an interesting article about competitive video game players. Some of these players express a strange attitude towards a game's "kill screen", the point at which the game breaks down because the authors didn't anticipate anyone getting that far.

From a programmer's perspective the kill screen is something to be decoded and understood. It is comprehensible and can be fixed. But when the kill screen comes at the end of a grueling ritual there seems a temptation to see it in esoteric and mystical terms.

With Pac-Man, there has always been a powerful appeal surrounding the notion of "The Doorway" -- a prospective passageway to the other side, a way past level 256... the final prize Pac-Man collects is not a fruit but a key, the last of nine--and why are there keys if there is nothing to unlock?

Before I go on, let me make my position clear: I am a total video game nerd (though not a particularly angry one). Songs have I written and stories that draw from this pixelated well. My cohort has a fascination with video games: old ones, new ones, the people who make them, the ones we make ourselves, their distribution mechanisms, their similarities and basic building blocks, the ways we push ourselves to best them, the stories we tell about them, the relationships they create and mediate.

So don't take it as "Get a life!" when I say there's nothing special about the games themselves. Like books, they only have the power we give them. Pac-Man has a bug. It's not even an Easter Egg. There's nothing to unlock. The kill screen is not in the realm of the meant. If you spend years mastering Pac-Man and prefer it to Ms. Pac-Man because it's totally deterministic, why get mystical about the way it crashes at the end? This is real life, not Lucky Wander Boy.

I could see writing fan fiction about bugs, the same kind of fan fiction that explains what it means to make the Kessel Run in twelve parsecs, but this kind of talk about the bug's mysterious meaning leaves me cold.

PS: Feel free to apply this attitude to everything else in life! Other people will love it!

PPS: The keys are for use in Super Pac-Man.

Das Komputermaschine Ist Nicht

Thu, 21 Aug 2008 13:04:55 GMT

I saw a German ad for the Commodore 64 and it got me thinking about how many C64s made it into East Germany. The PolyPlay was made in the mid-80s, and it was extremely popular despite the games being terrible by 1985 standards, especially compared to the C64. (You can play the PolyPlay games on MAME, or in Flash here.)

Well, in a move that can only be described as "astounding" I did some searching and found this translated essay on the topic. It's excellent. It takes you through the adolescence of an East German computer nerd, with its PolyPlays and its Robotron computers and its sections in the back of state-sponsored ham radio magazines. Once nerds got their own computers they were able to clone the PolyPlay games and distribute the clones through the mail.

Another interesting aspect is the difference between West Germany, which took a Music Man anti-pool-table approach to video games, and the East, which coopted the mania (or, I suppose, cöopted it).

Computer gaming was made a matter of the State – and computer gaming was officially labelled "Computersport", which means "computer sports" [thank you. -LR]. Connecting computer gaming to politics had a huge impact on the future development of computing in East Germany... In West Germany, in 1984 a new youth protection law prohibited gaming computers at public places. The new technologies were denounced to have a bad influence on young people, who from then had to go to bars if they wanted to play. In East Germany, the government had realised that computers were to become an important economic means in the future... Inspired by the Polyplay, or by visits at relative's places, many youths started putting together their own computers, or to program their own games.

And at the end, the essay answers my question.

Even before the borders were opened, and in spite of an import prohibition, advertisings in the Funkamateur [the aforementioned ham radio magazine -LR] offered C64 computers for 5000 East Mark, a horrendous price. But nevertheless, the commodore took over in the Berlin scene.

After that, unfortunately, "programming was less important here than cracking and copying C64 games." Thanks for the essay, Thilo Mischke and Kerstin Grosch. And Melanie Swalwell, for hosting the essay and also creating some sort of interactive oral history of video games in New Zealand.

One thing that didn't occur to me until I started writing this entry was that the people who made PolyPlay must have played a bunch of decadent Western games to figure out which ones to clone.

"A large bubble rises to the surface and pops"

Wed, 20 Aug 2008 21:12:43 GMT

The bathroom sink was draining slowly. I didn't want to buy a plunger or a specialized drain-cleaning chemical so I tried some Bill Nye the Science Guy-level science and made a fourth grade science fair in the sink. I poured some baking soda into the sink and washed it down with a minimum of water. Then I poured in some vinegar. It bubbled. Eventually I shut the drain in hopes that the CO₂ would build up pressure in the pipes and eject whatever was clogging the drain, but it didn't seem to be necessary. The sink works fine now.

Unlike more common uses of the deadly vinegar/baking soda combo, this would actually be a good fourth-grade science fair project.

Wed, 20 Aug 2008 03:23:59 GMT

While I wait for the dubbed edition of GameCenter CX to come out on DVD, I've been watching Frankomatic's "Obscure Game Theater", a mile-long series of YouTube videos in which the aforementioned Frankomatic gives retro games his ganbatte best, except with more cursing than you get from Shinya Arino.

While watching one of these videos I realized that A Boy and His Blob, possibly the NES game to combine the coolest concept with the worst execution, takes place in New Jersey. You start off looking at what seems to be the Empire State Building across a body of water, and as you run to the right (ie. south) you see the Manhattan skyline with the World Trade Center at the far right. It's a very nice graphic, actually, probably digitized from a photo (Wikipedia and the SEC agree that Absolute Entertainment was based in New Jersey). I think the game is supposed to take place in Brooklyn instead of Hoboken, because it's got a subway station, but basic directionality says the game board is to the west of Manhattan.

Tue, 19 Aug 2008 19:28:44 GMT

That one deserved its own entry. Onwards! (Again, here's the document I'm responding to.)

Is Statelessness Required? The universe will not end in an ontological segfault if you violate statelessness. Instead, you will give up a lot of scalability potential, and you will hide state where your clients can't get at it, which will annoy them. So why do it? Maybe because you're using a framework that makes it really easy to write programs that violate statelessness, and then tries complicated tricks to get scalability back, because that's the way web frameworks have been written for ten years.

We took a pretty hard anti-cookie line in RWS (pages 252-253), but we made it clear that cookies themselves are not the problem. Cookies, like links, are a way of serving application state. They have one big RESTfulness problem but it's moot because nobody follows the cookie standard that slavishly. The real problem is the session IDs that go into those cookies.

A session ID is a key into a big chunk of state kept invisibly on the server. That's what causes the problems. It's just as wrong to serve that session ID as a query variable in a link as it is to serve it in a cookie. You need to turn the hidden state into application state by incorporating it into links and forms, or you need to turn it into resource state by exposing it as part of your web service and letting clients manipulate it directly.

Is Being Message-Centric Good Enough? I say yes. I'm young enough that the Web is the first distributed programming environment I ever used. I've never felt like I was missing something that justified switching to a competitor. The more I've learned about its design the more impressed I've been. When something better comes along, I predict it will look more like the Web than it will look like DCOM or CORBA or WS-*, whether it's "better" in a global sense or in some application-limited sense.

Are PUT and DELETE Essential? The universe will not end in an ontological segfault etc. Also, you won't even stop being RESTful, assuming you have proper resource design, as per my first entry in this series. What you'll give up is the ability to make various reliability guarantees. I was going to say you also give up the ability to avoid the lost-update problem, but I guess you could do that with POST, so long as (yes) you were POSTing to the same URI you sent GET to earlier.

I don't know how good an argument this is anymore, but: designing with PUT and DELETE forces you to think in terms of resources. It's a form of discipline, like eschewing cookies. When you've got "read data" and "whatever!" it's very easy to think in terms of operations, flex your ten-year-old web application muscles, and end up with a mess like the Flickr web service. The vocabulary is negotiable. The underlying idea, (that a URI designates an object which responds to a standard vocabulary) is essential. That's the Architecture of the World Wide Web.

Is "Do Like the Web" a Good Argument? Not in and of itself, because parts of the Web are lousy. You need a tool to separate the good from the bad. I follow the admonition of Paul: "Prove all things; hold fast that which is good." Look at what people love about the Web and see if you can bring that same joy to distributed programming. Look at what people complain about and try to eliminate those things, whether you're building web applications or web services. When you find something that improves the distributed programming side of things, bring that knowledge back into the field of web application design.

Where Are The Tools? Traditionally this question has been answered by assertions that the (HTTP-level) tools already exist. As annoying as this answer is, it's technically correct. Without additional abstractions on top of HTTP/URI/*ML, there's nothing to write tools about. As we invent those abstractions (AtomPub, ROA, the ActiveResource control flow, OAuth, WADL, etc.) we write more tools. I'm writing tools now. As we get more practice the higher-level tools will get better, and as they get better they'll consolidate (as will the abstractions beneath them) and there will be fewer.

One thing I'd like to add is that it would be cool to see the existing frameworks for web applications apply some of the principles people have come up with while using the Web as a distributed programming environment. Make it easy to publish resources rather than operations, easy to support conditional GET, easy to write client interactions that respect statelessness. Rails has the right idea here.

'What Does "Hypermedia as the Engine of Application State" Mean, Really?'

Tue, 19 Aug 2008 19:04:38 GMT

It means that you operate the web service by following links and filling out forms. (For definitions of "link" and "form" that depend on the media type.) It means you can use a generic "web service browser" that doesn't have a bunch of hard-coded knowledge about one particular web service, the way PyAmazon does. When the web service changes, user behavior may need to change with it, but the client itself isn't instantly rendered useless, the way PyAmazon was. I called this "connectedness" in RWS, and Roy Fielding himself slammed me for it, but when Tim Bray, after years of real-world experience, isn't sure what "hypermedia as the engine of application state" means, I think that's a sign it needs to be explained in different words.

Saying that RESTful services "work like the web" is not just saying that things have URIs or you can cache GET requests. Ideas as confusing as hypermedia-as-the-engine-of-application-state become understandable when you think about the way you use the web.

If a website is well-designed, you don't need to mess around with the URL bar to get where you want to be: you fill out forms and follow links. You don't need a custom web browser to use crummy.com. My website is distinguished from all other websites by the HTML documents I serve. If I change my weblog software, it'll serve you different documents, and you may have to change the way you use the website, but your browser doesn't crash.

You know what to do (how to drive the crummy.com "application" into the desired state) because the document I send you lays out your options. And because the document represents the options as hypermedia links and forms, you know how to build an HTTP request that will carry out your desires.

Request Weblog #Frog.Frog

Tue, 19 Aug 2008 15:52:38 GMT

OK, now on to the specific weblog entry on which Rachel asked me for my thoughts. These are mostly based on the work I've done designing and building services since RWS was published.

"Has REST Been Fortunate in Its Enemies?" I really hope the whole "enemies" thing goes away and that the period 2000-2005 is seen in retrospect as a big misunderstanding. Just in the period since RWS was published there have been really interesting developments (OAuth, PATCH) that have advanced the state of the HTTP art, and we lost five years of that kind of thing going down the wrong path and then having a huge argument about which path was right. That said, when your enemies turn out to be big corporations with lots of money (as always seems to happen to me), I would not describe the situation as fortunate.

Schema-driven mapping: probably hopeless if you want a really detailed mapping to plug-and-chug into your strongly-typed language. Even if possible, very boring. Not too difficult if you just want an index that points out the interesting parts of a document.

Contracts: My personal crackpot theory that I can't get anyone to believe is that hypermedia is contracts. A link or form is a promise that if you make a certain HTTP request, you'll get a certain result. HTML is a primitive contract language. The AtomPub service document format and WADL are more advanced contract languages. That said, I think contracts are less useful than often supposed, because the ELIZA effect artificially narrows the distance between the syntax of the contract document and its semantics. An AtomPub service document works like magic because a human programmer did some work beforehand understanding the AtomPub standard and programming in the semantics.

Registries: We have registries on the human web; we should be able to have them for web services. How much of their imagined benefit is due to the ELIZA effect? How much work will a human need to put in to find the one of 10000 AtomPub implementations they want to post to? I don't know. It depends on the conventions and standards we come up with.

Payload wrappers: HTTP is a sufficient payload wrapper for my needs, and if I ran into a problem with it I'd extend it (a la OAuth), not that I feel competent to do that. I can kind of see how you'd consider Atom a payload wrapper, but I'm just a simple caveman and I prefer to think of it as a representation format.

Message-level signing and security: yes! to the former. OAuth works for me. I don't know if I'd ever use message-level security. In general, experimentation is good when it happens on top of HTTP where everyone can use the new standards.

"Is Getting HTTP Right Good Enough?" No, not in the way we mean "HTTP" when we have this never-ending argument about verbs. I don't even think it's the first step. The first step is to expose resources: to give a distinct URI to every object you want to publish. That's the first of the RESTful rehabilitation suggestions on page 303 of RWS. That's the first three steps of the ROA design procedure on page 148. The first document to understand is not the Fielding thesis, not RFC 2616, not even RWS, it's Architecure of the World Wide Web, Volume One. That document is mostly about URIs. HTTP shows up as a URI scheme and as a popular protocol for dereferencing URIs.

This is why Flickr and del.icio.us put up terribly-designed web services and people love them and use them all the time: they stick URIs on everything. People love URIs! Their problem with those services is they don't know when to stop. So the first step is to get on the web, to not ignore the intuitively obvious usefulness of URIs. It's terrible that these web services self-identify as "RESTful", but at least they've taken the first step.

The second step is to know when to stop handing out URIs. Picking a vocabulary lets you move some of the information out of the URI and into the HTTP method. It lets you expose objects that have methods, instead of a big C library of del.icio.us or Flickr-style functions. This is the step where you master HTTP. This is where the arguments are happening right now. These arguments are not based on intuitively obvious things. They're arguments about scalability and fiasco avoidance and client and network capabilities. They're also arguments about whether using all of HTTP would improve the Web.

The third step is the mastery of hypermedia. (Note that there's one step for each of the Web technologies: URI, HTTP, HTML.) This is about making it possible to program a computer to do the mapping of options to actions that happens in our brains when we surf the web. The advantage here is the same as in the other steps: loose coupling and flexibility. This is the part that confuses everybody, because we don't usually write programs like this unless we're scripting or screen-scraping. And the ELIZA effect is in play, because we've moved past HTTP requests and responses to what happens inside us when we look at response N and decide to make request N+1.

To the extent that there is a "religious" aspect to RESTful thinking, I think it's on this level, in the intuition that there's something in the high-level way we use the Web that we can use when programing a client. Maybe it will turn out to be a bust like AI in the 80s. I've had some encouraging results on my current project. I think it's likely that we're only now figuring this out because we spent five years arguing over whether to use URIs and then another three arguing over whether to use GET/POST or GET/POST/PUT/DELETE.

Request Weblog #Frog: REST Request

Mon, 18 Aug 2008 23:20:49 GMT

I wasn't going to get involved in the latest REST brouhaha because... well, because I've had to deal with nearly-identical brouhaha at work, and I'm kind of tired of it. But Rachel Chalmers asked for my thoughts, and I have a conference talk I need to start thinking about, so sure.

It looks like this is going to be a multi-part entry, so here is the single most important point I'd like to make. I mentioned it here, last year, and also mentioned it in RESTful Web Services (see especially pages 220-221), though not as prominently as I probably should've. Maybe I'll work this here into the second edition.

REST says you should use a uniform interface, but it doesn't say which one. How do you choose? You pick a set of verbs that 1) gives you the semantics you need for your application, and 2) lets you tie into network effects.

On one end there's the degenerate uniform interface of XML-RPC and SOAP: [POST]. Like a language with only one word, this is pretty useless. Seeing that "POST" gives you absolutely no information. POST means: "whatever!" You're just pushing the complexity somewhere else.

The one thing I want to say about this degenerate vocabulary is that your decision to use it is orthogonal to your decisions about resource design. XML-RPC and SOAP services provide a single "endpoint" resource, a monster message-processor at a single URL that responds to a wide variety of POST requests. That's bad resource design. But you could expose a tiny "message processor" for every part of your application you want clients to manipulate, and have them serve hypermedia documents (in response to POST) that linked these resources together. (See RWS page 303.) It would just suck, because clients could only communicate with those resources through POST.

Then there's the uniform interface of the human web: [GET POST]. Now that there are two verbs, you can give them separate meanings and split the complexity. GET means "read data" and POST means "change data." Actually, POST still means "whatever!" but since it's defined in opposition to GET, it's convenient to think of it as the opposite of GET.

Choose this set of verbs and you can perform a series of nifty optimizations on GET, which probably make up the bulk of your requests anyway. The GET optimizations work because GET means something. Its meaning is constrained, and constraints can be starting points for optimizations.

But, now that the words have real meanings, they can be misused. Use of GET where you mean POST will make you a victim of the next Google Web Accelerator type fiasco. Use of POST where you mean GET will ruin addressability and annoy your users.

And again, your decision about vocabulary is orthogonal to your decisions about resource design. Whence my rule of thumb: "POST to the same place you GET." If you had well-designed resources that responded to the degenerate vocabulary of [POST], you could convert your "read data" operations to GET and get a much better web service without changing your resource design. (again, see RWS page 303.)

Moving further down the scale of complexity is the uniform interface most often seen in discussions of REST: [GET POST PUT DELETE]. This vocabulary isolates certain classes of "change data" operations, rips them out of POST, and gives them their own names. POST still means "whatever!"

Why would we do this? Splitting out PUT and DELETE means giving them a meaning distinct from "whatever!" And a meaning is a set of constraints, and you can optimize around constraints, etc. Your decision to use this vocabulary is a decision about which constraints are useful.

And for the third time this is orthogonal to resource design. If you had a RESTful web service that used GET and POST for everything, you could PUTify any POST operations that fit the PUT constraints, DELETEify any operations that fit the DELETE constraints, and then you'd have a four-verb RESTful service. You could go the other way, too: fold PUT and DELETE back into POST and you'd have a RESTful two-verb service. What you'd lose is the ability to say, "make sure the state of the resource reflects the submitted document" or "make sure the resource goes away", instead of "whatever!"

A little more complex is the uniform interface we used throughout RWS. We use the same four verbs, but we told POST to stop meaning "whatever!" When we use POST, it's only allowed to mean "create a new resource underneath this one." We did this because "whatever!" is a linguistic rug under which to sweep things. If you write a book about home organization where you say "and here's how to sweep anything inconvenient under the rug!", readers will suspect that there are systemic flaws in your organization techniques. So we wrote a book where we organized complex things like queues and transaction systems without recourse to "whatever!"

But on a technical level, the point is not that "whatever!" is evil. The point is that if you chip a piece off "whatever!" and make it mean something specific, you can optimize around the constraints and reap the benefits. It's pretty clear from the history of the web that you need to separate "read data" out from "whatever!" Because there's less history, there's active disagreement about the cost/benefit tradeoffs of PUT, DELETE, PATCH, et al., though I find them very useful. But the vocabulary you use is an interoperability shibboleth, not a RESTfulness shibboleth.