Challenges When Programming for Home
by John C. Ardussi, President of Game Mechanics LLC
The title of this article reflects the popular assumption that programming PlayStation Home is somehow difficult, which has led to all the bugs and crashing. The reality is that this is partially true, but far from explaining all that we see happening.
The Macintosh Way
Let me start from a seemingly unrelated yet very pertinent experience. Back in the 1980’s, I programmed Macintosh computers exclusively. I even started a company to sell a library of routines that made it a lot easier to program Macs. The common thought at the time was that the Macintosh was so complex that it was doomed to crash. There was nothing you could do to avoid it. Saving your work a lot was the only option.
It turned out to be a complete lie used to explain bad software. Some of the bad software was built into the system and some was created by third party developers.
The library I created for programming Macintoshes
In 1988, I was hired to work on tape backup software for the Macintosh. The project manager was the first person to convince me that software never had to crash on any platform. This idea of “too complex not to crash” was just a cover for sloppy programming and not enough testing. And he was right. They tested our software doing crazy things like physically unplugging the computer from the network while backing up a server over the network. And yet, no crash.
Finding My Way Home
When I first started working on PlayStation Home in Sept, 2009, it was fragile compared to today. There were things you could do that the manual suggested you do that you should never do. They would crash the system. In Nov, 2009 came the first sign of hope — a new version that was much more solid. That trend continues to this day. We have had a few buggy versions, but they patched those pretty quickly.
What was great was while they were working on stability, they added Active Items. Before Active Items we just had Mini-Games. One of the big differences between a Mini-Game and an Active Item was that they quarantined off Active Items into their own memory space. I won’t go into the specifics of why that’s so important from a stability standpoint, but trust me: it is.
What just happened?
One of the crash-prone challenges with Home was the issue of insufficient memory. Why? It was difficult, in the early days, to know how much memory you had left to use, and any library call you might make could try to allocate memory and fail. Now there are lots of tools that help you see what the memory situation is and make it more likely you will see the problem and create code that avoids a crash.
The next major area of difficulty is networking. The setup can best be described as “CLIENT-CLIENT-server.” While there is a server in the mix, it is mostly a dumb server that passes packets around and mostly keeps track of who is here and who has left. So it is not a true “Client-Client” situation, but for many things it is.
24 players in a game at once
The big time this comes into play is when two players want to do the same thing at the same time. For instance, the “Cutthroats” QA testers spent an amazing amount of time setting up a boat with three out of four players on the ship, and having two or more people all trying to jump on the boat at the same time to see what happened. If we had had a real server, the server would have let whomever’s message they received first get the last spot and rejected everyone else. But we cannot run code on the server. So the players have to fight it out to see who gets the spot. This is really complex. And add to that the fact that any or all of the players on the boat can leave the boat while the packets are all flying around the network to determine who gets the last spot.
Where are they?
One of the challenges Home faces is the system for knowing if a user has arrived or left the space. I can’t give you the specifics of why this is such an issue, but suffice it to say that it takes up a significant amount of allocated memory. It may seem simple to the consumer, but behind the scenes a lot of horsepower is diverted to it.
Here is another scenario from “Cutthroats” that will give you an idea of what I am talking about: A player calls a mutiny, then quits. Resolving this situation is not easy. If no one has voted, you can just cancel the mutiny. But what if the other players vote for the mutiny before receiving the message that the mutiny is cancelled? We actually had situations where the Captain was mutinied, the player who did it left before being moved to Captain, so no one was aware there was no Captain. If the player had become Captain, then left, everyone would have gotten a message saying the Captain had left. But since they never actually became Captain, no message.
Now you might say, “The Internet is pretty fast. What are the odds of this happening?” When 100,000 people are playing your game, it happens every day at least once. And that is just one of the many things that can happen in a game.
When this happens, it is possible for code to fall apart. We made an assumption that there would always be a Captain who was in charge of keeping track of boat damage. Well, now there is no Captain. People who played the game a lot might remember the invulnerable boat bug; this gives you some insight into what was behind it.
Without a server to be the benevolent dictator, it becomes a game by committee. So many unexpected things can happen in those situations — and you have to handle them all.
But You Said…
Where complexity and bug free software meet
Okay, now wait a minute. I started this article by saying that bugs and crashing are avoidable. Now I have just made the case that the complexity of the Home networking system makes unexpected situations almost impossible to cover. While it may seem that there is no common area in that Venn diagram, the common area is created by testing and effort. You can handle all complexity issues with enough effort and lots of good QA.
“Cutthroats” had outstanding QA, but the effort was not enough based on the complexity that the game acquired by the time it shipped. Towards the end, new features were being added at almost every meeting. You cannot deal with the complexity you have if it is increasing weekly. To get an idea of the effort it takes to add a feature, let’s assume you have a game with five features. You have to make sure the features all work together. So draw five dots on a piece of paper and draw a line from every point to every other point. While you have five features, you have ten lines. To add one more feature adds 5 more lines. So those simple last features that are added at the end are the most complex features to add.
One more dot equals lots more lines
So What Do You Do?
The solution is to keep the feature set low. Don’t add an extra feature that almost no one will use. When my partner (Mike Gehri) and I started Game Mechanics, we agreed to make simple furniture items so the complexity level was such that we could handle it. We had visions of making a giant lake space surrounded by buildings that you could go in and out of. The problem became that people would want to do things in the space and the complexity grew extremely fast.
So instead of a giant space with little to do, we decided to make the best apartment ever. I had worked on the first Mansion and had listened to all the feedback that people gave. In fact, it was my suggestion that led to people being able to sit at the dining table. Luckily the deciders thought it was a good idea and we did it. Which later led to accessible appliances in other spaces and more. But everything was simple and not interconnected. We added dots and stayed away from lines.
When you decide to do a project, you need to actually sit down and draw up this dots and lines chart to see how complex what you want to do is. The problem is that most developers try to draw that graph up in their head. I have to admit that when we did “Cutthroats”, the diagram was only in my head. So I knew, but could not explain why adding power ups was hard and a chat system was easy. Lesson learned: if you don’t want to be viewed as an obfuscating pain, map things out clearly for everyone to see.
The key is that developers need to do projects that are within their ability to test and fix. Once they get beyond that, you get games and spaces that are broken from day one. How could they let such obvious bugs out the door? Now you know.
Why does this happen?
Every project is under scrutiny. The goal of every project is to make money. Sometimes a project is released because it is done and sometimes it is released because it ran out of money. Most times it is easy to identify which projects are which. Some projects are being pushed out the door even though there are known issues. Anything come to mind?
It is worth the effort
Luckily for our customers, Game Mechanics is not trying to make money at this. We do this because we love it. So by the time our items get to QA, the most common bug is that we forgot to put our company name on the item. Rarely does anything come up more complex than that. We test like crazy. We have found a few issues after the item passed QA because we keep testing the items once they go live.
Most companies would go out of business if they had our business model. Let’s just say it works for us.
Personally, I think people should demand better quality for things they have spent real money on. So far we have been able to stay out of the Home Support forums. I would try to convince other developers to try and do the same. I bet that for many that is not a stated goal like it is for us. And if it is not a stated goal, it only happens by chance.
Can you break our items? Absolutely. The complexity of the environment versus the amount of revenue coming back means that no one, not even Game Mechanics, can afford to find and fix every bug. But I am arguing that I think developers can do better. It is my hope that by reading this, some of them are inspired to do the right thing. They need to be honest about the complexity versus effort equation in the design phase and resist feature creep until it ships.
Home, as a platform, has never been as robust as it is now. And it has far more tools to play with than it did in its infancy. This in turn raises consumer expectations, driving developers to create increasingly complex Rube Goldbergs to entice consumer wallets. But I firmly believe that it is ultimately better for user retention and increased monetization if some features are sacrificed in exchange for greater stability if there is insufficient time to thoroughly code and test those features. Perhaps, now that Home seems to have moved beyond its experiment of being a gaming platform, this philosophy can be revisited.
September 17th, 2013 by John C. Ardussi | 8 comments
John C. Ardussi (deuce_for2) is a developer for PS Home and other platforms. He recently started a new company, Game Mechanics who is now making items and games. Be sure and tell him what you think of what he is doing. He truly listens and adjusts based on input from the community.
LinkedInShare
| Tweet |
Excellent article John, I am glad you decided to write this and explain to people just how much it takes to do things in Home. It is easy to complain a lot when something goes wrong, but understanding what is involved gives one pause and makes you want to adopt a wait and see attitude instead.
Thanks for writing this in layman’s terms too for those of us who aren’t technologically adept.
Great great read. I know have a better understanding why having friends at my new place increase the issue with the built in teleport pad.
I always been the type to prefer simplicity and stability over complex and unstable. Thanks to your little graph I can explain why now.
Every one of your article are informative and goes beyond the simple sales pitch.You just prooved it’s possible to explain to customer what the issue are without giving any “industrial secrets”. Too often we are left in the dark like if this industry was rated “secret defence”
Most 3rd party rep do a good job… whithin the specification they are given. You visit us on the week-end! That’s going the extra miles.
Only one question. Is it possible to explain a little further the “client-client-server”? A for dummy version.
Most games that cost $60 or an MMORPG have servers that handle all transactions. When you press fire on your weapon, you don’t really fire until the server gets the message and decides what to do.
On Home game creators cannot run any code on the servers. While they are there, they blindly pass packets around without looking at the contents. They do facilitate some tasks like players entering the space and joining the game, but that is it.
So developing for Home is like being the chairs in a game of musical chairs. You have to make sure that is two people or more are trying to do a conflicting thing that you resolve it by negotiating between the clients.
Let’s say you have a chair and two people try and sit in it at one time. The server is no help. The clients have to send messages to each other and ask if the other intends to sit. Then they have to decide between them who wins.
Luckily, most of the time there is no opportunity for a conflict. But doing the right thing when there is a conflict separates the amateurs from the pros in my book.
Now I can also understand the “5 pilots crash” we had in Novus. Thankfully it was fixed promptly.
As for my teleport problem… I’ll have to ask an oracle I guess
Yet another outstanding behind the curtain peek John!
We have grown so accustomed to out of the box and endless updates and patches, not to mention plug-ins crashing our browsers that it is the new normal and like your Mac example, people just assume it’s going to happen.
Thanks for the article John
Interesting article. Thanks for giving us a developer’s perspective.
Speaking of Cutthroats…ahem…is it ever going to be fixed? Loved that game.
And could you write something about how users glitching in public and private spaces impacts server performance or otherwise adversely affects other users’ experience? I’ve heard conflicting information about that. Some glitches occur through no fault or deliberate action on the part of the user while other times the glitch is quite intentional.
Cutthroats is now Sony’s responsibility. I left around the time it shipped version 1.0.
I have no idea whether glitching effects server performance. That is back end code that we have no access to.
;^) Thanks for taking the time to write this. Insights into Home’s inner workings and limitations are very informative and always welcome.