In embedded systems, every cycle counts. It would seem that in this day and age of high performance mobile devices, there would be no need to concern ourselves with every cycle, like we did as little as a decade ago. Yet there are still many situations where this is the case. Even many of today’s games and applications could be improved significantly with some tweaks to the underpinnings of their codebase.

For example, the often overlooked thread system. Most of the time, making your application threadable is an instant win. Suddenly you can make use of the often easily available multiple core devices that are available. However, it is well known that preemptive multitasking does come at a certain cost. There is some overhead for the processor to decide when to swap tasks, as well as a cost for the actual swap itself. Some of this cannot be avoided, but much of this can be considerably optimized, with very little additional work.

Consider cooperative multitasking. Not at all a new concept, and it has drawbacks to be sure. But it is worth considering when you are writing a time-critical codebase that is limited to a known number of cores at compile time. Rather than forcing the OS to guess at what is the most important code for you, managing it yourself allows you the freedom to use every cycle exactly how you see fit.

Let us assume your game has some logic modules that can largely be independent of each other. They are able to read from each other’s state, but there is no need to write to each other. For argument’s sake, we will use the following modules: player (P), enemy (A), enemy (B), and rendering (R). We will add overhead (O) for the threading needs.

In a preemtive threading model, on a single processor, time gets sliced between all of the modules in a fairly haphazard way.

Frame Start                                                               Frame End
OPPPPPOOOAAAAOOBBBBOOAAAAAAOBBBBBOOPPPPPOOAAAAOOPPPPOOBBBBOORRRRRRRRROOPPOORRRRRRRR

Obviously this haphazardness can be reduced by using mutexes to indicate that certain things need to be complete before other things, but that still ends up creating more overhead.

On the other hand, if you were to use cooperative multitasking, you can optimize the overhead.

Frame Start                                                               Frame End
OPPPPPPPPPPPPPPPPOAAAAAAAAAAAAAAOOBBBBBBBBBBBBBOORRRRRRRRRRRRRRRRROO---------------

Look at all that time you’re saving, simply by reducing the overhead. All those -‘s of extra time that you could use to support more objects onscreen at once.

Now consider how much simpler your code can be, knowing that you no longer have to manage mutexes to keep your data in sync across modules. This leads to simpler code, often leading to less potential bugs.

What’s even more exciting is because scheduling is now under your control, you can support features that are difficult to express in current operating system threading environments. You can prioritize certain modules, or even delay other modules to be run every other frame so that you can effectively extend the available frame time.

Actually implementing a cooperative system is fairly easy. Almost all platforms come with the necessary stdlib calls to save and restore processor register state. Even without, setting up caller expectations around those scheduler calls would be simple to do.

There’s nothing stopping you from supporting both styles of thread management in the same codebase. You simply design your modules with this expectation, calling a method to let the scheduler know that you are at a safe point to be interrupted. Wrap the mutex and thread calls in your own wrapper. On systems that work better with threads, they simply pass through to the OS. On systems where you only run on a single thread, they translate to No-ops and don’t affect anything at all.

Just because you are on a machine with multiple cores does not automatically mean you are allowed to use them all. Acting in a single threaded way in such a machine may actually turn out to be more efficient than trying to be multithread aware.

Incidentally, John Carmack certainly considers this model. Rather than splitting everything up for any core possible, the game code is split across rendering vs. game logic. This separates and simplifies both systems considerably in the Doom 3 engine.

Many ideas from the past are still just as relevant today as they once were. It would be a mistake to underestimate cooperative multithreading environments just because we have all these cores available.

Office Space

In this day and age, making games is more accessible than ever. Top notch tools are available cheaply (or free!) to just about anyone who wants them. That said, it takes significantly more than just tools to make great games. It takes talent, money, and time. What we don’t need is offices. Even big AAA game studios could improve their bottom line with this concept. Here is a proposition for why game builders, big and small, should seriously consider making games with remote workers as a deliberate choice.

The most obvious difference is cost. When you are already paying for a safe place to sleep, why let that space go to waste while you are away at work eight or nine hours a day? While it is still true that a dedicated space for work is required, the additional square footage cost is far below the cost of a completely separate space. This also takes into account the shared space of multiple people in one office.

Even the concept of a typical modern office is difficult to understand for a creative person. At some point in the the last decade someone suggested that the “open office” plan promoted collaboration. While us game makers are all about trying new ideas, it has been proven through multiple studies that open offices only end up creating more distraction. While distraction can be an issue at home as well, there are ways to minimize it. Home offices are not a good strategy for parents of young children, for example. However, once they are school-age you will have a house that is quiet and filled with the love of your family, ready for creativity to flow.

This suggests another point about our office environment. At places like IBM in the late 70’s, They had a job to do, and wanted to look good doing it. So it was no concern at all to put on a suit and a tie. Therefore the dress code was less about a form of control by the corporation, and more of a concession to what the people working in that company wanted. Why should that be any different today? And this extends to decorating our working space to any degree we like, without fear of offending anyone.

Game developers typically have two modes: they are at the keyboard making stuff happen, or they are thinking about what they will do the next time they are at the keyboard. Working at home allows us to make things happen when we think of them, not have to wait until the next morning. By then, we’ve already forgotten the idea!

This is one of the many myths about working at home: that it reduces productivity. It is true that it takes a certain kind of mindset to not take advantage of the situation. But one of the things that defines us as developers is that this is what we want to be doing, more than anything else. This isn’t “work” to us. It is freedom, a kind that is so much more rewarding when you get to escape to that place any time we like.

Setec Astronomy

In the interest of full disclosure, let me tell you a little about myself. In the five years I’ve worked for my last company, I have walked into the office twice. Technically I was based out of the San Francisco office, but I live and work from my home in Kansas City, Missouri. Before that, I worked remotely for people all over the world including New York and London since 2003.

For most of my last project, I was the sole programmer creating the entire user interface experience outside of the core gameplay itself. Clearly I was a necessary and valued member of the team. Unfortunately me and the other two remote workers in the company in other cities were recently laid off. At the time of the layoff, all of us were working on critical path features of a newly released game. Less than a month ago our company entered into an agreement with a company well-known for extreme secrecy.

It is my opinion that this is a direct result of one of the myths of remote workers: that they are more likely to reveal company or investor secrets, that we are somehow less secure because our computers are allowed to leave the office for extended periods of time. Yet the three of us worked for the last 12 months on a project owned by very big name intellectual property owners, without a single incident of leaked information or stolen equipment.

It could be argued that houses are more secure than an office. Every person in a household knows every other person, intimately. Every unusual face sticks out like a sore thumb. In an office’s business hours, who can pay attention to all the people who must come and go in a normal day? Sure, you could hire guards or install cameras. But all of that is quite expensive compared to the built-in security of a household. And if something ever does happen, you’ve lost only one machine, instead of having twenty machines available to a thief all in one place.

Make it Work

For those of you who have never attempted such an adventure, here are a few easy guidelines that have proven extremely useful over the years.

Treat every email, every instant message, every phone call as though you were standing in front of that person. Be respectful, don’t interrupt, and remember that this isn’t like the internet: these people do know who you are.

Team gatherings are just as important for remote workers as it is in an office. When starting a major milestone on a project, we gather the team to hash things out. Start projects by bringing everyone to a central location for a week to get as much on paper as you can. That way as we follow up with weekly online meetings there is a basis of understanding. Not to mention it breeds the camaraderie needed to get through the hard times.

Video conferencing is not as necessary as it used to be, with the advent of things like Google’s online presentation and document editing software. It’s not perfect, but it is a huge step in the right direction. On the other hand, quality audio conferencing is a must. Do not underestimate the importance of being able to interrupt someone to make a supporting comment. Unfortunately there is not much good software that can both keep a solid connection and also allow people to interrupt each other. Whatever you do, use headsets. The built-in computer microphones and speakers add a lot of unnecessary noise.

Have work accounts for email and instant messengers separate from personal ones. This way you can “tune out” work or personal life at appropriate times. A major point is reducing distractions, and having your inbox ping at you 20 times a day is the opposite of intended.

You might think there is more to it, but in truth there is not. It is as simple as ??. Therefore I bid you great game-making!

 

Gamasutra has been gracious enough to publish this blog entry as well! I’m so stoked!

10:43 AM mr.ed Hey guys! How you doing?
4:39 PM mr.ed Hey! Are you guys using this thing? :) #general
4:40 PM dan Just saw this, sorry. This afternoon has been crazy.

 

This happens every time I start with a new team. A communication tool is only useful if it gets a response from the other end. It takes some effort on the part of the main office, not only by the remote worker. And every team I’ve had the pleasure of working with have used this simple set of rules.

Act as though the person you are talking to is right in front of you. Do you like it when the person you are talking to constantly turns away to answer the phone, or get interrupted by other people? No, you don’t. So when you are talking to someone online, don’t just step away and leave the remote person hanging. Either ignore the interruption, or at least excuse yourself from the remote person. That way they know that your attention is elsewhere. If you try to do both, it becomes very obvious that you are not paying attention.

Likewise, don’t say anything you would not say to their face. Anonymity does not apply here – these are people that you work with, respect, and will remember how you act. That said, this is a rule that should really apply to any online interaction.

 

There are three main channels of communication available to remote workers. If everyone on the team has the same expectations of the methods, then all requests will be handled in timely manners.

The first channel you might think of is instant messaging. This is an extremely useful tool. However, it’s often overused. It is perfect for having ongoing conversations. It is also an interruption. If the servers are on fire, or someone is stuck and can’t get their job done, the an interruption is completely appropriate.

The rest of the time, email is the proper channel to start new conversations in. Email can be ignored until there is a break in your current task. There is no expectation of an immediate response. But also don’t let the email thread get twenty replies. If a conversation needs more than a single response, it makes sense to set a time for an instant message conversation to follow up with and hash everything out.

Which brings us to the third channel: phone. Nothing will replace the ability to talk out issues. It is often much faster and easier to understand nuance than instant messaging. A particularly powerful addition is using instant messenger at the same time, to post links and references to the other participants during the conversation.

Some additional notes about phones: high quality is a must. Do not use computer built in speakers and microphones, always use a headset or headphones with built in microphones. Computer speakers often create terrible distortion. Also, do not underestimate the importance of being able to interrupt another person, to provide a supportive comment. Many phone systems only allow the loudest participant to be heard, meaning no one can interrupt.

 

These guidelines have served me well for many years. May they also serve you well.