Unity3d Asset Bundle Data Sharing

This article is part of a series. If you’d like to skip ahead, you can go to the master post that links to them all.
 

Last week on our asset bundle journey we got started converting our old asset bundle system to the new Unity 5 one. It was actually really simple. But there is still a big problem that needs to be solved.

When you build bundles, you mark each asset that the game is going to ask for when you run the game. When you flag a single texture, it works just fine. But what about shared assets? For example a prefab holding a sprite that uses a texture atlas. Unity will gladly pick up the texture for the atlas and add it to the bundle for you. No problem there. But what happens when you add a different prefab in another bundle that wants the same atlas?

That’s when things get confusing. Will unity add the texture to both bundles? Yes. Uh-oh. Then you have wasted some memory on a duplicate texture. Worse, Unity seems to get confused by loading the same texture twice! No, Unity does not figure out that it is the same texture for you, loading it only once. Bad times. But I thought Unity handles dependencies for you in the new system? It does, but only considers each bundle separately. It does not figure out sharing data to other bundles.

But you can do something about it. All you have to do is mark the “shared” atlas as another asset bundle. In our example, the first prefab goes in bundle A, the second prefab in B. Even though the game does not ask directly for it, we set the atlas to be in new bundle C. Unity will then properly take over, making sure to load only one copy of the atlas, and only unload it after both prefab bundles are unloaded.

The more I think about it, the more it feels like there should be some automatic way of dealing with this resource sharing. Both in new projects and legacy ones that we are upgrading. However, this is something Unity does not provide. Why not? It’s easy enough to determine what assets are shared and used by what other assets. The catch is, what’s the most efficient way to assign which asset to which bundle? That will be different for every project.

That said, you can still write some tools that help in the process. For example, something that examines your bundle assignments and works out what should be shared. It could even go so far as to suggest what should share to what.

So there you go! That is how to deal with shared asset bundle data. Join us next week as we explore a useful ability that asset bundles make super simple: automatic texture scaling.

Upgrading to Unity3d 5 Asset Bundles

This article is part of a series. If you’d like to skip ahead, you can go to the master post that links to them all.
 

In my first installment of this asset bundle series, let’s talk about upgrading an older project to Unity 5 bundles.

On this latest project I’ve been working on with ClutchPlay Games, I was asked to help straighten out a project that was based on the pre-Unity 5 bundle system. We did a lot to help it. First we added the Android GPU class support first, making sure everything worked. But our build times were taking forever, even with code that checksummed the source assets and only building what changed. So I decided to solve it with a crazy idea: let’s move this project up to Unity 5’s new bundle system.

In Unity 5’s asset bundles, they take care of tracking all the source asset changes for you. It has many upsides: it’s really fast to make small changes. It’s super easy to add assets to any given bundle with the new UI. Building them is a single call, and it works reliably. The sample code for loading the bundles is clean and useful, took less than a day to integrate, and only a week to make it solid. If you’re starting a new project, I highly recommend adding support for this early, it will definitely make certain distribution problems very easy.

But what is the value added for projects that still use the older system? On the surface they look completely incompatible. Assets were always added from code, which is perfect for build automation, very wise in large projects. You don’t want to manually set every asset’s bundle identifier through the UI. That alone would simply take too long. Then you notice the new BuildAssetBundles call that takes the AssetBundleBuild objects. Surely this is how.

But lo! ‘Tis not true. It’s simply not well documented. You can have the best of both worlds. Simply replace your old calls to BuildPipeline.BuildAssetBundle() with a method that changes the asset bundle assignments for each asset that you would pass to BuildAssetBundle. How, you say? Simple: it’s now part of the AssetImporter interface.

AssetDatabase.StartAssetEditing();

for ( string assetPath in desiredAssets ) {
	var importer = AssetImporter.GetAtPath(assetPath);
	// note that you must set the bundle name before the variant.
	// If you want to clear the bundle name, set it to null - it will clear variant for you.
	if (importer.assetBundleName != bundleName)
		importer.assetBundleName = bundleName;
	if (importer.assetBundleVariant != variant && bundleName != null)
		importer.assetBundleVariant = variant;
}

// this might be a little extreme but we've had issues with old versions being used in builds
AssetDatabase.StopAssetEditing();
AssetDatabase.SaveAssets();
EditorApplication.SaveAssets();
AssetDatabase.Refresh(ImportAssetOptions.ForceSynchronousImport);

At that point, everything will be assigned to the bundle names you want. As a separate step you then actually build the bundles through the simplified BuildPipline.BuildAssetBundles(). As a bonus, you can then examine and tweak the bundle assignments through the Editor UI!
 

Not all is wine and roses, however. There are a few not-so-obvious rules with the names you use for the bundles.

  • Periods confuse the loading system for some reason. The asset extension is ignored during loading, but extra periods in the asset filename or the bundle names confuse things.
  • Underscores have special meaning in bundle names. It’s safest to avoid using them. Likewise, be sure not to use them in variant names.
  • Bundle names will always be stored lowercase. This prevents case confusion, and is not a bug.
  • Forward slashes create a “group” in the bundle name selection popup. This is highly useful. Be careful not to accidentally put them right together, like “textures//thing2”. It breaks the menu.

There we are! With the new bundle system in place, our build times are down to a few minutes instead of an hour. With this new design, you can do a pass with the bundle assignments, and then edit a specific asset many times, doing only the bundle update over and over again. Because Unity tracks it for you, each bundle update takes only seconds.

There is one more major piece to upgrading an old project. Join us next week as we work out how to handle shared assets!

Unity3d Asset Bundle Series

Welcome back! I’ve been neck-deep in Unity3d asset bundles for the last few months. They are potentially so super useful, and yet historically they are difficult to make and use. We used to use them to get around the download limits imposed on mobile app downloads. These days you can split the application binary to get a similar effect. Even then, it’s a tricky proposition. But with an asset bundles, you can change your live game just by telling the client to download a new data set. Did you know you can build different asset bundle sets for each of the Android GPU classes? The client would never have to know anything was different, and magically you have the most efficient texture format possible for each device, with just a little logic to load the right bundle directory file.

I used to use the pre-Unity 5 bundle system, with full automated dependency tracking. Even created a general purpose system where you simply add what you wanted and it figured out the details for you. Took over a year to build and test. Did get used in a shipping game. Right about then Unity came out with the new fangled Unity 5, with built-in simplified asset bundle support!

So stay tuned. Over the next few weeks I’m going to tell you all kinds of useful stuff about how to work with asset bundles. First one, next week, will tell the tale of upgrading a project to the new Unity 5 bundle system. See you there!

———–==========———–
The story thus far….

ModernCoalition

logo

  • Product description: Free-to-play “Group Decision” game for iOS and Android
  • Release Date: December 2015
  • Website: ModernCoalition
  • Product history/genesis: A startup focused on improving voter engagement
  • Team: Small group, ten total
  • Core Technologies: C# in Unity3D with a .Net based server
  • Most proud of: Implemented majority of project in roughly two months
  • Responsibilities: Client and Server coding

In March 2015 I was laid off from DeNA, just a few days after releasing Marvel Mighty Heroes. This was the end of an era for me, over a decade. I worked my way up from an independent developer porting Windows games to Macintosh, to writing next-gen cross platform games on mobile.

During the previous era, each opportunity was offered to me, without really having to “reach” for them. But this layoff was a hard stop. I had no idea what I was going to do next. I really thought I was out of the game development business. All the work I’d done was remotely for other companies. Where I live, there aren’t any game companies. And, other than the people I worked with, nobody had any idea I existed. I liked to joke “I’m the best game coder nobody’s ever heard of.”

I don’t recall how exactly this startup had heard of me, but we connected just before my layoff, asking only for counseling. They were not a gaming company in a classic sense, but one that was working on solving a problem by making a game. The idea was to make voting on issues into a game. One that you had to earn the right to vote for by playing minigames.

After a few months it became clear that I wasn’t finding reasonable options for non-game work where I live, so I offered my coding services to this startup in hopes that it would lead somewhere. The project itself wasn’t particularly complicated, but I did invest in learning enough server side .net to make the gameserver with it.

Eventually we hit a wall with funding, and it was either “ship it or close shop.” With so much of my own time and money invested, I didn’t want to let it go without a fight. In two months I tightened up what we had and shipped a working game to iOS and Android app stores.

In the end we couldn’t get enough players in the game to make it worth continuing, shutting the servers down a mere three months later. To this day I believe that with the right team this project could soar. Until then, adieu.

Threading in Real-time Systems

In embedded systems, every cycle counts. It would seem that in this day and age of high performance mobile devices, there would be no need to concern ourselves with every cycle, like we did as little as a decade ago. Yet there are still many situations where this is the case. Even many of today’s games and applications could be improved significantly with some tweaks to the underpinnings of their codebase.

For example, the often overlooked thread system. Most of the time, making your application threadable is an instant win. Suddenly you can make use of the often easily available multiple core devices that are available. However, it is well known that preemptive multitasking does come at a certain cost. There is some overhead for the processor to decide when to swap tasks, as well as a cost for the actual swap itself. Some of this cannot be avoided, but much of this can be considerably optimized, with very little additional work.

Consider cooperative multitasking. Not at all a new concept, and it has drawbacks to be sure. But it is worth considering when you are writing a time-critical codebase that is limited to a known number of cores at compile time. Rather than forcing the OS to guess at what is the most important code for you, managing it yourself allows you the freedom to use every cycle exactly how you see fit.

Let us assume your game has some logic modules that can largely be independent of each other. They are able to read from each other’s state, but there is no need to write to each other. For argument’s sake, we will use the following modules: player (P), enemy (A), enemy (B), and rendering (R). We will add overhead (O) for the threading needs.

In a preemtive threading model, on a single processor, time gets sliced between all of the modules in a fairly haphazard way.

Frame Start                                                               Frame End
OPPPPPOOOAAAAOOBBBBOOAAAAAAOBBBBBOOPPPPPOOAAAAOOPPPPOOBBBBOORRRRRRRRROOPPOORRRRRRRR

Obviously this haphazardness can be reduced by using mutexes to indicate that certain things need to be complete before other things, but that still ends up creating more overhead.

On the other hand, if you were to use cooperative multitasking, you can optimize the overhead.

Frame Start                                                               Frame End
OPPPPPPPPPPPPPPPPOAAAAAAAAAAAAAAOOBBBBBBBBBBBBBOORRRRRRRRRRRRRRRRROO---------------

Look at all that time you’re saving, simply by reducing the overhead. All those -‘s of extra time that you could use to support more objects onscreen at once.

Now consider how much simpler your code can be, knowing that you no longer have to manage mutexes to keep your data in sync across modules. This leads to simpler code, often leading to less potential bugs.

What’s even more exciting is because scheduling is now under your control, you can support features that are difficult to express in current operating system threading environments. You can prioritize certain modules, or even delay other modules to be run every other frame so that you can effectively extend the available frame time.

Actually implementing a cooperative system is fairly easy. Almost all platforms come with the necessary stdlib calls to save and restore processor register state. Even without, setting up caller expectations around those scheduler calls would be simple to do.

There’s nothing stopping you from supporting both styles of thread management in the same codebase. You simply design your modules with this expectation, calling a method to let the scheduler know that you are at a safe point to be interrupted. Wrap the mutex and thread calls in your own wrapper. On systems that work better with threads, they simply pass through to the OS. On systems where you only run on a single thread, they translate to No-ops and don’t affect anything at all.

Just because you are on a machine with multiple cores does not automatically mean you are allowed to use them all. Acting in a single threaded way in such a machine may actually turn out to be more efficient than trying to be multithread aware.

Incidentally, John Carmack certainly considers this model. Rather than splitting everything up for any core possible, the game code is split across rendering vs. game logic. This separates and simplifies both systems considerably in the Doom 3 engine.

Many ideas from the past are still just as relevant today as they once were. It would be a mistake to underestimate cooperative multithreading environments just because we have all these cores available.