Audience: Level Designers, UnrealScript Programmers, C++ Programmers.
Last Updated: 07/21/99
This document has been written with our licensees in mind (Microprose, Ion Storm, Legend, and others). However, much of the info is useful to the Unreal community, so I am releasing it here unedited. The sections that refer to C++ code are not relevent to the community because we aren't releasing the C++ source. But, there is still a lot of useful information here.
An Unreal "package" is a collection of related objects such as textures, sound effects, and scripts which are grouped together for convenient distribution and use.
For example, in Unreal, textures are stored in texture package files, which use the ".utx" extension. A .utx file contains many (tens or hundreds) of textures which are part of a similar theme, such as the "SkyCity.utx" texture package.
Unreal's file format is based on packages because packages enable the game engine and content to be developed modularly. Artists maintain their textures in .utx files; the programmers working on the engine and game maintain separate packages of UnrealScript classes; people in the Unreal community can create and distribute their own packages online; and so on.
One package may refer to objects which are contained in another package. For example, an .unr level package may refer to textures which reside in various external .utx files; sounds in other .uax files; and music in a .umx file. We say that the level file is "dependent" on the other packages. To load that .unr level file successfully, you need to have all of the other .utx, .uax, and .umx files which that level refers to. If one of those packages is missing, you will not be able to load the .unr file.
All Unreal package files are stored in the same binary file format, but they use different extensions to identify their type:
When the Unreal engine needs to load a package, it looks in the subdirectories listed in the [Core.System] section, in the Path#= list. The relevent lines of the clean (unmodified) Unreal.ini file look like this:
[Core.System] CachePath=..\Cache CacheExt=.uxx Paths[0]=..\System\*.u Paths[1]=..\Maps\*.unr Paths[2]=..\Textures\*.utx Paths[3]=..\Sounds\*.uax Paths[4]=..\Music\*.umx
When you try to load a package by its raw name (such as "SkyCity"), the engine first searches in the CachePath for a matching package downloaded from the Internet; then it searches in each of the numbered directories in order (0, 1, 2...) for matching files. For example, it looks for \Unreal\Cache\[some funky name].uxx, then \Unreal\System\SkyCity.u, then \Unreal\Maps\SkyCity.unr, then \Unreal\Textures\SkyCity.utx -- and the file \Unreal\Textures\SkyCity.utx does exist, so it uses that file.
Because of this storage and search scheme, you must not create two packages with the same base name, which only differ by extension. For example, you must not create a texture file named SkyCity.utx and a music file named SkyCity.umx. If you do that, the engine will always fail to load one of the packages and give an error.
All Unreal package files (.utx, .uax, .umx, .unr, and .u) use the exact same file format, and are processed in exactly the same way by the engine. The only reason we define different extensions is for convenience, and to interoperate with Windows cleanly. For example, .unr files are known to contain levels, so we associate the Unreal icon with them, and set them up so right-clicking on an .unr file in Windows launches Unreal. And, UnrealEd knows that .utx files contain textures. The meanings of these file extensions are purely superficial. They all use the same file format.
Whenever you save a package file to disk, the file is internally stamped with a GUID ("Globally Unique Identifier"), which is a 128-bit number which is guaranteed to be unique. In single-player play, this GUID is never used and is irrelevent. However, when you connect to an Internet-based server, the server sends a list of packages that are required for gameplay there (for example, the level it uses, any texture packages it uses, any sound packages, etc). These packages are identified by name and GUID. If you don't have one of the packages required by the server, the package file is automatically downloaded to your \Unreal\Cache directory, and given a filename based on its GUID.
A typical \Unreal\Cache directory looks something like this:
Directory of F:\Unreal\Cache 06/02/98 03:37a <DIR> . 06/02/98 03:37a <DIR> .. 06/02/98 03:37a 8,932 2306087A11D1E2FD4F00DBA5CAA00349.uxx 06/02/98 03:38a 8,932 2401042A11D1E2FD4F00DBA5CAA00381.uxx 3 File(s) 8,932 bytes 698,601,472 bytes free
Packages stored in the \Unreal\Cache directory are identified by their GUID rather than the package name given to them by their creator, to avoid conflicting names. For example, there are bound to be a ton of levels on the net called Test.unr, a ton of textures named Walls.utx, etc. The GUID scheme avoids conflicts.
In UnrealEd, each time a package file is saved, it is assigned a new GUID. This way, each version of a package is guaranteed to be tracked and cached separate from other versions. This is good, because it prevents people from trying to play on a server using different versions of a level. This is bad, because if you simply load a texture file, resave it, and copy it to your UnrealServer, all of the players who come to your server will have to re-download the package. Therefore, we recommend modifying your package files only when necessary, and releasing them only after enough testing and use that you know they're not going to change and force redownloading a day later -- especially with large package files.
| Core | The high-level framework Unreal is built on, similar to the MFC core or the Java Virtual Machine. The Core package contains the definitions of object, class, subsystem, script, property, name, and the object manager. The core is indepenent of any notion of a game or 3d engine. |
| Engine | The Unreal 3d engine, including support for vector math, actors, physics, collision, interaction, cameras, and networking. |
| Editor | The UnrealEd 3d editing environment with geometry editing, realtime lighting, and actor editing support. |
| Render | The Unreal rendering engine implementation. |
| Window | The platform-specific code supporting Unreal. |
| Galaxy | The implementation of Unreal's sound system (UAudioSubsystem) running on top of the Galaxy sound system for Windows. It is possible (for example, when porting Unreal to another platform) to completely replace the Galaxy package with an implementation of another sound system. |
| GlideDrv | Unreal 3d hardware support code for 3dfx's Glide library. |
| UnrealI | The Unreal I game implementation. |
| WinDrv | Windows viewport and client driver. |
| SoftDrv | Software rendering driver. |
| Fire | Fractal fire and water effects. |
| IpDrv | TCP/IP networking driver. |
Licensees may use all our .u packages except UnrealI verbatim or you may modify them to fit your needs. Bear in mind that it is attractive to use them in an unmodified form, as that makes merging with future code updates much easier and makes it easy to isolate problems in each others' code.
Licensees may use the UnrealI package as a reference and for internal development use, but the content in UnrealI is specific to our particular game, and it mustn't be used in your final product.
Licensees should create one more more new packages, for example "WheelOfTime" for their game-specific content. Keep in mind that your package and UnrealI can exist side-by-side during your development cycle until you've fully migrated away from the UnrealI specific content.
Here is an example of the directory structure for all the files that are related to the "Engine" package. The other packages follow a similar structure.
Distributable files:
| \Unreal\System\Engine.dll | The package's C++ DLL file. |
| \Unreal\System\Engine.u | The package's UnrealScript class file. |
Developer files:
| \Unreal\Engine\Inc\Engine.h | Main public header file for the engine package. |
| \Unreal\Engine\Inc\*.* | Various public header files for the engine package, included by Engine.h. |
| \Unreal\Engine\Src\EnginePrivate.h | Private implementation header file. |
| \Unreal\Engine\Src\EngineClasses.h | Scripted classes exported to C++. |
| \Unreal\Engine\Src\Engine.cpp | Generates the package's precompiled header file. |
| \Unreal\Engine\Src\*.* | Various C++ files comprising the engine's code. |
| \Unreal\Engine\Lib\Engine.lib | The export library for linking with the engine. |
| \Unreal\Engine\Classes\*.uc | The UnrealScript classes used in building Engine.u. |
| \Unreal\Engine\Textures\*.* | Graphics files used in building Engine.u. |
| \Unreal\Engine\Models\*.* | Model files used in building Engine.u. |
| \Unreal\Engine\Sounds\*.* | Sound files used in building Engine.u. |
When you declare an UnrealScript class as "intrinsic", the engine looks in the .DLL file corresponding to the class's package for the class's corresponding C++ code. For example, here is the declaration of the "PlayerPawn" class in the "Engine" package:
//============================================= // PlayerPawn. //============================================= class PlayerPawn expands Pawn intrinsic;
(You can find the above code in the file \Unreal\Engine\Classes\PlayerPawn.uc.)
When you declare a class as "intrinsic" in UnrealScript, you are telling the engine that you have created some corresponding C++ code for the class. In this way, you can mix and match UnrealScript and C++ code when you write your game code.
When you declare "intrinsic" classes, Unreal looks for the .DLL file in the \Unreal\System directory whose name matches your package's name. For example, to find the C++ implementation of the PlayerPawn class, Unreal looks in the "\Unreal\System\Engine.dll" file.
In C++, you must use the IMPLEMENT_INTRINSIC macro (defined in UnObjBas.h) to tell the engine that you have implemented a new class in C++. Internally, this macro generates some __declspec(DLL_EXPORT) declarations that Unreal looks for in your DLL. For example, the PlayerPawn class is exposed in the source file "\Unreal\Engine\UnPawn.cpp" as follows:
IMPLEMENT_CLASS(APlayerPawn);
In addition, once per package you need to use Unreal's package declaration macro to expose the package:
IMPLEMENT_Package(Engine);
So, to summarize:
You can start working with the "baseline" version of the Unreal engine (that is, the engine with all game-specific content removed), with:
unreal.exe level.unr?class=engine.playerpawn?class=engine.gameinfo
While you're playing with the baseline configuration, you won't be able to see the player model and you won't be able to use any weapons because, of course, all of those things are game-specific and stored in the UnrealI package. But, the point is, you can edit and play levels with only the baseline engine, without any of our game-specific content.
See below for a description of what the Unreal.ini [EditPackages], [DefaultGame], and [DefaultPlayer] settings mean.
The first question is, "How do I create my own game package like your UnrealI.u?"
The easiest way to create a new package is by creating your first new class within UnrealEd. When creating a new game using the Unreal technology, you will need to create a custom "GameInfo" class to define high-level gameplay rules. Here's how:
The next question is, "How do I have my new package used in the editor and during gameplay?" The Unreal.ini file defines many of these project-specific settings you will want to modify for your title:
The most important thing you need to realize while working with classes is that each class may be stored redundently in two separate places:
There are two ways you can create and edit UnrealScript classes:
To summarize:
You can rebuild individual .u files, or all of the .u files, from the command line as follows:
When Unreal rebuilds a package, it switches into the \Unreal\packagename\Classes directory (for example, if you're rebuilding the Engine package, it switches into \Unreal\Engine\Classes), and it imports all of the *.uc files it finds there.
The .uc files are text files containing UnrealScript code. There is one .uc file for each class. The .uc filename must correspond to the class name. For example, the PlayerPawn class must reside in the file "PlayerPawn.uc".
The .uc files may contain special UnrealEd commands which begin with the keyword "#exec". These "#exec" commands import various objects as the script is compiled. For example, #exec commands exist to import textures, sounds, meshes, and music. The "#exec" commands are not currently documented; you will need to see the example .uc files in the UnrealI package for examples of usage.
Rebuilding certain packages, such as the Engine package and the UnrealI package, requires that you have the source artwork, sounds, and meshes, which are available on the UnEdit ftp site as a separate download. The source data is not included in the regular source distribution because of its huge size. The source artwork can be extracted with WinZip into the various subdirectories of the Unreal\Engine\ and \Unreal\UnrealI\ directories (for example, the engine sounds go into \Unreal\Engine\Sounds).
When cleanly rebuilding, any UnrealScript compiler errors cause the rebuild to halt, and an appropriate error message is displayed on the screen.
To summarize:
When Unreal rebuilds your package using "Unreal -make", it tries to load a text PackageName.upkg definition file from the same \Unreal\PackageName\Classes directory. If not found, the default package options are used. A sample package definition file looks like this:
[Flags] AllowDownload=False ClientOptional=False ServerSideOnly=True
The options that appear in a package definition file are:
If you create any intrinsic UnrealScript classes, or if you need to access any UnrealScript classes from C++ code, then you can have Unreal generate a C++ header file which mirrors the classes you've defined in UnrealScript.
For example, the Actor class any many of its child classes are defined by scripts in the Engine package (such as \Unreal\Engine\Classes\Actor.uc). The C++ code in the engine often needs to access objects in the Actor class. Early in development, we manually created C++ definitions which mirrored all of the UnrealScript classes. This process became tedious and error-prone as the number of classes increased. So, we added an option to Unreal to automatically generate a C++ header file, based on your UnrealScript definitions.
For an example of an automatically-generated C++ header file, check out "\Unreal\Engine\Inc\EngineClasses.h". This file was not written by a human. Unreal generated it.
To automatically generate a C++ header for a package, do the following:
Because there are many ways which UnrealScript and C++ can share information back and forth, the automatically-generated header file (for example EngineClasses.h) contains a variety of definitions:
For a more concrete example, carefully examine the UnrealScript file "\Unreal\Engine\Classes\TcpLink.uc" and the following definitions in the file \Unreal\Engine\Inc\EngineClasses.h": enum ETcpLinkState, enum ETcpMode, and class ATcpLink. This provides an example of a fairly isolated UnrealScript class which defines several enumerations and intrinsic functions.
Finally, to avoid a chicken-and-egg scenario, you should create new classes in the following order: first create a class in UnrealScript, then generate the C++ header, then recompile the engine code (adding in the IMPLEMENT_CLASS macro for any new intrinsic classes you've added).
To summarize the important points here:
For programmers who like to keep their work archived under Microsoft Visual SourceSafe, here are some tips. We use SourceSafe internally for our development, though it is purely optional -- nothing in the Unreal source code assumes that you have SourceSafe:
When you add or remove variables in an UnrealScript class, that changes the binary layout of objects belonging to that class. For example, if you have a script that contains some variable declarations like this:
class TestClass expands Actor; var int i, j; var texture t;
And you remove one of the variables like this:
class TestClass expands Actor; var int i; var texture t;
Then any objects in memory belonging to TestClass will become corrupted as soon as you recompile the script. The solution is to only add or remove variables from a script in an empty level. However, empty levels contain an actor belonging to the LevelInfo class, as well as one or more Camera actors, so it's never safe to add or remove variables from any of the following classes in UnrealEd: Actor, Info, LevelInfo, Camera, PlayerPawn, Pawn. To work with these files, edit their .uc files on-disk, and do a clean rebuild.
Also, bear in mind that Unreal makes heavy use of the Actor, Pawn, PlayerPawn, LevelInfo, and other intrinsic classes in C++ code. So, adding and removing variables to any intrinsic classes will cause the C++ code and UnrealScript code to disagree about the binary layout of classes. Whenever you change an intrinsic class, the safe thing to do is a clean rebuild with the "Unreal -make -h" option to generate new C++ header files for your classes, then recompile the Unreal C++ source.
The Unreal package file format is only useful to people who want to write external utilities that read and write Unreal files. Level designers and UnrealScript programmers don't need to know this info. But, the Unreal package file format is quite complex compared to what Quake, Quake 2, and Jedi Knight utility writers are accustomed to. It's so complex, in fact, that I doubt many utility writers are going to want to deal with it directly. It is complex for several reasons:
The coming release of the enhanced version of UnrealEd in 3+ months will include a very modular plug-in interface for creating built-in UnrealEd tools, file importing tools, and file exporting tools, which will operate directly on the engine's C++ data. That release will greatly simplify the task of programmers who want to write Unreal utilities, because it eliminates the need to read and write package files directly. Rather, with the UnrealEd plug-in interface, you will be able to use C++ to create all of your tools and compile them into DLL's. You'll make calls to the Unreal engine to allocate objects, save package files, load package files, etc. with the hairy file-format issues taken care of by the engine.
Each object and name stored in a package file can contain any combination of the following bitflags:
Any other bitflags stored in package files are irrelevent, should be set to zero when saving a package file, and should be ignored when loading a package file.
Compact indices exist so that small numbers can be stored efficiently. An index named "Index" is stored as a series of 1-5 consecutive bytes with the following C++ code. Basically, the "Ar << B0" type code serializes the byte stored in the variable B0. Serialize can mean read or write, depending on the internal implementation of the archive object Ar.
//
// FCompactIndex serializer.
//
FArchive& operator<<( FArchive& Ar, FCompactIndex& I )
{
INT Original = I.Value;
DWORD V = Abs(I.Value);
BYTE B0 = ((I.Value>=0) ? 0 : 0x80) + ((V < 0x40) ? V : ((V & 0x3f)+0x40));
I.Value = 0;
Ar << B0;
if( B0 & 0x40 )
{
V >>= 6;
BYTE B1 = (V < 0x80) ? V : ((V & 0x7f)+0x80);
Ar << B1;
if( B1 & 0x80 )
{
V >>= 7;
BYTE B2 = (V < 0x80) ? V : ((V & 0x7f)+0x80);
Ar << B2;
if( B2 & 0x80 )
{
V >>= 7;
BYTE B3 = (V < 0x80) ? V : ((V & 0x7f)+0x80);
Ar << B3;
if( B3 & 0x80 )
{
V >>= 7;
BYTE B4 = V;
Ar << B4;
I.Value = B4;
}
I.Value = (I.Value << 7) + (B3 & 0x7f);
}
I.Value = (I.Value << 7) + (B2 & 0x7f);
}
I.Value = (I.Value << 7) + (B1 & 0x7f);
}
I.Value = (I.Value << 6) + (B0 & 0x3f);
if( B0 & 0x80 )
I.Value = -I.Value;
if( Ar.IsSaving() && I.Value!=Original )
appErrorf("Mismatch: %08X %08X",I.Value,Original);
}
return Ar;
}
Names are stored a compact indices with values >= 0 which index into the file's name table.
Objects are stored as compact indices with values defined as follows:
The following tables reside in the file at the offset specified by the header. If a table contains zero entries (as specified in the header), the offset value is meaningless and should be ignored.
Contains a list of human-readable Unreal names (which correspond to the UnrealScript "name" datatype and the C++ "FName" data type). Each name is stored consecutively in the following format:
Contains a list of objects contained in (a.k.a. "exported by") this file. Similar to a Windows DLL's export table.
Contains a list of objects in other packages which this packages refers to. Similar to a Windows DLL's import table.
Contains a list of GUID's which this package is compatible with. In the current version of Unreal, this table always contains exactly one GUID, because the backwards-compatible package management code has not yet been written. In the future, it may contain more than one GUID. Each entry contains:
Extremely complex. Different for each class. Hard to read. Not yet documented.