Building System For Large C#/XNA Projects


This Post will talk about our custom Building System used to create the PloobsEngine MSI Package. Lots of time ago, when the engine code and the project number were smaller, we could afford to build our packages by hand, but when things got bigger and bigger the necessity for something fully automatizated became obvious.

The main funcionalities of our Building System are:

  1. Download Latest/Specific version of your repo (Mercurial)
  2. Compile all Projects, Tests and Demos
  3. Run Some Tests
  4. Compile Documentation
  5. Create and compress the Package (generate the Msi file)
  6. Upload to a Server (ftp)
  7. Notify People (send emails)

Here we will describe with details how each part was implemented, showing some parts of the used scripts.  (Everything is hosted in a Windows 7 machine).

Download Latest/Specific version of your repo.
As said, We used Mercurial for source code management. To download the repo and update to the latest version we used the following script.

set RepoPath=c:/PATH_WHERE_WE_WILL_DOWNLOAD_OUR_REPO
@echo off
echo Downloading Ploobs HG Repo
cd /
if exist %RepoPath%/port-ploobsengine   goto NEXT
md %RepoPath%\
cd %RepoPath%\
call hg clone https://thiagodiaspastor@code.google.com/p/port-ploobsengine/
if %errorlevel% 1 (
   echo Failure Reason Given is %errorlevel%
   chdir /d %OLDDIR%
   exit /b %errorlevel%
)
cd /
:NEXT

cd %RepoPath%port-ploobsengine
call hg pull
call hg update --clean
cd /

The script is very simple, it just creates the repo folder and clone the repo (if it does not exists yet) and then pull and updates its contents to the latest remote version. (the bin folder of the TortoiseHg must be on the Windows Path. If you use SVN or git or …. just adapt the script).

To those that are curious, our public repo is: http://code.google.com/p/port-ploobsengine/

Compile all Projects, Tests and Demos
We have lots of different types of projects in PloobsEngine, then we need to use different tools to build each of them. The following script shows how to deal with these situations:
First of all: Set some Environment Variables (MSBuild Path, XNA PATH, .Net Framework Path …)

SET PATH=%*;%XNAGSv4%Tools;%XNAGSShared%XnaPack;%XNAGSShared%Device Center;C:\Windows\Microsoft.NET\Framework\v4.0.30319\;C:\Windows\Microsoft.NET\Framework\v3.5\;C:\Windows\Microsoft.NET\Framework\v2.0.50727;%PATH%

Using Msbuild for content pipeline and main XNA project

The content pipeline / XNA project are just “regular” Visual Studio projects, to build them we just used the following lines (adapt to your needs)

msbuild %RepoPath%\PloobsEngine\PloobsEngineContent\PloobsEngineContent.contentproj /p:XNAContentPipelineTargetPlatform=Windows;XNAContentPipelineTargetProfile=HiDef;TargetFrameworkVersion=v4.0;Configuration=Release
msbuild %RepoPath%\PloobsEngine\PloobsEngine\PloobsEngine.csproj /p:Configuration=Release

Using the Visual Studion IDE compiling ability directely (msbuild cant compile Visual Studio Installer projects !!!)

Some projects (like those that ends with .vdproj) cant be built using MSBuild, then we call directely the Visual studio line comand IDE utility.

echo Compiling Ploobs Updater
"%VS100COMNTOOLS%..\IDE\devenv.com" /build Release %RepoPath%\PROJEC_PATH\PloobsInstallerSetup.vdproj
rd /s/q FOLDER
md FOLDER
copy %RepoPath%\PROJECT_PATH\*.* FOLDER /Y

Just building and copying the generated contents to a folder (deleting if it already exists)

Using Msbuild to Compile/Clean Full Solutions

The script:

echo Copy Others Demos
rd /s/q SourceCodeDemos
md SourceCodeDemos
msbuild %RepoPath%\PloobsUpdater\PloobsUpdater.sln
msbuild %RepoPath%\AdvancedDemos\AdvancedDemos.sln /t:Clean
msbuild %RepoPath%\IntroductionDemo\IntroductionDemo4.0\IntroductionDemo4.0.sln /t:Clean
msbuild %RepoPath%\ReachDemos\ReachDemos.sln /t:Clean
msbuild %RepoPath%\PhoneSilverLightDemo\GraphicsApp1.sln /t:Clean

Just building the PloobsUpdater.sln Solution and cleaning some others (removing the Bin and Obj folders -> removing previous builds contents, we dont need to send then to the clients !)

Calling Some Custom Scripts

PloobsEngine depends of LOOTS of Dlls, for the end user is easier to have just one with everything inside of it, the following code shows how this can be achieved (using the Microsoft ILMerge Program)

ilmerge /out:PloobsEngineDebug.dll SharedOsiris.dll PloobsEngine.dll FarseerPhysicsXNA.dll Lidgren.Network.dll Library.dll TomShane.Neoforce.Controls.dll BEPUphysics.dll XNAnimation.dll DPSF.dll  /lib:"C:\Program Files (x86)\Microsoft XNA\XNA Game Studio\v4.0\References\Windows\x86" /targetplatform:v4,"C:\Windows\Microsoft.NET\Framework\v4.0.30319" /keyfile:XNAnimation.snk  /xmldocs  /ver:0.0.0.1 /log

For more informations about the arguments, look at the ILMerge Documentation. In this line we just merged Lots of dlls (and the xml docs of each one) and signed the generated assembly with a keyfile.

Running Some Tests

To make sure that we are delivering something that works, our building system also implements some custom unity and Visual Tests. The following script calls our Test Module and checks its answer. (if somethings wrong happens, errors are directed to us !)

echo Executing Visual Tests
cd AdvDemosBin
call AdvancedDemos.exe VisualUnitTests
echo Return Code is %errorlevel%
if %errorlevel% == 0 goto LB
   echo Tests Failed, Check the Logs
   set erro=1 #### Pack the logs and mail the programmers =P ....
   exit /b -1
:LB
del *.log
cd ..

Compiling the Documentation
We use the marvelous SandCastle project to create our documentation.
Sandcastle produces accurate, MSDN style, comprehensive documentation by reflecting over the source assemblies and optionally integrating XML Documentation Comments. It also let us to create some custom pages in order to make a full professional like manual.
The amazing thing is that SandCastle compilation can be triggered using msbuild. (the output is a .chm file), the following script shows how:

if %NODOC% == "FALSE"  goto ND
echo Compiling Ploobs Documentation
msbuild %RepoPath%\Documentation\EngineHelper.shfbproj
rd /s/q Help
md Help
copy "%RepoPath%\Documentation\Ploobs Engine Help.chm" Help /Y
:ND

Creating the Installer (using the marvelous InstallShield Visual Studio Extension)

Each piece are put together with the Install Shield Installer (ends up up a MSI package). The building process can also be triggered using msbuild !!!

echo Creating the Installer
rd /s/q PloobsEngine
md PloobsEngine
rd /s/q %RepoPath%\PloobsInstaller\PloobsInstaller\Express
msbuild %RepoPath%\PloobsInstaller\PloobsInstaller.isproj /p:Configuration=Release;Build=Complete;BuildCompressed=true;BuildSetupExe=true
xcopy %RepoPath%\PloobsInstaller\PloobsInstaller\Express\Release\DiskImages\DISK1\*.*  PloobsEngine /Y /e /i /h

We also run some custom scripts to set the right minor/minor version of the release.

Compressing with Rar (using the classic Winrar)

After the package generation, we need to compress it to a single file (not a requirement in most cases). We just used the classic Winrar for this task.

set PATH="C:\Program Files (x86)\WinRAR";%PATH%
cd PloobsEngine
rd /s/q Package
md Package
cd ..
rar a -r "PloobsEngine/Package/PloobsEngine.rar" "PloobsEngine/*"
copy %DeployPath%\Changes.pdf %DeployPath%\PloobsEngine\Package\ /Y

Uploading to FTP server (generated from the previous pass)

Then we make a simple script to connect to our FTP server to upload the rar file to the righ folder

@echo off
echo Uploading Installer to the Main Server
echo user USER> ftpcmd.dat
echo PASSWORD>> ftpcmd.dat
echo cd Web/Updater>> ftpcmd.dat
echo mkdir teste>> ftpcmd.dat
echo cd teste>> ftpcmd.dat
echo bin>> ftpcmd.dat
echo mput piece.bat>> ftpcmd.dat
echo y>> ftpcmd.dat
echo quit>> ftpcmd.dat
ftp -n -s:ftpcmd.dat FTP_URL
del ftpcmd.dat

The script just generates a dummy conf file (to be able to log in the remote ftp server) and uses it as a parameter to the FTP command.

You can download our packages here: http://www.ploobs.com.br/Updater/0.4.4/PloobsEngine.rar

Sending some Emails and notifications

The last step is notifying everyone involved about the new release. For this we just send some email to the righ people (We made a custom .exe to perform this task, unfortunaly the code is not avaliable but this is a very simple thing to build)

We also have some things to adjust the version number of the releases that were not shown here.

The whole proccess takes more than 4 hours to be completed and is fully automatized. (We remotely calls it and waits for the email containing the Download Link !!!)

There are lots of good build systems avaliable like Team City. I always encourage people to use them. Exceptionaly in our case, i prefered to do all by hand because of the large amount of customization we needed.