GPU Programming for Video Games (2014)

GPU Programming for Video Games (2014)
<!–
–>

Summer 2014

News

  • Almost over!

Assignments

Schedule

The shader programs in the slides and unitypackages below
are intended to
illustrate various aspects of shader programming; they are not necessarily
the most efficient or congruent with Unity’s workflow.

  • 5/13: Session 1 – Introduction
    (PDF slides,
    PDF slides 4-up)

  • 5/15: Session 2 – History of gaming hardware
    (PDF slides,
    PDF slides 4-up)

  • 5/20: Session 3 – 3D Coordinates & Transformations
    (PDF slides,
    PDF slides 4-up)

  • 5/22: Session 4 – 3D to 2D Projection
    (PDF slides,
    PDF slides 4-up)

  • 5/27: Session 5 – Lighting & Rasterization
    (PDF slides,
    PDF slides 4-up)

  • 5/29: Session 6 – Texturing & Blending
    (PDF slides,
    PDF slides 4-up)

  • 6/3: Session 7 – GPUs – Under the Hood
    (PDF slides,
    PDF slides 4-up)

  • 6/5: Session 8 – Materials in Unity
    (PDF slides,
    PDF slides 4-up)
    (this lecture will consist of a live
    demonstration
    with Unity; the slides are more of an outline than something I intend
    to show)

    • Vertex vs. pixel
      shading demo      : Create a new Unity project and drag the unitypackage
      file onto the Asset folder to import it, and then run the
      “Barrels VvD Lighting Demo” scene. Number keys 0 through 5 change the speed
      of rotation, and the x, y, and z keys toggle on and off rotations around those
      axes.
  • 6/10: Session 9 – Introduction to Shader Languages, Part 1
    (PDF slides,
    PDF slides 4-up)

    • GPU14
      Introductory Shaders (June 14 revision)      :
      Run “GPUXXIntroShadersScene” to see two spinning treasure chests
      (model
      by Mobile Design Lab      ,
      free on Unity Asset store), along with some other spinning objects. There
      is an “original” version and a “GPUXX” version of each of the materials,
      so you can experiment with comparing different shaders and textures.
      I replaced the treasure chest normal map with a normal
      map based on the Georgia Tech
      logo; you can put the treasure chest normal map back in to see something
      more subtle.
      There is also a non-visible game object
      that is just a host for a script I wrote to create some matrices and
      multiply them in various ways. I used this when I was trying to figure out
      what conventions Unity uses, since the official documentation is sometimes
      contradictory
      and confusing.
  • 6/12: Session 10 – Introduction to Shader Languages, Part 2
  • 6/17: Session 11 – Environment Mapping
    (Updated 6/21: PDF slides,
    PDF slides 4-up)

    • GPU14 Environment Mapping
      Demo      : This contains three scenes. One compares the “original Phong”
      (displayed as red) and “Phong-Blinn” (displayed as blue)
      specular models, so pixels where the two models are give similar results
      appear as purple. By adjusting the powers, you can get them to match pretty
      closely. The other two scenes illustrate environment mapping, one with
      per-vertex and one with per-pixel computations. A “crossfade” slider lets
      you blend reflections and refractions. The per-vertex version shows
      clear triangulation artifacts, so I didn’t bother implementing Fresnel
      effects for the per-vertex shader, but I did for the per-pixel shader.
      Depending on which “return” statement in the per-pixel shader is in effect
      (comment out the other one), you can use either the original crossfade slider,
      or play with the Fresnel parameters. (The
      horse
      model and animations      , free
      on Unity Asset store, are by
      Dootsy Development. The
      basilica
      cube map       is by Humus.)
  • 6/19: Session 12 – Projective Textures & Shadow Mapping
    (PDF slides,
    PDF slides 4-up)

    • GPU14 Projective Texturing
      Demo      : Run “GPUXXProjTexScene” — have fun! (For some reason, the
      Scene view sometimes comes up as pure black when you first load the scene;
      just press “play” and then “stop” — that seems to fix it.) (Base
      textures from Lost Garden.)
  • 6/24: Session 13 – Introduction to Surface Shaders
    (PDF slides,
    PDF slides 4-up)

  • 6/26: Session 14 – Case Study:
    Portalarium’s Multi-Shader (with
    Shroud of the Avatar
    pre-alpha demo)

  • 7/1: Session 15 – Surface Shaders with Vertex Modification
    • GPU14
      Vertex Modification Demo      :
      Run “GPUXXVertexModScene.” One of the squares has a 100×100
      tessellation; the other, which often demonstrates spatial aliasing, has
      a 10×10 tessilation. A custom material inspector lets you use a checkbox
      to switch between planar and circular waves. The “direction” setting is
      ignored if circular waves are active.
  • 7/8: Session 16 – Postprocessing (demo to be posted)
  • 7/10: Session 17 – G-buffers
    • Run the “GPUDepthNormalsTest” scene in the Deferred Rendering
      unitypackage from Session 18 below.
  • 7/15: Session 18 – Deferred Rendering
  • 7/17: Session 19 – Advanced Post Processing with G-Buffers
    • GPU14
      Advanced Post Processing Demo      :
      Make sure “process” in the AdvPostProc component attached to the Main
      Camera is checked, then
      experiment with the “return” lines in the GPUXXFancyPostProcTest.shader.

Details and descriptions

This webpage constitutes the class syllabus.


Course description: (2 hours lecture, 3 hours unscheduled lab,
3 credit hours total) 3-D graphics pipelines.
Real-time simulation concerns. Game engine architectures.
GPU architectures. Graphics APIs. Vertex, pixel, and compute
shader programming. Deferred rendering. Performance analysis.


Course objective: This class provides the GPU programming
concepts needed to meet timely demands of the multimedia,
visualization, and gaming industries.
The course also bridges the gap between our
current generic computer
architecture courses and the video game design courses offered by
the College of Computing,
the School of Electrical and
Computer Engineering, and the
School of Literature, Media, and Culture.
The class covers state-of-the-art general purpose graphical processing unit architectures (GPGPUs) from application and hardware design perspectives. The course considers programming models using examples from the algorithmic needs of modern 3-D games (e.g. geometry processing, shading algorithms, physical modeling, collision detection, and artificial intelligence).

Tentative topics

The topics and
the order in which they are covered is subject to change.

  1. Introduction and historical context
  2. Classic 3-D rendering pipeline: geometry transformation, lighting, texturing
  3. Overview of 3-D APIs (ex: OpenGL and Direct3D)
  4. Simulation loops and game engine components
  5. Object-oriented game engines (ex: Unity, Unreal 4, C4)
  6. GPU architectures and GPU assembly code
  7. Architecture comparison: Playstation 4 vs. Xbox One vs. Wii U vs.
    “gaming PC”

  8. Introduction to shading languages (HLSL/Cg), vertex and pixel shaders
  9. Per-pixel vs. per-vertex lighting
  10. Advanced 3-D shading effects (ex: bump mapping, environment mapping)
  11. Postprocessing effects (ex: bloom, motion blur)
  12. Compute shaders (ex: decompression, physics simulations)
  13. Deferred rendering
  14. Screen space techniques (ex: ambient occlusion)
  15. Real-time shadow techniques
  16. Profiling and performance analysis
  17. Adapting to the limitations of hand-held devices
  18. Possibly occasional guest speakers from industry
    (previous speakers)


Note that the course
does not cover OpenCL or CUDA, and unlike its
predecessor, “Multicore and GPU Programming for Video Games,”
does not cover multicore CPU programming.
OpenCL, CUDA, and general
multicore programming are well covered in many other classes in ECE and CoC, whereas the use of GPU architectures
for their native application of computer graphics
is not extensively
covered in many other classes, either here at Georgia Tech or other schools.

Course history

This is a heavily revised version of an earlier special topics class called
“Multicore and GPU Programming for Video Games,”
which
was taught
six times,
in the Fall semesters from 2007 to 2012. Prof.
Sean Lee
and I co-created
the class in 2007; we co-taught it the first three times, and I ran it on
my own the last three.


Game industry trends, from both technological and business
perspectives, demanded that the class be re-imagined to remain
relevant in 2014. The dropping
of “Multicore and” from the title — effectively
refocusing the entire course on GPUs — responds
to the trend of GPUs taking up
more and more of the die on modern
high-performance game platforms than CPUs,
along with their increased flexibility
compared with earlier dedicated graphics hardware.


From a more practical standpoint,
“Multicore and GPU Programming for Video Games”
leaned heavily on Microsoft’s
XNA
Game Studio
platform, which provided a
mechanism for students to deploy code
on retail Xbox 360s, and potentially
sell games on the
Xbox
Live Indie Games (XBLIG)
service. At the start of 2013, Microsoft
announced that it was sadly
discontinuing
development of and support for XNA
(although the “writing was on the wall” for
some time before that),
and XBLIG never
“took
off” the way,
for instance, Apple’s iOS “App Store” did.
Given the amount of material we had developed using XNA, I initially
looked at
MonoGame,
but scrapped it when I was
unable to readily
get it running on my Mac.
In searching for alternatives,
the
Unity engine,
the C4 Engine,
and
Unreal Engine 4
were found to have numerous appealing features.

Administrivia

Instructor:

Prereqs:
((ECE2036 or CS1372) and ECE2031) or CS2110 or CS2261 or ECE3090 or ECE3035.
(Reasoning behind the pre-requisites.)


Course materials: Primarily lecture notes and literature. We may
also draw material from textbooks such as:

  • Mathematics for 3D Game Programming and Computer Graphics, Third Edition, by
    Eric Lengyel,
    2011 (note: Lengyel is the creator of the C4 game engine).

  • Unity Shaders and Effects Cookbook, by Kenny Lammers, 2013
  • Game Engine Architecture, by Jason Gregory, 2009

Assessment:
Grades will be based on several major,
intensive programming projects, as well as a few
smaller “warm-up” assignments
designed to get students comfortable with various toolsets.
There may also be a few
short essay-type (around 1/2 page) questions and “pencil and paper” exercises.
Various assignments may have different weights in the final grade calculation;
these weights will be noted on each assignment.


On some assignments, students signed up
for the graduate section
will have additional
requirements beyond those signed up for the undergraduate section.
They may also be required to perform additional assignments that are
not assigned
to the undergraduates.


There will be no
traditional paper-and-pencil exams,
either midterms or finals.
This is a class about programming;
my philosophy is that any time you
would spend taking or studying for such exams
is better spent in front of a computer actually programming!


Projects: Students will be undertake
several projects to gain real programming
experience in C# using the
Unity
game engine (and maybe
C++ within
Unreal
Engine 4
and/or the
Terathon
C4 Engine; details TBD),
with emphasis on HLSL/Cg vertex and pixel
shader programming and more general GPGPU
programing using technologies such as DirectCompute
and OpenGL compute shaders.
In one of these assignments,
students will code a basic graphics pipeline in a scripting language
of their choice without
the aid of a 3-D API (to ensure that they understand
what a GPU does for them).


Some of the assignments will be individual assignments;
however, a few of them may provide the option
to work in groups of two (such
assignments may still be completed
individually if you prefer to work alone). For variety,
we will ask that you not
partner with the same person on more than one assignment.
This is a class on programming, so
even on team assignments, you
should feel confident that you could complete an assignment on your own if you needed to.


A lot of the course will use C#,
but we will not assume that you’ve seen C# before,
and hence will present some introductory material on C#.
(If you’ve seen Java, C++,
or any other “curly-brace” language, you’ve pretty much seen them all,
but there are some “gothchas” in moving from one to another.)


On-line discussions: We will use
Piazza
to facilitate class discussions.
I will try
to check Piazza at least once a day. You are welcome to post questions about
anything
related to the course material, and also answer other student’s questions, as
long as
you don’t “give away the answer” or post chunks of code that are more than a
few lines.


T-square usage: In spite of its awfulness, we will use T-square for
most homework submission and for posting grades, and occassionally for
distributing resources for assignments.


Office hours:
Shortly before assignments are due, I will post an announcement on Piazza describing when and where I will be sure to be available for questions. This will tend to change slightly from week to week, so look for that announcement. (Also, if you walk by my office and happen to see my VL431 office door open, you are welcome to pop in with questions about the class and/or life in general.)


Of course
you are always welcome to e-mail me and we can set up a specific time to meet. Again, put “GPU” in your subject.


Honor code:
This course
will be conducted under the rules and guidelines of the Georgia Tech Honor
Code;
infractions will be reported to the Dean of Students.
The “ground rules” for each assignment,
which may vary from assignment to assignment, will be given in each
assignment description. Please ask if any aspects of the given “ground
rules” seem unclear.


Backfile policy:
Use of homework
solutions and from previous versions of this class is forbidden.
The material is highly complex,
so it is extremely difficult to come up with 100%
new project descriptions on each offering.
Please be fair to students who may not have access to the same old materials.
Detection of the use of backfiles will result in significant wrath.


Major emergencies:
If you have some sort of major life emergency – serious illness or injury,
death in the family, house burns down or is flooded, etc. – that seriously
impedes your progress in the class, please let me know as soon as possible
so we can work something out.
You will find professors can be quite reasonable if you keep us in the loop.
Please don’t disappear with no warning half way through,
making me think that you dropped the class, and then reappear out of
nowhere the week before finals asking what you can do to make things up.
(Yes, this has happened quite a bit, in both undergrad and grad classes.)


On things that beep and blink:
Please silence all cell phones and pagers
before entering class. If you forget to do so and receive
a call, please shut the noisemaking device down as quickly as possible,
and return the person’s call after class. (Of course, there
are reasonable
exceptions, i.e. if your wife is in the 9th month of her pregnancy and may
give birth at any moment or your
kid isn’t feeling well but he or she
went to school anyway and their school nurse
may need to call you, leave your
phone on vibrate, and answer it as quickly as possible and
immediately step out
of the room to handle the call.)


In general,
please do not instant message, websurf, Facebook (can I use it as a verb?),
e-mail,
play games, etc. during class.
It can be quite distracting.
Unless I say otherwise, the preferred position for laptops during
class is in your backpack.


The Twitter exception:
If
Prof.
Lanterman says something particularly brilliant and clever during lecture,
you are allowed to
use your phone to Tweet it or post
it to Facebook.

Assorted relevant and fun links