Simon schreibt.

• Added new possible problem-source about batching here.
• Added a note about the pipeline simplification here. Madsy9 pointed it out here.
• This article was honored with two reddit threads. Here and here.
• Added better FIFO explanation here. Thx sccrstud92, tmachineorg and koyima
• Cort mentioned this these slides about “How Shader Cores Work”.
• NVidia specifications with actual core counts. Thx Jan, Volkan and Hakan Candemir
• Mr.Yeah also wrote actual core counts in his comment.
• Corrected some misspellings. Thx OddEyesCG for mentioning!
• Videos are now only auto-played when you scroll to them (to prevent another server overload). Thx Flannon & RXMESH for the code! And garblefart for all the feedback.

1. Copy the data into system memory for fast access

At first all necessary data is loaded from your hard drive (HDD) into the system memory (RAM) for faster access. Now the necessary meshes and textures are loaded into the memory on the graphic card (VRAM). This is because the graphic card can access the VRAM a lot faster and mostly doesn’t have direct access to the RAM.

If a texture isn’t needed anymore (after loading it into the VRAM), it can be thrown out of the RAM (but you should be sure, that you won’t need it again soon, because reloading it from HDD costs a lot time). The meshes should stay in the RAM because it’s most likely that the CPU wants to have access to them e.g. for collision detection.

Before the render-party can start, the CPU sets some global values which describe how the meshes shall be rendered. This value collection is called Render State.

2. Set the Render State
A render state is kind of a global definition of how meshes are rendered. It contains information like:

“vertex and pixel shader, texture, material, lighting, transparency, etc. […]” [b01 page 711]

Important: Each mesh, which the CPU commands the GPU to draw, will be rendered under these conditions! You can render a stone, a chair or a sword – they all get the same render values assigned (e.g. the material) if you don’t change the render state before rendering the next mesh.

After the preparation is done, the CPU can finally call the GPU and tell it what to draw. This command is known as: Draw Call.

3. Draw Call
A draw call is a command to render one mesh. It is given by the CPU. It is received by the GPU. The command only points to a mesh which shall be rendered and doesn’t contain any material information since these are already defined via the render state. The mesh resides at this point in the memory of your graphic card (VRAM).

After the command is given, the GPU takes the render state values (material, textures, shader, …) and all the vertex data to convert this information via some code magic into (hopefully) beautiful pixels on your screen. This conversion process is also known as Pipeline.

4. Pipeline
As i said at the beginning, an asset is more or less just a list of vertex- and texture data. To convert those into a mind blowing image, the Graphic Card has to create triangles out of the vertices, calculate how they are lit, paint texture-pixels on them and a lot more. These actions are called states. Pipeline states.
Depending on where you read, you’ll find that most of the stuff is done by the GPU. But sometimes they say, that for example the triangle creation & fragment creation is done by other parts of the graphic card.

This pipeline example is extremely simplified and shall just be seen as a rough overview. I tried to visualize it as good as i could but as a non-programmer i have problems to judge when it gets too simply to not lead you into a wrong direction. So please don’t take it too seriously and consider all the other beautiful sources i linked at the bottom of this article. Or feel free to mail, tweet or facebook me so i can improve the animation/explanation.

Here is an example with only one GPU core:

Rendering is basically doing an immense number of small tasks such as calculate something for thousands of vertices or painting millions of pixels on a screen. At least in (hopefully) 30fps.

It’s necessary to be able to compute a lot of that stuff at the same time and not every vertex/pixel one after another. In the good old days, processors had only one core and no graphic acceleration – they could only do one thing at the same time. The games looked … retro. Modern CPUs have 6-8 cores while GPUs have several thousands (they aren’t that complex like CPU-Cores, but perfect for pushing through a lot vertex and pixel data).

Exact GPU core numbers can be found in [a38], [a39], [a40], [a41], [a42] or in Mr.Yeah’s comment.

When data (e.g. a heap of vertices) is put into a pipeline stage, the work of transforming the points/pixels is divided onto several cores, so that a lot of those small elements are formed parallel to a big picture:

Now we know, that the GPU can work on stuff in parallel. But what’s about the communication between CPU and GPU? Does the CPU has to wait until the GPU finished the job before it can receive new commands?

Thankfully not! The reason is, that such a communication would create bottlenecks (e.g. when the CPU can’t deliver commands fast enough) and would make parallel working impossible. The solution is list where commands can be added by the CPU and read by the GPU – independent from each other! This list is called: Command Buffer.

5. Command Buffer
The command buffer makes it possible that CPU and GPU can work independent from each other. When the CPU wants something to be rendered, it can push that command into the queue and when the GPU has free resources, it can take the command out of the list and execute it (but the list works as a FIFO – so the GPU can only take the oldest item in the list (which was first/earlier added than all others) and work on that).

By the way: there are different commands possible. One example is a draw call, another would be to change the render state.

That’s it for the first book. Now you should have an overview about asset data during rendering, draw calls, render states and the communication between CPU and GPU.

The End

Welcome to the second book! Here we’ll checkout some problems which can occur during the rendering process. But first, some practice:

To know about a problem is useful. To actually feel the problem is even better for understanding. So let’s try to feel like a CPU/GPU.

Experiment
Please create 10.000 small files (e.g. 1 KB each) and copy them from one hard drive to an other. It will take a long time even if the data amount is just 9,7 MB in total.

Now create a single file with a size of 9,7 MB and copy it the same way. It will go a lot faster!

Why? It’s the same amount of data!

That’s right but for every copy-action there’s some stuff to do, for example: prepare the file transfer, allocate memory, read/write heads move back and forth in the HDD, … which is overhead for every write action. As you painful feel, this overhead is immense if you copy a lot small files. Rendering many meshes (which means executing many commands) is a lot more complex, but it feels similar.

Let’s now have a look at the worst case you can get during the rendering process.

Worst Case
To have many small meshes is bad. If they’re using different material parameters on them, it gets even worse. But: Why?

1. Many Meshes

The GPU can render faster than the CPU can send commands.

“The main reason to make fewer draw calls is that graphics hardware can transform and render triangles much faster than you can submit them. If you submit few triangles with each call, you will be completely bound by the CPU and the GPU will be mostly idle. The CPU won’t be able to feed the GPU fast enough.” [f05]

In addition, every draw call produces some kind of overhead (like mentioned above):

“There is driver overhead whenever you make an API call, and the best way to amortize this overhead is to call the API as little as possible.” [a02]

2. Many Draw Calls
One example for such an overhead is the command buffer (explained above). Do you remember that the CPU fills the command buffer and the GPU reads from it? Well, they have to communicate about the changes and this creates overhead too (they do this by updating read/write pointers – read more about it here)!
Therefore it might be better, to not hand over one command after another but first fill up the buffer and then hand over a complete chunk of commands to the GPU. This increases the risk that the GPU has to wait until the CPU is done with building the chunk, but it reduces the communication overhead.

The GPU would (hopefully) have stuff to do (e.g. working on the last chunk of commands) while the CPU builds up the new command buffer. Modern CPUs can also fill several command buffers in parallel and hand them over to the GPU later one after another.

This was only one example. In the real world not only CPU, GPU and Command Buffer are speaking with each other. The Api (DirectX, OpenGL), the driver and whatnot are also integrated in the process which doesn’t make it easier.

We only spoke about many meshes with the same material parameters (render state). But what happens when you want to render meshes with different materials?

3. Many Meshes and Materials

Flush the pipeline.

“When changing the state, there is sometimes a need to wholly or partially flush the pipeline. For this reason, changing shader programs or material parameters can be very expensive […]” [b01 page 711/712]

You thought it can’t get worse? Well … if you have different materials on different meshes, you can’t send their render-commands as chunks. You set a render state for the first mesh, command to render it, set a new render state, command the next mesh-rendering and so on.

I colored the “change state” commands in red because a) they are expensive and b) for better overview.

Setting the render state sometimes (not always, depends on what parameters you want to change) results in a “flush” of the whole pipeline. This means: every mesh which is currently processed (with the current render state) has to be finished before new meshes can be rendered (with the new render state) . It would looks like in the image above. Instead of taking a huge number of vertices (e.g. when you combine several meshes of the same render state – an optimization I’ll explain later), you would render a small amount before changing the render state which – this should be clear by now – is a bad thing.

By the way: Since the CPU needs a minimum time for setting up a draw call (independent of the given mesh size), you can assume that there’s no difference in rendering 2 or 200 triangles. The GPU is crazy fast and before the CPU has prepared a new draw call, the triangles are already freshly baked pixels on screen.
This “rule” changes of course when we talk about combining several small meshes into one big mesh (we’ll look at this in a second).

I wasn’t able to get latest values for how many polygons you render “for free” on current graphic cards. If you know something about that or did some benchmarks recently, please tell me!

4. Meshes and Multi-Materials
What when not only one material is assigned to a mesh but two or more? Basically, your mesh is ripped into pieces and then fed piece by piece into the command buffer.

This of course creates one draw call per mesh piece.

I hope i could give you a small insight of what is bad about a lot meshes and materials. Let’s now look at some solutions because even all of this sounds really really bad: There are beautiful games out there which means that they solved the mentioned problems somehow.

The End

1. Sorting
Firstly you can sort all your commands (e.g. by render state) before you fill the command buffer. This would reduce the necessary state changes to the minimum since you go through all meshes of the same kind before changing the state.

But you would still create a lot overhead by rendering every mesh one after another. To reduce this overhead, a technique called Batching seems to be useful.

2. Batching
When sorting your meshes, you kind of pile them together to heaps of the same kind. The next step would be, to tell the GPU to render such a heap at once. This is what batching is about:

“‘Batching’ means to group some meshes together before calling the API to draw them. This is why it takes less time to render a big mesh than multiple small meshes.” [a36]

So, instead of using one draw call per mesh (which share the same render state)…

…you would combine the meshes (with the same render state) and render them as one draw call. This is a really interesting topic because you can render different meshes (stone, chair or a sword) at once as long as they use the same render states (which basically means that the use the same material setup).

It’s important to mention, that you combine the meshes in the system memory (RAM) and then send the newly created big mesh to the graphic card’s memory (VRAM). This takes time! Therefore batching is good for static objects (stones, houses, …) which you combine once and let them stay in the memory for long time.
You can also batch together dynamic objects like for example laser-bullets in a space game. But since they’re moving you would have to create this bullet-cloud-mesh every frame and send it to the GPU memory!

A better solution for handling dynamic objects is Instancing.

3. Instancing
Instancing means, that you send only one mesh (e.g. a laser bullet) instead of many and let the GPU duplicate it several times. Having the same object at exact the same position with the same rotation or animation would be a bit boring. Therefore you can provide a stream of extra data like the transformation matrix to render the duplicates at different positions (and in different poses).

“Typical attributes per instance are the model-to-world transformation matrix, the instance color, and an animation player providing the bones used to skin the geometry packet.” [a37]

Don’t nail me down on that, but as far as i know, is this data stream just a list in the RAM where the GPU has access to.

This would result in only one draw call per mesh type! The difference in comparison to Batching is, that all instances look the same (because they’re copies of the same mesh) while a batched mesh can consist of several different meshes as long as they use the same render state parameters.

Now it gets a bit more creative. I think the following tricks are very cool, even if they are only suitable for special cases:

4. Multi-Material-Shader
A shader can access several textures and therefore it’s possible to not only have one diffuse/normal/specular/… map but e.g. two of them – which basically means, that you have two materials combined in one shader. The materials are blended into each other controlled by a blend-texture. Of course, this cost’s GPU power because the blending is expensive, but it reduces the draw call count because a mesh with two or more materials would not be ripped into pieces anymore (explained under “4. Meshes and Multi-Materials”).

Read more about this here.

The documentation says that a higher draw call count is still better than this expensive technique. Anyway, i found it very interesting and if you need some good numbers for statistics, you can argue that layer materials reduce the draw call count (even if this says nothing about the performance … but psssssssst!).

5. Skinned Meshes
Do you remember the laser-bullet-mesh i talked about? I said that this mesh would have to be updated every frame since the bullets constantly move. Batching them together and sending the resulting mesh every frame would be expensive.
An interesting approach to this problems is to automatically add a bone to every bullet and give it skinning information. With that you would have one big mesh which could stay in the memory and you would only update the bone data for every frame. Of course, if a new bullet as shot or an old gets destroyed, you would have to create a new mesh. But it sounds like a really interesting idea to me.

Read more about this here.

Feel free to send me more links about creative solutions to reduce draw calls!

Almost done! You should now have a vague understanding of what can be done to render assets a bit faster. Don’t worry, the next book will be short.

The End

Avoid small meshes
Check if small meshes are necessary or if you could connect several small to one big mesh. If you have small, talk to a graphic programmer to get infos about the polycount “sweet spot” (meshes below that triangle count aren’t rendered faster). So maybe yo want add some tris to make things round if you need to keep a small mesh. You should also care about multi materials. If you have one big mesh but with 5 sub-materials assigned, the big mesh is ripped apart during the rendering and this means you now have 5 small meshes again. Maybe an atlas texture could help?

Avoid too many materials
Speaking of materials: think about the material management. Sharing materials between assets might be possible if you plan ahead before the asset creation. Bigger atlas textures can help.

Debug Tools
Talk to you programmers if you can get ingame statistics so that you can assume how problematic your asset might be. Sometimes it’s hard to have the overview over complex assets. But if a tool can warn you, that this asset could be a potential performance problem, you might solve the problem before the asset is committed as final.

Ask the coder
As you see this topic is highly technical and very context dependent (hardware, engine, driver, game perspective…). So it might be a good idea to ask your programmer how assets should be setup. Or just wait, because if performance drops because of your assets, programmers will find your office and poke you until your optimized your stuff.

You have more tips I should add here? Let me know!

Wow, you read until here? You’re crazy! Thanks a lot! Let me now what you think. I hope you learned something.

The End