How Live Gift Effects Render and Composite On-Device: Decoding, Drawing, and Blending SVGA/VAP/PAG

Earlier posts in this series covered format selection, performance tuning, asset delivery, and queue scheduling — how a gift effect arrives on the device and when it plays. This post tackles the piece we haven’t opened up yet: once an SVGA/VAP/PAG file reaches the device, how does the client actually turn it into smooth on-screen animation, layer it seamlessly over the live video, and do so without hurting the stream’s own frame rate?

This is the rendering and compositing pipeline — from decoding, frame-by-frame drawing, offscreen rendering, and texture upload, all the way to GPU compositing and the blend with the live picture. Understand this chain and you can pinpoint the bottleneck when stutter, dropped frames, or corruption show up in production.

1. The starting point: from binary to drawable data

Mint-Flavored Afternoon: decoding an SVGA/VAP/PAG binary into drawable data is the starting point of rendering, and the richness of the data structure determines decoding complexity (XingJi Shop)

Decoding SVGA

SVGA is essentially a protobuf binary describing a vector animation — it records, per frame, the vector paths, transform matrices, and image references. When the client receives a .svga file, the first step is to deserialize it into in-memory objects.

// Typical decode flow with the official SVGA SDK
SVGAParser parser = new SVGAParser(context);
parser.decodeFromAssets("gift_rocket.svga", new SVGAParser.ParseCompletion() {
    @Override
    public void onComplete(SVGAVideoEntity videoEntity) {
        // videoEntity contains:
        // - videoSize: canvas dimensions
        // - frames: total frame count
        // - FPS: frame rate
        // - sprites: per-layer draw instructions (paths, images, transforms)
        svgaImageView.setVideoItem(videoEntity);
        svgaImageView.startAnimation();
    }
    @Override
    public void onError() { }
});

The resulting SVGAVideoEntity is a pure data structure — no bitmaps or GPU textures yet. The real rendering happens during playback, generating a bitmap or drawing paths frame by frame.

Decoding VAP

VAP (Video Animation Plugin) is essentially an MP4 container: the video track holds RGB data, the audio track is repurposed to carry an alpha mask, plus a JSON config describing animation parameters. Decoding has two steps:

1. Video decode: use the system MediaCodec or FFmpeg to decode video frames (YUV → RGB);

2. Alpha blend: pull alpha data from the audio track and blend it with the RGB frame to produce an RGBA bitmap.

VapView vapView = findViewById(R.id.vap_view);
vapView.setVideoSource("gift_heart.mp4");
vapView.startPlay(new IVapListener() {
    @Override
    public void onVideoStart() { }
    @Override
    public void onVideoRender(int frameIndex, Bitmap frameBitmap) {
        // frameBitmap is already the RGBA bitmap after RGB + Alpha blending
    }
    @Override
    public void onVideoComplete() { }
});

VAP’s decoding cost is dominated by the video decoder — on devices without hardware decode support it falls back to software, spiking CPU usage.

Decoding PAG

PAG (Portable Animated Graphics) is Tencent’s open-source vector animation format, with After Effects export support. Its decoding relies on a C++ rendering engine that outputs OpenGL textures directly, skipping the Bitmap stage.

PAGView pagView = findViewById(R.id.pag_view);
PAGFile pagFile = PAGFile.Load(assetManager, "gift_planet.pag");
pagView.setComposition(pagFile);
pagView.setProgress(0.0);  // 0.0 ~ 1.0
pagView.play();

PAG’s advantage is that it runs fully on the GPU with minimal CPU overhead, at the cost of requiring OpenGL ES 2.0+.

2. Frame-by-frame drawing: CPU vs GPU

Rose Rendezvous: a multi-layer petal effect where each layer needs its own transform and blend per frame — the scenario where CPU vs GPU drawing paths differ most visibly (XingJi Shop)

When an effect plays, each frame can be produced one of two ways: a CPU software path, or a GPU hardware-accelerated path.

CPU drawing (Canvas path)

Early SVGA versions drew vector paths with Android Canvas:

@Override
protected void onDraw(Canvas canvas) {
    super.onDraw(canvas);
    SVGAVideoEntity entity = ...;
    int currentFrame = (int) (animator.getAnimatedValue());

    for (SVGAVideoSpriteEntity sprite : entity.getSprites()) {
        // Pull the current frame's path and transform
        Path path = sprite.getFrameEntity(currentFrame).getShapePath();
        Matrix matrix = sprite.getFrameEntity(currentFrame).getTransform();

        canvas.save();
        canvas.concat(matrix);
        canvas.drawPath(path, paint);
        canvas.restore();
    }
}

The upside is compatibility — it runs on any Android version. The downside is that it draws entirely on the main-thread CPU; a complex animation (dozens of layers, hundreds of paths) pegs the CPU and stutters the main thread.

GPU drawing (OpenGL textures)

The modern approach is to render offscreen into a texture, then upload it to the GPU for compositing. Using SVGA as an example:

1. Offscreen draw: render the current frame onto a Bitmap (Canvas or Skia);

2. Texture upload: pass the Bitmap data to the GPU via glTexImage2D, turning it into a texture object;

3. GPU composite: the live picture is one texture, the effect is another, and an OpenGL shader blends them.

// Offscreen-render the current frame
Bitmap offscreenBitmap = Bitmap.createBitmap(width, height, Bitmap.Config.ARGB_8888);
Canvas offscreenCanvas = new Canvas(offscreenBitmap);
drawSVGAFrame(offscreenCanvas, currentFrame);

// Upload to a GPU texture
GLES20.glBindTexture(GLES20.GL_TEXTURE_2D, textureId);
GLUtils.texImage2D(GLES20.GL_TEXTURE_2D, 0, offscreenBitmap, 0);
offscreenBitmap.recycle();  // release the Bitmap immediately, keep only the GPU texture

PAG goes further and skips the Bitmap entirely, generating paths, gradients, and masks directly on the GPU and outputting a texture, with the CPU barely involved.

3. Compositing with the live stream: three approaches

Tipsy Hour: a semi-transparent glass effect that requires alpha blending when composited with the live picture — a stress test for GPU fillrate and blend strategy (XingJi Shop)

The live picture is already a video stream. Layering a gift effect on top can be done three ways.

Approach 1: View-layer stacking (simplest, worst performance)

The live picture renders in a SurfaceView or TextureView, and the gift effect uses a separate View (SVGAImageView / VapView) placed on top.

<FrameLayout>
    <TextureView android:id="@+id/live_video" />  <!-- live picture -->
    <com.opensource.svgaplayer.SVGAImageView android:id="@+id/gift_effect" />  <!-- effect layer -->
</FrameLayout>

Pros: simple to build; effect and stream are fully decoupled.

Cons:

• Double compositing cost: the system composites the live picture, then the effect layer, then the screen — the GPU does redundant work;

• Overdraw: the part of the live picture hidden by the effect is still drawn, wasting bandwidth;

• Deep View hierarchy: Android’s View-tree traversal, measure, and layout all cost time, so the main thread jitters when many gifts play.

This approach only suits low-frequency scenarios (the occasional gift). In a busy gifting room, frame rate collapses.

Approach 2: composite offscreen, then present once

Render both the live picture and the gift effects into one offscreen FBO (FrameBuffer Object), composite there, then present the result in a single pass.

// Pseudocode: OpenGL render thread
void onDrawFrame() {
    // 1. Bind the offscreen FBO
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, offscreenFBO);
    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT);

    // 2. Draw the live picture first (the texture from SurfaceTexture)
    drawTexture(liveVideoTextureId, fullScreenQuad);

    // 3. Then draw the gift effects (possibly several, sorted by Z-order)
    for (GiftTexture gift : activeGifts) {
        GLES20.glEnable(GLES20.GL_BLEND);
        GLES20.glBlendFunc(GLES20.GL_SRC_ALPHA, GLES20.GL_ONE_MINUS_SRC_ALPHA);
        drawTexture(gift.textureId, gift.bounds);
    }

    // 4. Unbind the FBO, draw to the screen
    GLES20.glBindFramebuffer(GLES20.GL_FRAMEBUFFER, 0);
    drawTexture(offscreenFBO.colorAttachment, fullScreenQuad);
}

Pros: composites only once, minimal overdraw, stable frame rate.

Cons: you manage the OpenGL context, FBO, and texture lifecycle yourself — high complexity, hard to debug when things go wrong (black screen, corruption, texture leaks).

Approach 3: the live SDK’s effect callback (recommended)

Mainstream live SDKs (Tencent Cloud, Alibaba Cloud, Agora) all provide a video-frame callback + custom rendering interface that lets you blend a custom texture directly onto the video frame.

Using Tencent Cloud TRTC as an example:

trtcCloud.setLocalVideoProcessListener(TRTCCloudDef.TRTC_VIDEO_PIXEL_FORMAT_Texture_2D,
    TRTCCloudDef.TRTC_VIDEO_BUFFER_TYPE_TEXTURE, new TRTCCloudListener.TRTCVideoFrameListener() {
    @Override
    public void onProcessVideoFrame(TRTCCloudDef.TRTCVideoFrame srcFrame,
                                     TRTCCloudDef.TRTCVideoFrame dstFrame) {
        // srcFrame.textureId is the current video frame's OpenGL texture
        // blend the gift-effect textures onto it here, and write to dstFrame
        int mergedTextureId = blendGiftEffects(srcFrame.textureId, activeGiftTextures);
        dstFrame.textureId = mergedTextureId;
    }
});

Pros:

• The SDK already manages the GL context and render thread; you only supply the blend logic;

• The encoder receives the composited frame directly, so the stream pushed to the CDN carries the effect baked in — remote viewers don’t render it themselves;

• Best performance, because it removes one CPU ↔︎ GPU copy.

Cons: tight coupling to the live SDK — switching SDKs means a rewrite.

4. Performance bottlenecks and fixes

Bottleneck 1: texture upload bandwidth

Every frame, the effect’s Bitmap has to upload from CPU to a GPU texture, and glTexImage2D is an expensive operation. A 1080×1920 RGBA bitmap is 8 MB per frame; at 60 fps that’s 480 MB/sec, easily saturating memory bandwidth.

Fixes:

• Lower resolution: the effect doesn’t need to match screen size — 512×512 is plenty, dropping upload to 1 MB/frame;

• Reuse textures: when the same effect plays repeatedly, reuse the texture object and only update its content (glTexSubImage2D) instead of recreating it;

• PBO (Pixel Buffer Object): async upload — the CPU writes the PBO, the GPU reads from it, reducing stalls.

Bottleneck 2: alpha-blend fillrate

Gift effects are usually semi-transparent, so the GPU runs alpha blending (glBlendFunc). With a large effect area and many layers, every pixel is read and written multiple times, and fillrate becomes the bottleneck — especially on low-end GPUs.

Fixes:

• Premultiplied alpha: export assets with premultiplied alpha so the blend formula simplifies to glBlendFunc(GL_ONE, GL_ONE_MINUS_SRC_ALPHA), cutting multiplies;

• Clip invalid regions: don’t draw the transparent border; shrink the bounding box;

• Tiered degradation: on low-end devices, render only P0-tier effects and skip P1/P2.

Bottleneck 3: main-thread stalls

Even with GPU rendering, the prep work for decoding and texture upload still runs on the main thread, and high-frequency gifts freeze the UI.

Fixes:

• Async decode: offload SVGA/VAP decoding to a background thread, callback to main when done;

• Texture-upload queue: the main thread only enqueues; the render thread batch-uploads;

• Object pools: reuse high-frequency objects like Bitmap, Paint, and Matrix to cut GC pressure.

5. Common rendering bugs and how to debug them

Black screen / nothing shows

Causes:

• Texture ID wrong or already released;

• GL context not current on this thread;

• FBO bind state corrupted.

Debug:

int error = GLES20.glGetError();
if (error != GLES20.GL_NO_ERROR) {
    Log.e("GL", "OpenGL error: " + error);
}

Corruption / tearing

Causes:

• Texture data written past its bounds (buffer overflow);

• Multiple threads touching the same texture concurrently;

• Bitmap recycled early while the texture still references it.

Debug: make sure the Bitmap is valid before glTexImage2D and recycle it right after upload; lock texture operations or keep them all on the same GL thread.

Memory leak

Cause: texture objects (IDs from glGenTextures) are never freed with glDeleteTextures, so GPU memory keeps climbing.

Debug:

class TexturePool {
    private final Set<Integer> allocatedTextures = new HashSet<>();

    int acquire() {
        int[] ids = new int[1];
        GLES20.glGenTextures(1, ids, 0);
        allocatedTextures.add(ids[0]);
        return ids[0];
    }

    void release(int textureId) {
        GLES20.glDeleteTextures(1, new int[]{textureId}, 0);
        allocatedTextures.remove(textureId);
    }

    void checkLeaks() {
        if (!allocatedTextures.isEmpty()) {
            Log.w("Leak", "Unreleased textures: " + allocatedTextures);
        }
    }
}

6. Rendering metrics to watch after launch

• Effect frame rate (FPS): track the effect layer’s frame rate separately from the stream’s, to tell whether the effect or the stream itself is dropping frames;

• Texture upload latency (P50/P95): instrument around glTexImage2D — anything over 5 ms is a bottleneck;

• GPU utilization: check via Android Profiler or adb shell dumpsys gfxinfo; effect playback shouldn’t push GPU usage past 70%;

• Main-thread stall rate: the share of main-thread frames exceeding 16 ms after an effect fires — reflects whether decode/upload is blocking the UI;

• Render anomaly rate: black screens, corruption, and crashes, broken down by device model and GPU vendor (Mali/Adreno/PowerVR behave very differently).

long uploadStart = System.nanoTime();
GLES20.glTexImage2D(GLES20.GL_TEXTURE_2D, 0, bitmap, 0);
long uploadCost = (System.nanoTime() - uploadStart) / 1_000_000;  // ms
tracker.track("gift_texture_upload", Map.of("cost_ms", uploadCost, "size", bitmap.getByteCount()));

Wrapping up

The core chain of live gift effect rendering and compositing:

1. Decode: SVGA deserializes protobuf, VAP decodes video + alpha blend, PAG outputs textures directly;

2. Draw frame by frame: CPU Canvas (compatible but slow) vs GPU offscreen (fast but complex);

3. Composite with the live picture: View stacking (simple but poor frame rate) vs offscreen FBO (optimal but you manage GL) vs SDK callback (recommended);

4. Optimize: lower resolution, reuse textures, PBO async upload, premultiplied alpha, clip invalid regions, async decode, object pools;

5. Debug: black screen → check GL error codes, corruption → check thread contention, leak → check texture release.

In one line: rendering an effect is the job of turning an animation file into a GPU texture every frame, blending it with the live picture, and doing so without choking the main thread, exhausting VRAM, or dropping frames.

Worth adding: a large share of these problems is decided long before any code runs — at the design and export stage. An effect authored at a sensible resolution, with clean transparency and a reasonable layer count, is dramatically cheaper to render than one that merely looks the same. That’s what we focus on at XingJi: gift effects built to render smoothly on real devices, delivered in whichever format (SVGA, VAP, or PAG) fits your stack. If you’re weighing which format or compositing approach suits your app, we’re happy to talk it through.