Building a Content Moderation Queue with Laravel Jobs and the AI SDK

Manual moderation queues don't scale, and simple keyword filters miss anything that isn't a blocklisted word. This tutorial builds a middle ground for PhpArchitect's community comment section: every new comment is queued for an AI classification pass immediately after submission, and only comments flagged as risky wait for a human. Everything else publishes instantly.

Step 1: The schema

Schema::table('comments', function (Blueprint $table) {
    $table->string('moderation_status')->default('pending')->after('body');
    $table->json('moderation_result')->nullable()->after('moderation_status');
});

moderation_status moves through pending → approved or pending → flagged, and moderation_result keeps the raw classification for a human reviewer to see why something was flagged.

Step 2: The classification job

namespace App\Jobs;

use App\Models\Comment;
use Illuminate\Contracts\Queue\ShouldQueue;
use Illuminate\Foundation\Queue\Queueable;
use Laravel\Ai\Facades\Ai;
use Laravel\Ai\Messages\SystemMessage;
use Laravel\Ai\Messages\UserMessage;

class ModerateComment implements ShouldQueue
{
    use Queueable;

    public int $tries = 3;
    public int $backoff = 10;

    public function __construct(private readonly Comment $comment) {}

    public function handle(): void
    {
        $response = Ai::json()->generate([
            new SystemMessage(
                'You moderate community comments for a developer forum. '.
                'Classify the comment and respond with JSON only: '.
                '{"flagged": boolean, "reason": string|null, "category": "harassment"|"spam"|"none"}'
            ),
            new UserMessage($this->comment->body),
        ]);

        $result = $response->json();

        $this->comment->update([
            'moderation_status' => $result['flagged'] ? 'flagged' : 'approved',
            'moderation_result' => $result,
        ]);
    }
}

Requesting JSON mode explicitly (Ai::json()) matters here — without it, a model will sometimes wrap its answer in explanatory prose, and parsing that reliably is exactly the kind of brittle string-matching this approach is meant to avoid.

Step 3: Publish optimistically, dispatch the check

The comment should appear to the author immediately — don't make them wait on an AI round trip to see their own comment. Publish it in a provisional state and let the job catch problems shortly after:

namespace App\Http\Controllers;

use App\Jobs\ModerateComment;
use App\Models\Comment;
use Illuminate\Http\Request;

class CommentController extends Controller
{
    public function store(Request $request)
    {
        $validated = $request->validate(['body' => 'required|string|max:2000']);

        $comment = Comment::create([
            'user_id' => $request->user()->id,
            'body' => $validated['body'],
            'moderation_status' => 'pending',
        ]);

        ModerateComment::dispatch($comment);

        return back()->with('status', 'Comment posted!');
    }
}

Comments render as visible-to-everyone once moderation_status is pending or approved, and are hidden from anyone but the author and moderators once flagged:

public function scopeVisibleTo($query, $user)
{
    return $query->where(function ($query) use ($user) {
        $query->whereIn('moderation_status', ['pending', 'approved'])
            ->orWhere('user_id', $user?->id);
    });
}

Step 4: Handling the retry and failure case

AI providers time out and rate-limit. $tries = 3 on the job gives the classification call a few attempts before giving up, but decide deliberately what happens if all three fail — leaving a comment stuck in pending forever, invisible to everyone but its author, is a worse outcome than failing open:

public function failed(\Throwable $exception): void
{
    // Fail open: publish the comment and flag it for a human to spot-check,
    // rather than leaving it stuck in limbo.
    $this->comment->update([
        'moderation_status' => 'approved',
        'moderation_result' => ['flagged' => false, 'reason' => 'moderation_check_failed'],
    ]);
}

Step 5: A lightweight review queue for flagged comments

// routes/web.php
Route::get('/moderation', function () {
    return view('moderation.index', [
        'comments' => Comment::where('moderation_status', 'flagged')
            ->with('user')
            ->latest()
            ->paginate(20),
    ]);
})->middleware('can:moderate-comments');

A moderator sees only the comments the model actually flagged, with the stored moderation_result explaining why, rather than reviewing every comment that comes in. From here, the natural extension is feeding moderator overrides (approve a false positive, confirm a true positive) back into a running accuracy log, so you have real data on whether the flagging threshold needs adjusting.