As a software engineer, we've all gone through this phase: you write a new feature, test it in a local environment with dummy data for 10 or 20 rows, and it all goes very fast. The API response time is only around 50 milliseconds. You are satisfied, do a commit, make a Pull Request, and the code is finallydeploy to production.
One week later, a spike in traffic occurred. Suddenly, your monitoring dashboard is filled with red. The app's response time jumped to 5 seconds. CPU usage on database server touched 90%. Users started complaining that the app felt very slow. Once you do the debugging process and look at the query log, you find thousands of the same SQL queries executed over and over again in one request.
Welcome to the real world. You've just fallen victim to one of the most classic and deadly architectural mistakes in software development: N+1 Query Problem.
This article will not only discuss what N+1 is superficially. We'll dissect it down to a fundamental level, looking at how this plays out in the PHP ecosystem—both using vanilla PHP (PDO) and modern ORMs like Eloquent in Laravel—and how we as engineers detect, prevent, and manage trade-offof existing solutions.
Anatomy of an N+1 Problem: Understanding the Root of the Problem
By definition, the N+1 problem is a database access inefficiency condition where your application executes 1 initial query to retrieve the parent record, and then executes N additional queries to retrieve each child record or relation from each parent individually in a loop.
Let's translate it into human language with an analogy.
Imagine you are a manager (PHP Application) who needs report data from 100 employees (Database).
The N+1 approach is that you call the 100 employees one by one to your room. The first employee comes in, you ask for his report, then he leaves. The second employee comes in, you ask for his report, then he walks out. You do this 100 times (plus 1 initiation of summoning). How much time is wasted just opening and closing doors and walking back and forth? That's what Network Latency and Database Round-trip Overhead.
The correct approach is for you to gather all 100 employees in a meeting room at the same time, and ask for all their reports at one time. This is equivalent to executing 1 or 2 efficient queries SQL.
Mathematically, if you have 100 parent records (N=100), then your application will do:
-
1 query to retrieve all parent
-
100 query to retrieve each parentrelationshipTotal: 101 SQL Queries executed to render only one page. If your traffic is 1,000 request per minute, database You should process 101,000 queries per minute. This is the perfect recipe for Downtime.
Let's Talk Code: Real Examples in Vanilla PHP (PDO)
Before we blame ORMs, we have to understand that this problem stems from a basic structural mindset. Let's see how a junior developer usually writes code to display a list of articles along with their author names using raw PHP and PDO.
Bad Case (The Anti-Pattern)
PHP
// Assuming a PDO connection already exists in the $pdo variable
// 1. Fetch the 100 most recent articles (This is the "1" part of N+1)
$stmt = $pdo->query("SELECT id, title, content, author_id FROM articles ORDER BY created_at DESC LIMIT 100");
$articles = $stmt->fetchAll(PDO::FETCH_ASSOC);
$result = [];
// 2. Loop through each article (This is the "N" part of N+1)
foreach ($articles as $article) {
// Execute query for EVERY loop iteration!
$authorStmt = $pdo->prepare("SELECT id, name, email FROM authors WHERE id = :author_id");
$authorStmt->execute(['author_id' => $article['author_id']]);
$author = $authorStmt->fetch(PDO::FETCH_ASSOC);
$result[] = [
'title' => $article['title'],
'content' => $article['content'],
'author_name' => $author['name']
];
}
// Render JSON or HTML...
Procedural logic, the code above is very easy to read. "Take the articles, then for each article, take the author." However, infrastructure-wise, this code is a disaster. If there are 100 articles, there will be 101 queries sent to MySQL/PostgreSQL over the network (TCP/IP).
Each query requires:
-
Process parsing and compilation by database engine.
-
Temporary memory allocation.
-
Data transfer (I/O) time from database server to application server.
Senior Solution: Eager Loading with WHERE IN
As engineers who think about performance, we have to do batching (grouping). We're going to fetch the entire article, collect all the necessary
author_ids, then do JUST ONE additional query to retrieve all those authors, then map them (mapping) at the application level (PHP).PHP
// 1. Fetch 100 latest articles
$stmt = $pdo->query("SELECT id, title, content, author_id FROM articles ORDER BY created_at DESC LIMIT 100");
$articles = $stmt->fetchAll(PDO::FETCH_ASSOC);
// If there are no articles, immediately return an empty array to prevent errors
if (empty($articles)) {
return [];
}
// 2. Collect all unique author_ids from the previous article
$authorIds = array_unique(array_column($articles, 'author_id'));
// 3. Create a placeholder for the WHERE IN query (?, ?, ?)
$placeholders = implode(',', array_fill(0, count($authorIds), '?'));
// 4. Execute ONE additional query to get all authors at once
$authorQuery = "SELECT id, name FROM authors WHERE id IN ($placeholders)";
$authorStmt = $pdo->prepare($authorQuery);
$authorStmt->execute(array_values($authorIds));
$authors = $authorStmt->fetchAll(PDO::FETCH_ASSOC);
// 5. Indexing author array based on ID for fast search (O(1) time complexity)
$authorMap = [];
foreach ($authors as $author) {
$authorMap[$author['id']] = $author;
}
// 6. Combine data in PHP memory
$result = [];
foreach ($articles as $article) {
$authorName = isset($authorMap[$article['author_id']]) ? $authorMap[$article['author_id']]['name'] : 'Unknown';
$result[] = [
'title' => $article['title'],
'author_name' => $authorName
];
}
Comparison:
The first code executes 101 queries.
The second code executes exactly 2 queries (regardless of the number of articles). We move the computing load from the Database (I/O) to the PHP Application's Memory (CPU), which is much cheaper and faster. The hash map array operation in PHP takes microseconds, compared to database network latency which can take tens of milliseconds per query.
The Illusion of ORM Ease and the Pitfalls of Lazy Loading
Today, it is rare to write complex applications using pure PDO. We use a framework with an ORM ecosystem (like Laravel with its Eloquent). ORMs were created to speed up the development process, making the database feel like object-oriented (OOP).
However, ORMs sweep the complexity of database under the carpet, and this often catches developers off guard. The main problem with N+1 in ORM stems from a feature called Lazy Loading.
Lazy Loading: The Silent Killing Feature
Lazy loading means that relational data is only loaded from the database when it is accessed (called) in the code.
Let's look at an example in Laravel (Eloquent):
PHP
// Controller
$articles = Article::latest()->limit(100)->get(); // Execute 1 Query
// View (Blade Template)
@foreach ($articles as $article)
{{ $article->title }}
Written by: {{ $article->author->name }}
@endforeach
The above code looks very clean, beautiful and elegant. No messy SQL syntax. But here's the danger: Behind the scenes, Eloquent is executing 101 queries!
Eloquent detected that the
author relationship has not been loaded. So, in the first iteration, it runs SELECT * FROM authors WHERE id = 1. In the second iteration, it runs SELECT * FROM authors WHERE id = 2, and so on. Because this happens magically in the background, it's very easy for developers to not notice until the app feels slow.ORM Solution: Eager Loading
The solution in Eloquent is very simple. We use a feature called Eager Loading using the
with() method.PHP
// Controller
// We instruct Eloquent to load the 'author' relationship AT ONCE.
$articles = Article::with('author')->latest()->limit(100)->get();
// View
@foreach ($articles as $article)
{{ $article->title }}
Written by: {{ $article->author->name }}
@endforeach
By adding
->with('author'), Eloquent behind the scenes will do the exact same thing as the manual WHERE IN code that we wrote about in the previous Vanilla PHP section. It will run 2 queries:-
SELECT * FROM articles ORDER BY created_at DESC LIMIT 100 -
SELECT * FROM authors WHERE id IN (1, 2, 3, ...)
Then Eloquent will assemble the relations (hydrating the models) into PHP's memory. Simple, right?
Advanced Problem: Nested N+1 Relationships
As senior engineers, our job is not only to solve problems that appear on the surface, but also to anticipate edge cases. What if author has a profile relationship (such as a profile photo or bio)?
Anti-Pattern:
PHP
$articles = Article::with('author')->get();
foreach ($articles as $article) {
// author has been loaded, it's safe.
// BUT, author->profile has NOT been loaded yet! N+1 happens again here.
echo $article->author->profile->avatar_url;
}
Solusi: Eager load bertingkat (Nested Eager Loading).
PHP
$articles = Article::with('author.profile')->get();
// Menjalankan 3 Query: Articles, Authors (WHERE IN), Profiles (WHERE IN)
Trade-off: Eager Loading vs Memory Exhaustion (Kelelahan Memori)
Dalam dunia software engineering, tidak ada silver bullet. Eager loading memecahkan masalah latensi dan jumlah koneksi, namun menciptakan potensi masalah baru: Penggunaan RAM (Memory) berlebih.
Misalnya Anda memiliki fitur ekspor data. Anda ingin mengekspor 50.000 artikel beserta penulis dan kategorinya ke dalam file Excel.
Jika Anda melakukan:
PHP
$articles = Article::with(['author', 'category'])->get(); // Mengambil 50.000 data
Ini memang hanya akan mengeksekusi 3 query. Namun, PHP harus mengalokasikan RAM untuk menampung 50.000 objek Article, ditambah objek Author, dan objek Category. Jika memori limit PHP Anda adalah 128MB atau 256MB, aplikasi akan crash dengan error:
Fatal error: Allowed memory size of X bytes exhausted.Solusi: Chunking (Pemotongan)
Untuk operasi dalam jumlah masif, Anda tidak boleh me-load semua data ke memori sekaligus. Anda harus menggunakan teknik Chunking.
PHP
Article::with(['author', 'category'])->chunk(500, function ($articles) {
// Proses 500 artikel pada satu waktu
foreach ($articles as $article) {
// Tulis ke CSV atau jalankan proses logik
}
// RAM akan dibebaskan oleh Garbage Collector PHP setelah scope function selesai
});
Dengan metode ini, Anda menjaga pemakaian memori tetap rendah dan stabil, sambil tetap menghindari N+1 problem pada setiap bongkahan (chunk) sebesar 500 baris data tersebut.
Bagaimana Senior Engineer Memastikan N+1 Tidak Lolos ke Production?
Seorang engineer yang berpengalaman tahu bahwa manusia bisa membuat kesalahan. Mengandalkan ingatan untuk selalu menulis
with() sangat rentan terhadap kegagalan. Kita harus membangun sistem pertahanan (defense in depth).Berikut adalah praktik standar industri untuk mendeteksi dan mencegah N+1:
1. Gunakan Tooling Profiling di Development
Jangan pernah mengembangkan aplikasi backend tanpa profiler. Di ekosistem Laravel, Anda WAJIB menginstal paket seperti:
-
Laravel Debugbar: Akan memunculkan bar di layar bawah browser yang menunjukkan persis berapa query yang dijalankan per halaman, lengkap dengan SQL syntax dan eksekusi waktunya.
-
Laravel Telescope atau Clockwork: Untuk memonitor request API secara mendalam.
Jika Anda membuka sebuah halaman dan melihat "150 queries executed", insting Anda harus langsung menyala: "Ini pasti N+1".
2. Fitur 'Prevent Lazy Loading' (Strict Mode)
Sejak Laravel versi 8.43, framework ini memperkenalkan fitur yang revolusioner: menonaktifkan lazy loading secara paksa.
Tambahkan kode ini di
AppServiceProvider.php pada metode boot():PHP
use Illuminate\Database\Eloquent\Model;
public function boot()
{
// Hanya matikan lazy loading di environment lokal dan testing.
// Jika ada yang mencoba mengakses relasi yang tidak di-load via with(),
// Laravel akan melemparkan Exception yang membuat aplikasi error dengan jelas.
Model::preventLazyLoading(! app()->isProduction());
}
Dengan fitur ini, jika anggota tim Anda lupa menambahkan
with('author') dan mencoba mengakses $article->author->name, layar akan langsung menampilkan error exception saat tahap development. Bug ini dipaksa muncul ke permukaan sebelum sempat di-commit ke repositori. Ini adalah game changer dalam menjaga standar codebase dalam tim yang besar.3. Log Slow Queries di Level Database
Di level infrastruktur production, pastikan MySQL/PostgreSQL Anda dikonfigurasi untuk mencatat slow query logs. Namun ingat, N+1 problem terkadang tidak tercatat di slow query log karena masing-masing query individual tersebut dieksekusi sangat cepat (misal 1ms). Masalahnya adalah akumulasi dari jumlahnya. Untuk itu, Application Performance Monitoring (APM) seperti New Relic, Datadog, atau Sentry jauh lebih efektif karena mereka bisa melacak "Database Time" secara agregat per Request.
Kapan N+1 Itu Diperbolehkan? (The Edge Cases)
Apakah N+1 selalu 100% haram? Sebagai senior, kita diajarkan untuk bersikap pragmatis, bukan dogmatis. Ada kasus yang sangat spesifik dan langka di mana membiarkan lazy loading / N+1 mungkin lebih menguntungkan, yaitu ketika diiringi dengan strategi Aggressive Caching di level aplikasi.
Katakanlah Anda memiliki objek relasi yang sangat berat dan kompleks, tetapi data tersebut sangat jarang berubah, dan Anda menggunakan Redis atau Memcached. Terkadang, melakukan query individual untuk mencari di Cache lebih dulu (dan jika luput, baru ke database) bisa lebih efisien ketimbang memaksakan tabel JOIN yang sangat masif di database relasional yang mengakibatkan table lock. Namun, kasus seperti ini adalah pengecualian yang harus dibuktikan dengan uji beban (load testing), bukan aturan praktis sehari-hari.
Kesimpulan: Mindset Seorang Engineer Profesional
Menulis kode yang berjalan (works) adalah tugas seorang pemula. Menulis kode yang terukur (scalable), efisien, dan dapat dirawat (maintainable) adalah tugas seorang engineer profesional.
N+1 Query Problem adalah cerminan dari kurangnya pemahaman developer mengenai batas antara kode aplikasi dan interaksi infrastruktur basis data. Ketika kita menulis kode:
-
Selalu asumsikan database berada di benua yang berbeda (untuk melatih kepekaan terhadap latensi).
-
Gunakan alat (tooling) yang tepat agar kelemahan arsitektur terlihat jelas selama pengembangan.
-
Pahami bagaimana ORM Anda menghasilkan query SQL di belakang layar. ORM adalah alat bantu, bukan pengganti SQL.
Mencegah masalah performa sejak dari fase development jauh lebih murah (baik dari segi waktu maupun biaya server) daripada melakukan proses debugging aplikasi yang sedang down di production pada jam 2 pagi. Lindungi aplikasi Anda dari N+1 problem, gunakan pemuatan awal (eager loading) dengan bijak, pantau memori Anda, dan pastikan pengguna Anda selalu mendapatkan waktu respons aplikasi dalam hitungan milidetik.

Sigit Wasis Subekti
Software Engineer & Tech Educator
Software Engineer and Tech Educator sharing insights on web development and software architecture.