🐧 Custom Linux Kernel with a New System Call

This project demonstrates the process of customizing the Linux kernel by:

1. Printing a custom boot message.
2. Adding and implementing a new system call.
3. Building a minimal initramfs with BusyBox.
4. Running the kernel in QEMU.
5. Testing the syscall using a custom user-space program.

It is intended as a learning project for understanding Linux internals.

---

🔹 1. Cloning the Linux Kernel

We start by cloning the Linux kernel source from the official repository:

git clone --depth=1 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux.git
cd linux

Why use --depth=1?

• The full Linux kernel repository is massive (tens of gigabytes).
• A shallow clone with --depth=1 downloads only the latest snapshot instead of the complete history.
• This makes the clone much faster and saves disk space.
• Perfect for learning and experimentation when you don’t need historical commits.

---

🔹 2. Custom Boot Message

To confirm that we are running our own kernel, we modify the kernel entry point to print a message.

• Edit init/main.c:

asmlinkage __visible void __init start_kernel(void)
{
     printk(KERN_INFO "CUSTOM KERNEL BOOT BY PRAZWAL NIBHRIT AAYUSH ASHUTOSH\n ");
    ...
}

When the kernel boots in QEMU, this message will appear in the logs (dmesg).

EXPECTED RESULT
[    0.000000] CUSTOM KERNEL BOOT BY PRAZWAL NIBHRIT AAYUSH ASHUTOSH

---

🔹 3. Adding a Custom System Call

(a) Register the syscall in the syscall table

Edit arch/x86/entry/syscalls/syscall_64.tbl and add:

548    common   my_syscall    sys_my_syscall

• 548 → syscall number (unused slot).
• my_syscall → user-visible name.
• sys_my_syscall → kernel implementation function.

(b) Implement the syscall

Create kernel/my_syscall.c:

#include <linux/kernel.h>
#include <linux/syscalls.h>

SYSCALL_DEFINE0(my_syscall)
{
    printk(KERN_INFO "my_syscall(): called by userspace (Prazwal nibhrit Aayush Ashutosh)\n");
	        
    return 1234; // return something 
}

• SYSCALL_DEFINE0 → means syscall takes 0 arguments.
• printk → logs a message in the kernel log.
• Returns 1234 to user space for verification.

(c) Add to the kernel build system

Edit kernel/Makefile and append:

obj-y += my_syscall.o

This ensures our syscall implementation is compiled.

---

🔹 4. Building the Kernel

Compile the kernel:

make defconfig
make menuconfig   # (optional, for custom configs) DESELCET NETOWORK , USB DRIVERS AND OTHER DRIVERS FOR FASTER COMPILATION 
make -j$(nproc) bzImage

The compiled kernel image will be created at:

arch/x86/boot/bzImage

---

🔹 5. Creating a Minimal Initramfs

We need an initramfs so the kernel has something to run after boot.

(a) Create folder structure

mkdir -p ~/initramfs/{bin,sbin,etc,proc,sys,dev,usr}

(b) Add BusyBox

BusyBox provides basic utilities like sh.

cp /bin/busybox initramfs/bin/
chmod +x initramfs/bin/busybox
ln -s /bin/busybox initramfs/bin/sh

(c) Create init script

File: initramfs/init

#!/bin/sh
echo "Prazwal Nibhrit Aayush Ashutosh  Custom Linux Kernel is now booting!"
exec /bin/busybox sh

Make it executable:

chmod +x initramfs/init

---

🔹 6. User Program to Test the Syscall

Inside initramfs, we place a small program to invoke our new syscall.

File: initramfs/bin/test.c

#include <unistd.h>
#include <sys/syscall.h>
#include <stdio.h>

#define __NR_my_syscall 548  // Must match syscall_64.tbl

int main() {
    long result;

    printf("Calling custom system call...\n");
    result = syscall(__NR_my_syscall);
    printf("Syscall returned: %ld\n", result);

    return 0;
}

Compile statically so it runs inside initramfs without shared libraries:

gcc -static initramfs/bin/test.c -o initramfs/bin/test

---

🔹 7. Building the Initramfs Image

cd ~/initramfs
find . -print0 | cpio --null -ov --format=newc | gzip -9 > ~/initramfs.cpio.gz

This generates initramfs.cpio.gz, which will be loaded by QEMU.

---

🔹 8. Running the Kernel in QEMU

Boot your custom kernel with initramfs:

qemu-system-x86_64   -kernel ~/linux/arch/x86/boot/bzImage   -initrd ~/initramfs.cpio.gz   -append "console=ttyS0 init=/init loglevel=7"   -nographic

• console=ttyS0 → routes kernel logs to QEMU terminal.
• init=/init → tells kernel which init script to execute.
• loglevel=7 → maximum log details.
• -nographic → disables GUI, runs in terminal.

---

🔹 9. Testing the Custom Syscall

Inside QEMU:

/bin # ./test

Expected output in user-space:

Calling custom system call...
Syscall returned: 1234

Check kernel logs:

dmesg | tail

Output should include:

called by userspace (Prazwal nibhrit Aayush Ashutosh)

---

🔹 Common Errors & Fixes

❌ Error: Kernel panic - not syncing: No init found
✔️ Fix: Ensure initramfs/init exists and is executable (chmod +x initramfs/init).

❌ Error: ./test: not found inside QEMU
✔️ Fix: Compile with gcc -static so the binary doesn’t need shared libraries.

❌ Error: Wrong syscall return / Invalid argument
✔️ Fix: Ensure syscall number in test.c matches entry in syscall_64.tbl.

❌ Error: BusyBox not working
✔️ Fix: Ensure BusyBox binary is copied into initramfs/bin and symlinked to /bin/sh.

---

✅ Summary

• Modified start_kernel() → printed a custom boot message.
• Added a new syscall (my_syscall) → returns 1234 and logs a message.
• Built a minimal initramfs with BusyBox + test program.
• Booted the kernel in QEMU and successfully invoked the syscall.

✨ With this setup, you now have a custom Linux kernel and a working system call running inside QEMU.
Project created by Prazwal ,Nibhrit and Aayush