+4 votes
1.4k views
Fix kernel panic - not syncing error on Ubuntu server

in Guides by (409k points)
reopened | 1.4k views

2 Answers

+5 votes
Best answer

The origin of the error.
Recover the system once the kernel panic - not syncing error has been identified with Finnix.
Recovering the system and booting with Grub.
Booting Finnix on a PC.
Booting Finnix in Vultr.
Using the Finnix terminal to make a backup of the files.

I did not know very well what title to give to this article, since I will collect a series of instructions with which to recover a system that returns a kernel panic error . This type of error is well known and quite frustrating, since the operating system does not start, remaining inaccessible. Specifically we will talk about the solution when we find the error message:

kernel panic - not syncing: VFS: Unable to mount root fs on unknown - block (0,0).

image

The origin of the error.

In my case the problem came after updating an Ubuntu server with the usual commands: sudo apt-get update and sudo apt-get upgrade . After the update, I saw a message on the screen asking me to restart the system.

My surprise was that after rebooting, I came across the error message "kernel panic - not syncing: VFS: Unable to mount root fs on unknown - block (0,0)" and there began the stress. It was a VPS server with the operating system Ubuntu Server 16.04 (64 bits) hired in the company Vultr.

Although it is a server, these instructions will work on home computers and will also work on servers from other companies. The solution was found in this post from the Ubuntu forums, but I will try to give more details on the subject.

Recover the system once the kernel panic - not syncing error has been identified with Finnix.

Once I had the error identified, the first thing that went through my head is to create a backup of those important files . This step is almost mandatory , because if for some reason the recovery process goes wrong, we will always have a backup.

As the operating system does not boot, we must resort to a LiveCD recovery. There are many, but the one I liked the most is Finnix , based on Debian.

Booting Finnix on a PC.

If we are going to use it on a PC, we will have to create a bootable USB pendrive with this LiveCD or we can burn it directly to a CDROM. Once it starts, we must follow the instructions on the screen to get a shell in the system.

Booting Finnix in Vultr.

I have chosen Finnix, because it is also available in Vultr and we can upload it as a custom ISO . To load Finnix, we access the Vultr control panel, then select the server we want to recover and go to the Settings → Custom ISO and select Finnix - 111 x86 . Finally we press the Attach ISO and Reboot .

image

Now we will wait for the ISO transfer to finish and click on the View console button.

image

At this point we will have on screen something similar to what we see in the following image.

image

We only have to select the architecture with which we want to start Finnix . In my case and in most modern systems, we will select "Boot Finnix (64-bit)" . We wait for all modules to load in memory and after a few seconds, we will have a terminal from which to work to recover the system.

Recovering the system and booting with Grub.

We start by mounting the operating system files that we want to recover. For this, we execute the sudo fdisk -l command and see what the system boot partition is.

image

In my case the partition is /dev/vda1 , so now I have to mount this partition with the command:
sudo mount /dev/vda1 /mnt

Using the Finnix terminal to make a backup of the files.

In my case the important files will be inside the folder /mnt/home/zeokat/www , so the first step is to compress this folder with the command:
tar -zcf backupfiles.tar.gz /mnt/home/zeokat/www

The next step was to transfer the backup file to my PC, for which we can use SFTP or another method you can think of. To be able to access via SFTP, we have to assign a password to the root user , for which we execute the passwd command.

image

Next, we start the SSH server with the command:
/etc/init.d/ssh start

image

Now we can use our favorite SFTP client to download the backup files.

We already have the important files backed up, so we will try to recover the system so that it starts correctly. Now we execute the following commands:

sudo mount --bind /dev /mnt/dev
sudo mount --bind /dev/pts /mnt/dev/pts
sudo mount --bind /proc /mnt/proc
sudo mount --bind /sys /mnt/sys

To finish, we execute:

sudo chroot /mnt
update-initramfs -u -v
grub-install --recheck /dev/vda
update-grub
reboot

If you are on a Vultr server, when you execute the reboot command, a message will surely appear on the screen saying "Running in chroot, ignoring request" . So you must first execute the exit command and then you can execute the reboot command. Finally, we use the button that says Remove ISO in the Settings → Custom ISO of the Vultr control panel.


by (2.6m points)
+4 votes

How to troubleshoot kernel panic error in linux

How to troubleshoot kernel panic error in linux

I'm trying to run Debian Linux on a Nokia IP710 firewall, but I can't boot properly - the kernel panics all the time.

The firewall has a 1Ghz Pentium III and 512MB of RAM. It has no vga, a very limited bios (ie no pxe / net boot ), and it has no cdrom drive or usb ports. Its operating system was Nokia IPSO, based on FreeBSD.

IPSO was started from a 32MB CF card that the bootloader was on, and the OS was on the 40GB hard drive.

I installed Debian (Wheezy) from within qemu on a 4GB compact flash, and then on a 32MB hard drive as / boot. Both OS work fine when booted within qemu or on an x86 thin client.

However, they are blocked on every startup on the firewall with those logs:

[ 2.676304] BUG: unable to handle kernel paging request at ffff8000 [ 2.680166] IP: [<c1010aec>] io_apic_write+0xb/0x13 [ 2.680166] *pde = 01442067 *pte = 00000000 [ 2.680166] Oops: 0002 [#1] [ 2.680166] Modules linked in: [ 2.680166] [ 2.680166] Pid: 1, comm: swapper Not tainted 3.2.0-4-486 #1 Debian 3.2.32-1 [ 2.680166] EIP: 0060:[<c1010aec>] EFLAGS: 00010086 CPU: 0 [ 2.680166] EIP is at io_apic_write+0xb/0x13 [ 2.680166] EAX: ffff8000 EBX: 00000031 ECX: 01000000 EDX: ffff8000 [ 2.680166] ESI: 01000000 EDI: 0001a929 EBP: 00000002 ESP: dec2dea0 [ 2.680166] DS: 007b ES: 007b FS: 0000 GS: 00e0 SS: 0068 [ 2.680166] Process swapper (pid: 1, ti=dec2c000 task=dec1ec00 task.ti=dec2c000) [ 2.680166] Stack: [ 2.680166] 00000010 00000002 c1010b5f 00000246 0001a929 01000000 c1010b95 0001a929 [ 2.680166] 01000000 00000010 66858080 dec2df20 c1315489 00000010 c1011974 0001a929 [ 2.680166] 01000000 00000001 00000000 00000002 0001a929 01000000 c1448d44 00000010 

and after call trace:

 [ 2.680166] EIP: [<c1010aec>] io_apic_write+0xb/0x13 SS:ESP 0068:dec2dea0 [ 2.680166] CR2: 00000000ffff8000 [ 2.680166] ---[ end trace 6f619dcb164ee6d0 ]--- [ 2.680206] Kernel panic - not syncing: Attempted to kill init! [ 2.684175] Pid: 1, comm: swapper Tainted: GD 3.2.0-4-486 #1 Debian 3.2.32-1 

I also tried Kernel 3.2.35, with the same result.

There are also a bunch of PCI-related error messages:

 [ 2.467037] pci 0000:00:02.0: address space collision: [mem 0xff000000-0xff03ffff pref] conflicts with 0000:00:01.0 [mem 0xff000000-0xff03ffff pref] [ 2.488165] pci 0000:04:06.0: no compatible bridge window for [mem 0xff000000-0xff03ffff pref] [ 2.620180] pci 0000:00:03.0: enabling device (0000 -> 0003) [ 2.624179] pci 0000:00:03.0: can't find IRQ for PCI INT A; probably buggy MP table [ 2.628186] pci 0000:00:03.1: enabling device (0000 -> 0003) [ 2.632174] pci 0000:00:03.1: can't find IRQ for PCI INT B; probably buggy MP table 

And 512 of these posts (with pin 0-255, twice):

 [ 1.172083] Unable to reset IRR for apic: 255, pin :3 

I would like to know how to fix these lockings. Most of the documentation I've read (like Documentation/oops-tracing.txt) doesn't mention the case when the kernel fails to boot.

The full login log saved from the console is here.

Edit: As per @jordanm's suggestion, I ran memtest overnight (11 hours). It worked fine without any errors:

      Memtest86+ v4.20 | Pass100% ####################################### Pentium III 999.6MHz | Test 81% ############################### L1 Cache: 16K 9342 MB/s | Test #8 [Modulo 20, Random pattern] L2 Cache: 256K 4253 MB/s | Testing: 196K - 512M 512M L3 Cache: None | Pattern: ab46c0e3-17 Memory : 512M 406 MB/s |------------------------------------------------- Chipset : CNB20HE (ECC : Detect / Correct) Scrub+ WallTime Cached RsvdMem MemMap Cache ECC Test Pass Errors ECC Errs --------- ------ ------- -------- ----- --- ---- ---- ------ -------- 11:06:12 512M 0K e820 on off Std 9 0 ----------------------------------------------------------------------------- 

    by (409k points)

    Related questions

    +5 votes
    1 answer
    asked Aug 10, 2019 in Guides by backtothefuture (409k points) | 190 views
    +5 votes
    1 answer
    asked Aug 9, 2019 in Guides by backtothefuture (409k points) | 124 views
    +3 votes
    1 answer
    asked Aug 10, 2019 in Guides by backtothefuture (409k points) | 153 views
    +3 votes
    1 answer
    asked Aug 10, 2019 in Guides by backtothefuture (409k points) | 191 views
    +3 votes
    1 answer
    7,849 questions
    7,949 answers
    400 comments
    2 users