si accettano suggerimenti per il titolo

storie, pensieri e progetti tra chimica, informatica, attualità e cazzeggio puro

Recover a non-booting QNAP TS-11x device

Overview

Recently Debian testing has received a kernel update configured in a way that prevents QNAP devices based on the kirkwood platform from booting. If you have installed this update and your NAS is therefore broken, this guide will help you to recover it without any data loss.

While it’s best to be sitting in front of the offended device, this is not compulsory. As long as there’s someone near the NAS who’s capable to at least press a button, this recovery procedure can be executed remotely, provided you have ssh access to the machine that you’ll use to recover the device. The author of this guide has fixed his device while sitting at the other end of the world from the device.
The simplest configuration for this operation is to have the NAS directly connected to the recovery machine with a wire, but this guide also work if there’s a router/switch between them, provided the router’s dhcp server can be disabled. This is the only way to work if you have to recover the device from remote, because you’ll need an Internet connection.

Requirements and Preparation

In order to recover your QNAP device, you need the following:

  • A non-booting QNAP TS-110, TS-112, TS-119, TS-119P+, TS-119P II, TS-120 or TS-121. This guide should also apply to TS-2xx and TS-4xx devices, provided they have a similar hardware.
  • A second machine on which a tftp and dhcp server can be installed, together with a ssh client. You will also need some UNIX console programs, like cat and dd. This machine can run any operative system, but this guide will assume that Linux is executed on the machine. It is beyond the aim of this guide to explain how to install these software on different operative systems.
  • A backup of your flash partitions, as explained here. If you’ve made a backup of your flash partitions after installing Debian on the device you’ll need little work to recover your NAS, while if you have only made a backup of the original QNAP firmware (as it was in my case) then you’ll need to work a little bit more. If you don’t have any backup then you’re out of luck, and you’ll need to find someone with your same device who can give you those files before you proceed. Before tampering with flash partitions, you should really have made backups!

Prepare the image to be flashed on the device

If you have the backup of the flash partitions, then you have six files that will be called in ths guide mtdX, with X varying from 0 to 5. If you made the backup from a running Debian installation, you can use all your files to create the recovery image with this command (pay attention to the order of the numbers):

cat mtd0 mtd4 mtd5 mtd1 mtd2 mtd3 > F_TS-XXX

The name of the generated file varies depending on the model of your device. In the following table is explained the exact string to use depending on the model:

Device String
TS-110 F_TS-110
TS-112 F_TS-112
TS-119 F_TS-119 or (for older revisions) F_TS-119PE
TS-119P+ F_TS-119P+
TS-119P II F_TS-119P2+
TS-120 F_TS-120
TS-121 F_TS-121

If you are unsure of which string you should use, you can discover it with the following command:

strings mtd4 | grep bootp_vendor_class

If you didn’t make your backup from a running Debian installation, but you still have the backup of your original QNAP firmware, you first need to download the Debian installer init.rd and kernel files. After that, you have to modify the kernel file to match the size of the mtd1 partition:

dd if=kernel of=kernel.pad ibs=2097152 conv=sync

After modifying the kernel, you can create your recovery image:

cat mtd0 mtd4 mtd5 kernel.pad initrd.gz mtd3 > F_TS-XXX

Again, see the table above for the exact name of the generated file.

Configure your Linux recovery machine – Install the software

In order to be able to recover your NAS, some software need to be installed on the machine you’ll use to recover your device. If the machine will remain connected to a router and/or a LAN during the recovery, please disable the dhcp server of the router/LAN before enabling the one on the machine.
For the recovery you’ll need to install three packages: xinetd (or inetd), tftpd and the dhcp server. If you’re running Debian on the machine, you can simply execute this command as root:

apt-get install tftpd xinetd isc-dhcp-server

Configure your Linux recovery machine – Configure the tftp server

For some reasons, the tftp server package doesn’t come with a configuration file, and you’ll have to create one. To do so, create the file /etc/xinetd.d/tftp and write the following inside, without chaging anything:

# default: off
# description: The tftp server serves files using the Trivial File Transfer
# Protocol.  The tftp protocol is often used to boot diskless
# workstations, download configuration files to network-aware printers,
# and to start the installation process for some operating systems.
service tftp
{
    socket_type     = dgram
    protocol        = udp
    wait            = yes
    user            = root
    server          = /usr/sbin/in.tftpd
    server_args     = -s /tftpboot
    disable         = no
}

You now have to create the tftpboot directory and restart xinetd in order for the changes to become active:

mkdir /tftpboot
/etc/init.d/xinetd restart

You can now move the recovery image into the tftpboot directory and you can test if the tftp server is working installing the tftp client and downloading the recovery image with the command tftp F_TS-XXX.

Configure your Linux recovery machine – Configure the dhcp server

The dhcp configuration varies depending on if you connect the NAS directly to the machine or you connect both to a router/LAN. It is beyond the aim of this guide to help you configure the dhcp server, however a working sample configuration file (/etc/dhcp/dhcpd.conf) is presented here with comments as reference:

option domain-name "local"; # if you are on a domain, put the domain name here
option domain-name-servers 208.67.222.222, 208.67.220.220; # OpenDNS servers
default-lease-time 600;
max-lease-time 7200;
authoritative; # force the use of this dhcp server
option subnet-mask 255.255.255.0;
option broadcast-address 192.168.0.255;
option routers 192.168.0.1; # the IP of your router/gateway, if you have one
subnet 192.168.0.0 netmask 255.255.255.0 {
  range 192.168.0.3 192.168.0.200;
}
host ts110 {
  hardware ethernet 00:08:9b:be:75:26; # mac address of your NAS, see below
  filename F_TS-XXX; # change XXX according to the table
  fixed-address 192.168.0.70; # the IP address of your device during recovery
  next-server 192.168.0.2; # IP of the recovery machine
}

After you’ve modified the config file, restart the dhcp server for the changes to take place. You can do so with the following command as root:

/etc/init.d/isc-dhcp-server restart

To test if your configuration file works try to connect another machine to your recovery one or, if everything is connected to a router/LAN, disable the dhcp server of the router and add a machine to your LAN. In both cases, the new machine should get an IP address and you should have a DHCPDISCOVER message in /var/log/syslog.

Flash the recovery image on the device

If your device is still switched on, switch it off now (you may have to pull the power plug), connect it to the recovery machine (or to your router) and put it in recovery mode. To do so, find the reset button on the back of your device, press it and keep it pressed, switch on the device again and continue to press the reset button until you hear two short beeps. Once the device is in recovery mode check again /var/log/syslog in your recovery machine for DHCPDISCOVER messages: it’s the NAS trying to get an IP address. From these messages you’ll be able to find out the mac address of your device, which will probably start with 00:08:9B or 00:50:43. Once you’ve found it, edit again the dhcpd.conf to add the mac address to the host section of the file, switch off again the NAS and restart your dhcp server.

Switch on again the NAS in recovery mode. This time, it’ll get the IP address you chose for it and it’ll start downloading the recovery image from the tftp server. You can read the messages in /var/log/syslog to check if the download from the tftp server is working. Once the downloading is finished, the device will beep twice and restart.

Recover your device – Debian backup

If your recovery image came from the backup of a working Debian system, you’re done. When the NAS reboots it’ll load the kernel inside the recovery image and start the system. At this point, after reconnecting your NAS to the Internet, you may want to download this kernel package from the sid repository which contains the fix to make it work with your device and flash it with flash-kernel to get an up-to-date kernel on your device.

Recover your device – QNAP backup

If your backup came from the QNAP firmware and you needed to download the Debian installer kernel and initrd files to create the recovery image, you now need to access the Debian installer to recover your system after the NAS reboots. To do so, connect to the Debian installer on the device via ssh. First of all, if this is not the first time you connect via ssh to your NAS, remove the ~/.ssh/known_hosts file from your home directory on the recovery machine, because the Debian installer will create a new ssh key for the device. Rather than delete it, you can just move it to a different position and put it back when everything’s done. You can connect to the installer with the following command:

ssh installer@192.168.0.70

The password for the user is install. Of course, you’ll have to change the IP address with the one you wrote in your dhcpd.conf file.
When you connect you’ll be given three choices: select the last one to start a shell. Please keep in mind that from now on command autocompletion using the tab key will not work and you’ll have to type your command entirely. Copy/paste still work though, so you can use the text below to speed up the process.

Once you’ve loaded the shell, type the following commands to create a chroot environment, access your system and recover it. The following commands assume that the operative system in your NAS hard drive is installed in the partition /dev/sda2 – change it accordingly to your configuration.

mkdir -p /target
mount /dev/sda2 /target
mount /dev/sda1 /target/boot # only needed if /boot is in a different partition on the drive
mount --bind /dev /target/dev
mount -t proc none /target/proc
mount -t sysfs none /target/sys
chroot /target /bin/sh
# the prompt will change because you've entered the chroot environment
cd /root
wget http://ftp.debian.org/debian/pool/main/l/linux/linux-image-3.12-1-kirkwood_3.12.8-1_armel.deb
dpkg -i linux-image-3.12-1-kirkwood_3.12.8-1_armel.deb
flash-kernel
exit
# the prompt will change again as you're leaving the chroot
umount /target/sys
umount /target/proc
umount /target/dev
umount /target/boot
umount /target
reboot

The wget command above assumes that you can access the Internet on your device while recovering. If this is not the case, you need to download the deb package on your recovery machine first while it’s still connected to the Internet and then access the file from the NAS. You can do so by configuring a web server like LAMPP on your recovery machine, setting up a NFS share or any other way you may think of. It is beyond the aim of this guide to help you with this.

If your NAS is not connected to the Internet and you don’t want to set up anything on your recovery machine to install the newer kernel, you can still recover your device by deleting the broken kernel from the /boot directory and flashing the last-working kernel, provided you didn’t manually delete them all before installing the kernel that broke your system.
To do so, replace the wget and dpkg commands above with the following:

cd /boot
rm System.map-3.12-1-kirkwood config-3.12-1-kirkwood vmlinuz-3.12-1-kirkwood initrd.img-3.12-1-kirkwood
ls # to find the suffix of the last-working kernel
ln -sf initrd.img- initrd.img
ln -sf vmlinuz- vmlinuz
ln -sf System.map- System.map
ln -sf config- config

After you’ve finished recovering your device, it’s still recommended to download the newer kernel and install it.

Cleaning up

When the NAS reboots your original system will be accessible again, with the new, fixed kernel installed (or you can install it later on). At this point, you should uninstall the tftp and dhcp servers on your recovery machine and re-enable the dhcp server of the router if you need to. You can also delete the /tftpboot directory on your recovery machine.

Since this problem may happen again in the future, it is recommended to make a periodical backup of your flash partitions so that you’ll have fresh images to do the recovering.

8 commenti per "Recover a non-booting QNAP TS-11x device"

Please note that if, like me, you were foolish enough to run debian on an ext4 partition, you’ll need the unstable initrd.gz and kernel, or else you won’t be able to mount your harddisks.

    Oh, good to know. I used an ext3 partition, so I didn’t notice that. I’ll add it to the guide, thanks.

Joachim Petermann
rispondi

Very helpful to have the whole steplist in on place as it avoids to gather all steps from all over the web. Thanks for your work.
I have one addition to “Requirements and Preparation” , item 3: if there is no backup neither a debian related nor a firmware related available you can give “http://wiki.qnap.com/wiki/Firmware_Recovery” a try. For me it worked on a TS-119PII using “http://eu1.qnap.com/Storage/tsd/live_cd/live-cd-20130730.iso”.
BTW: I proceeded it via Method 2 but using a VirtualBox VM.

Hello friends, its impressive paragraph regarding teachingand fully defined, keep it up all the time.

Thank you so much for this, it helped me get my TS-419 back to life.

I had to use quotation marks for the file name, like this: “filename F_TS-XXX”;

I also had to try both ethernet ports before it worked.

If this thread is still alive:
Is the above “I had to use quotation marks for the file name, like this: “filename F_TS-XXX”;” correct? Since no changes in your guide I suppose the aformentioned is incorrect…?
You write, “If your backup came from the QNAP firmware and you needed to download the Debian installer kernel and initrd files”… What if I do not want to work with Debian anymore and I wish to give QNAP’s OS one more chance, what about that…? How do I proceed…?
Thanx!
George

    Hi George,
    I’m not sure the quote thing is an error – I didn’t have to do it when I recovered my nas. It could be a difference between models, it could be due to a software update. I guess you can try both commands, at worst one will fail.
    If you want to restore the original qnap firmware you can follow the instructions in this page. If you have a backup of the qnap firmware (which you should have) it’s super easy. If you don’t, there is a firmware recovery procedure described on the qnap website (and linked in the page I mentioned) but it’s a bit more complicated.
    A note of warning, though: I’ve put QTS back on my TS-110 three weeks ago and I wished I hadn’t: the qnap firmware is really a piece of crap. If you want to remove debian, by suggestion is that you keep debian (maybe do a clean install of stretch, the current stable) and then you install the openmediavault package. At the moment I’m testing it on a VM until I will be able to buy my new nas, and it’s waaaaay better than QTS.

Scrivi un commento

L'indirizzo e-mail non sarà pubblicato. I campi obbligatori sono marcati con *


Codice HTML consentito: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <s> <strike> <strong>