OpenStack/Nova Rescue Mode

The SysEleven Rescue Image

The rescue images is built and tested using Ubuntu 18.04 LTS as core system.
They contain a variety of pre-installed tools to rescue your instances and supports most filesystems out of the box.

Goal

• Launching Nova rescue mode for an instance
• Repairing the (file)system and extract data if required

Prerequisites

• You should be able to use simple heat templates, like shown in the first steps tutorial.
• You know the basics of using the OpenStack CLI-Tools.
• Environment variables are set, like shown in the API-Access-Tutorial.

Optional: temporary work environment

wget https://raw.githubusercontent.com/syseleven/heat-examples/master/kickstart/kickstart.yaml
...
openstack stack create -t kickstart.yaml --parameter key_name=<ssh key name> <stack name> --wait
...

ssh syseleven@<server-ip>

source openrc

Launch Nova rescue mode

To be able to connect to the instance volume we need to launch nova rescue mode.
For this we use a rescue image that SysEleven provides in OpenStack.

First we stop the instance

openstack server stop <server uuid>

Then we need to get the current ID of the rescue image

$ openstack image list --public | grep -i rescue

| ace507db-7ac.......... | Rescue Ubuntu 18.04 sys11   | active |

Now we launch the instance into the rescue mode using the rescue image

# openstack server rescue --image < Image name / ID > <server name / uuid>

openstack server rescue --image "Rescue Ubuntu 18.04 sys11" app001

Please wait a few seconds while the rescue image is started.

Login to nova rescue instance

Once you are logged in you can work with the rescue mode.

via SSH

If you used a SSH key while deploying the instance initially you can login via ssh

ssh ubuntu@<Instance floating ip>

via Console

The rescue image generated a random password with every start, which can be viewed on the console. The root user is only allowed to login via the console. Once logged in you can define a password for the system user "ubuntu" if no SSH key is available and login via SSH is preferred.

Get the root password:

Login with the root password:

Work with the nova rescue instance

Become root user and check for connected block devices. Then mount the device.

$ sudo -i
$ lsblk

NAME   MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
vda    253:0    0   5G  0 disk
└─vda1 253:1    0   5G  0 part /
vdb    253:16   0  50G  0 disk
└─vdb1 253:17   0  50G  0 part

$ mount /dev/vdb1 /mnt/

Now we can start the repairs.
We could check and repair the filesystem using fsck.

# Using the option -y, fsck will repair without asking.
$ sudo fsck -f /dev/vdb1
[...]

We could use testdisk and search for files in damaged filesystems.

$ sudo testdisk /dev/vdb1

TestDisk 7.0, Data Recovery Utility, April 2015
Christophe GRENIER <grenier@cgsecurity.org>
http://www.cgsecurity.org

  TestDisk is free software, and
comes with ABSOLUTELY NO WARRANTY.

Select a media (use Arrow keys, then press Enter):
>Disk /dev/vdb1 - 53 GB / 49 GiB

>[Proceed ]  [  Quit  ]

Note: Disk capacity must be correctly detected for a successful recovery.
If a disk listed above has incorrect size, check HD jumper settings, BIOS
detection, and install the latest OS patches and disk drivers.

[...]

Stop Nova rescue mode

First we "unrescue" the instance.

openstack server unrescue <server uuid>

Then we start the instance again.

openstack server start <server uuid>

Conclusion

• We have access to the system/files via the rescue mode.
• The data is accessible on /mnt/.
• If it's an ext filesystem you should have a look in /mnt/lost+found.