[TuxOnIce-devel] Issue with toi_get_nonconflicting_page() and TOI disabled

[Oleksandr Natalenko]

Hi, Nigel.

One of my user has reported the following issue with TOI disabled:

===
kernel/power/snapshot.c: In function ‘get_image_page’:
kernel/power/snapshot.c:105:31: error: expected expression before ‘do’
return (void *) toi_get_nonconflicting_page();
^
scripts/Makefile.build:258: recipe for target 'kernel/power/snapshot.o' failed
===

That happens because of explicit cast of macro result to (void*), but in case
of TOI disabled that macro is just noop do{}, so the cast is invalid.

The user has suggested to protect toi_get_nonconflicting_page() call with
ifdefs, but I have doubts about that (why one would want to cast unsigned long
to void*?). Probably, there should be struct page
*___toi_get_nonconflicting_page() call protected with ifdefs?

[Oleksandr Natalenko]

Proposed fixes [1], [2].

[1] https://github.com/pfactum/pf-kernel/commit/46287730bcf118135a9d4988769d69f7666c2f79

[2] https://github.com/pfactum/pf-kernel/commit/56de8e280db35ff1f1ba34e7463a7dbc35b20d61

Post by Oleksandr Natalenko
Hi, Nigel.
===
kernel/power/snapshot.c:105:31: error: expected expression before ‘do’
return (void *) toi_get_nonconflicting_page();
^
scripts/Makefile.build:258: recipe for target 'kernel/power/snapshot.o'
failed ===
That happens because of explicit cast of macro result to (void*), but in
case of TOI disabled that macro is just noop do{}, so the cast is invalid.
The user has suggested to protect toi_get_nonconflicting_page() call with
ifdefs, but I have doubts about that (why one would want to cast unsigned
long to void*?). Probably, there should be struct page
*___toi_get_nonconflicting_page() call protected with ifdefs?

[Nigel Cunningham]

Hi Oleksandr.

Thanks for the report and the patches.

I've just cherry-picked some commits from HEAD that deal with this. I'm
currently doing test compiles before pushing to github.

Regards,

Nigel

Post by Oleksandr Natalenko
Proposed fixes [1], [2].
[1] https://github.com/pfactum/pf-kernel/commit/46287730bcf118135a9d4988769d69f7666c2f79
[2] https://github.com/pfactum/pf-kernel/commit/56de8e280db35ff1f1ba34e7463a7dbc35b20d61

Post by Oleksandr Natalenko
Hi, Nigel.
===
kernel/power/snapshot.c:105:31: error: expected expression before ‘do’
return (void *) toi_get_nonconflicting_page();
^
scripts/Makefile.build:258: recipe for target 'kernel/power/snapshot.o'
failed ===
That happens because of explicit cast of macro result to (void*), but in
case of TOI disabled that macro is just noop do{}, so the cast is invalid.
The user has suggested to protect toi_get_nonconflicting_page() call with
ifdefs, but I have doubts about that (why one would want to cast unsigned
long to void*?). Probably, there should be struct page
*___toi_get_nonconflicting_page() call protected with ifdefs?

[Oleksandr Natalenko]

Hello, Nigel.

I've encountered hibernation issues while using TOI and btrfs.

The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk layout
and same kernel version (4.2).

First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second stacktrace
[2] (it occurs immediately after first stacktrace) shows us that there is some
btrfs activity happening. So, on may conclude that btrfs is not frozen
properly by TOI.

Also, I've noticed that while userui helper reports writing progress on screen
(I've set console debug level to 1), in fact nothing is being written to disk
(I see no HDD LED activity during TOI writing operation).

Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].

Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).

How that issue could be fixed? Feel free asking me for any additional info.

Regards,
post-factum

--

[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg

[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg

[3] https://gist.github.com/abaaeb5b26f338e978a6

[4] https://gist.github.com/b22a4eca266a4958b0be

[5] https://gist.github.com/cc8ef85d394efe5a7aab

[Oleksandr Natalenko]

Correction: the right kernel config is here: [1]

[1] https://gist.github.com/405afe6e4dea4640baed

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk
layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us that
there is some btrfs activity happening. So, on may conclude that btrfs is
not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

[Nigel Cunningham]

Hi Oleksandr.

Just a quick note to let you know I'm looking at it.

Sorry for the delay in replying.

Nigel

Post by Oleksandr Natalenko
Correction: the right kernel config is here: [1]
[1] https://gist.github.com/405afe6e4dea4640baed

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk
layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us that
there is some btrfs activity happening. So, on may conclude that btrfs is
not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

[Oleksandr Natalenko]

OK, looking forward to some results.

Meanwhile, I've prepared VM for VirtualBox with similar disk layout and
custom TOI-enabled kernel, and it fails as well in the same manner. So,
I expect it to be generic, non-hardware issue. Also, it would be much
easier for me to test changes without being afraid of real data
corruption.

Please, let me know if you need such a VM as well.

Post by Nigel Cunningham
Hi Oleksandr.
Just a quick note to let you know I'm looking at it.
Sorry for the delay in replying.
Nigel

Post by Oleksandr Natalenko
Correction: the right kernel config is here: [1]
[1] https://gist.github.com/405afe6e4dea4640baed

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk
layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us that
there is some btrfs activity happening. So, on may conclude that btrfs is
not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs
partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

[Nigel Cunningham]

Hi Oleksandr.

Please find attached a patch that should help. I would like to get your VM if it's a reasonable size (I'm not on the fastest connection), as I'd like to see how the btrfs thread is managing to get to run - it looks to me like it should be properly frozen.

Regards,

Nigel

Post by Oleksandr Natalenko
OK, looking forward to some results.
Meanwhile, I've prepared VM for VirtualBox with similar disk layout and custom TOI-enabled kernel, and it fails as well in the same manner. So, I expect it to be generic, non-hardware issue. Also, it would be much easier for me to test changes without being afraid of real data corruption.
Please, let me know if you need such a VM as well.

Post by Nigel Cunningham
Hi Oleksandr.
Just a quick note to let you know I'm looking at it.
Sorry for the delay in replying.
Nigel

Post by Oleksandr Natalenko
Correction: the right kernel config is here: [1]
[1] https://gist.github.com/405afe6e4dea4640baed

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk
layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us that
there is some btrfs activity happening. So, on may conclude that btrfs is
not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

[Oleksandr Natalenko]

Hello, Nigel.

Unfortunately, this patch does not help.

First of all, I had to modify it in order to compile kernel successfully as
shown via [1].

Then, I've got the same behavior as without it:

1) no SSD led activity while saving image (so, in fact, nothing is being
written to disk despite of userui showing some progress);
1) the same first stacktrace indicating the same line of blk-core.c;
2) the same second stacktrace (after a while) showing some btrfs calls,
indicating that btrfs wasn't freezed.

I will able to provide you with VM image on Monday, as it is located on
computer at my work (I'm on vacation right now). The size of it IIRC is ~3
GiB. If it is reasonable to you, I'll host it somewhere. Probably, it could be
compressed, but I doubt the size of it could become lesser than 1 GiB (it
consists of swap area as well and kernel sources/binaries, so that's why it is
large enough).

[1] https://gist.github.com/447c97e907d00ca351ee

Post by Nigel Cunningham
Hi Oleksandr.
Please find attached a patch that should help. I would like to get your VM
if it's a reasonable size (I'm not on the fastest connection), as I'd like
to see how the btrfs thread is managing to get to run - it looks to me like
it should be properly frozen.
Regards,
Nigel

[Nigel Cunningham]

Hi again.

Post by Oleksandr Natalenko
Hello, Nigel.
Unfortunately, this patch does not help.

Okay. I'm pleased to hear that as I wasn't totally sure it would be effective.

Post by Oleksandr Natalenko
First of all, I had to modify it in order to compile kernel successfully as
shown via [1].

That's fine. I was using a recent HEAD.

Post by Oleksandr Natalenko
1) no SSD led activity while saving image (so, in fact, nothing is being
written to disk despite of userui showing some progress);

Userui shows progress because it displays the amount of I/O that's been submitted. It doesn't reflect the amount that's made it to the hard drive so far.

Post by Oleksandr Natalenko
1) the same first stacktrace indicating the same line of blk-core.c;
2) the same second stacktrace (after a while) showing some btrfs calls,
indicating that btrfs wasn't freezed.

Ok. I need to see what's going on there as the btrfs thread that appears to be running looks like it should be frozen.

Post by Oleksandr Natalenko
I will able to provide you with VM image on Monday, as it is located on
computer at my work (I'm on vacation right now). The size of it IIRC is ~3
GiB. If it is reasonable to you, I'll host it somewhere. Probably, it could be
compressed, but I doubt the size of it could become lesser than 1 GiB (it
consists of swap area as well and kernel sources/binaries, so that's why it is
large enough).

Ok.

Thanks,

Nigel

[Oleksandr Natalenko]

Hello, Nigel.

I've prepared a little bit smaller VM image to play with. You may grab it here
[1].

md5sum: bfd52fec5a22ffa3dcad6fab773293a6
sha256sum: ac292f9e79054f5eed670abc0dfad5486a4176233b363641898f4c2905bd38d2

It contains latest Arch Linux, installed on exactly the same disk layout that
fails for TOI. Disk layout consists of:

* block device (8G)
* RAID10, f2 layout (degraded, because it has only one block device, feel free
to add one more 8G image and resync RAID, but it won't change anything)
* LUKS on top of RAID10
* LVM on top of LUKS with /boot (ext4), swap and / (btrfs)
* / has 1 extra subvolume (@home), mounted under /home

On boot GRUB2 asks for LUKS password. It is "qwerty". The system has 2 users:
"root" and "user", they also have "qwerty" passwords.

I've recompiled vanilla 4.2.2 kernel with Arch config. The only thing I've
added is TOI itself and changed version information (kernel version has -pf
suffix not to conflict with stock Arch kernel). Both stock and recompiled
kernels are available via GRUB boot menu.

I've also modified kernel command line, so it looks like:

cryptdevice=/dev/md0:system root=/dev/mapper/system-root resume=/dev/mapper/
system-swap consoleblank=0 no_console_suspend.

Also, there are pre- and post-hibernation hooks in /etc/systemd/system folder.
post-hook is dummy and does nothing, pre-hook allows to enable/disable TOI, it
also enables Escape key and changes default console level to make stacktrace
visible. Also I've added sync and caches drop.

You may launch VM using QEMU (I guess you may use other tools like VBox,
because VM kernel is universal):

qemu-system-x86_64 -enable-kvm -vga qxl -cpu host,+vmx -drive
file=vda.img,if=virtio -smp 2 -m 1024

To get the stacktrace, call the hibernation:

systemctl hibernate

Also there are 2 packages in /home/user folder, containing recompiled binary
kernel as well as headers (I've build kernel outside of VM). If you need full
source tree, let me know.

Feel free to contact me if you need more info.

[1] https://natalenko.name/myfiles/vda.img

Post by Nigel Cunningham
Hi again.

Post by Oleksandr Natalenko
Hello, Nigel.
Unfortunately, this patch does not help.

Okay. I'm pleased to hear that as I wasn't totally sure it would be effective.

Post by Oleksandr Natalenko
First of all, I had to modify it in order to compile kernel successfully as
shown via [1].

That's fine. I was using a recent HEAD.

Post by Oleksandr Natalenko
1) no SSD led activity while saving image (so, in fact, nothing is being
written to disk despite of userui showing some progress);

Userui shows progress because it displays the amount of I/O that's been
submitted. It doesn't reflect the amount that's made it to the hard drive
so far.

Post by Oleksandr Natalenko
1) the same first stacktrace indicating the same line of blk-core.c;
2) the same second stacktrace (after a while) showing some btrfs calls,
indicating that btrfs wasn't freezed.

Ok. I need to see what's going on there as the btrfs thread that appears to
be running looks like it should be frozen.

Post by Oleksandr Natalenko
I will able to provide you with VM image on Monday, as it is located on
computer at my work (I'm on vacation right now). The size of it IIRC is ~3
GiB. If it is reasonable to you, I'll host it somewhere. Probably, it
could be compressed, but I doubt the size of it could become lesser than
1 GiB (it consists of swap area as well and kernel sources/binaries, so
that's why it is large enough).

Ok.
Thanks,
Nigel

[Oleksandr Natalenko]

Hello, Nigel.

I haven't modified TOI source tree except of changing kernel version
suffix. I've merged tuxonice-4.2 branch [1] into kernel stable branch.

So you may just clone Greg's stable tree [2] and merge tuxonice-4.2 into
it to get the same source I have.

[1] https://github.com/NigelCunningham/tuxonice-kernel/tree/tuxonice-4.2
[2]
https://git.kernel.org/cgit/linux/kernel/git/stable/linux-stable.git/log/?h=linux-4.2.y

Post by Nigel Cunningham
Hi Oleksandr.

Post by Oleksandr Natalenko
Hello, Nigel.
I've prepared a little bit smaller VM image to play with. You may grab it here
[1].

Thanks for the image. Is your kernel source modified from the TOI
tree? If not, could I get the commit ID you used when building the
kernel? If not, can I get a URL to pull from and a commit ID to
checkout?
Thanks!
Nigel

[Oleksandr Natalenko]

tuxonice-kernel/tuxonice-4.3 git branch has the same issue.

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've reproduced
the following stacktraces reliably on 2 machines that use similar disk
layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows us
that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us that
there is some btrfs activity happening. So, on may conclude that btrfs is
not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied except
TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.jpg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

[Oleksandr Natalenko]

Unfortunately, this patch does not fix the issue as well.

Post by Nigel Cunningham
Hi Oleksandr.
Please try this patch.
Regards,
Nigel

[Oleksandr Natalenko]

Any success on TOI+btrfs?

Oh oops.
Sorry; I somehow got in my head that you were reporting the same issue as
Martin (freeing memory).
Yes, the btrfs issue is entirely different, and I need to get back to that
one. Too much going on at once.
I'll see if I can look at it on the train to Melbourne this morning. I did
mange to reproduce it in a VM. The 'fun' part has always been getting gdb
to be helpful. I'll have another try.
Regards,
Nigel

Post by Oleksandr Natalenko
tuxonice-kernel/tuxonice-4.3 git branch has the same issue.

Post by Oleksandr Natalenko
Hello, Nigel.
I've encountered hibernation issues while using TOI and btrfs.
The problem is that TOI refuses to work properly with btrfs. I've
reproduced the following stacktraces reliably on 2 machines that use
similar disk layout and same kernel version (4.2).
First stacktrace [1] occurs in block/blk-core.c:1994. That BUG_ON() shows
us that there is 3rd-party bio activity while hibernating. The second
stacktrace [2] (it occurs immediately after first stacktrace) shows us
that there is some btrfs activity happening. So, on may conclude that
btrfs is not frozen properly by TOI.
Also, I've noticed that while userui helper reports writing progress on
screen (I've set console debug level to 1), in fact nothing is being
written to disk (I see no HDD LED activity during TOI writing operation).
Here is my disk layout [3]. /dev/mapper/system-root is btrfs partition with
subvolumes [4].
Also here is my kernel config [5]. No additional patches were applied
except TOI from git tree (4.2 branch).
How that issue could be fixed? Feel free asking me for any additional info.
Regards,
post-factum
--
[1] http://i.piccy.info/
i9/99adc1e49adcbea1641cc7cc334e4fd4/1442696236/723905/951663/toi_fail_1.j
pg
[2] http://i.piccy.info/i9/12ddd0e30317965a9713e5250209f7af/
1442696258/666221/951663/toi_fail_2.jpg
[3] https://gist.github.com/abaaeb5b26f338e978a6
[4] https://gist.github.com/b22a4eca266a4958b0be
[5] https://gist.github.com/cc8ef85d394efe5a7aab
_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel

_______________________________________________
TuxOnIce-devel mailing list
http://lists.tuxonice.net/listinfo/tuxonice-devel