Performance issue copying disk to disk?

[stokkes]

Hey there,

I'm quite interested in this project - been using Unraid for over a decade, but over the last few years have been using it virtualized under Proxmox. Performance is fine (15 disks, 2 parity) - I can max out my LSI HBA.

I attempted to use nonraid, so moved the super.dat, and it imported just fine (my disks are XFS), array started, etc.

However I am seeing something very odd happen - I went to do a bit of cleanup on one of my disks and was doing an rsync from disk1 to disk2 and the performance was absolutely abysmal.. Here's what was happening:

• Performance really bad (14-25MB/s)
• The rsync operation with the --progress flag would literally not update the speed in real time, only every 30-60 seconds
• If I cancel the rsync (CTL-C), it would pause for a good 30-40 seconds while I assume it's flushing the data from memory (I can tell as doing an iostat -x 2 shows writes keep happening on parity even after CTRL-C has been hit.

What's curious, on the same proxmox system I purposefully invalidated my parity and just mounted disk1 and disk2 manually, did the same rsync operation and the performance was 160-180MB/s, which is about as fast as I can consider (yes I know this isn't including parity, see below), and the --progress showed real-time speed, and cancelling it would cancel immediately without hanging/flushing from memory.

Under Unraid, I would do this rsync operation (with valid parity) quite often (using unbalance and directly with rsync), and performance was always 70-80MB/s, which I believe is what you can expect when doing a disk to disk copy with parity being calculated.

Spent 3-4 hours last night trying to troubleshoot this, but the common thread appears to be nonraid, granted I don't really have any other way to validate this.

Any thoughts? Thanks!

[qvr]

I've also seen variable performance, but so far it's been "good enough" for me. I've also noticed that nonraid seems to take a bigger performance hit when using luks and having the FS on top of that (compared to performance penalty of luks just on top of raw disks). Something about nested blockdevices possibly not playing nice with how the i/o operations eventually get queued to nonraid parity operations.

Anyway, my hunch is that the performance difference between unraid and nonraid is not the driver itself, but instead there might be some blockdevice / kernel tuning options that a nonraid system might benefit from, which unraid (the OS) might be using by default. If you have access to an unraid system, checking out differences between possibly relevant sysctl's and blockdevice tunables under /sys/block/ could be interesting.

You can also try playing with the nonraid driver's tunable performance settings which you can see with nmdctl set:

  md_queue_limit [1-100]  I/O queue limit as percentage (1-100, throttles normal I/O)
  md_sync_limit [0-100]   Sync queue limit as percentage (0-100, throttles parity operations)
  md_num_stripes [COUNT]  Number of stripe cache entries (default: 1280)

The driver's hardcoded defaults for these might not be the same which unraid is using by default.

Setting the driver in turbo mode can also improve things (md_write_method 1), but especially with 15 disks, you probably don't want to have all drives spinning when copying just between 2 disks.

[stokkes]

Hey thanks for the reply.

So yes I do have access to another unpaid system, there are some sysctl (under vm.dirty) that I replicated on Proxmox. I also played with md_queue_limit trying various permutations without much luck. Turbo writes on/off doesn't make a difference either.

It's tough based on my knowledge to sift through the /sys/block devices differences however..

Copying to the disk (from a different disk not in the array, e.g.: SSD) appears fine and is responsive to CTRL-C, etc.. I can see the real-time progress.. Also parity check is running quite fast (I restarted a parity check after invalidating my parity).

It's really just the disk-to-disk copy that slows everything down. Actually I also booted my unraid system with the same disks and started a parity check there, and was still able to hit 80MB/s disk to disk while parity was being calculated.

So yeah something somewhere (as its the same hardware just different OS) is causing this and for the life of me I can't figure it out.

Here's the painful results of disk-to-disk transfer:

File1.mkv
  3,144,218,887 100%   19.04MB/s    0:02:37 (xfr#1, to-chk=32/87)
File2.mkv
  2,071,431,427 100%   17.53MB/s    0:01:52 (xfr#2, to-chk=31/87)
File3.mkv
  1,886,822,146 100%   17.84MB/s    0:01:40 (xfr#3, to-chk=30/87)
File4.mkv
  1,919,080,840 100%   16.56MB/s    0:01:50 (xfr#4, to-chk=29/87)

Pretty abysmal.

[qvr]

I don't really have a good test setup where I could test this with any kind of consistence, but for me setting the block device read-ahead larger for all raw disks and nonraid blockdevices seems to help with array to array copies, and I managed to rsync at least 55MB/s (my test system has very old and slow disks to begin with).

Default readahead was 256 on the systems I checked, setting them to 2048 seems to improve the performance:

blockdev --setra 2048 /dev/sdX
blockdev --setra 2048 /dev/nmdXp1

(If you are using luks, also set the larger readahead for the mapper device.)

From the unraid forums, speeds of 40 - 60MB/s are apparently what you can expect from array to array copies:
https://forums.unraid.net/topic/185546-horrendously-slow-transfer-speed-from-one-share-to-another/#comment-1515701

don't expect more than 40 to 60MB/s for array to array transfers with parity enabled.

[stokkes]

Thanks, lol at this point i'd take 40-50MB, that'd be a 200-300% speed increase, pretty huge when transferring multiple TBs.

I did check the unraid system, which has the same read ahead setting (256) but the raw device is 65520.. so unraid/nonraid set the same, yet unraid shows full performance.

I will try this first in a VM (with a vdisk on each of the raw spinning disks) - not ideal, but I am re-running a parity check and would prefer not cancel it. I'll report back.

Appreciate the time on this!

[qvr]

Did you manage to test this further? I have a couple of other things to try:

• Unraid seems to disable the native NCQ from array disks by default: https://github.com/limetech/dynamix/blob/master/plugins/dynamix/DiskSettings.page#L124-L128
  This is basically echo 1 > /sys/block/sdX/device/queue_depth for all array disks

• Unraid also seems to use larger nr_requests by default for array disks: https://gist.github.com/syaifulnizamyahya/234901cfdb177173e748eaba28597464#file-gistfile1-txt-L4693
  Usual kernel default seems to be 64, so this can be increased with echo 128 > /sys/block/sdX/queue/nr_requests

• Unraid does not seem to change the default io scheduler (mq-deadline) settings, but these might still help write performance:

echo 1 > /sys/block/sdX/queue/iosched/writes_starved
echo 500 > /sys/block/sdX/queue/iosched/write_expire

I'd be interested if you can test again between unraid and nonraid with the same disks, especially with the NCQ and nr_requests tunings.

[stokkes]

Hey, I did try the blockdev from your Feb 27 comment, but that didn't do anything noticeable. I did switch back to pure Unraid temporarily as I do a lot of moves disk to disk to consolidate data/cleanup/etc and this was just unbearable - also i believe it did cause some issues with locking the rest of the array when a disk to disk move was happening.

I am planning to try it again near the end of the month, just waiting on a new case. Appreciate you keeping this open, I will try some of the things you mention to see if it helps. Just a pain as when I switch from Unraid to this, going back to Unraid requires me to re-run a parity check which takes 2-3 days on 28TB parity drives.

[stokkes]

Getting ready to try this soon, but some things are not the same as Unraid:

• queue_depth is set to 32 on all my disks, not 1
• nr_requests is set to 256, not 128
• writes_starved is set to 2, not 1
• writes_expire is set to 5000, not 500.