Dariush Debian Diary

echo "blacklist intel_agp" >> /etc/modprobe.d/blacklist

1. Divide your flash-based block device into >3 parts, for example let's use 12 sections.
2. Raid5 (or Raid6 if you need more protection) them.
3. use resulting block device as base for your filesystems/swap.

This way you can run without wear-levelling layer (which AFAIK can't work in general case, because you would need Size/blocksize bytes/nibbles of additional wear-resistant storage to keep statistics of your wear), AND you gain additional protection, for very little price ( less then 10% of your storage ).

Conventional wisdom says you shouldn't raid on single device, but since flash devices are not mechanical and don't suffer noticable seek penalty you can do this with very little loss of performance. If you'd do this in hardware inside your card, there would be no performance penalty.

Another problem is that raid5/raid6 are famouse for their poor write speed, but, since you're already writing whole blocks-at-a-time when dealing with flash devices, you're already paying the prices (writing block-at-a-time is the best case scenario for raid5/6, there's no need to recalculate checksums then)

1. Effective protection for wear problems, plus additional protection for other block errors
2. Very small price storage-wise (<10%)
3. Very small performance penalty when you synchronise flash blocks with raid blocks (the penalty for writing less-then-block-at-a-time is already payed when dealing with flash, raid penalty is virtually non-existant then)
4. Ideal for embedding in hardware, but doable on software-side

Note: it doesn't make much sense to use raid5 instead of raid4 in this case. And raid4e for double protection. TODO: implement RaidF as dm layer?

Problem:

homenode:/home/eyck# pvcreate /dev/cciss/c0d2p5
  Physical volume "/dev/cciss/c0d2p5" successfully created
homenode:/home/eyck# vgscan
  Reading all physical volumes.  This may take a while...
  Found volume group "sys" using metadata type lvm1
homenode:/home/eyck# vgextend sys /dev/cciss/c0d2p5
  Physical volume /dev/cciss/c0d2p5 is of different format type (lvm2)
  Unable to add physical volume '/dev/cciss/c0d2p5' to volume group 'sys'.
homenode:/home/eyck# pvscan
  PV /dev/cciss/c0d0p3   VG sys   lvm1 [66.34 GB / 0    free]
  PV /dev/cciss/c0d1p5   VG sys   lvm1 [130.34 GB / 6.19 GB free]
  PV /dev/cciss/c0d2p5            lvm2 [33.91 GB]
  Total: 3 [230.60 GB] / in use: 2 [196.69 GB] / in no VG: 1 [33.91 GB]

Solution:

homenode:/home/eyck# vgconvert -M2 sys
  Volume group sys successfully converted
homenode:/home/eyck# pvscan 
  PV /dev/cciss/c0d0p3   VG sys   lvm2 [66.34 GB / 0    free]
  PV /dev/cciss/c0d1p5   VG sys   lvm2 [130.34 GB / 6.19 GB free]
  PV /dev/cciss/c0d2p5            lvm2 [33.91 GB]
  Total: 3 [230.60 GB] / in use: 2 [196.69 GB] / in no VG: 1 [33.91 GB]
homenode:/home/eyck# vgextend sys /dev/cciss/c0d2p5
  Volume group "sys" successfully extended
homenode:/home/eyck# pvscan 
  PV /dev/cciss/c0d0p3   VG sys   lvm2 [66.34 GB / 0    free]
  PV /dev/cciss/c0d1p5   VG sys   lvm2 [130.34 GB / 6.19 GB free]
  PV /dev/cciss/c0d2p5   VG sys   lvm2 [33.91 GB / 33.91 GB free]
  Total: 3 [230.59 GB] / in use: 3 [230.59 GB] / in no VG: 0 [0   ]

homenode:/home/eyck# lvextend -L+5G /dev/sys/v
  Extending logical volume v to 194.50 GB
  Logical volume v successfully resized
homenode:/home/eyck# xfs_growfs /fs/v/
meta-data=/dev/sys/v             isize=256    agcount=61, agsize=819200 blks
         =                       sectsz=512   attr=1
data     =                       bsize=4096   blocks=49676288, imaxpct=25
         =                       sunit=0      swidth=0 blks, unwritten=1
naming   =version 2              bsize=4096  
log      =internal               bsize=4096   blocks=6400, version=1
         =                       sectsz=512   sunit=0 blks
realtime =none                   extsz=65536  blocks=0, rtextents=0
data blocks changed from 49676288 to 50987008

gnoll2:~# mdadm --create /dev/md1 --level 1 --raid-devices=2 missing /dev/hdh5
mdadm: array /dev/md1 started.
gnoll2:~# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 hdh5[1]
      77152064 blocks [2/1] [_U]
            
unused devices: <none>

gnoll2:/fs/tmp# mdadm --add /dev/md1 /dev/hde5 
mdadm: added /dev/hde5
gnoll2:/fs/tmp# cat /proc/mdstat 
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10] 
md1 : active raid1 hde5[2] hdh5[1]
      77152064 blocks [2/1] [_U]
            [>....................]  recovery =  0.0% (18560/77152064) finish=414.6min speed=3093K/sec
	          
unused devices: <none>
gnoll2:/fs/tmp#