Dear Bill
It is great that you are experimenting with file systems and large partitions. This is an area where things start to get quite complex and there is no single best practise solution. Linux is a great tool but is by no means the only option. Some higher end NAS products and SAN products may provide a better and easier to administer solution. 3ware make good cards, but if performance is not a high priority, Linux software raid is also fairly capable.
I noticed a few things: - msdos partition format is actually limited to 2TB partitions. So you might want to try using GPT (use mklabel gpt). - fdisk doesn't support large partitions. you should be exclusively using parted at this point.
There is lots of information about creating large partitions on linux:
Alternatively you might want to consider using LVM. Given the size of the volume, it might make things easier to manage.
(I'm sorry if the following is offensive, but I feel it must be said - this is supposed to be constructive criticism!)
I noticed that you are working for a lab and that you are the "Sr. Technical Advisor". As a consultant who deals with the deployment of various storage technologies on a regular basis, the story below is personally frightening to me..
You mention that you've already lost data, and that you've decided to move to RAID6 because of this... RAID is not a backup solution. It never was and never will be. Using RAID6 is not going to improve the situation very much. Some kind of frequent reliable off-line backup is the bare minimum in my mind for actual data backup. If data is important (and it sounds like you have a fair amount of it at ~5.5TB) you need to have a policy in place to manage it. To not have this in place when dealing with important data might be construed as negligence, especially in the event of data loss. How much trouble is it if you loose the data? Does it represent man-years or man-hours?
The [apparent] lack of experience could potentially cause a lot of problems down the road. For your own peace of mind, you might want to seriously consider hiring a consultant who specialises in storage if what you are putting together is intended for production - even it is to simply confirm you are on the right track both reliability and performance wise.
Other things you might want to consider: Are you using a UPS to protect the system including hard-drives? Is the unit located in a secure cabinet in a secure room? [is the data confidential?] Is there suitable air conditioning and environmental monitoring? Is there suitable fire protection? Is it possible to have an online backup to a different building/lab on campus?
I do wish you the best of luck but "thar be the dragons".
Kind regards, Samuel On 19/08/2009, at 3:56 AM, Sherman, William R wrote: Hello,
I have a largish (~5.5TB) partition onto which I thought I had successfully
created a filesystem -- and have been using it, but recently I've been
given cause for concern that there are grave issues that have the potential
for the loss of a large amount of data.
I was made aware of my potentially dangerous situation after rebooting
my machine and finding that not only did my large partition not mount,
but that the partition was listed by the OS as only begin about 1.5TB.
I managed to convince the system that the partition was okay, and it
mounts again, but I get this error (which I also get when rebooting):
"Aug 14 19:07:33 angelico kernel: I/O error in filesystem ("sdb1") meta-data dev sdb1 block 0x2ae7bffff ("xfs_read_buf") error 5 buf count 512"
I'm convinced that this error is related to the incongruous partition
size issue that I am seeing.
For some background information, here are some details about my system
and what I learned when first creating the partition:
I am running Fedora-11 with a 3ware 9650SE RAID controller card with
8 Seagate 1TB drives configured as a ~5.8TB RAID-6 system. Using the
RAID bios controls, I configured the system to have two logical units.
One unit is for the OS (or actually OSes) and is only 96GB, the other
is for all the data, and is (or is supposed to be) ~5.7TB.
The 96GB unit shows up as drive "/dev/sda" and I've partitioned it
into four partitions to hold multiple OSes (though thus far, I've
only put Fedora-11 on the machine). The 2nd logical unit is referred
to as "/dev/sdb", and I've only created one partition to consume
the entire unit.
Of course the reason I've created the RAID-6 system to for the protection
of my data -- got burned once with RAID-5 when I was under the gun to get
some data off a system with one bad drive -- a 2nd drive went bad, and
all was lost. So, putting my data in peril is something I'm not
interested in (if that isn't a tautology).
When creating the partitions, I first tried to use the partition manager
that Fedora 11 provides, and it reported that it had successfully created
a "5623675 MB" partition. But after the system came up, "/proc/partitions"
disagreed, reporting:
# cat /proc/partitions
major minor #blocks name
8 0 100663295 sda
8 1 33554432 sda1
8 2 33554432 sda2
8 3 12582912 sda3
8 4 1 sda4
8 5 20964352 sda5
8 16 5758648320 sdb
8 17 1463676507 sdb1
So somewhere the partition size got messed up.
I then went to the partitioning tool I'm familiar with: fdisk. But
when creating the partition, it only allowed me to create a partition
of 267349 cylinders. If I tried anything larger, it reported "Value
out of range." The man page for fdisk didn't report anything about
partition size limitations, so not sure why that's failing.
I then tried the "cfdisk" program, and it seemed to accept my larger
partition size, but after writing the partition table and quitting,
"/proc/partitions" reported the same values as above -- no change.
I then tried "parted", but didn't find an option that would allow
for the "xfs" filesystem (or even ext3).
So I moved on to "sfdisk" which also seemed to accept my desired
partition size, but again after writing the partition table and
quitting, there was no change in the values of "/proc/partitions".
So I decided to give "parted" another try. And it seemed to work,
though now when I look back at my notes there was something that I
should have been concerned about in the output.
The essential commands I gave to "parted" are:
# parted /dev/sdb
(parted) rm 1
(parted) mkpart primary 0 -0
(parted) print
(parted) quit
At this point, I checked "/proc/partitions" and was happy to see
the results I had been shooting for:
# cat /proc/partitions
major minor #blocks name
8 0 100663295 sda
8 1 33554432 sda1
8 2 33554432 sda2
8 3 12582912 sda3
8 4 1 sda4
8 5 20964352 sda5
8 16 5758648320 sdb
8 17 5758648320 sdb1
From there I was able to create an 'xfs' filesystem and mount it:
% df
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/sda1 33027952 6217352 25132880 20% /
tmpfs 5076716 1820 5074896 1% /dev/shm
/dev/sr0 388 388 0 100% /media/Bluebirds
/dev/sdb1 5758517248 4320 5758512928 1% /mnt
And I proceeded happily along until after a reboot, I got the above
'meta-data' error, and "/proc/partitions" was again reporting the
partition to be only 1463676507 1K blocks.
So thinking my data might be lost anyway (and fortunately still having
the original data available), I did an experiment. I tried just
doing a re-partition with "parted", and then mounting it -- and
that worked!
# parted /dev/sdb
(parted) print
(parted) rm 1
(parted) mkpart primary 0 -0
(parted) print
(parted) quit
# cat /proc/partitions
[...]
8 17 5758648320 sdb1
# mount /raid (renamed from previous)
And it worked, and the files were still there!
But then I looked back and noticed a discrepancy I'd missed before.
That final 'print' command to "parted" responded with:
Model: AMCC 9650SE-8LP DISK (scsi)
Disk /dev/sdb: 5897GB
Sector size (logical/physical): 512B/512B
Partition Table: msdos
Number Start End Size Type File system Flags
1 512B 1499GB 1499GB primary xfs
That "1499GB" isn't right. In fact, I compared the two 'print'
outputs (before removing and recreating the partition, and after),
and they were identical. So "parted" reported that it had created
the exact same partition as I started with -- but for some reason
unknown to me, it still causes "/proc/partitions" to recognize
the full disk as the partition, and the mount proceedure works
(whereas it failed before).
So, I'm at a loss for what the problem is. I looked at the data
from the 3Ware RAID system, and it is reported as expected.
So, I'm hoping someone more familiar with "parted", and even
more familiar with Linux partitions might be able to shed some
light into the situation and illuminate my mind.
My appologies for the length of this email, but I figured all
this would be requested (and useful) in determining the situation.
Thank you in advance,
Bill
--
Bill Sherman
Sr. Technical Advisor
Advanced Visualization Lab
Indiana University
address@hidden
_______________________________________________
bug-parted mailing list
address@hidden
http://lists.gnu.org/mailman/listinfo/bug-parted |