Through the progression of my professional carrier I have learned a lot about interpersonal interactions. The most significant being, how to incentivize people. Giving people a reason to excel is fundamental to enabling your employees to achieve their best. A person with no reason to exceed their current situation will not. More importantly, they will seek that fulfilment elsewhere which can take away from the tasks you wish them to perform.
However, and this is the most important thing to remember, everyone is a special butterfly. An important aspect of any successful manager is to recognize this and adapt to each and every employee. This can be extremely difficult for managers with large teams or many remote workers. At the end of the day, the manager's primary function is to ensure the company is getting the most out of its employees.
I can't speak for most people but I'd like to share some of my experience dealing with management. Personally, I do not like verbal praise. I have always felt it hollow and basically just a tool they are using to make me work harder then I am paid for. I always do the best I can at my job regardless of my position. Please, I don't need someone telling me that. I also understand that if I want a raise I need to work at the next pay level but this must be a temporary situation. If you truly believe I am excelling at my job, PAY ME MORE.
Unfortunately, paying people seems to be the bane of management. My ideal position would be where when the manager compares my work to my pay they should think, "yep he is doing exactly what we pay him for" and never "WOW your doing such amazing work for the penance we grudgingly give you". Because if that is the thought you are having, I not being properly incentivized. Remember I will seek my fulfilment elsewhere.
Wednesday, April 3, 2013
Sunday, March 31, 2013
Ububtu netoworking hell.
Or how to stop hating routing and get on with your life.
As some of you may know, when our love, ubuntu, encounters a problem setting up network interfaces during upstart it tries to "failsafe". As a result, it waits two minutes with little information about what went wrong.
In most cases where DHCP is configuring the interface this is the correct thing to do. However, when the interface is static no amount of waiting is going to safe your sorry butt.
Unfortunately the networking subsystem "ifup/down" can be less then scrutable in it's error messages.
Take the following config as an example of a fail.
auto eth0
iface eth0 inet static
address 10.10.10.200
netmask 255.255.0.0
network 10.10.0.0
broadcast 10.10.255.255
gateway 10.20.0.1
dns-nameservers 10.20.0.1
up ip route add 10.20.0.0/16 proto static dev eth0
down ip route del 10.20.0.0/16 dev eth0
Seems legit? Ok so ya, this system is running on different subnet then the gateway but that's FINE. If the route exists then the packets flow. The problem here is the gateway but at this point we don't know that.
This config results in the 2 minute wait at startup because ifup is returning an error "no such process".
Figuring out what was causing the problem was a bit.. hard
Looking into how upstart configures the interfaces is what worked.
Upstart uses the ifup/down utilities and if it detects an error calls failsafe.
Ok that works. Let's bring down the interface and try it again. EXCEPT the interface isn't really up so use --force.
ifdown --force eth0
After that going line by line in the config with ifup and ifdown --force until we don't get an error.
In my case it was the gateway line. Turns out you need the route to the gateway BEFORE it can be configured but the line which adds the route happens POST up.
Since we can't add a route pre-up we must configure the gateway post-up. It could be done at up but there might not be a guarantee of execution order so keeping it explicit seems like a good idea.
The new config which works,
auto eth0
iface eth0 inet static
address 10.10.10.200
netmask 255.255.0.0
network 10.10.0.0
broadcast 10.10.255.255
dns-nameservers 10.20.0.1
up ip route add 10.20.0.0/16 proto static dev eth0
post-up route add default gw 10.20.0.1 eth0
down ip route del 10.20.0.0/16 dev eth0
As some of you may know, when our love, ubuntu, encounters a problem setting up network interfaces during upstart it tries to "failsafe". As a result, it waits two minutes with little information about what went wrong.
In most cases where DHCP is configuring the interface this is the correct thing to do. However, when the interface is static no amount of waiting is going to safe your sorry butt.
Unfortunately the networking subsystem "ifup/down" can be less then scrutable in it's error messages.
Take the following config as an example of a fail.
auto eth0
iface eth0 inet static
address 10.10.10.200
netmask 255.255.0.0
network 10.10.0.0
broadcast 10.10.255.255
gateway 10.20.0.1
dns-nameservers 10.20.0.1
up ip route add 10.20.0.0/16 proto static dev eth0
down ip route del 10.20.0.0/16 dev eth0
Seems legit? Ok so ya, this system is running on different subnet then the gateway but that's FINE. If the route exists then the packets flow. The problem here is the gateway but at this point we don't know that.
This config results in the 2 minute wait at startup because ifup is returning an error "no such process".
Figuring out what was causing the problem was a bit.. hard
Looking into how upstart configures the interfaces is what worked.
Upstart uses the ifup/down utilities and if it detects an error calls failsafe.
Ok that works. Let's bring down the interface and try it again. EXCEPT the interface isn't really up so use --force.
ifdown --force eth0
After that going line by line in the config with ifup and ifdown --force until we don't get an error.
In my case it was the gateway line. Turns out you need the route to the gateway BEFORE it can be configured but the line which adds the route happens POST up.
Since we can't add a route pre-up we must configure the gateway post-up. It could be done at up but there might not be a guarantee of execution order so keeping it explicit seems like a good idea.
The new config which works,
auto eth0
iface eth0 inet static
address 10.10.10.200
netmask 255.255.0.0
network 10.10.0.0
broadcast 10.10.255.255
dns-nameservers 10.20.0.1
up ip route add 10.20.0.0/16 proto static dev eth0
post-up route add default gw 10.20.0.1 eth0
down ip route del 10.20.0.0/16 dev eth0
Thursday, August 11, 2011
Beer Chiller
Due to popular demand I've decided to do a writeup on the beer chiller I created for the Beer Chilling Contraption Contest at DEFCON 19.
In previous years I had used a coiled pipe submerged in a chilled liquid consisting of a mixture of dry ice, alcohol, lN2, water, and salt. These chillers where very effective at getting beer cold but could not particularly fast or capable of handling capacity.
This year the rules changed to include scores for speed, capacity, and reaching a target temperature of 36 degrees. Unfortunately, the contest became a cluster fuck and the winner was chosen on lowest temperature so I didn't win.
Deviant has decided to stop hosting beer chilling so I'll be taking over the contest next year. Stay tuned if you want to participate for updated rules, sponsor opportunities, and othershit useful information.
Onto the meat of this post. The team Ad-Hoc DEFCON 19 beer chiller.
Description and General theory of operation.
This chiller operates with a similar concept as a water jacketed pipe. However, unlike the water jacket which relies on flowing liquid to move heat this chiller uses a stagnant reservoir of of liquid CO2 in the jacket cavity which it boils off to move thermal energy away from the center pipe (where the beer is)and into the operators face. I'll be moving the vent location on the next revision.
Operating the chiller is the simple process of filling the liquefaction chamber with dry ice. Melting the dry ice under pressure, and then transferring the liquid CO2 into the cooling cavity. In reality this ends up being a lot harder then it sounds. Just figuring out if you have a chamber of liquid CO2 ready for transfer is very difficult.
In previous years I had used a coiled pipe submerged in a chilled liquid consisting of a mixture of dry ice, alcohol, lN2, water, and salt. These chillers where very effective at getting beer cold but could not particularly fast or capable of handling capacity.
This year the rules changed to include scores for speed, capacity, and reaching a target temperature of 36 degrees. Unfortunately, the contest became a cluster fuck and the winner was chosen on lowest temperature so I didn't win.
Deviant has decided to stop hosting beer chilling so I'll be taking over the contest next year. Stay tuned if you want to participate for updated rules, sponsor opportunities, and other
Onto the meat of this post. The team Ad-Hoc DEFCON 19 beer chiller.
Description and General theory of operation.
This chiller operates with a similar concept as a water jacketed pipe. However, unlike the water jacket which relies on flowing liquid to move heat this chiller uses a stagnant reservoir of of liquid CO2 in the jacket cavity which it boils off to move thermal energy away from the center pipe (where the beer is)
Operating the chiller is the simple process of filling the liquefaction chamber with dry ice. Melting the dry ice under pressure, and then transferring the liquid CO2 into the cooling cavity. In reality this ends up being a lot harder then it sounds. Just figuring out if you have a chamber of liquid CO2 ready for transfer is very difficult.
Beer chiller operation overview.
Why / How it works.
In a liquid only chiller the wall temperature of the inner pipe is dependent on the thermal conductivity of the cooling liquid and the rate which it can be moved from the edge of the pipe. Most systems rely on the thermal conductivity of the fluid rather then circulating. It is difficult to find a pump which operates at cryogenic temperatures. This greatly limits the cooling capacity of the system as most fluids don't conduct heat very well and rely on convection.
The phase change CO2 solution solves the removal of cooling mass from the edge of the pipe through boiling action. This greatly increases the thermal transfer capacity of the unit. Further, unlike the liquid coolers the wall temperature is regulated to the boiling point of the CO2 under the pressure set by the operator. Therefor if liquid is freezing in the pipe the pressure can be increased thus increasing the boiling point and the temperature of the inner pipe.
Results.
In the end we where able to chill a beer from 90 degrees to 45 degrees in about 4 seconds. I feel we can get lower temperatures if more care is taken when charging the unit to ensure all the dry ice has melted before starting a transfer.
Future Work.
The biggest changes I will be making are to the pipe's connecting the two chambers. There was significant leaking at the compression fittings. Normally they would have worked but as the unit was moved around they started to leak.
A way to determine the fill level of the cooling cavity and the liquid state of the liquefaction chamber would be extremely helpful. We still have no idea when the chamber is ready for transfer or if all the CO2 has been moved to the cooling cavity.
I feel it was a good first run and I look forward to giving this thing another go next year at DEFCON 20.
All in all the whole thing cost me about $300 and resulted is tones of fun. Totally worth every penny and now I have a beer chiller which looks like a weapon of mass destruction.
All in all the whole thing cost me about $300 and resulted is tones of fun. Totally worth every penny and now I have a beer chiller which looks like a weapon of mass destruction.
BE THERE, GET FREE BEER. SEE YOU AT DEFCON 20!
Now for pictures. Ill try to get more from other people who have them.
Tuesday, April 26, 2011
[Odd Space] : There is no Spoon
Odd Space - A series on the universe
I have decided to write a series of entries on some of my better ideas about the nature of the universe and the physics which govern it. Let's get started.
There is no Spoon.
I would like to begin by putting forward a fairly strange but pivotal idea. Time does not exist. To be specific, there is no past or future, only the infinitely small instance which is now.
However saying now is an infinitely short period of time is misleading as it still maintains the concept of time. The present, as you will, is better described as 1. The number 1 has absolutely no dimension, no width, height, nor any other measure then that of it self, 1. This very aptly describes the present as it has no dimension either.
If this is held true how do we progress through existence? Even taking an infinite number of steps we would never progress forward as each step is infinitely small.
Imagine a rocket (the Paradox 1) moving through free space, its engines are currently turned off. Its state at the present is the sum of all previous states from some predefined initial state. Therefor at each instance the state of the rocket is a sum of an infinite number of states. This is important because it allows us to solve the changing state of the rocket, or more allows it to have a state which changes.
Let R(i) be the state of the rocket after it as accumulated i number of previous states or i number of information.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N ; Where N=infinity
To show how this allows state to exist, let R(0) = 1.
Therefor, R(i) = (1 + 1 + 1 + ... + 1)/N results in inf/inf or 1 as this limit goes.
However, let the rocket engage its engine such that it is moving by of 1 for each instance.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N + 1 ; Where the additional 1 is the velocity of the rocket.
This results in inf/inf + inf or 1 + inf. As such the rocket moves from its initial location to its final location in an infinitely small instant. Normally, this would be considered a bit of a problem as in our universe your can't move an infinite speed let alone do so infinitely fast.
But the truth of the matter is you can, and we do all the time. The trick is to realize that the rate at which state propagates in our universe is only observable within the universe. If you where outside the universe, where your state does not propagate you would "observe" the existence (state) of our universe propagate from its initial to its final state instantaneously. I say "observe" specifically as since your state does not propagate, the passing of our universe has no effect on your extra-universal state. Therefor, you would not know that it occurred, or more, could not in fact observe it happening in the sense that your state would incorporate any information about the state of the universe which just flashed in and out of existence.
However, inside the universe state propagates and information exchanges at a rate observable to objects within the universe as their state is propagating at equal rate even if that rate is infinite. This is where we get the concept of time.
Going back to the good ship Paradox 1, and now someone S(i) is observing the rocket moving through space.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N + 1
S(i) = R(i)
However, here S at any i has knowledge of R(i) at the same instance. Even if the observer's existence flashes, well, in and out of existence from the view of an extra-universal observer the intra-universal observer will observe the rocket moving through space the same as any of us.
I believe this is enough for the first post. I will continue next Monday by beginning to delve into how the speed of light works it self into all of this.
I have decided to write a series of entries on some of my better ideas about the nature of the universe and the physics which govern it. Let's get started.
There is no Spoon.
I would like to begin by putting forward a fairly strange but pivotal idea. Time does not exist. To be specific, there is no past or future, only the infinitely small instance which is now.
However saying now is an infinitely short period of time is misleading as it still maintains the concept of time. The present, as you will, is better described as 1. The number 1 has absolutely no dimension, no width, height, nor any other measure then that of it self, 1. This very aptly describes the present as it has no dimension either.
If this is held true how do we progress through existence? Even taking an infinite number of steps we would never progress forward as each step is infinitely small.
Imagine a rocket (the Paradox 1) moving through free space, its engines are currently turned off. Its state at the present is the sum of all previous states from some predefined initial state. Therefor at each instance the state of the rocket is a sum of an infinite number of states. This is important because it allows us to solve the changing state of the rocket, or more allows it to have a state which changes.
Let R(i) be the state of the rocket after it as accumulated i number of previous states or i number of information.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N ; Where N=infinity
To show how this allows state to exist, let R(0) = 1.
Therefor, R(i) = (1 + 1 + 1 + ... + 1)/N results in inf/inf or 1 as this limit goes.
However, let the rocket engage its engine such that it is moving by of 1 for each instance.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N + 1 ; Where the additional 1 is the velocity of the rocket.
This results in inf/inf + inf or 1 + inf. As such the rocket moves from its initial location to its final location in an infinitely small instant. Normally, this would be considered a bit of a problem as in our universe your can't move an infinite speed let alone do so infinitely fast.
But the truth of the matter is you can, and we do all the time. The trick is to realize that the rate at which state propagates in our universe is only observable within the universe. If you where outside the universe, where your state does not propagate you would "observe" the existence (state) of our universe propagate from its initial to its final state instantaneously. I say "observe" specifically as since your state does not propagate, the passing of our universe has no effect on your extra-universal state. Therefor, you would not know that it occurred, or more, could not in fact observe it happening in the sense that your state would incorporate any information about the state of the universe which just flashed in and out of existence.
However, inside the universe state propagates and information exchanges at a rate observable to objects within the universe as their state is propagating at equal rate even if that rate is infinite. This is where we get the concept of time.
Going back to the good ship Paradox 1, and now someone S(i) is observing the rocket moving through space.
R(i) = R(i-1)/N + R(i-2)/N + ... + R(i-N)/N + 1
S(i) = R(i)
However, here S at any i has knowledge of R(i) at the same instance. Even if the observer's existence flashes, well, in and out of existence from the view of an extra-universal observer the intra-universal observer will observe the rocket moving through space the same as any of us.
I believe this is enough for the first post. I will continue next Monday by beginning to delve into how the speed of light works it self into all of this.
Thursday, December 31, 2009
Backing up like the cool kids (linux)
So you want to backup a bunch of files or a few very large files to DVD. Should be simple enough right? Unless you have files larger then a single DVD or you want some redundancy with your backup. IE if you loose one or two DVDs you can still get all your data back.
The solution to this problem is fairly simple.
You will need the following...
1) Enough free disk space to hold all the data + 4GB. I recommend NOT using the disk you are backing up from. If you have eSATA attached externals like I do they work FANTASTICALLY!
2) mdadm
3) sha256sum [OPTIONAL]
Method:
To start out we need to make the block files that will be written to the DVDs. Normally you will see people making these with dd. Which is how we will be doing it however unlike most we will make use of the seek option which reduces the creation time from minutes to seconds.
However before we start making images we need to determine how much space we need. For this example we will be backing up 20GB of data. This means we need 6 block files at 4GB each. The attentive of you will realize that many block files is enough to hold 24GB of data. That is correct! However one of thoes blocks will be the parody block which allows us to loose any single DVD and still get back all our DATA.
To create the block files you will need to run the following commands
cd /my/backup/scratch/directory
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block0
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block1
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block2
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block3
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block4
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block5
Whats going on here: We are using dd to create block files of 4GB in size. The seek option tells dd to call truncate on the out file (of=block#) and then write count=1 blocks to the end. Using the seek option has the benefit of 1) taking much less time then if omitted and 2) since we request the whole 4GB at once greatly reducing fragmentation.
So now we have our block files, lets use em. Now we need to make linux turn these block files into virtual disks. For this we will use losetup.
First we need to make sure that none of the loop0-5 devices are in use. To check if a loop device is currently in use run the command
sudo losetup /dev/loop# replacing # with the number of the device 0-5
Assuming none of the devices are in use we can run the following commands
sudo losetup /dev/loop0 block0
sudo losetup /dev/loop1 block1
sudo losetup /dev/loop2 block2
sudo losetup /dev/loop3 block3
sudo losetup /dev/loop4 block4
sudo losetup /dev/loop5 block5
Now that we have our virtual block devices lets combine them into one large raid5 array so we can begin our backup. To do this we will employ the linux software raid provided by mdadm
Before you create a mdadm disk you should check and see which IDs are free. Most likely if you have never used mdadm before /dev/md0 will be free. But, its always good to make sure.
cat /proc/mdstat
will print out all the active mdadm disks. I have a raid1 running on /dev/md0 in my system so this outputs
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0] sda1[1]
488383936 blocks [2/2] [UU]
Therefor I cant use /dev/md0 and must use /dev/md1 however on your system /dev/md0 is likely free so we will continue assuming /dev/md0 is free.
So lets create that raid5 disk at /dev/md0
sudo mdadm --create /dev/md0 -n 6 -l 5 -x 0 -f --assume-clean /dev/loop[0-5]
Whats goin on here?:
--create - create a new disk at /dev/md0
-n 6 - six "physical disks" /dev/loop[0-5]
-l 5 - level 5 raid or raid5
-x 0 - no spare disks - by default mdadm will spare one of your disks. WE DONT WANT THIS
-f - force mdadm will bitch if you have -x 0 and dont specify force
--assume-clean - mdadm will attempt to sync the virtual disks and we dont want it to
/dev/loop[0-5] - expands to /dev/loop0 /dev/loop1 /dev/loop2 ... /dev/loop5
This should create a new disk /dev/md0 and running cat /proc/mdstat should list it.
Now lets shove a file system on our new virtual array. I like XFS because it formats FAST.
sudo mkfs.xfs /dev/md0
Lastly we need to mount this disk somewhere so we can put files on it.
sudo mkdir /mnt/backup
sudo mount /dev/md0 /mnt/backup
FILE COPY TIME - go get some tea...
Copy your files to /mnt/backup
TIME PASSES!
Once our backup is complete we need to break apart our array so we can copy the parts to DVDs
Unmount the virtual disk
sudo umount /dev/md0
Stop the mdadm array
sudo mdadm --stop /dev/md0
Turn off the loop devices
sudo losetup -d /dev/loop[0-5]
Now if all went well we should have 6 files sitting around waiting to be backed up.
BUT WAIT.
[OPTIONAL STEP - this takes quite a bit of time]
If your paranoid like me you will want some assurance later on that your files did not get corrupted when you go to restore them.
sha256sum data* > sha256sums
will generate sha256 sums for all the block files and dump them into the sha256sums file. COPY THIS FILE TO EVERY DVD YOU MAKE!
OK! Now we have our block files and possibly a sha256sums file.
Start BURNIN!
Put a blank DVD into the drive and burn the first block file. If you made a sums file copy it as well.
Do this for each block file.
Once you have copied all the block files you can delete them.
YOUR DONE!
------------------- Restore --------------------------
OH GOD! [Insert catastrophe here] HAS OCCURRED! I need to restore my backup!
Easy as pie I say!
First get your backup DVDs
Copy the block files off of the DVDs onto a spare hardrive
If all went well your lucky, if not no worries assuming you only lost one block file.
If you generated a sha256sums file copy that too and run sha256sum -c sha256sums
If any of the files are corrupt DELETE them. If more then one file is corrupt or you lost a disk and also had a corrupt file try recopying the block file. If it still fails your likely hosed but you can try the following steps... maybe...
Now we need to reverse the process stated before with some modifications.
1) Create the loop devices - if any where lost omit the line to create its loop device
sudo losetup /dev/loop0 block0
sudo losetup /dev/loop1 block1
sudo losetup /dev/loop2 block2
sudo losetup /dev/loop3 block3
sudo losetup /dev/loop4 block4
sudo losetup /dev/loop5 block5
2) Assemble the array. Remember CHECK AND MAKE SURE that /dev/md0 is free and if not pick another number.
sudo mdadm --examine /dev/loop0
This will print a bunch of info. The bit you want is the UUID field. Copy the uuid of the disk.
sudo mdadm --assemble /dev/md0 --uuid=COPIED UUID
Assuming you did not loose more then one disk, got the UUID correct, and md0 was free this should have created /dev/md0
2) sudo mount /dev/md0 /mnt/backup
Yay! now you can copy your backed up files.
TIME PASSES
Now that you have restored your files you can remove your copied block files - If you lost a block file / disk please check the section below this before running these commands!
sudo umount /dev/md0
sudo mdadm --stop /dev/md0
sudo losetup -d /dev/loop[0-5]
rm data*
And your done!
--------- Even more! Oh no when doing a restore one of my disks / block files was damaged -------
Fear not fair citizen!
True, you DO have good reason to be worried because if you loose one more disk all is lost! But we can regenerate the lost disk!
Lets assume we lost block4
1) create a new block4
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block4
2) make a loop device for it
sudo losetup /dev/loop4
3) add it to the damaged md array
sudo mdadm --add /dev/md0 /dev/loop4
4) Wait for the array to resinc - this will take quite some time!
You can check the progress by running
cat /proc/mdstat
5) Once the resync has completed you can dissemble the array. You will need to copy the new block files and sum file (if you created one) to new DVDs. The regenerated block4 file WILL NOT WORK with the old block files. You need to make A COMPLETELY NEW set of DVDs.
Well thats it friends. If you have questions or comments please feel free to contact me or leave a comment... you know, for comments.
The solution to this problem is fairly simple.
You will need the following...
1) Enough free disk space to hold all the data + 4GB. I recommend NOT using the disk you are backing up from. If you have eSATA attached externals like I do they work FANTASTICALLY!
2) mdadm
3) sha256sum [OPTIONAL]
Method:
To start out we need to make the block files that will be written to the DVDs. Normally you will see people making these with dd. Which is how we will be doing it however unlike most we will make use of the seek option which reduces the creation time from minutes to seconds.
However before we start making images we need to determine how much space we need. For this example we will be backing up 20GB of data. This means we need 6 block files at 4GB each. The attentive of you will realize that many block files is enough to hold 24GB of data. That is correct! However one of thoes blocks will be the parody block which allows us to loose any single DVD and still get back all our DATA.
To create the block files you will need to run the following commands
cd /my/backup/scratch/directory
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block0
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block1
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block2
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block3
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block4
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block5
Whats going on here: We are using dd to create block files of 4GB in size. The seek option tells dd to call truncate on the out file (of=block#) and then write count=1 blocks to the end. Using the seek option has the benefit of 1) taking much less time then if omitted and 2) since we request the whole 4GB at once greatly reducing fragmentation.
So now we have our block files, lets use em. Now we need to make linux turn these block files into virtual disks. For this we will use losetup.
First we need to make sure that none of the loop0-5 devices are in use. To check if a loop device is currently in use run the command
sudo losetup /dev/loop# replacing # with the number of the device 0-5
Assuming none of the devices are in use we can run the following commands
sudo losetup /dev/loop0 block0
sudo losetup /dev/loop1 block1
sudo losetup /dev/loop2 block2
sudo losetup /dev/loop3 block3
sudo losetup /dev/loop4 block4
sudo losetup /dev/loop5 block5
Now that we have our virtual block devices lets combine them into one large raid5 array so we can begin our backup. To do this we will employ the linux software raid provided by mdadm
Before you create a mdadm disk you should check and see which IDs are free. Most likely if you have never used mdadm before /dev/md0 will be free. But, its always good to make sure.
cat /proc/mdstat
will print out all the active mdadm disks. I have a raid1 running on /dev/md0 in my system so this outputs
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid1 sdb1[0] sda1[1]
488383936 blocks [2/2] [UU]
Therefor I cant use /dev/md0 and must use /dev/md1 however on your system /dev/md0 is likely free so we will continue assuming /dev/md0 is free.
So lets create that raid5 disk at /dev/md0
sudo mdadm --create /dev/md0 -n 6 -l 5 -x 0 -f --assume-clean /dev/loop[0-5]
Whats goin on here?:
--create - create a new disk at /dev/md0
-n 6 - six "physical disks" /dev/loop[0-5]
-l 5 - level 5 raid or raid5
-x 0 - no spare disks - by default mdadm will spare one of your disks. WE DONT WANT THIS
-f - force mdadm will bitch if you have -x 0 and dont specify force
--assume-clean - mdadm will attempt to sync the virtual disks and we dont want it to
/dev/loop[0-5] - expands to /dev/loop0 /dev/loop1 /dev/loop2 ... /dev/loop5
This should create a new disk /dev/md0 and running cat /proc/mdstat should list it.
Now lets shove a file system on our new virtual array. I like XFS because it formats FAST.
sudo mkfs.xfs /dev/md0
Lastly we need to mount this disk somewhere so we can put files on it.
sudo mkdir /mnt/backup
sudo mount /dev/md0 /mnt/backup
FILE COPY TIME - go get some tea...
Copy your files to /mnt/backup
TIME PASSES!
Once our backup is complete we need to break apart our array so we can copy the parts to DVDs
Unmount the virtual disk
sudo umount /dev/md0
Stop the mdadm array
sudo mdadm --stop /dev/md0
Turn off the loop devices
sudo losetup -d /dev/loop[0-5]
Now if all went well we should have 6 files sitting around waiting to be backed up.
BUT WAIT.
[OPTIONAL STEP - this takes quite a bit of time]
If your paranoid like me you will want some assurance later on that your files did not get corrupted when you go to restore them.
sha256sum data* > sha256sums
will generate sha256 sums for all the block files and dump them into the sha256sums file. COPY THIS FILE TO EVERY DVD YOU MAKE!
OK! Now we have our block files and possibly a sha256sums file.
Start BURNIN!
Put a blank DVD into the drive and burn the first block file. If you made a sums file copy it as well.
Do this for each block file.
Once you have copied all the block files you can delete them.
YOUR DONE!
------------------- Restore --------------------------
OH GOD! [Insert catastrophe here] HAS OCCURRED! I need to restore my backup!
Easy as pie I say!
First get your backup DVDs
Copy the block files off of the DVDs onto a spare hardrive
If all went well your lucky, if not no worries assuming you only lost one block file.
If you generated a sha256sums file copy that too and run sha256sum -c sha256sums
If any of the files are corrupt DELETE them. If more then one file is corrupt or you lost a disk and also had a corrupt file try recopying the block file. If it still fails your likely hosed but you can try the following steps... maybe...
Now we need to reverse the process stated before with some modifications.
1) Create the loop devices - if any where lost omit the line to create its loop device
sudo losetup /dev/loop0 block0
sudo losetup /dev/loop1 block1
sudo losetup /dev/loop2 block2
sudo losetup /dev/loop3 block3
sudo losetup /dev/loop4 block4
sudo losetup /dev/loop5 block5
2) Assemble the array. Remember CHECK AND MAKE SURE that /dev/md0 is free and if not pick another number.
sudo mdadm --examine /dev/loop0
This will print a bunch of info. The bit you want is the UUID field. Copy the uuid of the disk.
sudo mdadm --assemble /dev/md0 --uuid=COPIED UUID
Assuming you did not loose more then one disk, got the UUID correct, and md0 was free this should have created /dev/md0
2) sudo mount /dev/md0 /mnt/backup
Yay! now you can copy your backed up files.
TIME PASSES
Now that you have restored your files you can remove your copied block files - If you lost a block file / disk please check the section below this before running these commands!
sudo umount /dev/md0
sudo mdadm --stop /dev/md0
sudo losetup -d /dev/loop[0-5]
rm data*
And your done!
--------- Even more! Oh no when doing a restore one of my disks / block files was damaged -------
Fear not fair citizen!
True, you DO have good reason to be worried because if you loose one more disk all is lost! But we can regenerate the lost disk!
Lets assume we lost block4
1) create a new block4
dd if=/dev/zero bs=1024 seek=4194303 count=1 of=block4
2) make a loop device for it
sudo losetup /dev/loop4
3) add it to the damaged md array
sudo mdadm --add /dev/md0 /dev/loop4
4) Wait for the array to resinc - this will take quite some time!
You can check the progress by running
cat /proc/mdstat
5) Once the resync has completed you can dissemble the array. You will need to copy the new block files and sum file (if you created one) to new DVDs. The regenerated block4 file WILL NOT WORK with the old block files. You need to make A COMPLETELY NEW set of DVDs.
Well thats it friends. If you have questions or comments please feel free to contact me or leave a comment... you know, for comments.
Monday, November 23, 2009
Gratuitous First Post
Its the first post... yay. I would say something meaningful but then it would not be gratuitous. Therefore, I shall say nothing of any importance.
Subscribe to:
Posts (Atom)