Restoring an attachment a guber deleted when using DAOS (the manual way)
Category
In advance - by Guber I mean "user". While on the last and longest leg of my trip to the Philippines, and trying to stay awake so I can kick jetlag's ass when I land, working on bits and pieces was helping. So, I was at the "inbox 5000" milestone that means I am as up to date as possible, so time to do something else.
A question asked frequently when presenting on DAOS. "OK, I have read the instructions on DAOS from the slide deck on your resources page". It is all working and all attachments are now in the DAOS directory as .NLO files. I understand that when a user has a email with the attachment it is actually in the DAOS directory. If he deletes it, the refcount of the attachment in the daoscat.nsf database will decrease by 1. When the refcount for that .NLO attachment is at zero, that means that nobody has kept a copy of this attachment anymore. So now, the .NLO attachment sits in the DAOS directory for 30 days (as per the prune setting in the server document). After that, it is deleted. I get it. It works. Its great.
Guber user though gets in touch with me 31 days later, saying that he wants that email (and attachment back). I can restore the .nsf fine, but how do I find out what .NLO I need to restore from the file system and how do I restore it?"
In this post, I will show you how to simulate this scenario for test purposes, and then how to recover the file. I am assuming you do NOT have backup software that supports DAOS. You may or may not have open file/.nsf supported backup for the mail databases.
More
In advance - by Guber I mean "user". While on the last and longest leg of my trip to the Philippines, and trying to stay awake so I can kick jetlag's ass when I land, working on bits and pieces was helping. So, I was at the "inbox 5000" milestone that means I am as up to date as possible, so time to do something else.
A question asked frequently when presenting on DAOS. "OK, I have read the instructions on DAOS from the slide deck on your resources page". It is all working and all attachments are now in the DAOS directory as .NLO files. I understand that when a user has a email with the attachment it is actually in the DAOS directory. If he deletes it, the refcount of the attachment in the daoscat.nsf database will decrease by 1. When the refcount for that .NLO attachment is at zero, that means that nobody has kept a copy of this attachment anymore. So now, the .NLO attachment sits in the DAOS directory for 30 days (as per the prune setting in the server document). After that, it is deleted. I get it. It works. Its great.
Guber user though gets in touch with me 31 days later, saying that he wants that email (and attachment back). I can restore the .nsf fine, but how do I find out what .NLO I need to restore from the file system and how do I restore it?"
In this post, I will show you how to simulate this scenario for test purposes, and then how to recover the file. I am assuming you do NOT have backup software that supports DAOS. You may or may not have open file/.nsf supported backup for the mail databases.
More
I have setup my demo server to use DAOS,
but only enabled one database (dnotes.nsf) to utilise the DAOS manager.
i.e. that database has attachments stripped out to the DAOS directory,
but the other mail files are still setup as "normal". That
has given me exactly 31 .NLO files in my DAOS directory 0001.
Now, I go to Bill Buchan's mail file through iNotes. This file is NOT DAOS enabled. From there, I send an email with an attachment to Doctor Notes.
This promptly arrives in the inbox of Doctor Notes.
DAOS immediately kicks in and the attachment is removed from the message, converted to a .NLO file and placed in the DAOS directory (Sorry, I know that screen grab is a bit compressed. The last file has the latest date - that's the new one).
FOR THE PURPOSE OF TESTING, at this stage, take a backup of the dnotes.nsf database and also the entire DAOS directory. All I did was shut down the server and do a file copy of both. Bring the server back up and on with the show.
Now, I promptly delete the email message, as I don't need it and it's never ever going to be important.
Then into my trash folder and empty the trash, which permanently deletes the messages.
Now, that attachment will have a refcount of zero. And 30 days from now, the .NLO will be deleted. I don't have 30 days to wait (its a long trip, but not that long) so FOR THE PURPOSE OF TESTING I will tell DAOS to delete any files that have a refcount of zero, regardless of how long they have been in that state. The command "Tell DAOSMGR prune 0" tells DAOS to do that. The zero is a variable indicating number of days being at refcount=0. Here is the output from the command.
And if I check the DAOS directory, I am back to 31 .NLO files. All is peachy and the attachment is no more.
Dr. Notes is a guber. He calls the help desk as that unimportant file he deleted ages ago is now the most important file in existence, and the lives of everyone on the planet, not to mention his bonus depends on getting it back now. And by now, I mean "you have 20 seconds to restore the file or he will report you to your boss". You are in the operations team. You are the firefighter. This is your fault. You should of known that this file that has nothing to do with you was vital even though the intended recipient didn't. You get the drift. If you don't, ask someone who has worked in operations or service desk delivery. Then give them a hug. Its like 'Nam. "You weren't there man!".
So you restore the back of the .nsf to a temporary directory and give it a new name. Of course, you will want to disable replication and agents in the database (if there are any). So I am restoring the mail file to a BACKUP directory on the domino server called guberuserrestore.nsf
Now, I can open this database on the server just fine. I can see the old mail message, but if I try to open the attachment, this happens...
(Note... At this stage if you look at the console, you will see the error listing what .NLO is missing. I'm going to ignore that although you could use it to resolve you issue there and then.)
That's because the .NLO is not in the DAOS directory. You now need to find out what .NLO is missing and what the filename is. Type this on the console.
TELL DAOSMGR LISTNLO MISSING BACKUP\GUBERUSERRESTORE.NSF
The following appears on the console
If you open the LISTNLO.TXT file in the data directory, the missing .NLO file is listed.
Copy that file back from the backup directory of the NLO's to the 0001 directory and the file is now available. You can now copy the old document back into the users' mail file.
I know there is backup API software available that can do this, or you can script it. I just thought the simple basics may help someone.
Now, I go to Bill Buchan's mail file through iNotes. This file is NOT DAOS enabled. From there, I send an email with an attachment to Doctor Notes.
This promptly arrives in the inbox of Doctor Notes.
DAOS immediately kicks in and the attachment is removed from the message, converted to a .NLO file and placed in the DAOS directory (Sorry, I know that screen grab is a bit compressed. The last file has the latest date - that's the new one).
FOR THE PURPOSE OF TESTING, at this stage, take a backup of the dnotes.nsf database and also the entire DAOS directory. All I did was shut down the server and do a file copy of both. Bring the server back up and on with the show.
Now, I promptly delete the email message, as I don't need it and it's never ever going to be important.
Then into my trash folder and empty the trash, which permanently deletes the messages.
Now, that attachment will have a refcount of zero. And 30 days from now, the .NLO will be deleted. I don't have 30 days to wait (its a long trip, but not that long) so FOR THE PURPOSE OF TESTING I will tell DAOS to delete any files that have a refcount of zero, regardless of how long they have been in that state. The command "Tell DAOSMGR prune 0" tells DAOS to do that. The zero is a variable indicating number of days being at refcount=0. Here is the output from the command.
And if I check the DAOS directory, I am back to 31 .NLO files. All is peachy and the attachment is no more.
Dr. Notes is a guber. He calls the help desk as that unimportant file he deleted ages ago is now the most important file in existence, and the lives of everyone on the planet, not to mention his bonus depends on getting it back now. And by now, I mean "you have 20 seconds to restore the file or he will report you to your boss". You are in the operations team. You are the firefighter. This is your fault. You should of known that this file that has nothing to do with you was vital even though the intended recipient didn't. You get the drift. If you don't, ask someone who has worked in operations or service desk delivery. Then give them a hug. Its like 'Nam. "You weren't there man!".
So you restore the back of the .nsf to a temporary directory and give it a new name. Of course, you will want to disable replication and agents in the database (if there are any). So I am restoring the mail file to a BACKUP directory on the domino server called guberuserrestore.nsf
Now, I can open this database on the server just fine. I can see the old mail message, but if I try to open the attachment, this happens...
(Note... At this stage if you look at the console, you will see the error listing what .NLO is missing. I'm going to ignore that although you could use it to resolve you issue there and then.)
That's because the .NLO is not in the DAOS directory. You now need to find out what .NLO is missing and what the filename is. Type this on the console.
TELL DAOSMGR LISTNLO MISSING BACKUP\GUBERUSERRESTORE.NSF
The following appears on the console
If you open the LISTNLO.TXT file in the data directory, the missing .NLO file is listed.
Copy that file back from the backup directory of the NLO's to the 0001 directory and the file is now available. You can now copy the old document back into the users' mail file.
I know there is backup API software available that can do this, or you can script it. I just thought the simple basics may help someone.
Comments
Posted by Oliver Regelmann At 08:04:15 On 16/11/2009 | - Website - |
Im not a sales rep for them, but Commvault Galaxy was always one of my favourite Domino backup solutions.
Posted by Paul Mooney At 08:09:23 On 16/11/2009 | - Website - |
Posted by Dennis van Remortel At 08:28:02 On 16/11/2009 | - Website - |
Posted by Oliver Regelmann At 10:35:01 On 16/11/2009 | - Website - |
Posted by David Bailey At 15:22:08 On 16/11/2009 | - Website - |
Posted by Chris Blatnick At 16:21:40 On 16/11/2009 | - Website - |
Posted by Mike Kinder At 18:16:08 On 16/11/2009 | - Website - |
What I do not know is - tada - is DAOS circular (like TX logging)? By that I mean that each folder (0001 thru 9999) can hold 40,000 attachments. During a purge one or more NLO files are removed from a given folder, lets say fodler 0001. If I have 40,000 attachments in folder 0001 and two NLO files are now removed during a purge(now I have 39,998 NLO files in 0001). So....does DAOS ever re-add two new NLOs to the 0001 folder? If so (and I hope not) you maybe screwed. Or does it (hopefully) just continue to folder xxxx, where xxxx is the newest NLO folder in the DAOS repository?
Posted by Darren Duke At 20:17:26 On 16/11/2009 | - Website - |
From my experience, DAOS will quite happily re-use the folders when files are removed, to keep the total number of folders to a minimum.
I don't see it being an issue though, if the required file is in the path it should be found. 40k is a soft limit, not a hard limit, so I don't see having 40,001 files being a major issue. A resync may even fix this, I doubt anybody will ever see it in production.
In our case, we matched DAOS with out backup retention. We backup each night and 2 tapes are written, one for on-site, one for off-site vaulting. Our on-site retention period is 90 days, so we just extended the purge interval to 93 days. If Guber deletes an email then wants the attachment back after 90 days, we couldn't even give him the .nsf so have the .nlo files would be pointless.
Posted by Ben Rose At 23:32:57 On 19/11/2009 | - Website - |
Posted by Andy Dennis At 12:52:13 On 23/11/2009 | - Website - |
1. Restore NSF (s) (use lotus API's)
2. FInd missing NLO's for these NSF's you just restored and generate a text file containing the list of missing NLo's.
3. Provide the path of the text file generated in step2 and restore utility will restore missing NLO's.
The problem here is that the step2 is a manual process, how can we automate this. My executable should be able to take step1, genrate file automatically (take step2) and take step3.
We know that we can run nserver.exe -d "tell daosmgr listlno.txt missing dbname.nsf" , in a batch command from our application and generate this file. But this is dirty implementation, as error returning and reliablity is questionable. Also we will have to run it for each and every database, so if user is restoring 50 dbs then we will have to execute it 50 times.
Posted by Navneet At 06:46:28 On 21/12/2009 | - Website - |