How Do Email Servers Work? Please dont PARF...
 

I will fully admit that the whole email/internet, etc., is kind of "magic" for me. I don't really know how it works, I just appreciate that it DOES - LOL!

When someone sends me an email, I may keep it for a long time depending on the subject matter. It is common for me to forward or resend an email that someone else needs who has deleted their copy. I still have my copy so I'll send it to them.

How did Clinton destroy 30,000 emails and no one anywhere has a single copy not even on a personal device? I figure that because I will sometimes save an email to my hard drive that other people probably do the same thing (I save technical type information and contract information).

I don't want to PARF this, I just want to know how this is physically possible that absolutely not one single person or one single device has one single item from 30,000 plus emails. Please remove Clinton from the discussion if it keeps it out of PARF.

thank you,

angela

Email servers typically hold the email until deleted. If you delete the message on your personal device, in most cases it will also be deleted from the server. Then, on top of that, the storage/drive of the email server could be "wiped, and not with a rag", to make sure deleted emails cannot be un-deleted. Deleting files really means nothing. Wiping your phone before trading it in really does nothing. Data can be recovered unless steps are taken to make sure it cannot.

It's not easy, but every instance of an email can be deleted. It's hard to do it by accident. Hitting the "delete" key just marks the space the email occupies in storage as available for future use. It doesn't remove anything. To wipe memory you have to "delete" then put something in the space. I just wiped a hard drive. It took several steps and then it took the computer about 24 ours to over write a one terrabyte disk.

that's what I have wondered too. chances are no one wants to go against the Clintons. if she was not going to be president it might be a different story. (no, not a fan of her).

They used Microsoft Exchange server which is a product intended for enterprise use with 10's of thousands of mailboxes (or more) depending on the version used. Exchange stores emails in a central database and applications like Outlook or a smartphone connect to that database to manipulate messages. If messages are deleted from the central store those messages will no longer be seen from the clients any longer. Once large chunks of messages are removed from the database Exchange "re-shuffles the deck" to shrink the database and they can be very difficult to recover.

Most phones and clients don't really hold a copy of the emails that can be preserved without the original data on the database. Clients can be configured to retain offline copies but in big businesses that is discouraged because the data becomes too hard to manage.

Angela,

There are copies of the e-mails, unless they were deleted as well, at the recipient (sent e-mails) and the sender if they were received e-mails.

I have multiple devices connected to an e-mail server. One device is setup to delete the e-mails after 30 days. Which means copies of the e-mails are on my PC as well as the server. The copies on my PC are kept until I manually delete them.

I use 30 days because I don't own the e-mail server (just a service) and they limit how much e-mail is allowed per account. 30 days does limit how much e-mail I can access on my mobile devices. They usually only keep a subset on the device. So the more that is left stored on the server, the more that is accessible to mobile devices.

If this is an Outlook server, it may keep everything on the server, until it is specifically deleted. At GE, this is how it was setup. Not many of us had remote access (I left GE in 2001!). At Cameron it was similar with Exchange servers (I left them in 2008).

There are 2 or 3 protocols involved with sending email.

The first is SMTP - this is the sending of mail, from the client (webmail or thunderbird or outlook or whatever on local machine) to the outgoing mail server, and then from that server to the destination server.

Once mail is on the destination server, a client (again, thunderbird, outlook, a webmail program, etc) accesses it via either the POP3 protocol or the IMAP protocol.

POP3 was designed around not being always connected. Connect to mail server, download all messages to local machine. Possibly delete them from the server at that point, or when deleted/removed from teh inbox on the client (this is configured client side).

IMAP was designed around being "always connected". With IMAP when the mail is checked it only downloads the header information. When a message is opened to be read, it downloads the rest of the message. Messages can be stored in folders on the server, or on the client, or both (ie at work my Inbox is on the server but any mail message over 30 days old is archived on my desktop).


I can't speak about Exchange or qmail, but both Exim and Postfix store messages on the file system, not in a database, so deletion is trivial to accomplish.

I think Angela was asking why there aren't copies on the recipients devices/servers. Mrs. C used Exchange and they could have burned that unit to charcoal where nothing could be recovered but there are logs/traces/actual emails on the devices those emails were sent to in most cases. There are apps you can add into email to make the email "expire" so there wouldn't be copies on those devices, but that usually only works with encrypted mail and as far as I know the FFL did not encrypt most of those. I could be wrong, unless I am not.

Good question.

Been waiting for that lightbulb to go off.

Every email that is sent or received has for an indeterminate amount of time at minimum one counterpart on the other end.

We have 30k emails missing.

There was a copy of each one of them somewhere for some period of time either in an inbox or sent items.

Since the crux of the complaint is the possibility that classified information was passed through her system it's pretty much a given that were that true X number of them would exist in a Fed archive.

The Fed can not legally delete them at will and the systems are backed up.

So really all anyone needs to / needed to do was search, starting with State as it's most target rich, the archive for emails sent to or received from her private account.

Match whatever you find with what she handed over and if there are unaccounted for matches it can be assumed those might be among the 30k.

At minimum you should be able to parse the log files for the Fed servers and still find evidence of bi directional exchanges that are not accounted for by "physical" copies.

OK, based on what you guys are telling me, it is possible depending on the server protocol for emails to be erased. For instance, when I was working at a large company a few years ago, the company had its own servers. Company email did indeed disappear - guaranteed. So that company server must have been similar in nature to what you are describing.

But the copies that I had loaded into my computer hard drive did not disappear nor did the backup that I store in a remote hard drive (my aww-**** contingency). No matter what happened, I had those and they proved to be most valuable on a couple of occasions...

So how did Clinton make all of those disappear for everyone who had received them and stored them either on a laptop hard drive, or backed up onto a remote hard drive? I realize that gov't employees are not supposed to back up items to remote hard drives, but it DOES get done and not everyone of those 30,000 deleted emails went to a gov't person... How the heck did she do this?

*edit - was still typing when stomach monkey wrote in - thanks

angela

And here's the other thing.

Spam.

This is one of my partners / clients spam filters.

It's the first line of defense in a 3 tiered system so it's not set too aggressively.

It represents about 7 months of filtering at this point.

Even with the triple filter approach the average users inbox is 1/3rd spam.

I'd say realistically only around 15% of what passes through that server is legit mail.

http://forums.pelicanparts.com/uploa...1476470360.jpg

She didn't. So either the people who have something incriminating don't want to get involved, have something that can be held over them, are on her side anyway, etc. I would imagine that quite a few of them were internal to others in the state dept. who may be employed only by the political will of the incumbent office holders... Also possibly people who work for other governments, etc.

You've hit on my point.

There is a lot of talk about the number deleted and the assumption is that some or all of them were work related.

You don't have to find all 30K or even 5K to prove that.

You just need to find X out of 30k.

And that is absolutely doable.

It's unrealistic to think that out of a potential 30K not one of them hit a government system and there will absolutely be a record of it somewhere.

Unless they were not work related.

It would be pretty funny after all this to find 30k emails about amazon gift cards or hot russian women looking for love.

Retention policies and how its implemented are a big part of this. Rotate old email out and screw that up and poof, mail if gone. Lots of potential ways to 'lose' data. Could be as simple as a misconfiguration, something got corrupt, intentional....

At first I was going to joke about those would be on Mr Clinton's server, then I thought more about it and seems appropriate for her as well. Not that there is anything wrong with that.

Oh, as far as misconfiguration, her people can only control her equipment. Unless the NSA was involved, then they might be able to scrub a lot of systems.

I haven't read most of this thread, but I'll add this...I've been out of the large corporate game for eight years now, but EVERY single email I ever sent or received could be retrieved if necessary going back over 20 years. NOTHING was ever deleted without a permanent backup on tape. Email was just the tip of the iceberg....I could also data mine VPN, firewall logs etc. and tell the legal dept. exactly what a user was accessing too...I didn't have to worry about big brother watching me...I was he :)

I've seen a few cases where something was supposed to be quarantined yet the target disk was full and things got hosed. I've also seen a backup routing fail leaving holes in you cant explain.

No parf intended, but this type of thing can be expected in a home brew system. Or, someone can rm * and everything is out the window.

I've seen some bizarre stuff.

This one goes really far back to the dial up days before companies started dropping T1's.

Our Japanese office was the absolute worst at communicating. Seemed like sometimes it took days for them to reply to things.

Our CEO used to get really hot under the collar when he'd send something to the President of our APAC office and not hear back till the next day.

He'd call to chew the guy out only to hear the excuse that he had replied right away even though his email was no where to be found.

I'm sitting in the VP of IT's office one day when the CEO calls and proceeds to give him a particularly unpleasant reaming when it dawned on me.

CEO hangs up and I look at VP and asked, how does the mail server deal with mail from the future?

He looked at me like I was nuts then went "ohhhh"

Hint, it was holding inbound mail till the emails "future" timestamp matched the local time on the receiving end.

My employer had an outlook email server crash this year and most of us lost everything. I assume my customers lost the same emails I lost. It was a PITA and embarrassing.

The DB nature of Exchange is a nightmare.

One of the earliest iterations of Outlook used a local DB, actually still does.

One day I send an email and immediately get a pop up that says "Your mail store has exceeded the DB limit, so sorry but you are ****ed" then it crashed and I lost two years of mail.

What was infuriating was they thought to create an alert for an after the fact condition but not one that said "Hey dip****, your DB is almost full, back it up or purge it or do something before you receive a piece of mail and lose everything" which would have been far more useful.

To add insult to injury two days later MSFT issued a patch that increased the max DB size and gave you the nearing capacity warning before hand.

From Reddit - a post from the tech who maintained Mrs. Clinton's server:

http://forums.pelicanparts.com/uploa...1476479138.jpg

I have received threatening emails in the past, breaking laws, but they disappeared before saving.

There is this:
https://www.cnet.com/how-to/this-e-mail-will-self-destruct-heres-how/
Self Deleting Email, How to Send Self Deleting Mail

Yes, but she is asking about the recipients.

For instance, you send me an email and then your system gets hosed. I still have my copy of the email.

Interesting.

HRC did make a joke a few month back about how she liked Snap-Chat, as the messages automatically delete themselves. --she's so funny corrupt fun. :D

Anyway, considering that she had her own server she certainly could have done the URL supported messaging (in your first link) but, clearly, as indicated in daepp's post, her IT guy never set up something so sophisticated.

I expect that she has something like that going now. Her whole team has to have squeezed a few brick on all of their email trail on display. I expect that they may use courier pigeons now. Possibly courier drones. ;)

Certified MS Exchange admin here. It would be a non-issue for HRC's staff to make mail disappear from her server. The more disturbing question is who made them disappear from the federal system. There is zero chance that those emails were not archived. Zero chance that they were not on a back up tape at some point.

Or they were never there to begin with.

I know squat about the Feds archiving system but It's unrealistic to think they don't get a ridiculous amount of emails a day.

To find and delete 30k emails spread over years would probably take quite some time.

Time I doubt they had.

It would seem to be like finding 30k needles in a million haystacks.

Since she was running a private server the users probably had accounts on her server so senders and/or recipients would not have emails from that server on any another servers. Also that way message retention policies could be put in place that would keep the messages on the server and not be allowed to store copies on the client's computer. No off-line retention of messages. These same retention policies can also be used to basically delete all the emails if you so desired.

You can also use retention policies to archive messages without any of the users knowing. Especially deleted emails. Only the top level administrator would know or have access to those archived emails unless the administrator gave access to another user. This is most likely why she had her own servers instead of using a service. A service would have top level access.

Retention and archiving policies are specifically to archive and recover messages to use in litigation. There is no reason why you couldn't use the same retention rules to make the server delete those messages. It just wouldn't be legal if your company was required to have archived messages. Probably why she had a personal server, it would not be required to provide legal access to messages for litigation.

The default install of an exchange server when you delete and email it stays in your deleted items folder for 14 days. Then it deleted from your client and no longer visible in your client. The server marks the message for deletion and holds it for 30 days. After 30 days it deletes it from the message store to save space in the message store/database.

Almost all operating systems now have a secure delete that writes zeros where the files was instead of just marking the space a usable again. There is also hard disk maintenance software that will defragment, or group the currently used files together and write zeros on the unused space.

Then...... where would the classified emails in question have come from?

the Russians uploaded them to WikiLeaks

Theory <> reality sometimes when it comes to document retention.

Once upon a time, the SEC was investigating the company I worked for, relating to communications we had with the management of a company shortly before that company disclosed information that sent its stock tumbling. We were required to produce all internal emails, documents, notes. We retained all such emails, of course, as required by Sarbannes Oxley and other post-Enron Federal law, and we had a whole IT and compliance team making sure of that. So naturally, our IT and compliance and legal people couldn't locate any of those emails. Fortunately I always archived my emails on my laptop, so we were able to produce them.

SQL doesn't lie. If its there, its easy to find.

Didn't say it was hard.

Just time consuming.

And even that's relative.

10 minutes is an eternity when all you have is 5.

Maybe some underpaid, overworked sysadmin misplaced tapes or rewrote drives or botched some settings or - my point above is that things aren't always done just because they are supposed to be done.

My impression is, whenever there is an inquiry like this, emails turn up missing, and that is the case for large and small corporations, for government agencies, for past administrations, etc.

I think that, as a general matter, document retention is about as foolproof as cyber-security.

True but in this specific situation the thought that a mistake would be so surgically precise as to effectively and efficiently eliminate all traces of specific activity spanning years is simply not realistic.

It would have caused massive data loss. Not something trivial that would or could have gone unnoticed.

Mistakes don't discriminate.

We are left to consider a coordinated intentional act.

Certainly possible but I think also very difficult.

We'd need to know more about the Feds backup infrastructure.

Tape is very efficient for this type of archiving.

You can't delete specific files from a tape archive.

To do so you'd first need to request an index dump of archives to find which tapes contain the data you want.

Then you need to access the physical media. Protocol is not going to let you stroll in and check out a tape or bunch of tapes and take them home like its Blockbuster on a Friday night.

You're going to have to tell them what you want retrieved and they are going to do it for you.

To delete it permanently you'd need to restore the archive to traditional disc media, delete the files then back up the altered source.

And then your time stamps will have a mismatch.

If the Fed is backing up to disc based media it's certainly easier but I'd be willing to bet those discs are also backed up, probably to tape.

Again all of this is certainly possible and doable given enough time which I don't think they had and even if you could pull it off I don't believe you could do it without leaving evidence all over the place.

It just does not seem plausible.

LOL Tapes...ask Nixon

I haven't paid that much attention to the details of the Clinton email thing. However, some quick searching shows that the 32,000 emails missing were claimed to be personal emails that were deleted, rather that government related emails. If these were in fact personal emails, to friends and family and other people not in the US government, then it would make sense that copies would not be found in the US government email systems. And if copies were in fact not found in those US government email systems, that would be consistent with the missing emails in fact being personal emails.

This... But was Hillary actually backing up her server to tapes?

Yep, when I was administrator of various email servers a backup was run each night and the tapes kept for a month. The end of week ones were kept for a year and the end of month ones were kept forever. This is fairly normal procedure. So in Hillary's case someone is not telling the truth.

Without actually knowing the infrastructure and policies of the systems, a lot of you obviously experienced techies and smart guys can only speculate :). It all just depends....

I was a systems programmer, systems designer, communications guy before for some large corps that were simply excellent in this type of stuff...they had to be.

Once mined some critical data (a much smaller needle, in a much larger haystack than this). It's potentially doable, but certainly not easy imo...assuming blah, blah, blah :)