I had every intention of publishing this a while ago, but I didn't. I forget why. I'm sure it was because I had something life-alteringly important to say. As always... Well I'm gonna just end this post and make a new one...

Now for the updates on the fun stuff. Things haven't been all bleak and computer oriented the last few weeks, there's been a lot of other stuff going on too.

First, my apologies to ladytikk. She's swedish, not Norwegian. I should have posted this a while ago, but it wasn't till she knocked on my door with a viking helmet, threatened with a thor hammer and banished me to the realm of Loki by the meade dribblings of Odin's beard did I take her seriously. She also dropped of some meatballs in case I fealt threatened (I didn't, I'm steel) or hungry (I did, I eat free food whenever I can). So swedish, not norwegian.

I also completely forgot to tell you guys that I gotta dog! Here's a picture of said kuli wearing his nicest ta'ovala. Never mind his junk, he just lets that hang out whenever the spirit moves him: He's really awesome. I named him 'Ifo, which, for those of you not in the know, is Tongan for "Delicious". I wasn't sure how the Tongan's would take it, being that they have been known on occasion to consume said delictable. But for the most part all the Tongan's I've met have gotten the biggest kick out of the name. It was a toss up between 'Ikale (first beer in the world everyday) and 'Ifo. I chose 'Ifo because its funny and NOT related to beer at all. I work at Tupou High School so I didn't want the kids calling out 'Ikale all day long. PC being PC. Ha! Ha!

So we've also lost one more volunteer from G66 this week. Lisiate was cool. I didn't really ask him why, because I'm sure they were absolutely valid in every way possible. That man was stud. I'm sure somewhere in the U.S. this man is getting a lap dance. Here's his picture. If you see him treat him well.

*Note - I don't remember what I was talking about below. Please don't read it*
I've got a great story about this, but I'm not going to share it as the freshmen that keep reading my blog get scared away. I'll recant it in all of its kai kuli glory later on. I think the regulars would get a kick out of it. Good stuff. 0 comments - Post a Comment

Here's a cute little email that I've sent whitireai (the college campus I work with in New Zealand) describing the problems I've been having here with my network and the internet and everything else. Hopefully, it will explain sort of why I haven't been posting as much as I should. Feel the love.

My apologies to everyone for not getting this to you sooner. I have been working around the clock for the last few weeks trying to get everything straightened out with the system and the email service has been intermittent as a result.

April 3rd

Mark and I notice that a couple of the computers have bad group policy objects and that virus protection has been failing reguarly on the workstations in the labs, infecting multiple computers on a daily basis. I find the root email account forwarders and point them towards my email address while looking around for some mail server antivirus applications.

April 4th

I get my first "Log Watch" log from the linux email server with an ftp connection from a reverse IP lookup of a german porn site. I wonder why ftp is turned on, and more importantly why this person has access to our sites ftp services. I start to investigate looking at the log files. My postmaster inbox begins to receive undeliverables in its inbox from various porn sites, sales sites, viagra companies, and spam IP addresses. I cannot locate the log files for more thorough checking

April 5th

I disable the ftp service and other non-important services and check the Sendmail Configuration. Tupou has been set up to allow internet relaying for the last 2 years, thus explaining why our connection is so slow evening during hours when students are not in the labs. I disable internet relaying and check the updates on the server. The server is running redhat 8.0 and as near as I can tell has not been update once since the initial install. I run the up2date service.

April 7th
The up2date service finishes installing litterally hundreds of secuity patches, fixes and enhancements for the Linux server. Sendmail is updated to a reasonably secure version and appears to be working correctly. Everything seems in order. I call it a day.

April 11th
During the next 4 days I begin doing a lot of installations and tightening up of services. I install spamassassin as well as the f-prot anti-virus suite, trying to head off viruses before the get into user mailboxes. FTP service starts itself backup after I disabled it. Completey remove all FTP services manually. I begin to get thousands of non-deliverable emails again in the postmaster account. 5 days after everything has been secured. I investigate the firewall settings and find out that IPChains, an older technology that was used as the original firewall software in Linux 8.0 has been removed from the system, effective destroying any filtering or firewall protection for our linux box (which also acts as the router for our two subnets) An nmap from a friend in the states confirms my suspicions. All of our ports are unprotected. IP tables in installed and properly configured. I post some messages on groups.google.com.
http://groups.google.com/groups?hl=en&lr=&ie=UTF-8&oe=UTF-8&threadm=28am705c57fg5bs1jf3lnbemdaq4svkaup%404ax.com&rnum=1&prev=/groups%3Fq%3DSendmail%2Bhacked%2Bgroup:comp.os.linux.security%26hl%3Den%26lr%3D%26ie%3DUTF-8%26oe%3DUTF-8%26group%3Dcomp.os.linux.security%26selm%3D28am705c57fg5bs1jf3lnbemdaq4svkaup%25404ax.com%26rnum%3D1

April 12th
I get a few replies from google with a tool called "chkrootkit" and some suggestions. Most believe that the system has been compromised and recommend that I build a new box from scratch using another mail server application. I choose QMail and begin building a new email server. chkrootkit never works properly because I don't have a copy of redhat 8.0 to boot from. Per the recommendations of the google post, I turn off sendmail for a few days to stop sending out spam (estimated at about 100,000 per day or so). I have 5 days until messages start being bounced for good.

April 15th
After 2 or 3 days of about 3 or 4 hours of sleep a night, I finally get the qmail server, along with spammassisin, IP chains, linux 9, clam-av antivirus software and a new safer webmail interface called squirrel mail setup and working properly on my created subnet. Afraid of sticking the server directly out onto the internet, I decide to install a firewall with DMZ support built in for an extra layer of protection. We don't have much money to buy a firewall solution, so I download Smoothwall (http://www.smootwall.org) and build a firewall. I get the firewall as well as the mail server running about 4 a.m. in the morning.

April 16th
I wake up and move the smoothwall system and the new mail system upstairs and begin doing some testing. I find out that old server is using identd for authentication to the proxy server that is running on the old mail server. I do some research and find out that it is contacting our server using clear text password and username authetication. It is an old version, not supported by redhat. I disable the service and allow all computers on the network to have access to the internet through the port without authentication, using just IPs. Seems to work fine. Spend the rest of the day trying to get the mail server to authenticate to the windows box using OpenLDAP and kerberos encrypted passwords. Work all night and into sunday, not sleeping.

April 17th
Cannot get the new linux qmail server to talk to windows 2000 server from the DMZ. Around 2:00 a.m. decide that the Students will arrive tomorrow and need access to their email as well as staff so I put the old email server back in place and let it start spamming again. Its been doing it for 2 years, what's a few more days. I add the firewall to the network, and everything seems like it is in place. Try to update the clients group policy objects to point to the new router and to point to the Squid proxy server. Clients refuse to update. Cannot understand why.

April 18th
Classes start. Miss the first day and sleep for the first time since Friday night. Tell the students that we'll make the class time up on Saturday and get everything squared away. No one can access the internet or their email. Uninstall firewall, cofigure everything back the way it was originally. Try to figure out what happened. About 80,000 emails in the message queue after 4 hours of being online.

April 19th - 23rd
Rebuild a new subnet in my house with old mail server and firewall configuration works. Can't understand why clients won't update settings nor why qmail will not authorize with samba and kerberos. Spam starts again, this time with a vegence. Internet grinds to a halt. During the week a number of students tell me that their passwords were changed. My personal password was changed as well during the week. Having my password changed convices me that by keeping the old mail server on the newtwork I'm running a major security risk. Take it offline that friday the 23rd. Start installing firewall again realize that the mail server is running an old, old version of bind and is acting as an internet DNS server. Move the DNS server settings over to the windows 2000 box inside the network and configure all the /etc/hosts files to point to the correct entries for DNS resolution.

April 25th
After working all weekend, I get the firewall installed and everything working properly. Still using the old mail server, but now I have a way of tracking who is coming into our server and from where. Start compiling log files to send to CERT and have friends starting to port scan. Network is finally safe. Ports 80, 137, 25 and 110 are stealthed. Portscans are being logged and IP addresses are being blocked by the new firewall. Bring the mail server back online. Spam starts immediately and I notice a number of ping requests coming from the mail server trying to get out on the internet.

April 26th
Get an email from one of the technical people here telling me that my network is sending out an insane amount of ARP requests and the local ISP cannot troubleshoot virus problems their having because we're flooding the network with ARP requests. Asks me to disable the server until the local ISP can track down the Netsky Virus Culprit. They do later on that day. I turn the mail server back on. ARP requests begin flooding the network again.

April 27th - 29th
Focus on getting prepared for my classes for the week and planning for the semester. Interrupted repeatedly by problems by the staff about the network problems. DNS is not updating on clients once again. Manually change the computer settings for most of the administrative staff so they can check their emails and get online. Spend rest of the week troubleshooting individual clients.

April 30th
Realize that DNS is not properly updating on the clients. Head to windows 2000 server to find out why. MS Knowledgebase suggests clearing out the Application Event log as a possible solution. I check the windows 200 server event logs and realize they haven't been configured properly and have effectively not worked for about 1 and 1/2 years. Clear event logs and reconfigure properly. Reboot server and find roughly 50 error messages on reboot from system and application log. Start the long tedious process of troubleshooting error messages.

May 1st
Reduced error messages to about 10. Got all major services working except for netlogon which won't start at boot. Clients will still not update their Group Policies. Student workstations GPO's start expiring all over the place. The kids begin running command prompts and installing game software on the computers. Spend the rest of the day troubleshooting the 10 major error messages. Find massive problems with the newly installed DNS server. Unistall Active Directory and DNS reboot. Problem goes away. Reinstall Active Directoy and DNS from the CD and reinstall service pack 4. Still no good. Everything fails for the clients again. Around 5:00 in the morning read a post that describes my problem exactly: Active Directory has been corrupted. Promote Backup Domain Controller and reinstall windows 2000. Problem arises. We have no backup domain controller. We have just one windows controller and the earliest domain backup copy is corrupt as well. Realize this is the original problem from the beginning of April. OUR ACTIVE DIRECTORY DATABASE IS CORRUPT, which explains why the server cannot find ANY of the user accounts on the system and won't update GPO's.

May 2nd - May 7th
Still teaching classes, but spending a lot of time working on solving computer problems. Have gotton most administrative clients set up and working but none of them are updated GPO's at all. Start buidling brand new Windows 2000 server on a brand new domain. Start looking for transfer scripts for porting old, verified accounts from old Windows 2000 server to new Windows 2000 server. We have roughly 600 - 700 accounts on the old windows server. Old mail server has slowly but surely reduced the amount of spam it has been sending. Realize that somewhre on the server, is a ping script that sends pings to assorted mail servers around the world to notify them that this server is open for relay. Because mail server is now on a DMZ, all ping requests are blocked by the firewall. Bandwidth finally returns to stable norms and log files are accumulating and recording a monstrous amount of hack attempts and buffer overflow runs. Mail server finally seems secure. Realize, however, that subnet from the high school is completely bypassing the security of the DMZ because it connects directly into the windows server and mail server. Disconnect and remove all subnetting. Put every computer in the network in the 192.168.0.255 subnet. Run into some major problems because old network is BNC and if one computer is disconnected the whole network goes down. Spend Tuesday and Wednesday of this week trying to find out which computer is failing. Find computer, fix network, everything starts working correctly again in terms of routing, subnetting and IP addressing provided the clients are properly, manually configured.

May 8th
Find script I need to transfer user accounts. Start writing script

May 9th
Around 12:30 or so, I finally get the script to transfer the accounts that have been logged onto during the last 90 days. Accounts decrease significantly from 683 to 250. Try three or four test accounts at random all work correctly. Get prepared for classes for the week and go to sleep.

May 10th
Realize that I havent' talked to anyone back in the states for at least a month. Figure this weekend I'll install the new Windows 2000 server when everyone is gone. Go through my email and start talking to friends and relatives again. Realize that I've missed some pretty important emails and have to spend the next 2 days doing damage control and apologizing to my sister for not calling her on her college graduation.

May 12th - Today
So Where I'm at today:

- I have two new servers that are almost ready to go. One Windows, One RedHat 9 with qmail
- The new windows server has all of the proper accounts set up, I just need to transfer all of the student, staff and administrations personal directories sometime when they are not here.
- I have the qmail server ready to go once the new Win2000 domain is established.
- Test runs indicate that the same corruped active directory problem that held me up before was the reason why I couldn't get samba and kerberos authentication to work correctly on the qmail server.
- Firewall is in place and is basically self healing and self updating. It is logging so many hack attempts that the /var/ and /tmp/ directories have been filled to capacity with hacking attempts and port scans. Often I must manually archive the logs because the firewall freezes up.
- Our server has been an open relay for so long that once everything is working again, I will have to change our IP address to something completely different.
- I also plan on moving our webhosting off site to completely close port 80. Most scans on port 80 are directed at our apache server. There is no reason why we need to expose 80 for our website.

Essentially, I have been inundated with poor system administration. I had no idea how vulnerable everything was and how unmaintained everything was. While the server 2000 and linux problems were unconnected, they happened at the same time which makes them seem related. The Windows box has been having this problem well over a month after I examined the log files on my laptop. The fact of the matter is that everything would have worked weeks ago if the Active Directory controller a.) Had a Backup Domain Controller or b.) Had an adequate backup strategy. Neither was inplace. The hacking could have been prevented long ago, by installing the latest service packs on the Redhat 8.0 box. While that wasn't an option before, it is now. I have two brand new servers, certified updated and certified installed by me. I have essentially rebuilt the entire school network without a single backup to work with. Things are still a mess because it took me this long to track everything down and find out what was happening.

Tevita has mentioned that you would like to send someone up here to help and fix the problems. I'm all ears. Please, this has been a burden I never wanted, just something I inherited. What I could realy use is someone that knows how to install and transfer accounts from sendmail to qmail and from Horde to SquirrelMail without losing emails. That's the one piece left in the puzzle that has escaped me.

In regards to grading, I would have like to have had the grading done weeks ago during the term, but the fact of the matter is I live on campus. My house is 100 yards away from the school. Because I'm the "computer guy" here on campus and everyone's problems are of the utmost, I couldn't even sleep because people would wake me up and wait for me to get dressed in order to get them back on the network. I'm not complaining, I'm just telling you how it is here. Hopefully, this explains in full detail the mess that I've been experiencing and the reason why I have not had time to get my grades submitted to you. I do apologize for any inconvenience that this may have caused. I have the hardest time explaining this to my students, more than anyone else.

Sincerely,

Jim Jawn 0 comments - Post a Comment