How can I trace Hacking attempt

Question

First of all I dont want to track the hacker. I want to track if the system is being compromised some way. For example when a website is under attack it'll receive malicious data. I want to check those data and if they are malicious I want to call assistent via mail or sms or phone call.

The bold phrase above is the tricky part. How to define if something is malicious? Hacker sends valid data from the apps point of view. But outside that application domain another tool can examine it.

Is there any tool like that? If not what are the common logic to trace intrusion? I can write a tool then.

Answer 1

As mentioned in the comments, try and IDS or IPS. My favorite and free one is hostbased and called OSSEC.

It will alert you of break-in attempts and other irregularities on your machine.

Answer 2

An IDS (intrusion detection system) or IPS (intrusion prevention system) is the standard solution. It generally works by watching traffic to your protected resources (such as a web server) and looking for characteristics associated with attacks. The big advantages is that it saves a vast amount of time and provides many more ways of scanning for attack vectors than you could on your own. With most systems of this type, you plan a way to get updates, so that attackers evolve, your system follows shortly after.

For any given type of resource, the attack vectors will vary and are always evolving. As you mention, for web servers, a pretty common vector is faulty inputs - code injection and buffer overflow being the two that come to mind most readily. But this is going to vary for any type of resource you want to protect and any protocol that can an attacker can use to access it.

A sophisticated system may combine:

• benchmarks for how the system currently behaves and observation of changes to those benchmarks
• analysis of protocols and misuse of aspects of those protocols
• repeated attempts to provide incorrect or invalid input
• any behavior pattern that shows a type of scanning
• detection of significantly unusual behavior

and more...

There's really no perfect answer to the eternal - "have we been successfully attacked?" question, the real key is knowing a given system, what "normal" is and thus what "abnormal" is, and knowing a given design or communication pattern well enough to know how and when it can be misused