How to define and remove outliers in a data?

Hi Marieta,

 

This is a pretty standard task in Statistics. You can basically follow two main basic approaches given your context of univariate data:

 

1) Check if you have gaussian or nearly gaussian data (by visual inspection and hypothesis testing)

 

If so, you could be using the Standard Deviation method to define outliers. This would typically be values falling beyond three standard deviations from the mean. In any case you can adapt your threshold to the context by having a check on the distribution of the values. A box plot is always a nice visualization technique. If you use R, it will even automatically identify the outliers for you in the boxplot function. 

 

2) Non-gaussian data or small sample sizes

 

If you cannot apply the previous method, you could be using the IQR (Interquartile Range), i.e, the difference between the 0.75 percentile and the 0.25 percentile. Typically, you would then set the factor of 1.5 times the IQR and define as an outlier any value smaller than (0.25 percentile - 1.5*IQR) or higher than (0.75 + 1.5*IQR). The factor 1.5 is a value usually used in practice but other values can turn out to be more adequate, explore the data and refine it.

 

These are two methods generally used in practice because they are straightforward but still effective. If you need something more powerful there is a whole lot of more recent techniques like proximity-based models (clustering) or information theoretic methods (outliers are data instances increasing the complexity of the data set) that you can look up online.

 

Still, in this particular case I think that one of the previous methods should be enough, especially if you are able to derive a nicely tuned-up threshold given out of functional experience with the problem of timing.

 

I hope it helps.

 

Best regards,

Pedro F. B. Silva

Marieta

be careful about moving "outliers" as this can easily distort your interpretation of the results.

There are statistical methods as others have commented.

However, before removing an outlier you really should look at that occurrence and try to find out WHY it is an outlier - why is this observation so different from the others? If there are genuine reasons for the difference then remove it If there aren't then you should leave it in the data set as it represents a real situation

You could also think about using quality control charts to monitor your data over time. These set pre-determined limits for outliers  

 

 

 

Hi Marieta,

 

if speaking about Excel, the Median formula might help you.

BR

Milena

Hello Marieta, 

Have you tried P80 în Excel?

Iuliana