Removing duplicates in lists

There are a few ways to remove duplicates from a list in Python. One of the simplest ways is to convert the list to a set, and then back to a list. This works because sets only allow unique elements, so any duplicates will be removed automatically when you convert the list to a set. Here's an example:

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(set(original_list))
print(unique_list)

This will print [1, 2, 3, 4, 5], which is the original list with duplicates removed.

Another approach to remove duplicates from a list is to create an empty list, and then iterate through the original list, adding each element to the new list only if it has not already been added. Here's an example:

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = []
for element in original_list:
    if element not in unique_list:
        unique_list.append(element)
print(unique_list)

This will also print [1, 2, 3, 4, 5].

Another approach is to use list comprehension, which can often make your code more concise:

original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(dict.fromkeys(original_list))
print(unique_list)

This code uses the dict.fromkeys() method to construct a dictionary from the original list with all values set to None. Because dictionaries are unordered collection of unique keys, This will give you the same output, with no duplicates.

It's worth noting that these examples will not preserve the original order of elements in the list. If you need to preserve the original order, you can use the collections.OrderedDict.fromkeys() method to achieve that:

from collections import OrderedDict
original_list = [1, 2, 3, 4, 2, 3, 4, 5]
unique_list = list(OrderedDict.fromkeys(original_list))
print(unique_list)

Keep in mind that while the above methods works great for small lists and simple types, if you are dealing with large lists or complex types this may not be the best approach, you might prefer using something like pandas, or numpy to handle such cases.