How do you remove duplicates from a list whilst preserving order?

Question

Is there a built-in that removes duplicates from a list in Python, whilst preserving order? I know that I can use a set to remove duplicates, but that destroys the original order. I also know that I can roll my own like this:

def uniq(input): 

output = [] 

for x in input: 

if x not in output: 

output.append(x)

return output

But I'd like to avail myself of a built-in or a more Pythonic idiom if possible.

Answer 1

There are many ways to remove duplicate from Python List some are:-

• We can remove duplicate from List using the more_iterable library.

• The iterable library comes under Python itertools module which is a collection of tools for handling iterators.

• iterators are data types that can be used in a for loop. The most common iterator in Python is the List.

• For installing this library you can run the following commands: (pip install more_itertools).

from more_itertools import unique_everseen

items = [1, 2, 0, 1, 3, 2]

list(unique_everseen(items)) 

• Another way we can solve this problem using the ordered_dict library.

• The dict subclass always remembers the entire order of any collection.

from collections import OrderedDict

items = [1, 2, 0, 1, 3, 2]

list(OrderedDict.fromkeys(items))