Python find similarity between two lists

Use collections.Counter[] perhaps; those are multi-sets, or bags, in datatype parlance:

from collections import Counter

counterA = Counter[listA]
counterB = Counter[listB]

Now you can compare these by entries or frequencies:

>>> counterA
Counter[{'apple': 3, 'orange': 2, 'banana': 1}]
>>> counterB
Counter[{'apple': 2, 'orange': 1, 'grapefruit': 1}]
>>> counterA - counterB
Counter[{'orange': 1, 'apple': 1, 'banana': 1}]
>>> counterB - counterA
Counter[{'grapefruit': 1}]

You can calculate their cosine similarity using:

import math

def counter_cosine_similarity[c1, c2]:
    terms = set[c1].union[c2]
    dotprod = sum[c1.get[k, 0] * c2.get[k, 0] for k in terms]
    magA = math.sqrt[sum[c1.get[k, 0]**2 for k in terms]]
    magB = math.sqrt[sum[c2.get[k, 0]**2 for k in terms]]
    return dotprod / [magA * magB]

Which gives:

>>> counter_cosine_similarity[counterA, counterB]
0.8728715609439696

The closer to 1 that value, the more similar the two lists are.

The cosine similarity is one score you can calculate. If you care about the length of the list, you can calculate another; if you keep that score between 0.0 and 1.0 as well you can multiply the two values for a final score between -1.0 and 1.0.

For example, to take relative lengths into account you could use:

def length_similarity[c1, c2]:
    lenc1 = sum[c1.itervalues[]]
    lenc2 = sum[c2.itervalues[]]
    return min[lenc1, lenc2] / float[max[lenc1, lenc2]]

and then combine into a function that takes the lists as inputs:

def similarity_score[l1, l2]:
    c1, c2 = Counter[l1], Counter[l2]
    return length_similarity[c1, c2] * counter_cosine_similarity[c1, c2]  

For your two example lists, that results in:

>>> similarity_score[['apple', 'orange', 'apple', 'apple', 'banana', 'orange'], ['apple', 'orange', 'grapefruit', 'apple']]
0.5819143739626463
>>> similarity_score[['apple', 'apple', 'orange', 'orange'], ['apple', 'orange']]
0.4999999999999999

You can mix in other metrics as needed.

The original list 1 is : [1, 4, 6, 8, 9, 10, 7]
The original list 2 is : [7, 11, 12, 8, 9]
Percentage similarity among lists is : 33.33333333333333

A while ago I wrote a guide on how to compare two dictionaries in Python 3, and how this task is not as simple as it might sound. It turns out comparing two lists in Python is just so tricky as comparing dicts.

The way we've been taught to compare two objects in Python is a bit misleading. Most books and tutorials teach object comparison by using either the == or the is operator. In reality, these two operators cover just a small fraction of the most frequent use cases.

For example:

• what if we want to compare a list of floating-point numbers considering a certain tolerance?
• what if we wish to contrast two lists but ignoring the order in which the elements appear?
• maybe we need to compare two lists and return the elements that intersect both
• sometimes we might want to get the difference between two lists
• what if we have two lists of strings and need to compare them by ignoring the string cases?
• what if we're given a list of numpy arrays to compare each other, what can we do?
• or maybe we have a list of custom objects, or a list of dictionaries.

The list goes on and on, and for all of these use cases using == doesn't help.

That's what we are going to see in this article. We’ll learn the best ways of comparing two lists in Python for several use cases where the == operator is not enough.

Ready? Let's go!

Comparing if two lists are equal in python

The easiest way to compare two lists for equality is to use the == operator. This comparison method works well for simple cases, but as we'll see later, it doesn't work with advanced comparisons.

An example of a simple case would be a list of int or str objects.

>>> numbers = [1, 2, 3]
>>> target = [1, 2, 3]
>>> numbers == target
True
>>> [1, 2, 3] == [1, 3, 2]
False
>>> ['name', 'lastname'] == ['name', 'lastname']
True
>>> ['name', 'lastname'] == ['name', 'last name']   
False

Pretty simple, right? Unfortunately, the world is complex, and so is production grade code. In the real world, things get complicated really fast. As an illustration, consider the following cases.

Suppose you have a list of floating points that is built dynamically. You can add single elements, or elements derived from a mathematical operation such as 0.1 + 0.1.

>>> numbers = []
>>> numbers.append[0.1 + 0.1 + 0.1]  # derive the element based on a summation
>>> numbers.append[0.2] # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target  # compares the lists
False
>>> numbers  # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]

Clearly, floating point arithmetic has its limitations, and sometimes we want to compare two lists but ignore precision errors, or even define some tolerance. For cases like this, the == operator won’t suffice.

Things can get more complicated if the lists have custom objects or objects from other libraries, such as numpy.

In [1]: import numpy as np

In [2]: numbers = [np.ones[3], np.zeros[2]]

In [3]: numbers
Out[3]: [array[[1., 1., 1.]], array[[0., 0.]]]

In [4]: target = [np.ones[3], np.zeros[2]]

In [5]: numbers == target
---------------------------------------------------------------------------
ValueError                                Traceback [most recent call last]
 in 
----> 1 numbers == target

ValueError: The truth value of an array with more than one element is ambiguous. Use a.any[] or a.all[]

You might also like to compare the lists and return the matches. Or maybe compare the two lists and return the differences. Or perhaps you want to compare two lists ignoring the duplicates, or compare a list of dictionaries in Python.

In every single case, using == is not the answer, and that's what we are going to see next: how to perform complex comparison operations between two lists in Python.

Comparing two lists of float numbers

In the previous section, we saw that floating point arithmetic can cause precision errors. If we have a list of floats and want to compare it with another list, chances are that the == operator won't help.

Let's revisit the example from the previous section and see what is the best way of comparing two lists of floats.

>>> numbers = []
>>> numbers.append[0.1 + 0.1 + 0.1]  # derive the element based on a summation
>>> numbers.append[0.2] # add a single element
>>> target = [0.3, 0.2]
>>> numbers == target  # compares the lists
False
>>> numbers  # Ooopppssss....
[0.30000000000000004, 0.2]
>>> target
[0.3, 0.2]

As you see, 0.1 + 0.1 + 0.1 = 0.30000000000000004, which causes the comparison to fail. Now, how can we do better? Is it even possible?

There are a few ways of doing approaching this task. One would be to create our own custom function, that iterates over the elements and compare it one by one using the math.isclose[] function.

Fortunately we don't have to reinvent the wheel. As I showed in the "how to compare two dicts" article, we can use a library called deepdiff for that. This library supports different types of objects and lists are one of them.

The example below starts off by setting up the two lists we want to compare. We then pass it to the deepdiff.DeepDiff constructor which returns the difference. That's great, the returned value is much more informative than a simple boolean.

Since we want to ignore the precision error, we can set the number of digits AFTER the decimal point to be used in the comparison.

The result is an empty dict, which means the lists are equal. If we try comparing a list with a float number that differs in more than 3 significant digits, the library will return that diff.

For reproducibility, in this article I used the latest version of deepdiff which is 5.6.0.

In [1]: from deepdiff import DeepDiff

In [2]: numbers = []

In [3]: numbers.append[0.1 + 0.1 + 0.1]  # derive the element based on a summation

In [4]: numbers.append[0.2] # add a single element

In [5]: target = [0.3, 0.2]

# if we don't specify the number of significant digits, the comparison will use ==
In [6]: DeepDiff[numbers, target]
Out[6]: 
{'values_changed': {'root[0]': {'new_value': 0.3,
   'old_value': 0.30000000000000004}}}

# 0.30000000000000004 and 0.3 are equal if we only look at the first 3 significant digits
In [7]: DeepDiff[numbers, target, significant_digits=3]
Out[7]: {}

In [8]: numbers
Out[8]: [0.30000000000000004, 0.2]

In [9]: target = [0.341, 0.2]

# 0.341 differs in more than 3 significant digits
In [10]: DeepDiff[numbers, target, significant_digits=3]
Out[10]: 
{'values_changed': {'root[0]': {'new_value': 0.341,
   'old_value': 0.30000000000000004}}}

Comparing if two lists without order [unordered lists] are equal

Lists in Python are unordered by default. Sometimes we want to compare two lists but treat them as the same as long as they have the same elements—regardless of their order.

There are two ways of doing this:

• sorting the lists and using the == operator
• converting them to sets and using the == operator
• using deepdiff

These first two methods assume the elements can be safely compared using the == operator. This approach doesn’t work for floating-point numbers, and other complex objects, but as we saw in the previous section, we can use deepdiff.

Sorting the lists and using the == operator

You can sort lists in Python in two different ways:

• using the list.sort[] method
• using the sorted[] function

The first method sorts a list in place, and that means your list will be modified. It's a good idea to not modify a list in place as it can introduce bugs that are hard to detect.

Using sorted is better since it returns a new list and keep the original unmodified.

Let's see how it works.

In [6]: numbers = [10, 30, 20]

In [7]: target = [10, 20, 30]

In [8]: numbers == target
Out[8]: False

In [9]: sorted[numbers] == sorted[target]
Out[9]: True

In [10]: sorted[numbers]
Out[10]: [10, 20, 30]

In [11]: sorted[target]
Out[11]: [10, 20, 30]

As a consequence, by sorting the lists first we ensure that both lists will have the same order, and thus can be compared using the == operator.

Converting the lists to a set

Contrary to lists, sets in Python don’t care about order. For example, a set {1, 2, 3} is the same as {2, 3, 1}. As such, we can use this feature to compare the two lists ignoring the elements’ order.

To do so, we convert each list into a set, then using the == to compare them.

In [12]: numbers = [10, 30, 20]

In [13]: target = [10, 20, 30]

In [14]: set[numbers] == set[target]
Out[14]: True

In [15]: set[numbers]
Out[15]: {10, 20, 30}

In [16]: set[target]
Out[16]: {10, 20, 30}

Using the deepdiff library

This library also allows us to ignore the order in sequences such as lists. By default, it will take the order in consideration, but if we set ignore_order to True, then we're all good. Let's see this in action.

In [11]: numbers = [10, 30, 20]

In [12]: target = [10, 20, 30]

In [13]: DeepDiff[numbers, target]
Out[13]: 
{'values_changed': {'root[1]': {'new_value': 20, 'old_value': 30},
  'root[2]': {'new_value': 30, 'old_value': 20}}}

In [14]: DeepDiff[numbers, target, ignore_order=True]
Out[14]: {}

Using deepdiff has pros and cons. In the end, it is an external library you need to install, so if you can use a set to compare the lists, then stick to it. However, if you have other use cases where it can shine, then I’d go with it.

How to compare two lists and return matches

In this section, we'll see how we can compare two lists and find their intersection. In other words, we want to find the values that appear in both.

To do that, we can once more use a set and take their intersection.

In [1]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [2]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [3]: set[t1].intersection[set[t2]]
Out[3]: {2, 7, 9}

# the & operator is a shorthand for the set.intersection[] method 
In [4]: set[t1] & set[t2]
Out[4]: {2, 7, 9}

How to compare two lists in python and return differences

We can the find difference between two lists in python in two different ways:

• using set
• using thedeepdiff library

Using set

Just like we did to determine the intersection, we can leverage the set data structure to check difference between two lists in python.

If we want to get all the elements that are present in the first list but not in the second, we can use the set.difference[].

On the other hand, if we want to find all the elements that are in either of the lists but not both, then we can use set.symmetric_difference[].

In [8]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [9]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [10]: set[t1].difference[set[t2]]
Out[10]: {0, 1, 3, 4}

In [11]: set[t2].difference[set[t1]]
Out[11]: {6, 11, 12, 23}

In [12]: set[t1].symmetric_difference[set[t2]]
Out[12]: {0, 1, 3, 4, 6, 11, 12, 23}

In [13]: set[t1] - set[t2]
Out[13]: {0, 1, 3, 4}

In [14]: set[t1] ^ set[t2]
Out[14]: {0, 1, 3, 4, 6, 11, 12, 23}

This method has a limitation: it groups what is different between the lists into one final result which is the set difference. What if we want to know which elements in that diff belong to what list?

Using deepdiff

As we've seen so far, this library is powerful and it returns a nice diff. Let's see what happens when we use deepdiff to get the difference between two lists in Python.

In [15]: t1 = [2, 1, 0, 7, 4, 9, 3]

In [16]: t2 = [7, 6, 11, 12, 9, 23, 2]

In [17]: DeepDiff[t1, t2]
Out[17]: 
{'values_changed': {'root[0]': {'new_value': 7, 'old_value': 2},
  'root[1]': {'new_value': 6, 'old_value': 1},
  'root[2]': {'new_value': 11, 'old_value': 0},
  'root[3]': {'new_value': 12, 'old_value': 7},
  'root[4]': {'new_value': 9, 'old_value': 4},
  'root[5]': {'new_value': 23, 'old_value': 9},
  'root[6]': {'new_value': 2, 'old_value': 3}}}

In [18]: DeepDiff[t1, t2, ignore_order=True]
Out[18]: 
{'values_changed': {'root[4]': {'new_value': 6, 'old_value': 4},
  'root[6]': {'new_value': 11, 'old_value': 3},
  'root[1]': {'new_value': 12, 'old_value': 1}},
 'iterable_item_added': {'root[5]': 23},
 'iterable_item_removed': {'root[2]': 0}}

Accordingly, deepdiff returns what changed from one list to the other. The right approach then will depend on your use case. If you want a detailed diff, then use DeepDiff. Otherwise, just use a set.

How to compare two lists of strings

Comparing two lists of string in Python depends largely on what type of comparison you want to make. That's because we can compare a string in a handful of ways.

In this section, we'll see 3 different ways of doing that.

The simplest one is using a == operator, like we saw in the beginning. This method is suitable if you want a strict comparison between each string.

In [1]: names = ['jack', 'josh', 'james']

In [2]: target = ['jack', 'josh', 'james']

In [3]: names == target
Out[3]: True

Things start to get messy if you want to compare the list of strings but ignoring the case. Using the == for that just doesn't work.

In [4]: names = ['Jack', 'Josh', 'James']

In [2]: target = ['jack', 'josh', 'james']

In [5]: names == target
Out[5]: False

The best tool for that is again deepdiff. It allows us to ignore the string by passing a boolean flag to it.

In [1]: import deepdiff

In [2]: names = ['Jack', 'Josh', 'James']

In [3]: target = ['jack', 'josh', 'james']

# ignoring string case
In [4]: deepdiff.DeepDiff[names, target, ignore_string_case=True]
Out[4]: {}

# considering the case
In [5]: deepdiff.DeepDiff[names, target]
Out[5]: 
{'values_changed': {'root[0]': {'new_value': 'jack', 'old_value': 'Jack'},
  'root[1]': {'new_value': 'josh', 'old_value': 'Josh'},
  'root[2]': {'new_value': 'james', 'old_value': 'James'}}}

We can also ignore the order in which the strings appear in the lists.

In [6]: names = ['Jack', 'James', 'Josh']

In [7]: target = ['jack', 'josh', 'james']

# ignoring the order and string case
In [8]: deepdiff.DeepDiff[names, target, ignore_string_case=True, ignore_order=T
   ...: rue]
Out[8]: {}

# considering the order but ignoring the case
In [9]: deepdiff.DeepDiff[names, target, ignore_string_case=True]
Out[9]: 
{'values_changed': {'root[1]': {'new_value': 'josh', 'old_value': 'james'},
  'root[2]': {'new_value': 'james', 'old_value': 'josh'}}}

You can also go further and perform advanced comparisons by passing a custom operator to DeepDiff.

For example, suppose you want to compare the strings but ignoring any whitespace they may have.

Or perhaps you want to perform a fuzzy matching using an edit distance metric.

To do that, we can write the comparison logic in the operator class and pass it to DeepDiff.

In this first example, we'll ignore any whitespace by trimming the strings before comparing them.

class IgnoreWhitespaceOperator:

    def match[self, level] -> bool:
        return True

    def give_up_diffing[self, level, diff_instance] -> bool:
        if isinstance[level.t1, str] and isinstance[level.t2, str]:
            return level.t1.strip[] == level.t2.strip[]
        return False

Then we can just plug into DeepDiff by adding it to the list of custom_operators, like so custom_operators=[IgnoreWhitespaceOperator[]].

In [6]: from deepdiff import DeepDiff

In [13]: names = ['Jack', 'James ', '  Josh ']

In [14]: target = ['Jack', 'James', 'Josh',]

# the operator will ignore the spaces in both lists
In [15]: DeepDiff[names, target, custom_operators=[IgnoreWhitespaceOperator[]]]
Out[15]: {}

In [16]: target = ['Jack', 'James', 'Josh', 'Jelly']

# if one of the list has an additional member, this will be flagged
In [17]: DeepDiff[names, target, custom_operators=[IgnoreWhitespaceOperator[]]]
Out[17]: {'iterable_item_added': {'root[3]': 'Jelly'}}

In [18]: target = ['Jack', 'Josh', 'James']

# by default, the library doesn't ignore order
In [19]: DeepDiff[names, target, custom_operators=[IgnoreWhitespaceOperator[]]]
Out[19]: 
{'values_changed': {'root[1]': {'new_value': 'Josh', 'old_value': 'James '},
  'root[2]': {'new_value': 'James', 'old_value': '  Josh '}}}

# if you don't care about order, be explicit
In [20]: DeepDiff[names, target, ignore_order=True, custom_operators=[IgnoreWhitespaceOperator[]]]
Out[20]: {}

How to compare two lists of dictionaries

Comparing two lists of dictionaries in Python is definitely intricate without the help of an external library. As we've seen so far, deepdiff is versatile enough and we can use it to compare deep complex objects such as lists of dictionaries.

Let's see what happens when we pass two lists of dictionaries.

In [1]: from deepdiff import DeepDiff

In [2]: first_list = [
   ...:     {
   ...:         'number': 1,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [3]: target_list = [
   ...:     {
   ...:         'number': 3,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [4]: DeepDiff[first_list, target_list]
Out[4]: {'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}}}

It outputs the exact location where the elements differ and what the difference is!

Let's see another example where a list has a missing element.

In [2]: first_list = [
   ...:     {
   ...:         'number': 1,
   ...:         'list': ['one', 'two']
   ...:     },
   ...:     {
   ...:         'number': 2,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [5]: target = [
   ...:     {
   ...:         'number': 3,
   ...:         'list': ['one', 'two']
   ...:     },
   ...: ]

In [6]: 

In [6]: DeepDiff[first_list, target]
Out[6]: 
{'values_changed': {"root[0]['number']": {'new_value': 3, 'old_value': 1}},
 'iterable_item_removed': {'root[1]': {'number': 2, 'list': ['one', 'two']}}}

It says the the second dictionary has been removed, which is the case for this example.

How to compare two list of lists

Comparing multidimensional lists—a.k.a list of lists—is easy for deepdiff. It works just like a list of dicts.

In the example below, we have two multidimensional lists that we want to compare. When passed to DeepDiff, it returns the exact location in which the elements differ.

For example, for the position [1][0], the new value is 8, and the old is 3. Another interesting aspect is that it works for deeply nested structures, for instance, deepdiff also highlights the difference in the [2][0][0] position.

In [1]: from deepdiff import DeepDiff

In [2]: first_list = [[1, 2], [3, 4], [[5]]]

In [3]: target_list = [[1, 2], [8, 4], [[7]]]

In [4]: DeepDiff[first_list, target_list]
Out[4]: 
{'values_changed': {'root[1][0]': {'new_value': 8, 'old_value': 3},
  'root[2][0][0]': {'new_value': 7, 'old_value': 5}}}

When feeding the library with two identical multidimensional lists, it returns an empty response.

In [3]: target_list = [[1, 2], [8, 4], [[7]]]

In [5]: second_list = [[1, 2], [8, 4], [[7]]]

In [7]: DeepDiff[second_list, target_list]
Out[7]: {}

How to compare two lists of objects

Sometimes we have a list of custom objects that we want to compare. Maybe we want to get a diff, or just check if they contain the same elements. The solution for this problem couldn't be different: use deepdiff.

The following example demonstrates the power of this library. We're going to compare two lists containing a custom objects, and we'll be able to assert if they are equal or not and what are the differences.

In the example below, we have two lists of Person objects. The only difference between the two is that in the last position Person object has a different age. deepdiff not only finds the right position - [1] - but also finds that age field is different as well.

In [9]: from deepdiff import DeepDiff

In [10]: first = [Person['Jack', 34], Person['Janine', 23]]

In [11]: target = [Person['Jack', 34], Person['Janine', 24]]

In [12]: DeepDiff[first, target]
Out[12]: {'values_changed': {'root[1].age': {'new_value': 24, 'old_value': 23}}}

In [14]: second = [Person['Jack', 34], Person['Janine', 24]]

In [15]: DeepDiff[second, target]
Out[15]: {}

How to compare two lists of numpy arrays

In this section, we'll see how to compare two lists of numpy arrays. This is a fairly common task for those who work with data science and/or machine learning.

We saw in the first section that using the == operator doesn't work well with lists of numpyarrays. Luckily we can use... guess what!? Yes, we can use deepdiff.

The example below shows two lists with different numpy arrays and the library can detect the exact position in which they differ. How cool is that?

In [16]: import numpy as np

In [17]: from deepdiff import DeepDiff

In [18]: first = [np.ones[3], np.array[[1, 2, 3]]]

In [19]: target = [np.zeros[4], np.array[[1, 2, 3, 4]]]

In [20]: DeepDiff[first, target]
Out[20]: 
{'values_changed': {'root[0][0]': {'new_value': 0.0, 'old_value': 1.0},
  'root[0][1]': {'new_value': 0.0, 'old_value': 1.0},
  'root[0][2]': {'new_value': 0.0, 'old_value': 1.0}},
 'iterable_item_added': {'root[0][3]': 0.0, 'root[1][3]': 4}}

Conclusion

In this post, we saw many ways to compare two lists in Python. The best method depends on what kind of elements we have and how we want to compare. Hopefully, you now know how to:

• check if two lists are equal in python
• compare two lists without order [unordered lists]
• compare two lists in python and return matches
• compare two lists in python and return differences
• compare two lists of strings
• compare two lists of dictionaries
• compare two list of lists
• compare two lists of objects
• compare two lists of numpy arrays

Other posts you may like:

• The Best Way to Compare Two Dictionaries in Python

• How to Compare Two Strings in Python [in 8 Easy Ways]

• 7 Different Ways to Flatten a List of Lists in Python

See you next time!

This post was originally published at //miguendes.me

How do you find the similarities between two lists in Python?

How do you find the similarity between two lists?

How do I compare two lists and differences in Python?

How do you find the cosine similarity between two lists?