I was getting the same UnicodeEncodeError when saving scraped web content to a file. To fix it I replaced this code:
with open(fname, “w”) as f:
f.write(html)
with this:
import io
with io.open(fname, “w”, encoding=”utf-8″) as f:
f.write(html)
Using io gives you backward compatibility with Python 2.
If you only need to support Python 3 you can use the builtin open function instead:
with open(fname, “w”, encoding=”utf-8″) as f:
f.write(html)
If your file is encoded in something other than UTF-8, specify whatever your actual encoding is for encoding.
I fixed it by adding .encode(“utf-8”) to soup.
That means that print(soup) becomes print(soup.encode(“utf-8”)).