Can my ISP take me to court for downloading torrents "illegally"?

My ISP wants to take me to court if I download another program or game "illegally", even though it is NOT illegal to download content from torrent websites that host servers in Canada and Sweden. They don't even know anything about right and wrong, yet they accuse me of copyright infringement. It's not copyright infringement and it never will be.