Indeed - never, ever rely on a "I saw other animals eating it" standard. It's a good way to get you killed, or at least sick.
For some reverse examples, chocolate which humans can consume in bulk is poisonous to most higher animals, while avocado is poisonous to many types of birds, etc. Some are even ironic - for example, horseradish is poisonous to horses.
Update: Concerning cassava, there are two main types, sweet and bitter. Both have cyanide-producing compounds (cyanogens), but bitter has far more (it is, however, a hardier crop). Bitter cassava can be as much as 0,1% cyanide-equivalent by mass. Neither should be eaten without proper preparation, but bitter can be very dangerous if improperly processed.