Many people have asked me this question before. It's sometimes hard to get a straight answer, but if we remove all of moot details, use cases, and focus simply on the architectural design details the difference becomes quite clear. This video attempts to explain this difference in a relatively simple and straight-forward manner.
The conversation doesn't just talk about PHP frameworks specifically. There's also some mention of things like jQuery from other programming communities. There's plenty of suggested definitions in the conversation but most revolve around two ideas: that the library is a "tool" to use when needed and the framework is the structure of the application (hence "framework") that "uses you".