Using inc and lib aren’t something that I say belong in just themes or belong in just plugins. Instead, they are directories that I’ve used in both themes and plugins.
The inc Directory
As my general rule, I use the inc directory primary to place collections of functions that are related to core functionality but aren’t necessarily meant to clutter up the primary core of the theme or plugin.
For example, if I’m working on a theme and I have a collection of functions for said theme that I use as helper functions, I’d much rather create inc/helpers.php than to drop them in functions.php. Over time, more helpers are added, the file gets longer, maintenance becomes a pain, and so on.
But in more specific cases, I also use the inc directory as a way to store core theme files such as theme-customizer.php or custom-header.php. This way, these files are focused solely on a single purpose and are easier to maintain over time. Plus, they are self-descriptive.
This keeps functions.php lean, and it keeps procedural programming files slightly more organized than having one giant “god-file” by the time of delivery.
In the case of plugins, I generally use object-oriented programming so the inc directory is normally used to hold additional classes that I write that are used as part of the core plugin file, but are dependencies.
This means that if I have the core plugin that depends on, say, a custom CSV parser or a serialization / de-serialization class, then these files would reside in inc.
The lib Directory
In short, the lib directory is used for third-party libraries. That is, these are used to make sure that I place code written by another author or team of developers in a place that I can easily retrieve (and attribute – don’t forget! :)) in my project.
Also, I know that an alternative to this is using a vendors directory. Though I’ve seen the convention and think it’s just as good, I’ve personally just stuck with lib.
references: developer.wordpress.org, tommcfarlin.com